Hardware overlays

Last night I managed to check in the first set of changes to add hardware video overlay support to the OS: A mostly-working OMAPVideo (for OMAP3 & OMAP4 machines), and a basic version of the VideoOverlay module (“Z-Order” overlays should work OK but handling of “basic” overlays is still rudimentary).

For OS developers:

• The VideoOverlay docs folder has up-to-date documentation for the new/updated APIs
• There’s a simple test program in the VideoOverlay sources that developers of other video drivers can use to help get them started in adding overlay support to their drivers
• For some real-world testing, I’ve also uploaded my modified KinoAmp sources which adds mostly-functional support for a couple of the YUV formats supported by OMAPVideo (plus RGB overlay support various pixel formats).
• For OMAP5 & Titanium the updated OMAPVideo sources should hopefully prove a useful reference for implementing overlays on those platforms. Just be aware that there are still some bugs left to fix (e.g. on OMAP4, YUV overlays which are both rotated & scaled are broken), and quirks to resolve (NV12 chroma subsampling seems a bit off)

For app developers:

• The VideoOverlay module should now start appearing in disc image builds, so you should be able to grab fresh builds from there (in !System). Similarly your code should be trying to load it from there, rather than going down the route of distributing copies with your apps.
• See the VideoOverlay docs folder for API details. Hopefully there won’t be any breaking changes to the API going forward, but there are still quite a few things which the module doesn’t deal with (software fallback, some edge cases with alpha/TCK management, Basic overlay quirks, etc.). Plus of course OMAPVideo can be a bit quirky as well. So don’t expect everything to work smoothly just yet.
• I’ll continue working on the KinoAmp changes alongside the OS-side changes, and start talking to André about folding the changes into the official sources once things become suitably stable.

For everyone else:

• I’d appreciate it of you could keep an eye out for any odd behaviour with the new OMAPVideo, since there were quite a lot of changes made in an effort to fix some bugs.
• I’m aware of several bugs in the updated version of KinoAmp, so I don’t think there’s much point reporting those just yet. But by all means feel free to play around with it and see how well it performs. You’ll probably want to use an alpha-channel screen mode (“LARGB” / “LABGR”) for the desktop otherwise (with the current OMAPVideo) the overlays will be appearing ontop of the desktop instead of underneath it.

Other stuff:

Now that OMAPVideo actually enables alpha-blending when alpha-channel screen modes are in use, it’s clear that there are some deficiencies in the OS that need to be tackled for things to work nicely:

• Currently there’s no way of controlling the alpha channel when setting a colour using ColourTrans – you have to use OS_SetColour to set the raw pixel/colour value directly
• DragASprite ignores the screen alpha
• Selection boxes will be transparent because the Wimp uses ColourTrans_SetGCOL with an EOR plot action, causing the screen alpha to be inverted to zero.
• Possibly the display manager should start preferring alpha channel modes over non-alpha modes, if they result in better overlay support on some drivers

Next steps for me will probably be to add overlay support to BCMVideo, which I’m hoping won’t be too painful, and should help shake out any remaining issues with the GraphicsV API. Then I should be able to start work on tidying up the loose ends with OMAPVideo/VideoOverlay, while working out whether anything useful can be done for IMX6. Iyonix support should also appear at some point as well, thanks to Adrian Lees making me realise that there’s some unused code for YUV overlays inherited from the haiku NVidia driver sources.

Selection boxes will be transparent because the Wimp uses ColourTrans_SetGCOL with an EOR plot action, causing the screen alpha to be inverted to zero.

As I’m sure you’re aware, the Wimp is far from the only system to use EOR while drawing. It will have to work sensibly everywhere.

ColourTrans certainly needs to become alpha aware. This will need a new flag. I wonder whether the existing ‘background is transparent’ bit is enough to switch between replacing the screen alpha and inverse multiplying it.

Next steps for me will probably be to add overlay support to BCMVideo,

Is overlay support fully enabled on BCMVideo. The overlay enabled version of Kino works OK on BBxM and Pandaboard ES but gives a green screen with parts of the video showing or duplicated on the Raspberry PI. Fullscreen errors as the LARGB/BGR modes also don’t seem to exist. The Basic test program works OK though.

Is overlay support fully enabled on BCMVideo. The overlay enabled version of Kino works OK on BBxM and Pandaboard ES but gives a green screen with parts of the video showing or duplicated on the Raspberry PI.

Yes. What does the KinoAmp film info window show? I think there are some combinations of format & dimensions which aren’t dealt with correctly yet. Also make sure cropping / pan & scan is disabled (I haven’t tested it, and I think it needs fixing).

Fullscreen errors as the LARGB/BGR modes also don’t seem to exist.

I’ve only been testing KinoAmp with the Multitasking mode – possibly the other modes are a bit broken (and VideoOverlay currently only supports using overlays when in the Wimp).

For LARGB/BGR modes to be available you need to explicitly set framebuffer_ignore_alpha to 0 in config.txt (it defaults to 1, with no way of overriding it at runtime via the mailbox interface).

For LARGB/BGR modes to be available you need to explicitly set framebuffer_ignore_alpha to 0 in config.txt

That was the bit I was missing. Thanks.

I’ve found switching to 64K colours displays the video OK. It’s a Bluray converted to 1080p mpg2. Switching back to 16M colours while it’s playing and then playing the film again in 16M colours seems to work.
Playback on a Pi 3B using overlays is smoother than on a Titanium.

OK, that sounds like the GPU might be a bit low on memory or is having trouble due to fragmentation. I’ll make a note to do some testing for that kind of situation.

I am integrating Overlays into my development version of KinoAmp. I managed to make it work in full-screen mode by creating a transparent window on top of the screen. Here are some results on my PI3:
- DVD scaled to 1920×1080, deinterlacing mode: Blend X2 (attempts to play 60 fps instead of 30 fps). Normal: 22 fps displayed, Overlays: 60 fps with a wait (= free) time of 25 to 30%.
- Raiks, 1920×1080 (30fps) test video from the FF_CUt thread. Normal 12.5fps, Overlay: 24.5fps.

I also noticed the green tint issue. For some files once you change mode once it nevers occurs again, for some othe files the issue is permanant.

Some more work to do:
- fix crashes when changing modes in the desktop
- hide the overlat when the window is pinned to the desktop.
- handle panscan/cropping
- show OSD in overlay
- handle brightness, contrast, color options in overlay.

For the later, I presume the overlays do not offer an option to control the YUV to RGB conversion?

By the way once I start using overlays I get ZeroPains (whcih continue to occur even after quitting KinoAmp):

Time: Thu Aug 9 22:41:39 2018
Location: Offset 00001878 in module BCMVideo
Current Wimp task: Unknown
Last app to start: Kino:Kino -fKinoChoices:Choices Fat32fs::Fatty.$.Wespe_HD_kpl/mpeg -d0x0030 2>Kino:log

R0 = 00000000 R1 = 88000211 R2 = 00000000 R3 = 00010017
R4 = 00000000 R5 = 00000001 R6 = 00000000 R7 = 0006cdd4
R8 = 20000113 R9 = 0000002a R10 = fc0197fc R11 = 20047cb0
R12 = 20047474 R13 = fa207ed8 R14 = fc225310 R15 = fc224e90
DFAR = 00000027 Mode SVC32 Flags nZCv if PSR = 60000113

fc224e48 : 0000001c : ANDEQ R0,R0,R12,LSL R0
fc224e4c : 00000000 : ANDEQ R0,R0,R0
fc224e50 : 0003000d : ANDEQ R0,R3,R13
fc224e54 : 00000004 : ANDEQ R0,R0,R4
fc224e58 : 00000004 : ANDEQ R0,R0,R4
fc224e5c : 00000000 : ANDEQ R0,R0,R0
fc224e60 : 00000000 : ANDEQ R0,R0,R0
fc224e64 : 0000001c : ANDEQ R0,R0,R12,LSL R0
fc224e68 : 00000000 : ANDEQ R0,R0,R0
fc224e6c : 0003000e : ANDEQ R0,R3,R14
fc224e70 : 00000004 : ANDEQ R0,R0,R4
fc224e74 : 00000004 : ANDEQ R0,R0,R4
fc224e78 : 00000000 : ANDEQ R0,R0,R0
fc224e7c : 00000000 : ANDEQ R0,R0,R0
fc224e80 : e92d400f : STMDB R13!,{R0-R3,R14}
fc224e84 : e24dd020 : SUB R13,R13,#&20 ; =" "
fc224e88 * e5d01027 * LDRB R1,[R0,#39]
fc224e8c : e1500001 : CMP R0,R1
fc224e90 : 2a000028 : BCS &FC224F38
fc224e94 : e28b8028 : ADD R8,R11,#&28 ; =“(”
fc224e98 : e7b8e280 : LDR R14,[R8,R0,LSL #5]!
fc224e9c : e37e0001 : CMN R14,#1
fc224ea0 : 1a00001a : BNE &FC224F10
fc224ea4 : e59b1000 : LDR R1,[R11,#0]
fc224ea8 : e1db21b0 : LDRH R2,[R11,#16]
fc224eac : e1db31b2 : LDRH R3,[R11,#18]
fc224eb0 : e3a00003 : MOV R0,#3
fc224eb4 : e5911000 : LDR R1,[R1,#0]
fc224eb8 : e88d000f : STMIA R13,{R0-R3}
fc224ebc : e3a00010 : MOV R0,#&10 ; =16
fc224ec0 : eb0002cc : BL &FC2259F8
fc224ec4 : 628dd020 : ADDVS R13,R13,#&20 ; =" "

R15 = fc224e90 = BCMVideo +1880
R14_svc = fc225310 = BCMVideo +1d00
Function call to fc224e80 = BCMVideo +1870

SVC stack:
fa207ed8 : 204e29e8 :
fa207edc : 30003e8c :
fa207ee0 : 204e29e8 :
fa207ee4 : 40000113 :
fa207ee8 : 0002003b :
fa207eec : fc17edd4 :
fa207ef0 : fa20021c :
fa207ef4 : 20513aa8 :
fa207ef8 : 00000000 : – R0
fa207efc : 88000211 : | R1
fa207f00 : 00000000 : | R2
fa207f04 : 00010017 : | R3
fa207f08 : fc225310 : | R14: fc225310 (ASM call to fc224e80)
: : | fc225310 = BCMVideo +1d00
: : | fc224e80 = BCMVideo +1870
fa207f0c : 00000000 :
fa207f10 : 88000211 :
fa207f14 : 00000000 :
fa207f18 : 00010017 :
fa207f1c : 00010017 :
fa207f20 : 23d7818c :
fa207f24 : 00000000 :
fa207f28 : 0006cdd4 :
fa207f2c : 20000113 :
fa207f30 : 00000034 :
fa207f34 : fc01945c : – Return to fc01945c?
: : | = +1945c in the Kernel
: : | = CallVecLoop +8
fa207f38 : fc019e14 : – fc019438 return to fc019e14?
: : | fc019438 = +19438 in the Kernel
: : | = CallVector +0
: : | fc019e14 = +19e14 in the Kernel
: : | = FIQKiller_ws +10
fa207f3c : 00000113 : – PSR?
fa207f40 : 00020034 : | SWI XOS_CallAVector
fa207f44 : fc17ee70 : | R14: fc17ee70
: : | = SharedCLibrary +13ee0
: : | = _swi +94
fa207f48 : fa20021c : | R10
fa207f4c : fa207fb8 : | R11
fa207f50 : 00020034 : | R12
fa207f54 : fa207fb8 :
fa207f58 : fa207f94 :
fa207f5c : 00000002 :
fa207f60 : 20513aa8 :
fa207f64 : 300f8f20 :
fa207f68 : 88000211 :
fa207f6c : fa207fbc :
fa207f70 : 23d7818c :
fa207f74 : 00000000 :
fa207f78 : 0006cdd4 :
fa207f7c : 20000113 :
fa207f80 : 0006659c :
fa207f84 : 2052cfe4 : – fc17ed7c return to 2052cfe4?
: : | fc17ed7c = SharedCLibrary +13dec
: : | = _swix +0
: : | 2052cfe4 = VideoOverlay +d70
fa207f88 : 00000000 :
fa207f8c : 00010017 :
fa207f90 : 0000002a :
fa207f94 : fa207fa0 :
fa207f98 : fa207f9c :
fa207f9c : 00000001 :
fa207fa0 : 000f5859 :
fa207fa4 : 00000000 :
fa207fa8 : 23f5249c : – R4
fa207fac : 00000000 : | R11
fa207fb0 : fa207fbc : | R12
fa207fb4 : 2052c524 : | R14: 2052c524
: : | = VideoOverlay +2b0
fa207fb8 : 2052cf7c : | APCS function: 2052cf74
: : | = VideoOverlay +d00
fa207fbc : 80000020 : | R0 \ CMHG veneer kernel_swi_regs?
fa207fc0 : 00000000 : | R1 |
fa207fc4 : 80000020 : | R2 |
fa207fc8 : 00000000 : | R3 |
fa207fcc : 0005659c : | R4 |
fa207fd0 : 00000000 : | R5 |
fa207fd4 : 00000000 : | R6 |
fa207fd8 : 0006cdd4 : | R7 |
fa207fdc : 000565a4 : | R8 |
fa207fe0 : 0006659c : | R9 /
fa207fe4 : fc0106b4 : – fc019438 return to fc0106b4?
: : | fc019438 = +19438 in the Kernel
: : | = CallVector +0
: : | fc0106b4 = +106b4 in the Kernel
: : | = VectorUserSWI +8
fa207fe8 : 40000110 : – PSR?
fa207fec : 00079cc3 : | SWI XVideoOverlayMapBuffer
fa207ff0 : fc17edd4 : | R14: fc17edd4
: : | = SharedCLibrary +13e44
: : | = _swix +58
fa207ff4 : 00050f90 : | R10
fa207ff8 : 000517a4 : | R11
fa207ffc : 00079cc3 : | R12

R14_usr = 00031520 = +29520 in application memory = player_redrawLastFrame +1ac
Function call to fc17ed7c = SharedCLibrary +13dec = _swix +0

I am integrating Overlays into my development version of KinoAmp.

Cool – feel free to use any of the changes in the sources I posted a link to above.

Also, I’ve been keeping the ModeVars doc up to date with the list of YUV formats that are supported by VideoOverlay / drivers. The Pi supports some RGB formats, YV12, YV16 and NV12. YV12 and YV16 look like they’d be good fits for KinoAmp (you might even be able to decode directly into the overlay buffers), but those KinoAmp sources don’t support them. Out of the formats that KinoAmp does support, YUY2 is supported by OMAP3/4/5, and NV12 is supported by OMAP4/5 (and Pi).

I guess it would be useful if I added that info to a doc, or the wiki, so that people testing overlay support know what to expect on each platform.

I managed to make it work in full-screen mode by creating a transparent window on top of the screen.

I’ll see if I can add proper support for full-screen / non-desktop use in the next VideoOverlay version (it’ll mostly be the same API, you just won’t attach the overlay to a window).

There is one noteworthy VideoOverlay bug that I’m aware of, which is that “basic” overlays remain visible when their window is iconised. I thought iconised windows would get closed, but it turns out they just get moved to the back of the window stack, behind the pinboard. So I need to fix VideoOverlay to detect when windows/overlays are fully obscured so that it can auto-hide the overlay (I’ll probably leave use this logic for both overlay types, since it’ll help on memory bandwidth).

- hide the overlat when the window is pinned to the desktop.

That’s a VideoOverlay bug I’ll be fixing.

- handle brightness, contrast, color options in overlay.

For the later, I presume the overlays do not offer an option to control the YUV to RGB conversion?

Some hardware does – I’ll take a closer look at what KinoAmp needs and see if there’s a way to support it.

By the way once I start using overlays I get ZeroPains (whcih continue to occur even after quitting KinoAmp):

OK, I’ll look into it.

Here is a test version of KinoAmp

I still have to had the OSD and image control (brightness, …) for overlays but the rest works.

Not that by using triple buffering to avoid tearing performance dropeed somewhat (instead of 60 fps with a 25% wait/free time I dropped to 56 fps.

Another issue GraphicsV do mention new mode variable 12 for specifying the numbe of required banks.
Other documents mention variable 13. This latest is the correct one. When I used variable 12, the image was flickering and some memory including the top of the image which serves as background for the pinboard gor overwritten by the content of the second buffer.

Not that by using triple buffering to avoid tearing performance dropeed somewhat (instead of 60 fps with a 25% wait/free time I dropped to 56 fps.

I’ll make a note to look into it (I’ve got a list of potential performance improvements to make to the overlay support in the OS)

Another issue GraphicsV do mention new mode variable 12 for specifying the numbe of required banks. Other documents mention variable 13. This latest is the correct one.

Yes, 13 is the correct number. I’ll correct the GraphicsV docs.

Now in CVS:

• VideoOverlay improved to attempt to crop overlays if there are windows infront. This’ll fix the “iconised overlay still visible” bug.
• Zero page access in BCMVideo fixed

I had a look at the green screen/corruption issue – it looks like it could be a firmware bug when it runs low on memory. I’ll try and investigate more to rule out the OS causing the corruption.

A couple more notes for the Pi driver:

• It looks like there’s a firmware bug relating to cropping of NV12 overlays (and potentially others). If the left edge is cropped then the UV plane will be offset from the Y plane.
• Overlays are only available for screen modes <= 2048×2048 (due to the hardware limits on the overlay coordinates – same problem that causes us to start using the software mouse pointer in high-res modes)

Here is a new test version of KinoAmp

I have now implemented all the types of overlays listed in the docs for 4:2:0, 4:2:2, 4:4:4 and monochrome (Y only).
I could not test UYVY and YUY2 on my PI3 so I am not sure if they are correctly implemented.

I also could verify that all [T|A][RGB][BGR] 32bit overlays worked, but encountered some curious results for the 16bit (64K colours). Only RGB16 is supported as screen-mode on the PI3, but for some reason both RGB16 and BGR16 are accepted as overlays and curiously the RGB16 shows weird colours.

Only RGB16 is supported as screen-mode on the PI3, but for some reason both RGB16 and BGR16 are accepted as overlays and curiously the RGB16 shows weird colours.

OK, looks like that’s another bug in BCMVideo – RGB16 should be the only supported 16bpp overlay format, but the driver isn’t checking the requested format correctly.

Tomorrow’s Pi ROM should contain fixes for the green screen/corruption, and the phantom 16bpp overlay formats.

Here is a test version of KinoAmp

This fixes the flickering sidebars reported here. Tested with OS5.24 and OS5.25 (14-Aug-18) on the RPi3B+

I confirm that the green screen/corruption disappeared as well as the phantom overlay format.

Regarding the RGB16 overlay colour issues it is my fault. I defined a seperate colour conversion to TBGR32 for sprite saving when overlays are in use since for most overlay the image remain in YUV like colorspace and sprites must be saved in an RGB format. The problem is that I forgot that both first fill the same conversion table so the overlay got 8-bit values from the conversion table instead of 5-bit one.

Here is a new test version of KinoAmp
which fixes that issue and a few minor ones.

Final (?) test version of KinoAmp .

This version adds brightness, contrast and colour control and OSD visibility on the YUV overlays (slows KinoAmp a bit though).

Can someone confirm that UYVY and YUY2 overlays work for all type of MPEG: YUV420, YUV422 and Y444? Please also check the monochrome option.

YUV420 & YUV422 seem to be working fine with UYVY & YUY2 overlays on OMAP3. However, I’m having trouble convincing ffmpeg to produce a YUV444 MPEG, and I can’t find any test clips online. Do you have any clips you can share?

I only have stream444.mpg .
Pretty much academic, since there is no commercial use.
You might won’t to play this very short clip with the “Continuous loop” option on or you won’t see anything.

Thanks – that clip plays fine too.

Thanks for tidying up my code!

Removed post, wrong diagnose on my part

While playing around with KinoAmp on the PI3, I noticed a few display issues with overlays of type YV16 and NV12 (YV12 seems fine).

When I size down the image of HD videos, at some point the display turns entirely black. I increase the size again and the screen reappears. For smaller videos that does not seem to cause an issue.

When I display a KinoAmp window with “Lock size to window” disabled and scroll within the window, the chroma becomes de-synchronized. It occurs when I scroll down for YV16 and when I scroll to the right for NV12.