Proposed GraphicsV enhancements

Shame the Paint bounty hasn’t really taken off (yet). Creating alpha sprites was one of the reasons it was started.

I’m hoping to upgrade to the latest version of ArtWorks when funds allow, and was pondering asking Martin if he would consider adding alpha sprites to the already excellent bitmap export. Creating new application icons is best done in ArtWorks anyway these days, and it already has good transparency support at the design stage.

Of course the problem there is that I don’t think there are any tools to convert PNGs, etc. to proper alpha sprites

Can NetSurf do it? Maybe someone could update either Png2Spr ¹ or Spr2Png ² to support alpha sprites. There at least seems to be source code ³ available for spr2png (GPLv2).

Shame the Paint bounty hasn’t really taken off

Agreed. That’d make a huge difference.

¹ http://www.pittdj.co.uk/software/index.htm

² http://old-www.moreofthesa.me.uk/progs.graphics.html#spr2png

³ http://old-www.moreofthesa.me.uk/progs.source.html#spr2png

NetSurf and Spr2Png are both able to create sprites with alpha, but they don’t set the mode word up correctly to indicate that the data’s there (they use an &ABGR pixel format, the same as a RISC OS 5 alpha channel sprite, but they only set a RISC OS 3.5 mode word).

For NetSurf this behaviour is understandable as that’s the way in which Tinct requires its alpha sprites to be set up. But for Spr2Png it’s pretty boggling – it accepts the RISC OS Select alpha mask format as input, but only produces the non-standard alpha channel format as output.

NetSurf could be changed to export sprites with the correct mode word for alpha sprites. Patches are welcome. :)

ArtWorks can export bitmaps with alpha channels, but you’d need to use something to convert a PNG to alpha sprite, I think.

I’m not sure, but I believe the default RISC OS 5 icon set by Richard Hallas was done in ArtWorks, so it may “just” be a case of re-exporting and converting them all.

For NetSurf this behaviour is understandable as that’s the way in which Tinct requires its alpha sprites to be set up.

Speaking of NetSurf and Tinct: Most people will be aware that Tinct isn’t ARMv7 safe and may be familiar with NetSurf going pop when you use it on a beagleboard and hit sites with graphic content that causes Tinct based dither to be used.

The fun bit is changing the NetSurf graphics handling from the default to the “Use OS” option. When you click on the “Images” option in choices it seems the display uses Tinct and so… Pop!

Anyway, the tip: Open !Boot.Choices.WWW.NetSurf.Choices and add the following two lines

plot_fg_quality:128
plot_bg_quality:128

and then start NetSurf. Check the “Images” option in the choices and note it now says “Use OS”

ArtWorks can export bitmaps with alpha channels

No offence to MW, but having to buy a copy of ArtWorks and then learning how to use it just to create a couple of icon sprites is still a rather high barrier to adoption of alpha channels in RISC OS icons…

No offence to MW, but having to buy a copy of ArtWorks and then learning how to use it just to create a couple of icon sprites is still a rather high barrier to adoption of alpha channels in RISC OS icons…

Sure, but I think there’s a fair chance that the sort of arty RISC OS folk who might design an iconset might also have ArtWorks.

Sure, but I think there’s a fair chance that the sort of arty RISC OS folk who might design an iconset might also have ArtWorks.

And the rest of us?
[hint – for my last icon, I used a program running under Windows to design the icon, but there was no fancy transparency mask or anything – didn’t seem to be an obvious way to convert that to RISC OS…]

To solve your design problems on the dreaded windows platform, this may be a solution:

www.gimp.org

Back on the topic of interlace & EDID – I’ve now fixed up all the drivers to make sure that they use the interlace control list item (not the sync/pol flags) and to get rid of support for GraphicsV 3. I’ve also implemented the EDID changes to disallow EDID writes and add support for automatically setting the segment pointer.

Does that mean https://www.riscosopen.org/bounty/polls/3 is done? Or partly done?

Partly done, potentially fully done once Steve is back from holiday and reviews the latest version.

A question that I’d like to be able to answer at the show on Saturday. Will the GraphicsV enhancements ultimately allow devices like the Pi be able to output display simultaneously to HDMI, DPI, DSI and USB?

That’s the plan, yes.

output display simultaneously to HFMI, DPI, DSI and USB

Could be rather interesting on something like the Sabre Lite (nice feature set on that if we have an iMX6 HAL)

That’s the plan, yes.

Great that’s what I thought, I was just worried that I may have misunderstood things and didn’t want to get it wrong in my Show Theatre talk on Saturday.

I’m very interested in the development of YUV sprite rendering.

The SDL library implementation on RISC OS provides virtual YUV (or YCbCr) overlays. The FFplay application is built on SDL and uses a YUV overlay to playback videos. Specifically, FFplay asks SDL to create a “YV12” overlay defined as planar mode (Y + V + U). FFplay decompresses video data into YUV420P format and writes onto the SDL-managed YV12 overlay, then calls SDL functions which in turn copy and transform the YUV frame into an RGB sprite before calling SWIs to blit the sprite to screen.

But if RISC OS 5 now supports YUV sprite modes and YUV graphics hardware, then it may be possible to significantly improve the performance of SDL’s YUV overlays, and therefore FFplay, by decompressing video data directly into a YUV sprite for immediate blitting by the new SWIs.

The key is to find a common YUV format that both FFmpeg and RISC OS support. And this is where I am having trouble. The only pairing I have found that produces recognisable frames are when I make FFmpeg output data in “uyvy422” pixel format with a prepended sprite header with the mode word specifying 4:2:2 YCbCr ITU-R BT.709 video range. But these frames are very poor compared to what FFplay achieves through SDL’s RGB conversion.

Am I just heading for grief by going down this road?

YUV (+ CMYK) support in RISC OS 5 isn’t implemented yet. The sprite types have been allocated, but there’s no rendering code. So if you’re trying to render a YUV/YCbCr sprite then I think what will happen is that the OS will panic and treat it as a 32bpp RGB sprite, giving a mostly garbage result. I’m not really sure why the OS does that instead of returning an error, but I daren’t try removing that logic just yet!

I think there are two interdependent development paths which will lead to YUV support:

1. Hardware overlay support. This is blocked by needing some pretty major changes to memory management within the OS & video drivers. But those changes are in progress, and once they’re complete it hopefully shouldn’t be too much extra work to get overlays working, at least for OMAP & Pi machines. It’s also good to know that SDL/FFplay is designed around the use of overlays – I can have a go at updating it to use the new API once it’s ready for testing.
2. Software rendering. This is potentially blocked by the JPEG support bounty since that involves touching the same bits of code (SpriteExtend handles both sprites and JPEGs), and there’s already YUV code within SpriteExtend for the purpose of JPEG rendering (and hopefully phase 2 of the bounty will involve NEON optimising that code, saving me the trouble of coming up with a reasonable implementation myself). I’m aiming for the YUV sprite support to be a suitable fallback for situations where hardware YUV overlays aren’t available, so it’s important to make sure the OS has high performance routines (hopefully good enough that video players won’t have to worry about providing their own fallbacks).

So YUV support is coming, but it’s still a few months off I expect.

Jeffrey: Thanks for that situation update. I have done quite a bit of work on the SDL library as well as FFmpeg and OSMesa. I may be in a position to assist in some way, perhaps by implementing the SDL-to-YUV hardware interface, when we get to that point. I’d be interested for others to look at the ARM code I have in the SDL library for converting YUV to RGB; though I’d be happier still if that code became redundant.

A couple of things to consider: (1) SDL and FFmpeg strongly favour planar formats. Is that going to be good or bad for RISC OS? I fear it will be bad for YUV sprites. (2) Writing directly to YUV overlays will, I expect, be fastest. But will writing to YUV sprites be simpler to code and only slightly slower?

I’m just trying to think ahead, consider strategies.

(1) SDL and FFmpeg strongly favour planar formats. Is that going to be good or bad for RISC OS? I fear it will be bad for YUV sprites.

Initially I’ll be focusing on packed formats, as that looks like it’s the lowest common denominator in terms of hardware support, and is a bit easier to get working with sprites.

I’m not sure how widely supported planar formats are in hardware. OMAP4/5 definitely support NV12, and I’m fairly certain the Pi supports it as well, but I’m not sure about YV12. I can certainly imagine planar formats being a bit trickier to support in sprites, but if there’s a planar format which is optimal for software rendering then we should certainly try our best to make sure it’s supported, even if most machines do only support it for the software fallback case. As long as the API makes it clear which formats are going to be handled in software and which are going to be handled in hardware we shouldn’t have much to worry about.

(2) Writing directly to YUV overlays will, I expect, be fastest. But will writing to YUV sprites be simpler to code and only slightly slower?

I’m aiming for a solution where the same API is used whether the overlay is handled by software or hardware. Maybe there’ll have to be a couple of extra calls which are needed for software overlays (e.g. to actually trigger the draw to the screen), but for the most part it should be the same.

In terms of performance I’d expect writing to the software sprite to be fastest, since you’ll be dealing with regular cacheable memory. Hardware overlays I’m not sure what the best cache policy will be yet – I suspect it’ll need a bit of experimentation for each platform. But any slowness writing to the overlay should be more than offset by the fact that there’s no software YUV → RGB conversion going on in order to render it to screen!

I’ve been spending the past few days focusing on getting custom overlays working on OMAP3. All the major features are now working (RGB, YUV, alpha, colour-key transparency, scaling, rotation), and dealing with the various quirks of the hardware has helped me to refine the design for the GraphicsV API. Next step is likely to be OMAP4 (which should hopefully be pretty straightforward, since most of the changes to OMAPVideo were to shared code), followed by the Pi. Then after that I’ll be looking to come up with the design/implementation for the “VideoOverlay” module which is what apps will be talking to. I had a look at KinoAmp’s sources, and it looked like it would be pretty straightforward to add hardware overlay support to that, so chances are I’ll be using that to help plan & test the VideoOverlay module.

At some point the drivers + VideoOverlay module also need to find their way into CVS, although I’m not quite sure when that will be. The GraphicsV API should be pretty stable once the Pi video driver has been updated (maybe some minor tweaks needed later for VideoOverlay’s benefit), so that’s likely to be the first point at which I’d consider checking things in. But there are bound to be at least a few driver bugs that slip through my initial testing, plus there are some issues affecting the system as a whole (PCI heap blocking, and some issues with alpha-channel desktops when alpha blending is enabled). So it might be best to wait until 5.24 is out, then everyone can have a fun few months post-release dealing with both graphics bugs and multi-core bugs in 5.25 (once the SMP code gets merged).

(note to self: make sure VideoOverlay is reasonably MP-safe)

and multi-core bugs in 5.25

!!!!!..?

!!!!!..?

Can you be more specific? There are bound to be bugs there! :)

Every core attempting a file operation at the same time leads to FileCore crapping itself messily and then promptly dropping dead?
(^_^)

Seriously, though, might be worth reading the PRMs again to spot re-entrancy gotchas, as I think those, more than anything, will restrict what can and can’t be done on other cores.

Every core attempting a file operation at the same time leads to FileCore crapping itself messily and then promptly dropping dead?

Perhaps that’s something where the user processes have to go through a scheduler to access the bowels of the system that actually talk to a file storage device?

Yes I know that it moves you the programmer a step away from direct interaction with the guts of the system but with a multiprocessor/core setup which particular core did you want to be close to?