YUV image formats

I think I’m now at the point where the unsolved problem of “how to describe the different pixel formats” is the only blocker that’s preventing me from releasing the first version of the overlay code.

Long-term, I think a descriptor based approach makes sense. But it could end up taking quite a while to develop a descriptor-based system that’s easy to use and also capable of describing everything we need.

So I’m wondering if a suitable short/medium term approach would be to go with an enum-based approach which can be used to identify a set of common formats which most/all current hardware supports. I.e. just define a few new NColour/ModeFlags/Log2BPP values. An NColour/ModeFlags/Log2BPP triplet is essentially just another type of descriptor, so as long as the API is capable of supporting different descriptor types/versions there’s nothing stopping a full descriptor system being introduced in the future, with VideoOverlay handling mapping between the different types depending on which descriptor types each client supports.

If it’s not part of the sprite mode identifier ‘namespace’, then a 32bit word has quite a lot of room. It sounds like the fields are likely to be

• BPP
• colourspace/ncolour
• shape (subtype of the above)

I like the idea of an identifier that tells you something about the content, even if the exact definition is not recognised (too new).

Candidates for new VduVariables:

• Colourspace
• Data format (subtype of colourspace/depth)

“shape”?

Introducing new VDU/mode variables sounds like a good idea – it would remove the need for people to implement their own code to pull apart NColour/ModeFlags/Log2BPP values. (Mode variables would probably make more sense than VDU variables – that way OS_ReadModeVariable can be used to read the values for a given input mode/format)

In fact, I should probably change the overlay create/vet API so that instead of using a custom “overlay descriptor” struct it uses a mode selector block (or a mode specifier). It’ll add a bit more complexity to drivers/VideoOverlay, and not all properties will be applicable (e.g. framerate), but it should simplify use by programs due to the consistency with how screen modes are described.

“shape”?

I meant the various combinations of macrocells of different resolutions in YUV, for example. I’m not convinced that a generalised way of describing such geometries is much more useful than merely enumerating them – any code generalised enough to cope with an arbitrary geometric description is probably going to be extremely suboptimal. The likely use will be “can you support my preferred format, or shall I default to RGB?”.

Mode variables

Indeed, quite so.

overlay descriptor

A Mode selector is preferable to a sprite mode word yes. And your ‘overlay selector’ can reasonably be a subset of the same parameter space.

Is there any requirement for a mode selector to also specify the overlay, or can any overlay be enabled on any framebuffer? Is there any advantage to OS_ScreenMode doing it all?

I’m not convinced that a generalised way of describing such geometries is much more useful than merely enumerating them – any code generalised enough to cope with an arbitrary geometric description is probably going to be extremely suboptimal.

Yes, it will be difficult to support completely arbitrary formats in an optimal manner. It seems like something that a code generator would be well-suited for, but creating a good code generator isn’t easy either.

However, just because a format can be described, it doesn’t mean that it has to be supported. Programs can still elect to support a fixed list of formats. The only difference is that they’ll be using a verbose descriptor to describe the format, instead of an opaque enum value.

A Mode selector is preferable to a sprite mode word yes. And your ‘overlay selector’ can reasonably be a subset of the same parameter space.

While looking at it in a bit more detail last night, I realised that they’re two overlapping sets. Some properties which make sense for selecting screen modes don’t make sense for selecting overlays and vice-versa. And (unless we define a new mode selector block format) the only way of adding extra data to a mode selector is via mode variables – but I’m not sure if it makes sense for everything to be a mode variable.

Is there any requirement for a mode selector to also specify the overlay, or can any overlay be enabled on any framebuffer? Is there any advantage to OS_ScreenMode doing it all?

I think you’ve lost me a bit.

Overlays are created & destroyed on the fly; when changing screen mode there’s no way of specifying a list of overlays that you want to create. Such a feature would probably be useful for drivers (since hardware configurations that are performed during the mode change may affect the availability of certain overlay features), but at the moment I don’t think it’s necessary.

At the GraphicsV layer, when you create an overlay you specify what head(s) you want it to appear on. At the VideoOverlay layer (i.e. the interface that apps should use) you don’t get to choose which heads are used – the API is designed around overlay use within the desktop, so (once we have a proper multi-monitor API) VideoOverlay will automatically perform whatever actions are necessary to make sure the overlay can be freely moved from screen to screen, in the same way that windows will behave.

two overlapping sets

That’s how it appeared to me. Hence my question as to whether both would ever be found in the same selector. To which I think the answer is ‘no’. So overlapping works fine.

when you create an overlay you specify what head(s) you want it to appear on

Yes, it would be a shame if a video window straddling two screens was only animated on one. And I think I’ve seen something that worked like that, I’ve forgotten what.

I think I’m close to finding an acceptable way of extending mode selectors so that they can be used as overlay descriptors.

Framerate

The mode selector framerate field can simply be ignored by the overlay system

Screen banks

Overlay descriptors require you to specify how many buffers/banks you need, so that the driver can guarantee that enough memory is available. This is different to screen mode changes where there’s no way of specifying ahead of time how many screen banks you want (apart from OS_Byte 114 and the numbered mode shadow bit – but they only allow you to select between 1 or 2 banks). Also, once you’re in a mode, there’s no mode variable to allow you to read how many banks are available. ROL’s OS_ScreenMode 7 comes close, but by my reading of the docs it’s an over-estimate of the number of banks you’re going to get, which might not always be useful.

So I’m thinking that a “guaranteed minimum # screen banks” mode variable would make sense. During screen mode/overlay vetting & selection the variable will have the expected effect. And after mode selection is complete, it’ll still be useful as a mode variable (even if it duplicates some of the functionality of OS_ScreenMode 7).

Rotate/flip transform

Three bits are required to specify the rotate/flip transform used with overlays. Currently there’s no corresponding way of specifying a rotate/flip transform when changing screen mode, and my current plans for adding display rotation support are for that to mostly be handled via a persistent configuration setting rather than a per-mode setting.

But, these are for two different purposes. The configuration setting is because the user has rotated his monitor to its side. The per-overlay (or per-mode) setting is because the app that’s writing to the overlay/mode wants to avoid a slow software rotate/flip operation when dealing with certain rendering algorithms. So it’s reasonable that e.g. an emulator for some obscure computer system might want to use a full-screen mode which has its framebuffer rotated 90 degrees. Anything that writes to this buffer will also need to be aware of the transform, so that e.g. text rendered by the OS will still be correct. And this is different to the user-specified rotation configuration setting, where the user’s head (or monitor) will be rotated.

So there needs to be the user-specified configuration setting, and a per-mode/per-overlay setting. The configuration setting is only relevant to the video driver (so it can program the hardware accordinly) and ScreenModes (so it can enumerate the modes and convert to mode timings properly). Ordinary software doesn’t need to know or care that a transform is in use. Meanwhile, the per-mode/per-overlay setting will be relevant to anything which accesses the screen, so that it knows that the memory view of the image is in a different orientation to the user’s view of the image.

So a new mode variable, or three extra bits in ModeFlags, would seem like a good way of supporting per-mode/per-overlay rotate/flip transforms. During mode selection/overlay creation the driver/ScreenModes will combine this with the user-configured transform setting in order to arrive at the real transform that will be used for programming the hardware.

Scaling

For overlays there’s a single “must support scaling” flag as part of the overlay descriptor. Scale factors are specified separately, to allow them to be changed on the fly as the user adjusts video playback settings or to cope with aspect ratio changes in the movie.

At some point, we’ll probably be wanting something similar for screen modes – e.g. on-the-fly tweaking of scale & offset parameters to allow the user to adjust the display to combat overscan issues. But I’m not sure if we actually need a “must support scaling” flag when selecting screen modes, or if it makes sense to have a “supports scaling” flag once mode selection is complete.

It’s only a single flag bit that’s needed, so using one of the flag bits in the mode selector block header, or a ModeFlag, would probably be fine. But since scaling is closely related to aspect ratio (and the eigen values are wholly insufficient for describing the aspect ratio of an image), maybe some more thought is needed before deciding what to do with this.

screen banks mode variable

Neat, for both screens and overlays.

Presumably one can create additional overlays on demand for temporary augmentations, eg to indicate system volume changes?

Rotation/flip

Orientation makes perfect sense, both from an output POV (I’ve rotated my monitor but you can’t detect that) and from an application POV (this arcade game was designed for a portrait CRT). However, resolving those might have to be a configurable strategy. This is analogous to printers, where there is the orientation of the content (A4 Portrait), the orientation of the actual paper (A3 Landscape), and a policy to work out what to do (do not rotate; rotate to fit; match orientation; match orientation and scale). In some cases you might want to add the rotations, in others you might want to match them.

I don’t know whether an 800×600 Portrait ‘mode’ can be implemented as a 600×800 framebuffer overlay within a 1600×900 screen (or some other timing trickery), but it does seem that there ought to be some difference of strategy required by those that have a rotatable monitor and those that do not, as an example. Tablets don’t suffer that problem.

As for flipping, that’s less likely to be a choice (I’m reflecting my monitor in a mirror but you can’t detect that), so I don’t think a configurable strategy is required. It’s just a rendering effect.

Scaling

Again, different scaling strategies may be required if scaling is to be used to match desired mode dimensions to supported monitor formats – centre, scale, stretch, fill. Actually that’s a general point – given this overlay geometry and that video content, how should they be fitted when scaling is available?

Presumably one can create additional overlays on demand for temporary augmentations, eg to indicate system volume changes?

Yes.

In some cases you might want to add the rotations, in others you might want to match them.

Yes; I realised soon after making that post that for both the rotate/flip transform and the scaling we need to support both program-specified settings and user-specified settings. And there’ll be some cases where you want the user settings to affect rendering and other cases where you won’t.

My current thoughts about how this will work is that if a property is going to affect software which renders into a framebuffer (whether that’s the screen or an overlay), then it should be included in the mode specifier during the vet/select process, and be available as a mode variable or VDU variable during rendering. And to allow for the “sometimes you do, sometimes you don’t” nature of the user-specified settings, ideally the mode/VDU variables need to be updatable while rendering is taking place.

Properties which aren’t intended to affect rendering code (i.e. they’re only relevant to the presentation of the framebuffer) will be specified via other APIs – APIs that are specific to the type of framebuffer, e.g. VideoOverlay SWIs if it’s an overlay.

In terms of my previous post, this means we’d probably still want the rotate/flip flags in ModeFlags (and by extension, in the mode selector block). But we’d also want VideoOverlay_Create & VideoOverlay_Vet (and their GraphicsV equivalents) to have an extra flags word so that the “only affects presentation” (i.e. user-specified) rotate/flip flags can be specified. The “must support scaling” flag could probably go here as well, since the scaling will (mostly) only affect the presentation of the overlay, rather than how software renders into it. Replacing eigen values with something better can be left to a later date.