Proposed GraphicsV enhancements (Rev #12, changes)

GraphicsV BeginPointerUpdate

This call is used to allocate a shape buffer. The stride value returned in R3 is the number of bytes from the start of one row to the start of the next. The caller is expected to fill in the buffer with data in the format requested, which is not guaranteed to match the format that was returned from an easlier call to GraphicsV PointerFeatures. Any pixels/bytes which lie outside the image (for if the stride in R3 is larger than width*bpp) must not be modified.

GraphicsV EndPointerUpdate

This call is used to return a buffer to the driver once it has been filled in. The active point of the image is also specified (as per OS_Word 21_0).

If the caller was updating the shape data for the image currently being displayed, it is the driver’s responsibility to ensure that the changes are buffered correctly such that there is an instantaneous transition from the old shape to the new; i.e. buffers which are currently being written to must not be displayed.

GraphicsV SetPointer

This call is used to select the displayed shape number and/or move the image on the screen.

GraphicsV RemovePointer

This call is used to hide the pointer.

Other changes

• Exact OS_SpriteOp and OS_Word changes are TBD.
• OS_Word 21 API is likely to remain as-is. However the implementation will need changing to add support for the new interface, including drivers which elect not to support 2bpp images and instead only support true-colour ones. Therefore conversion code will need adding to the kernel, or the kernel may elect to rely on OS_SpriteOp 36 for updating the image.
• OS_SpriteOp 36 is a prime candidate for automatically using true-colour/alpha-blending functionality when supplied a suitable sprite, if the hardware supports it. However additional flags to control behaviour may be desired. It’s expected that it will call GraphicsV internally in order to set up the new shape data (falling back to OS_Word 21 if support for the new GraphicsV pointer API is unavailable).
• VIDC20Video: Migrate shape image buffering code out of kernel and into the driver. Update to allow for taller images, up to at least 64 pixels.
• BCMVideo, OMAPVideo, NVidia: Add support for true-colour images, at least 64×64. Ensure changes are buffered appropriately.
• Kernel: Ideally we’d remove the code for dealing with the cursor image buffers. This would also free up some workspace. However the code may be required in order to support old drivers that don’t support the new API (e.g. if Aemulor or Geminus are in use and are hiding the true capabilities of a driver). The buffering code could potentially be split off into a support module, creating a compatibility layer to map old calls to new and vice-versa.

Potential future enhancements

Allow the scale of the pointer image to be specified.

Improved screen memory management

Overview

In order to fully support multiple active drivers, heads, and overlays, we need to overhaul how screen memory is managed.

At the moment the OS is in charge of memory management for the current driver:

• If the driver does not have its own framestore, the OS will use the screen dynamic area (DA 2) for screen memory. Internally this allocates from video-capable system memory that was identified by the HAL on startup. The dynamic area is doubly-mapped in order to support VIDC-style hardware scrolling. Given the current implementation, only one entity (driver, head, overlay) can sensibly use DA 2 at a time.
• If the driver has its own framestore, the driver will communicate its size and physical address to the kernel. The kernel will then map in this area using OS_Memory 13 and use it with the VDU drivers. Although this interface allows the driver to dictate where the buffer is located, the kernel is still in charge of things such as where each individual screen bank is located within that buffer.

Proposed changes

• Implement ROL’s OS_ScreenMode reason codes 7-10 (or something like them) to abstract screen banks away from memory layout and DA 2.
• Migrate screen memory management out of the kernel and into the drivers; it will be the driver’s responsibility to map in and manage its screen memory.
• GraphicsV APIs which previously used physical memory addresses will be updated to either use logical addresses, or perhaps offsets relative to the start of the driver’s buffer. This will allow the easy creation of drivers which do not require contiguous physical memory (e.g. software emulation of low colour modes, a VNC server implemented as a GraphicsV driver, USB displaylink devices, etc.)
• Generalise the code for managing DA 2 in order to allow other DA 2-like dynamic areas to be created. For example to support multiple concurrently active drivers which support VIDC-style screen scrolling.
• Within the kernel create a new dynamic area which accesses system VRAM – similar to DA 2, but singly mapped instead of doubly mapped. Drivers will be able to request blocks of memory from this dynamic area, for purposes such as display buffers (e.g. if the driver doesn’t support VIDC-style hardware scrolling and therefore doesn’t need doubly-mapped memory), or for hardware overlays.
• To avoid physical memory fragmentation, the kernel is at liberty to compact any of the VRAM dynamic areas (DA 2, DA 2-like DAs, or the singly-mapped DA) as it sees fit. A basic implementation of compacting the area after any block is freed may suffice. Drivers and clients will need to be notified of any buffers which are moved by compaction – whether they’ve moved in logical space, physical space, or both spaces.
• Add a new API (perhaps a new kind of dynamic area) which allows for physical pages to be claimed but no logical address space allocated to them. This will assist in the implementation of display rotation support for OMAP, where a private block of contiguous physical memory is required to store the frame buffer but the buffer must not be directly accessed by the CPU.
• Consideration is required for how these changes will impact software that provides multiple monitor support in the desktop; by putting too much control of screen memory into the hands of the drivers, it may become hard/impossible to set up screen memory in a way which makes multiple monitor support trivial from the OS’s perspective. I.e. it may be hard to conjoin framebuffers such that the OS can treat the two displays as one.

Potential future enhancements

• Cacheable screen memory

The OS will issue this call when it begins to use a head for the VDU driver. The driver should use this call as a hint that it should give this head preferential treatment over other heads, e.g. ensuring a hardware cursor overlay is available.

GraphicsV ReleaseOS

The counterpart to ClaimOS, this call will be issued when the OS stops using a head.

Other changes

The display manager and screen setup plugin will need updating to allow selection of the head to use. Rather than end up with two new drop downs (one for driver, one for head) it’ll probably make sense to combine them into one list.

Similar to with multiple driver support, consideration will be needed for how the default selection is stored in CMOS.

As most machines will be able to support multiple heads out of the box, display detection support will be required in order to ensure the OS picks the right head on startup.

Potential future enhancements

Once multiple head support is implemented it’s worth considering implementing a Geminus-style app to allow for multiple monitor use within the desktop. There are, broadly speaking, four ways of providing multiple monitor support, each with differing level of implementation complexity:

1. For drivers which are flexible in terms of their memory allocation (NVidia, OMAP) it should be possible to specify a memory layout where two or more framebuffers are placed side-by-side to create one framebuffer at the physical memory level. The OS will see this conjoined framebuffer, while the driver(s) will see two framebuffers with large stride values (specified using the ‘extra bytes’ control list item. However this technique will only work if the same pixel format is available across all heads. It’s expected that this approach would be implemented by creating a GraphicsV driver which acts as a man in the middle, so that the OS only directly talks to one driver bu that driver organises things with the other two.
2. For drivers with inflexible memory, allocate the driver framebuffers as normal, but restrict the line length to a multiple of the page size. Then utilise the MMU to create a conjoined logical mapping of the two buffers. This will require the driver to have greater control over memory management than is currently possible, so the “improved memory management” task will need implementing first. As above, this approach is expected to be implemented in the form of a man-in-the-middle GraphicsV driver, and is also subject to the pixel format limitations.
3. For other situations, implement a GraphicsV driver which allocates its own framebuffer from ordinary RAM, marks it as read-only, and uses an abort handler to trap writes. Then at regular intervals copy the contents of any changed pages to the buffers which are managed by the real drivers. This approach will allow any size or configuration of screen geometry to be used, and can also allow for transformation between different pixel formats (within reason – the driver would probably want to ensure the hardware isn’t using a lower colour depth than the user requested). However performace will be worse than the other approaches.
4. And finally there’s the most complex approach of them all – update the wimp to have full knowledge of multiple displays.

Support for hardware overlays

Overview

The primary goal is to provide a fully-featured API to allow hardware overlays (particularly YUV) to easily be used by user applications, e.g. movie players. The API must allow for both windowed and full-screen use.

Other, more complex use cases are not to be considered at this stage.

To allow for multiple monitor setups, the API is designed to allow for one overlay to appear on multiple displays, at different positions and scales. It is the driver’s responsibility to provide support for this feature as best it can.

Proposed changes

On some hardware (e.g. Raspberry Pi) we do not have direct access to the overlay hardware. We can only use a resource-based API which allows overlays to be allocated to us and their contents updated. Memory management is 100% under the control of the GPU. Due to this it’s proposed that the GraphicsV overlay interface uses a similar resource-based approach. Specifically, it’s expected that the following calls will be required:

GraphicsV CreateOverlay

In:

• Width, height, pixel format, rotation, transparency mode specified
• Usage pattern specified (e.g. write-only or read-write – to help driver decide on cacheability, RAM vs. VRAM, etc.)
• Flag for whether single or multiple buffers required (i.e. for flicker-free updates)
• Driver number specified in R4
• List of heads specified which the overlay is to be visible on
• Other flags as required – e.g. whether scaling may be required

Out:

• Properties updated to reflect the capabilities of the overlay that was created?
• Overlay ID returned
• Error returned if overlay can’t be created (unsupported features, no memory, etc.)

This call is used to allocate an overlay resource. The overlay is initially hidden and must be explicitly shown using the SetOverlayPosition call.

GraphicsV DestroyOverlay

In:

• Driver number specified in R4
• Overlay number specified

Out:

• All registers preserved

This call is used to free an overlay resource

GraphicsV SetOverlayPosition

In:

• Driver number specified in R4
• List of position information, one entry per head:
  • Visibility flag (on/off)
  • X, Y coords of position on head and scale factor
    • Potentially allow two ways of specifying position & scale – either as coords and scale factor or as a rectangle (with scale implicitly calculated from rect size)
  • Subrectangle of overlay to display
  • Depth value for Z sorting
  • Alpha level

Out:

• Position information updated with actual state

This call is used to update the position information of an overlay, specifying the position across all heads at once to allow the driver to reconfigure the overlay in the most efficient manner. On exit the actual used position information will be returned, to e.g. allow a window to resize itself correctly to avoid any gaps around the edge of the overlay if the driver wasn’t able to provide the exact scale factor requested.

GraphicsV LockOverlayBuffer

In:

• Driver number and overlay number specified in R4
• Usage pattern specified (R, W, RW – to allow driver to decide on cacheability, etc.)

Out:

• Pointer to buffer and row stride

This call is used to lock a buffer for update by the CPU. An overlay can only be locked by one user at a time; a subsequent lock attempt without an unlock will return an error.

GraphicsV UnlockOverlayBuffer

In:

• Driver number and overlay number specified in R4

Out:

• All registers preserved

This call unlocks an overlay buffer. However, in a multi-buffered overlay it does not schedule the new contents to be displayed.

GraphicsV FlushOverlayBuffer

In:

• Driver number and overlay number specified in R4
• Flag for memory persistency (true: ensure data is preserved ready for next lock request. false: next lock request can return uninitialised buffer)
• Flag for synchronisation? (true: block until buffer is visible, false: don’t block)

Out:

• All registers preserved

In a multi-buffered overlay this call will submit the updated buffer for display by the GPU. Even in a single-buffered overlay this call is important, as the driver may use it to e.g. flush data from the CPU cache.

Other GraphicsV calls

Calls will be needed for setting the palette and gamma tables used by the overlay. For the palette it makes sense to reuse the existing palette calls; therefore it’s proposed that overlay IDs are allocated by drivers such that they do not intersect the range of IDs used for heads. This will allow calls such as the palette ones to accept either a head ID or an overlay ID in bits 16-23 of R4.

On some hardware it’s possible to directly specify the YUV → RGB conversion matrix. Allowing this matrix to be specified at overlay creation time may be advantageous.

It may also be desireable to have the following calls available:

• A call to allow the active overlays and their state to be enumerated
• Calls to query the systems capabilities (e.g. get a list of available pixel formats)
• Allow GraphicsV_Render to be used on overlays

User-facing API

User software is to be discouraged from using the GraphicsV API directly. Instead, particularly for software running in the desktop, it’s expected that another API is to be used. Specifically the API used for the desktop must allow overlays to be associated with windows, such that the OS can automatically calculate the depth values of the overlays in response to the windows moving through the window stack. The overlay position will also be updated automatically as the windows are moved and scrolled, and the scale factor could be linked to the work area size.

A SWI call is also required to allow a window (or any other portion of the screen) to be filled with the correct colour to allow the overlay to be visible through it. For example to clear the alpha channel to zero or to write the correct colour value for systems which use a transparency colour key. For a software emulation of overlays this SWI could instead be used to directly plot the software overlay to the screen (e.g. use OS_SpriteOp calls to plot a YUV sprite).

For full-screen applications another API may be required, perhaps in the form of new OS_ScreenMode reason codes. To reduce code duplication it may make sense for user software (whether single or multi-tasking) to use OS_ScreenMode to create and update the contents of overlays. For multi-tasking software a Wimp SWI could be used to associate the overlay with a window, at which point the Wimp will automatically make SetOverlayPosition calls as appropriate. The Wimp could also be made to automatically destroy the overlay when the application dies.

Other notes

Some systems (e.g. OMAP) only have a handful of overlays available, and their abilities are further restricted by the fact that the desktop and mouse cursor are overlays as well. This means that that the overlays available can vary wildly by screen more or active head(s). Therefore it is permitted for drivers to destroy any overlays they wish when switching screen modes, or when the ClaimOS call is received. A service call or some other notification method will be required in order to allow drivers to communicate overlay destruction to the owner of said overlay – exact details TBD.

Potential future improvements

Improve screen output redirection support so that screen output can be redirected to overlay memory buffers. This may be a fairly simple API change, as redirecting output to an arbitrary memory buffer shouldn’t be much different to redirecting to a sprite – we just need the ability to use a buffer which doesn’t have a sprite header attached. The ability to redirect output to an arbitrary buffer will also ease the implementation of the new pointer API (at the moment, OS_SpriteOp 36 creates the pointer image by redirecting output to a 2bpp sprite. For the new interface, redirecting output directly to the pointer memory buffer would be desireable in order to avoid temporary memory allocations)

Add support for plotting of YUV sprites to SpriteExtend, and add a software emulation layer to the user-facing overlay API, so that software can elect to use software emulation of overlays if a suitable hardware overlay is unavailable.

Support for display detection

Overview

Provide a method to query whether a display is currently connected to a head. Provide a method to allow connection or disconnection of displays to be automatically detected and communicated to the system, e.g. via service call.

Proposed changes

Polling for connected displays

Add a new GraphicsV call to perform on-demand display detection:

• As input, the caller will provide a list of heads which he wants to check, and a buffer to place the results in
• As output, the driver will fill in the buffer with a sorted list of entries. The driver will sort the list by preference, such that the display which is most suitable for use as the default display will be first. This will allow the OS to easily decide which display to use on startup.
• Each list item will be as follows:
  • The head number
  • The connection state
• Different connection states are likely to include:
  • Connected
  • Disconnected
  • Detection not supported
  • Additional states may also be required, e.g. detection not supported due to driver being in power-saving mode

Note that under the current multi-head specification, the S-Video and Composite outputs of the Pandora are to be treated as separate heads. Display detection for these outputs is supported, but the hardware is incapable of determining which of the two outputs the display is connected to. It may therefore be desireable to be able to communicate these limitations to the caller in the result buffer, e.g. listing the connection state of multiple heads at once.

Automatic display detection

API TBD. Care needs to be taken with power saving; e.g. TV-out detection on OMAP is only possible if the TV-out circuitry is on and generating a video signal. Leaving this running all the time could place an unnecessary drain on the battery of portable systems. Therefore it’s suggested that automatic detection is performed either entirely at the discretion of the driver, or a method is available to enable/disable detection at the user’s request (i.e. discourage the development of software which enables automatic detection for long periods of time without the user’s permission). The service call which is used to notify the system of connection state changes can also be used to notify the system of automatic detection starting/stopping.

Other changes

The kernel should make use of the display detection functionality to decide which display to use on startup. The display manager and screen setup plugins can be updated to display the connection state of each display, potentially filtering out or listing separately any outputs for which no display is detected.

Support for display scaling and rotation

Overview

The overlay API will allow for scaling and rotation of overlays. However an API to allow scaling and rotation of the desktop (and implicitly the mouse pointer) is desireable.

Proposed changes

New feature flags (e.g. as part of GraphicsV 8) will be required to allow drivers to advertise that they support the feature.

Add several new VIDC control list items to allow the framebuffer size and rotation to be specified. The timing parameters of the VIDC list will remain as-is, specifying the actual timings to be used when generating the video signal.

Control list items will also be required to specify the method of scaling – i.e. the coordinates of the rectangle (in physical pixels) in which the frame buffer is to be scaled to fit.

New mode string attributes can be added to allow the user to select scaling and rotation when selecting screen modes. How this maps to mode specifier blocks is TBD – new mode variables, mode flags, or a new mode specifier block format may be required. The display manager and screen setup plugin will also need updating to allow selection of rotation.

How overlay positions are specified when dealing with a scaled/rotated desktop is TBD; are they specified relative to the desktop overlay, or are they specified relative to physical monitor pixels? Specifying relative to the desktop would be the most appropriate for most software, but specifying relative to physical pixels may be desireable for some use cases. A flag at the time of overlay creation could potentially be used to control the coordinate space that it used, and for completeness it may make sense to allow any arbitrary head/overlay to be used as the parent. Care may also be required with respect to rotation; what if the user wants an overlay to be rotated on one display, but unrotated on the other?

Potential future enhancements

If a driver supports scaling, allow the system to automatically scale the display, to allow almost any mode to be used without requiring it to be listed in the MDF – i.e. as per AnyMode

Mode vet feedback

Overview

Improve or replace the vet mode call, to allow drivers to provide feedback on why certain modes have been rejected. Feedback should be designed such that software can interpret it and make an informed decision on how to adjust the mode timings in order to arrive at a mode which is supported. E.g. the call could return a series of flags of the form “pixel rate too high”, “complex constraint between vertical sync with and vertical front porch”, etc. This feedback would be particularly useful when generating a set of screen modes from EDID.

A call should also be available to query which control list items a driver understands, and drivers should be updated to ensure they reject any VIDC list containing items which they don’t claim to understand.

Proposed changes

TBD.

Other

Analogue video formats

PAL/NTSC/etc. encoding and sending of a widescreen selection signal could be specified by VIDC control list item. See this thread for some related discussion.

Interlace handling

It’s recommended that the GraphicsV 3 (set interlace) be deprecated, and that interlace only be specified via the flags in the VIDC list during mode changes. See this post for related discussion.

TV offsets and overscan

TV offset (as controlled by *TV) and overscan settings (for any kind of interface – TV-out, HDMI etc.) need to be taken into account. At the moment the kernel modifies the VIDC list in order to apply the *TV setting, but it’s probably something that’s better done inside the driver. E.g. for the Raspberry Pi the overscan settings can be directly specified to the GPU, on OMAP they could be used to control the offset and scale of the desktop overlay, etc. Being able to configure these settings in the screen setup plugin would be nice.

Driver-specific configuration

The screen setup plugin could do with expanding to add support for driver-specific settings to be configured. E.g. overscan (as mentioned above), PAL/NTSC mode, compatibility options, red/blue swapping on Iyonix, etc.

Having a “configure” option in the display manager which launches the screen setup plugin for easy configuration of driver could also be nice.

Gamma

Gamma handling needs revising; at the moment the GraphicsV palette calls handle both the palette and the gamma table, with no clear indication as to whether the gamma should apply to just the desktop overlay or the pointer (and any other overlays) as well.

Splitting per-head gamma handling into a separate call would be best, along with potentially allowing gamma to be specified on a per-overlay basis, with the palette calls only used to set the palette.

EDID reading

GraphicsV 14 (IIC op) may need revising to make it clear how to access the extra banks of EDID that are present in some devices. If GraphcisV 14 is seen as a direct IIC interface then it will require the caller to manually program the bank prior to reading the EDID. However this could cause problems in threaded environments, as unlike OS_IICOp there’s no way of performing a repeated-start transfer to keep the bus locked for the duration of the operation.

Additionally the current implementation in BCMVideo doesn’t emulate the bank address register and instead allows the full > 256 bytes of data to be read from the main EDID IIC address.

Power controls

At the moment there’s no way of turning off a display once an OS/program has finished with it. The ClaimOS and ReleaseOS calls will help with this to a certain extent, but for situations where third party software is using a display separate calls to control power will be required.

Display ownership

Related to the above, an API is required to allow heads to be claimed by third-party software. Each head will have three states of ownership:

v3.00 26-Jan-2014

• Rewritten to reflect current development status and my plans for other areas