RGB Transfer Functions and CMYK, etc

Mapping functions are used with OS_SpriteOp and DrawFile_Render to process RGB before it is painted to the output target. In an all-RGB world, this is fine.

CMYK sprites raise a dilemma (or would if RO5 supported CMYK sprites). RO4 does support CMYK but uses the mapping function in exactly the same way – mapping RGB to RGB. It performs a terrible CMYK to RGB conversion first, and then passes the result to the function. This is a problem which could have been avoided with a better-defined API.

I have worked around that poor design in RO4 but it’s highly circumstantial code – it has to infer from the state of all the registers what kind of code SpriteExtend has built (which varies greatly by output mode), and therefore how to access the original CMYK. It’s fragile, and no doubt RO5 would be different again given half a chance.

So I propose an extension to the syntax of mapping functions to make them more flexible to avoid this API problem: bits 0 & 1 of the function pointer to encode the type of mapping function (that is, the pointer at +4 of the two-word mapping structure often referred to as “the mapping function”, which it isn’t).

Conventionally, SpriteExtend builds code that performs up to four steps:

1. Extract a pixel’s bits
2. Convert bits to RGB (possibly using the palette)
3. Call RGB mapping function
4. Paint result to output

Let’s call that a Type0 mapping function. There are two other places mapping functions would be useful: (only one at a time)

1. Extract a pixel’s bits
2. Call Type2 bit manipulation function
3. Convert bits to RGB or call Type1 colour converter function
4. Call Type0 RGB mapping function
5. Paint result to output

Type1 would be most useful for CMYK sprites – it would receive the raw CMYK pixel data, and be responsible for turning that into RGB (with any RGB processing that was required). For other sprite types it would perform (and replace) the action of the palette. [Note that when outputting to a CMYK sprite, it would be required to output CMYK, not RGB]

Type2 is less compelling, as it can only perform manipulations within the colour space of the sprite. That may still be useful, but I’ll come back to this one at the end.

This functionality would be valuable and easy enough to add to SpriteExtend, but isn’t sufficient for DrawFile – it applies the same mapping function to all sprites it finds regardless of source mode, and also to its text and path colours which are <waves hands equivocally> RGB. So Type1 and Type2 functions would be useless for DrawFile except for the most contrived of cases.

Instead, Type3 would be a pointer to a function table containing mapping functions per sprite type. eg:

+0  4  LDR PC,[PC] (or MOV PC,R14 if default is null)
+4  1  1 = format (of this table)
+5  1  n = number of entries (beyond the mandatory default)
+6  1  0 = _flags_
+7  1  0 = _reserved_
+8  4  default mapping function (which must be a Type0 or null)
+12 4  mapping function for 1bpp
+16 4  mapping function for 2bpp
... etc

This would allow SpriteExtend (and by extension, DrawFile) to select a mapping function by sprite type. This may seem over-specified, but there are certainly cases where one would want to do different things for CMYK, 24b, 15b and paletted sprites – and I wouldn’t want to restrict the option of different mappers for 2 and 4 colour (say) just for the sake of a few words in a single structure. DrawFile would just use the default mapper for its text/path colours.

Note that the initial LDR/MOV makes a Type3 behave exactly like a Type0 for the benefit of DrawFile, which would not have to be modified at all. All functions would use the same R12 from +0 of the two word mapping structure.

Final thought

I’ve long thought that mapping functions missed an obvious abstraction that would have been even more useful, and this is what I would like to use Type2 for – total abstraction of the pixel data. A function that maps not from RGB or pixel value, but from coordinates and sprite pointer. Given that SpriteExtend (sometimes) needs the width, height and mode of the sprite in order to calculate its extent and scaling, the sprite header must be as defined, but if SpriteExtend is going to call a user-supplied function for every pixel, why not (optionally) allow that function to supply the pixel data in the first place?

There are many scenarios that this would enable, including support for new types; tiling; 9-patch scaling; on-the-fly decompression; procedural content such as grads; and so on. It would require a more disciplined register allocation from SpriteExtend (no bad thing), which would also benefit from a rework of its scaling code if it’s still anything like RO4’s (which slavishly extracts a pixel, converts to RGB, and calls the colour mapping function, for every source pixel on the line regardless of how few output pixels are actually written… which surely can’t have been the intention).

P.S.

It’s also a shame that mapping functions can’t return -1 to mean ‘none’, which has useful applications for both SpriteOp and DrawFile (though the latter is easier). Perhaps the Type3 would enable a flag that switched on that API – this would require a change to DrawFile though. In the case of SpriteExtend there would be a small performance impact in some modes, but no different to that of using a masked sprite.

Conventionally, SpriteExtend builds code that performs up to four steps:

Remember that there’s also masking/alpha blending and logical operations (GCOL actions) to deal with. Arguably we should support a custom function for that stage as well.

So I propose an extension to the syntax of mapping functions to make them more flexible to avoid this API problem: bits 0 & 1 of the function pointer to encode the type of mapping function (that is, the pointer at +4 of the two-word mapping structure often referred to as “the mapping function”, which it isn’t).

Why use the low bits of the pointer as flag bits? Type 1 & 2 are used at different places in the pipeline than the current type 0, and presumably there are situations where you’d want to have multiple functions active at once. So forcing the programmer to pick one or the other to pass in seems like a bad choice. Type 3 is the only one that fits into the pipeline at the same place as type 0, and code can easily determine which is which by examining the first couple of code words.

There are many scenarios that this would enable, including support for new types; tiling; 9-patch scaling; on-the-fly decompression; procedural content such as grads; and so on.

How many of these would be properties of the sprite (e.g. compression) vs. rendering effects (e.g. warping coordinates to create a wavy image)? Sprites are an image interchange format; a program should be able to pass a sprite to any API which accepts arbitrary sprites and have it work correctly. So we’d need a system-wide way of registering support for new sprite format handlers at runtime, so that sprite properties like compression can be dealt with automatically instead of requiring code to manually pass the function pointer through all the different API layers.

So we’d need a system-wide way of registering support for new sprite format handlers at runtime,

It’s just a few more steps from where you are to having the ability to register an image format handler, so RISC OS can gain the ability to load and display BMP, GIF, PNG, etc etc once somebody writes appropriate handlers.

Sprites are an image interchange format

A subset are, but it’s certainly possible to construct sprites in memory that cannot exist in the file format. Some of these suggestions were of that form.

we’d need a system-wide way of registering support for new sprite format handlers at runtime

Well that already exists. On the one hand there’s already enough of an abstraction through SpriteV… but on the other hand much software groks the sprite format to some extent – eg who calls OS_SpriteOp,40 to read sprite mode when you can just peek the word at +40? !Paint is fairly well behaved, but other things delve right in there.

There are also inherent restrictions on some existing SOPs that would make complete abstraction impossible – for example, code understands that SOP57 will need more memory to insert rows, but nothing will expect SOP42 (write pixel) to fail due to lack of memory… so compressed sprites would have to be read-only.

The upshot is that no matter what you do, some programs wouldn’t notice if you made a massive change (e.g. my “PNGlu” PNGS-in-spritefile wheeze), while others fall over if you so much as include an extension area (which is part of the official format dammit). Sprites are venerable and ancient and OS support especially regarding colour has varied over the years, so there’s a lot of different usage patterns. So I was talking mainly about runtime rendering tricks, rather than file format stuff.

Arguably we should support a custom function for [output].

The problem with using a mapping function for my ‘9-patch’ or your ‘wavy’ examples is that it would be inherently limited to the dimensions of the sprite. I intended that the sprite in this sense was an artificial construction that would result in the desired coverage (and again, I stress, not something you could put in a file) rather than the source sprite, but this is not terribly easy to calculate in extremis (transformed source-rect plotting for example) and suffers from the absence of transparency in the mapping API, as mentioned.

Having implemented both ROP16s and ROP256s in code and hardware and all the Adobe blend modes, I’m well aware there’s more operations that one could apply than AND and EOR! That’s a separate question though; literally a different stage of the pipeline and not limited to OS_SpriteOp.

[Aside: Also the RISC OS clipping problem continues to restrict – OS_Plot, OS_SpriteOp and Font_Paint should be as clippable as Draw now is. I implemented Draw clipping a long time ago by intercepting HLine (a dangerous pursuit) which ought to have affected OS_Plot too but doesn’t because it calls the routine directly instead of indirecting as Draw does. It was not intended to be a replaceable interface, sadly.]

To backtrack and rethink key points:

The CMYK colour mapping function as implemented in RO4/6 is very silly. My work-around is horrific, and I wouldn’t want to have to repeat or further specialise it for RO5.

The ability to specify more than one mapping function for DrawFile is useful anyway, and probably necessary for the CMYK case. The backwards-compatible method I suggested (Type3) does not even require a change to DrawFile.

The ability to leverage the existing SpriteExtend output stage via a content provider (a shader, in effect) would be really useful for enabling compression, new pixel layouts and procedural content. Whether such a construction could be put in a file is an orthogonal question. This would be a half-way house between rendering to a (huge) sprite, and having to implement all the fiddly pixel modes that obscure ARM platforms support – code could provide 24b/32b pixels on demand and allow SpriteExtend to get that onto the screen/target. If SpriteExtend gained system-wide clipping, even better.

However, rendering tricks such as non-linear scaling (9-patch), tiling (I have code that tiles one axis while scaling the other), distortion (wavy) and so forth perhaps ought to be an entirely new call rather than overloading an existing SpriteOp, if only to avoid the need to build a fake sprite to circuitously define the effective bounding box.

The ability to return ‘-1=none’ from a mapping function would be genuinely useful, both for separation of DrawFiles and chromakeying of sprites, but can’t be imposed on the API as it exists, which has always been -1=white, and doesn’t extend to the CMYK-to-CMYK case.

RISC OS can gain the ability to load and display BMP, GIF, PNG, etc etc once somebody writes appropriate handlers.

RO4 has had that for a long time but it’s mostly bogus. PNG rendering is very useful indeed:

• Documented gamma
• Well-defined alpha
• Great compression

But RO4 can also cope with ICO, Clear, PCX, BMP, PNM, Sun and XBM… and frankly that’s silly.

RO4 has had that for a long time but it’s mostly bogus.

When you say RO4 are you referring to RO4.3x or have I forgotten about a feature of RO4.02?
Sometimes the.xx is important – like Windows 3.0 being pretty crap and 3.11 being a bit better and actually being usable.

and frankly that’s silly.

Why? Granted some of them aren’t commonplace, but I do have PCXs and BMPs around on older hardware. It’d be nice to open them in Paint without ChangeFSI’s over processing.
So I see nothing wrong with having some filters to load various image formats.

who calls OS_SpriteOp,40 to read sprite mode when you can just peek the word at +40?

Anybody who bothers to follow the API, that’s who.

The only time I’ve ever disobeyed the SpriteOp API was the init area call in one program, and that was after reading through the OS source to discover that it doesn’t actually do anything… But if that should change in the future, I’ll admit that I’m wrong.

so compressed sprites would have to be read-only.

A problem here is that there isn’t a call to read the amount of memory needed by a sprite that doesn’t yet exist. It’s up to you to work out the size, colour depth, headers, palette, blah blah, add that all together and hope for the best, while not forgetting “wastage”…

If there’s no API to get the size, there’s nothing that can lie to ensure there will be enough space to modify a compressed image.

Steve asked

When you say RO4 are you referring to RO4.3x

ImageFileRender was first released in 2002. That counts as “a long time” even in RISC OS terms.

So I see nothing wrong with having some filters to load various image formats.

Neither do I, but that isn’t what you said and what I described as silly. You said:

so RISC OS can…display BMP, GIF, PNG

I was pointing out that there’s no point having code to display BMP, for example, when you already have code to convert BMP to Sprite (ImageFileConvert). PNG rendering has merits for the reasons I gave, but all those other formats have no benefit to justify the overhead of conversion-during-rendering which is what ImageFileRender does with those formats. It’s the wrong thing to do.

It’d be nice to open them in Paint

ImageFileConvert, not ImageFileRender. These are different things.

here’s no API to get the size

Indeed.

Not unrelated: I was just combining some sprite code from the Vantage renderer with some more recent stuff and produced an updated index of all the different mask and content handlers to render sprites: 29 different types of mask and 37 different types of content. And that doesn’t include fancy stuff like colour management, separation or simulation (as the Vantage renderer has). Now although you can’t have every kind of mask with every kind of content, there’s still an awful lot of sprite formats, which is why SpriteExtend has to build code for the sprite in question, and Vantage uses a multistep modular rendering pipeline.

ImageFileRender was first released in 2002. That counts as “a long time” even in RISC OS terms.

My point wasn’t the length of time but rather the specific version being referenced. RO4 is rather broad brush and I was trying to ascertain whether the feature was there in 4.02 and I’d missed it.

RO4.02 is somewhat lacking in various feature areas, where RO4.3x is rather better blessed.

In the same fashion there are features of RO5.27 that simply don’t exist in RO5.19 never mind earlier releases. The original RO5 only had USB1 support and USB2 came as paid update, but now it’s standard code.
So, the specific version is important – which I seem to have said earlier.

but all those other formats have no benefit to justify the overhead of conversion-during-rendering […] It’s the wrong thing to do.

That probably depends on whether you prefer to render from the source, or to convert it once and render from a conversion.

In favour of conversion – for some formats it’s quicker. You’ll need to try to convince me in the case of BMP (which is basically an upside down sprite) but certainly back in the day it was notably laggy having JPEG backdrops on a slow machine. Less of an issue these days.

In favour of rendering on the fly – what you see is what you get. If it’s a 250K JPEG, it’s a 250K JPEG and not a massive sprite. An example here is Windows. XP, at least, makes a bitmap copy of the chosen backdrop image, and it must surely contain several images (at different colour depths) because it, which may be from a JPEG as small as 300K, generates a backdrop image of around 16MiB. Which is going to be always loaded in memory (so it can be redrawn quickly).

As for “the wrong thing to do”, there’s no “one option fits all use cases”.

ImageFileConvert, not ImageFileRender. These are different things.

I have no idea, I don’t have RISC OS 4.xx. I’m simply thinking that it’s not far away from the hooks for extra sprite formats mentioned earlier to having hooks for extra image formats.
Or to put it another way, why can’t RISC OS (our version) natively load things like BMP, GIF, etc? Not via a third party app, but natively, so everything can benefit.
Your choice right now is pretty much sprite or JPEG, and only one of those is known outside of RISC OS.

there’s still an awful lot of sprite formats

Indeed it is. But don’t forget the…uh…API quirk that old sprites have a mode number, and extension modes can offer extra numbers (with numbering essentially picked at random), and without the extension mode active and running (most probably won’t on VIDC2 or later!), it’s anybody’s guess as to what that mode actually was…and since that is what determines the colour depth (and likewise how to interpret the sizes).
One can guess, given we know the height in pixels and the bytesize of the image to try to work out what BPP would match the image width (given in words, not pixels), but this is a lot of bother isn’t it? ;-)

So, yeah, making sense of the sprite format indicator, involved bordering on convoluted.

why can’t RISC OS (our version) natively load things like BMP, GIF, etc? Not via a third party app, but natively, so everything can benefit.

Because you haven’t got ImageFileConvert (which, despite its name, is not limited to bitimages).

making sense of the sprite format indicator, involved bordering on convoluted.

You could say that, yes:

Cerilica VVS? The bazillion other sprite formats not good enough for ya? ;-)

Because you haven’t got ImageFileConvert

And we most likely won’t, so there are two options:

1. Think of a way to implement alien-format image loading
2. Do nothing

Option 2 is easier, but not so future friendly.

not good enough for ya?

Not when you need eight channels in a sprite, no. Comes out weird if you rely on SpriteExtend of course…

2. Do nothing

ImageFileConvert is absolutely not big or complicated. It’s a very simple interface. Of course you’d then have to write some converters, but “build it and they will come” no?