Paint won't accept a palette file

When editing a sprite that is in RISC OS 3.5 or RISC OS 5 format (i.e. the mode word doesn’t contain an old-style mode number) Paint will reject any palette file dragged to the sprite window. The error “Palette file does not match sprite palette” is produced even when a 16-colour palette is dragged to a 16-colour sprite. This bug has been around since at least RISC OS 3.7, so it’s not a recent thing.

To demonstrate the problem:

• Create a sprite with 16 colours, 180×180DPI, colour palette.
• With the sprite window open, choose “Save > Palette”.
• Drag the palette file you just saved back to the sprite window.

Partial workaround: For 16-colour sprites destined for a “!Sprites11” file, it is possible to create the sprite in mode 20 or 27, add the palette file, then hex-edit the sprite file to change the mode word. This only works for sprites without a mask, so if you want a mask you’ll have to add it after changing the mode word.

On a somewhat-unrelated subject: Is it really a good idea to still allow double-clicking a palette file to change the desktop colours? This isn’t that much of a problem for me as I’m an old codger who has been using RISC OS since the year dot, but I do feel we want to limit the number of “gotchas” likely to inconvenience a new user…

Just to confirm that Paint 2.31 (29 Feb 2020) fixes this bug.

A big “thank you” to Sprow for dealing with the issue.

Just to confirm that Paint 2.31 (29 Feb 2020) fixes this bug

…but is there still this extra one, or are you saying both are OK now?

No, “Artist-friendly colours layout” is still a problem.

No, “Artist-friendly colours layout” is still a problem

I’ll take a look at this when I can find a spare moment. Not sure what’s going on there.

No, “Artist-friendly colours layout” is still a problem.

It seems to be related to the “Rearrange user palettes” option. If I create a 256 colour palette with artistic layout and pick colour 7 (which is brown, 8 squares down from the top, 5th square in the from right) then go Paint→Edit palette and set it to 100% white, the colour is changed to white in the palette but now the colour number shown is 123.

Importing a previously exported palette at this point then jumbles the colours up, perhaps because the import step assumes that the sprite’s palette is numbered 0-255 in the sprite memory but that there’s some extra level of mapping not being applied by Paint.

Anyway, sounds like Andy’s on the case!

Another Paint bug…

Using Paint 2.31 on RISC OS 5.27 (01 Mar 2020 build).

If “Artist-friendly colours layout” is ticked, saving a 256-colour palette file will produce …interesting… results when that palette file is dragged back to the sprite.

Right, I just tested the above using my local copy of the Paint code which was before Sprow’s fix. I made a new sprite with the full 256 colour palette (no mask, and “Artist-friendly colours layout” and “Rearrange user palettes” ticked), edited one colour, saved that palette to disk then immediately dragged it back in to the sprite window, and yes, the sprite’s colours became jumbled.

However, I’ve repeated the same test several times using a fresh build of Paint 2.31 from source I downloaded from GitLab today, and the colours are no longer jumbling when I drag the palette file back onto the sprite window. There is however, another bug apparent:

If I edit colour 255 which is white (in the top right of the colours window), to become magenta, the new magenta colour remains in the top right, renumbered to 15 (which is by design) but if I save and reload this palette, it becomes apparent that the new colour 255 has also been changed to magenta, in addition to colour 15, so the edit has somehow affected both the old and new colour numbers. :(

I should mention that this artist-friendly “Rearrange user palettes” feature does give rise to one situation where the colours will inevitably get jumbled up.

If you edit one or more colours on your sprite, with “Rearrange user palettes” ticked, and then you drag in a palette file that was:

• Made with an old version of Paint without the artist-friendly features.
• Made with the “Rearrange user palettes” option unticked.
• Saved from Paint without editing any of the colours (which means it saved the default RISC OS 256 colour palette).

then that is always going to jumble up the sprite’s colours. This is by design because “Rearrange user palettes” means the sprite’s pixels, and the sprite’s palette entries will all be reordered into the artist-friendly layout which of course means that any palette in the OS default (non artist-friendly) layout will then be incompatible with that sprite.

Edit: This all seems horrendously user-unfriendly. It’s adding some weight to Chris’s view that rearranging the palette numbers of user palettes might not be the best idea.

Stuart:

I had made one or two alterations to the default palette then saved it. Things only went seriously wrong when I dragged the saved palette file onto the source sprite window.

Sorry to be a pain, but as I can’t reproduce this with Paint 2.31, only an older version, please would you be able to test it again and confirm whether it’s happening when you keep your sprite window open the whole time, so:

1. Make a new 256 colour sprite, palette type: Colour.
2. Do some painting on the sprite window in colours you can remember.
3. Menu→Paint→Edit Palette to change one of the colours.
4. Save the palette to disk.
5. Still keeping the sprite window open, drag the saved palette file back into the sprite window.
6. Let us know the results.

If that doesn’t jumble the colours, could you let me know the exact steps you take to reproduce the problem please.

The exact sequence (tested on RPCEmu and Raspberry Pi) is:

1. Create sprite (256 colours, colour palette)
2. Paint using the colour in the bottom right hand corner (pale cyan).
3. Save palette file (Palette1)
4. Edit palette: alter pale cyan (bottom right corner) to red. Notice that its colour number has changed from 251 to 255.
5. Save palette file (Palette2)
6. Drag “Palette1” back to the sprite. Colours are now wrong.
7. Drag “Palette2” back to the sprite. Colours are now (mostly) right but colours 251 and 255 both show as red.

Thanks. That’s good in as much as it matches the behaviour I’m getting.

6. Drag “Palette1” back to the sprite. Colours are now wrong.

As described above, that bit isn’t a bug. It’s an unfortunate side effect of “Rearrange user palettes”. It’s because when you edited the palette in step 4, Paint rearranged all the colours of the sprite and its palette to match the artist-friendly sequence that the colours window uses. Palette1 was in the previous (RISC OS default) sequence, so is no longer compatible with the sprite’s pixels and gives the wrong colours.

7. Drag “Palette2” back to the sprite. Colours are now (mostly) right but colours 251 and 255 both show as red.

They’re mostly right because the artist-friendly sequence of colours in Palette2 matches the sprite’s pixels. The issue with 251 and 255 both being red is the remaining bug I need to fix.

If you want to stop the problem in 6. happening, but still like the artist-friendly colours layout untick “Rearrange user palettes”.

As described above, that bit isn’t a bug.

I realised that shortly after posting. Sorry for the confusion.

Not to worry. This palette remapping feature is confusing. I had to double check my posts to make sure I was explaining it right as well.

6. Drag “Palette1” back to the sprite. Colours are now wrong.

As described above, that bit isn’t a bug. It’s an unfortunate side effect of “Rearrange user palettes”. It’s because when you edited the palette in step 4, Paint rearranged all the colours of the sprite and its palette to match the artist-friendly sequence that the colours window uses.

But would it be possible to take that into account?

1. Say I have a 4 colour sprite, whose palette is 0/1/2/3, and the image shows a colour 1 square on a colour 2 background.
   For artistic reasons the colours window shows them as 3/2/1/0. That’s OK as that’s just a display thing in the colours window, the data in memory is still a colour 1 square on a colour 2 background with a palette 0/1/2/3.
2. I export this palette.
3. Now, I edit (unused) colour 3 so the palette is no longer a default OS one, and so the palette is permanently reordered to be 3/2/1/0 and the sprite bitmap also reordered (such that it’s now a colour 2 square on a colour 1 background).
4. Now, I import the palette stored earlier. The palette file contains the colour numbers (though they tend to be ascending) encoded in VDU19 sequences. If I were to apply the same mapping function from step 3 to the palette from step 2 the result would be a palette ordered 3/2/1/0 which would then load satisfactorily.

So, isn’t all that’s needed is an intermediate loading step which applies the OS palette → artistic palette reordering transform? Or at least to flag a werr() if the colours were reordered in order to stop the load proceeding.

From a state point of view you do need some way to know (reloading the sprite on another day) that the colours are the default OS set but rearranged into artistic order, in order that you can jump straight to step 4 again. That’s probably where the scheme falls over because trying to distinguish a slightly edited artistic palette from an entirely custom one would need some shakey heuristics.

I think I just talked myself out of it there – there is an (optional) warning about the reorganise step after all.

It might be less confusing if a slight change was made to what “Rearrange user palettes” actually does.

If “Rearrange user palettes” meant “ALWAYS rearrange user palettes”:

• When the sprite is initially created, the palette is in “Artist-friendly” order, not RISC OS default order.
• Saving this palette will save the “Artist-friendly” order. This is the same order as a modified palette would have, reducing the chance of user confusion.
• Working on a pre-existing sprite will use the palette stored with the sprite, or RISC OS default (rearranged into artist-friendly order) if no palette was stored.

The only problem that springs to mind is that a paletted sprite using RISC OS order wouldn’t have the colours in artist-friendly order when an attempt was made to edit it. That gets us back into the argument over “how much deviation from the default can we permit when rearranging the palette”…

From a state point of view you do need some way to know (reloading the sprite on another day) that the colours are the default OS set but rearranged into artistic order, in order that you can jump straight to step 4 again. That’s probably where the scheme falls over because trying to distinguish a slightly edited artistic palette from an entirely custom one would need some shakey heuristics.

Yep and I already coded just such a shakey heuristic because I didn’t like most of the colours going wrong when the user deletes a sprite’s artist-friendly palette:

/*************************************************************************************
 * Checks if the sprite's palette has most colours matching the artist-friendly      *
 * order. If it does, they are rearranged to the OS default palette's order.         *
 *                                                                                   *
 * This is currently a necessary evil before deleting a palette that the user chose  *
 * to remap to artist-friendly, otherwise most of the colours will be wrong after    *
 * the palette is deleted. As a future improvement, it may be better if palettes     *
 * are never rearranged for the artist-friendly layout and instead the reordering    *
 * is only done on the Colours window, with a similar detection routine for palettes *
 * that are near enough to the default for the layout to be appropriate.             *
 ************************************************************************************/

BOOL colours_try_remap_from_artist_friendly (main_sprite *sprite)

/*************************************************************************************
 * Returns true if the sprite has a 256 colour palette where most of the colours     *
 * match those of the artist-friendly order, indicating it was remapped to that.     *
 ************************************************************************************/

static BOOL colours_remapped_to_artist_friendly (main_sprite *sprite)

In the case we’re discussing, where a palette in the RISC OS default order is being loaded into a sprite that has the artist-friendly order, wouldn’t we need a similar heuristic to decide whether the palette being loaded is similar to the RISC OS default palette (say with just a few colours edited)? The Holy Grail here would be some kind of best-fit remapping that just matches the sprite’s pixels to the closest colours it finds in any palette that’s loaded, although even then there are problems; like what do you do if there are two or more colours in the new palette all equidistant to the pixel colour you want?

When the sprite is initially created, the palette is in “Artist-friendly” order, not RISC OS default order.

Yeah I did think about that. You couldn’t do it though for sprites that don’t have a palette upon creation. There’s also a performance penalty in that the RISC OS default palette is faster because the OS doesn’t have to build a translation table.

I’m increasingly coming back to the view that this palette remapping stuff is pushing technicalities onto the user that they shouldn’t have to deal with and usually won’t have any interest in. Rather than repeatedly tweaking it, I still think replacing it with an algorithmic sort of the colours window is the best way forward.

I still think replacing it with an algorithmic sort of the colours window is the best way forward.

That would make the most sense. Leave the actual palette alone (let the user modify as necessary) and simply build a one for one table for the colours window that can be arranged however the user wants (which would also work with sprites without a palette by rearranging the default colours).

like what do you do if there are two or more colours in the new palette all equidistant to the pixel colour you want?

Pick one using some consistent method (first found, lowest number, whatever). After all, brown is brown is brown. If the new palette wants brown and there happen to be two, it probably won’t matter which is chosen.
The bigger question is what to do it there are no matching colours.

I still think replacing it with an algorithmic sort of the colours window is the best way forward.

If you do go down that path, may I suggest you relabel the option? Calling it “Artist-friendly” obscures the real reason for its existence: that the original layout doesn’t make much sense from a user perspective. Much better to rename it “Use backward-compatible palette order” (and invert the setting so that the option starts off unticked) – the option would flip between two static mappings without any second-guessing involved.

The phrase “artist-friendly” can still appear in the user manual and/or help text, of course. For example: “When editing some 256-colour sprites, you may find that the colours in the ‘Colours’ window are arranged in a confusing order. This can sometimes be improved by ticking or unticking ‘Use backward-compatible palette order’. You will usually want to have this option unticked, as that will arrange the colours in the default palette in an artist-friendly order.”

Pick one using some consistent method (first found, lowest number, whatever). After all, brown is brown is brown. If the new palette wants brown and there happen to be two, it probably won’t matter which is chosen.
The bigger question is what to do it there are no matching colours.

Yeah, for example the pixel might have been 50% grey but the new palette might only contain black and white. Or consider a pixel that was previously green and the new palette contains only red and blue. Also, the user might not want three different pixel colours in the source image to all map to one colour even though that’s the closest, because it would make those areas of the image indistinguishable and so lose geometric information!

If you do go down that path, may I suggest you relabel the option? Calling it “Artist-friendly” obscures the real reason for its existence: that the original layout doesn’t make much sense from a user perspective. Much better to rename it “Use backward-compatible palette order” (and invert the setting so that the option starts off unticked) – the option would flip between two static mappings without any second-guessing involved.

Good point. If I went this algorithmic route, I’d probably get rid of the singular option and replace it with a list of two or more named sort orders when clicking Menu over the colours window, as others have suggested.

My view is still that the artist-friendly layout is only more useful than confusing in the single case of a 256-colour sprite without a palette. In this case, the Colours window rearranges the colours to make it easier to find the right shade, but doesn’t do anything with the underlying colour numbers in the sprite. It’s purely a difference of icon ordering in the Colours window.

In cases where a sprite has a palette of its own, I think the confusion caused by trying to cope with an artist-friendly remapping outweighs its usefulness. If users want a palette with better colour ordering, they can define one themselves, save it, and load it when required (or there could be more presets in the Create Sprite dialogue than just ‘colour’ or ‘greyscale’). If a user adds a palette to a 256-colour non-palette sprite, then I think the artist-friendly layout option should be greyed-out, and the default ordering displayed in the Colours window.

There is however, another bug apparent:

If I edit colour 255 which is white (in the top right of the colours window), to become magenta, the new magenta colour remains in the top right, renumbered to 15 (which is by design) but if I save and reload this palette, it becomes apparent that the new colour 255 has also been changed to magenta, in addition to colour 15, so the edit has somehow affected both the old and new colour numbers. :(

Gah I see why it edits both the old and new colours by mistake now. When the colour picker gets spawned using dboxtcol(), that takes a pointer to an integer containing the colour to edit, which is also used to return the newly selected colour value. Paint has always dutifully copied the new colour it receives from the picker into the sprite’s palette. The only problem is that the colour pointer it passed to the colour picker in the first place was already a pointer into the sprite’s palette! So basically, the colour picker does the palette editing for Paint, and then Paint does it all over again! This never used to matter, but now we’re in the business of rearranging palettes into artist-friendly layouts, it means two colours get changed instead of one.

What muddies the waters even further is that this bug where two colours change instead of one isn’t apparent until you either change the desktop screen mode, save and reopen the sprite, or (as mentioned in this thread), export and re-import the palette. The reason it’s not apparent seems to be an out of date translation table for the sprite. You can see evidence of this if you edit colour 255 from white to red, after the colours window gets rearranged, the new colour 255 in the bottom right gets a red outline on its text. This is because the colour number text’s background colour is set directly from the RGB value in the palette, whereas the colour squares themselves and indeed the sprite’s pixels, are coloured using the translation table.

It’s not really clear at all to me at the moment why the translation table is getting out of date. It’s supposed to be updated when sprite_palette_has_changed() gets called, and this function’s called at the end of Apply_Colour(), after all the remapping and colour editing is complete, which should be exactly the right time to do it! Strange bugs.

Ah, can anyone tell me, would flashing colours confuse ColourTrans_GenerateTable?

When Paint passes the colour picker a pointer to the colour to edit and the colour picker then changes the colour, the adjacent colour (which would only be used for a flashing effect) doesn’t get updated. Even though a new translation table gets made for the sprite, the colours window cell and any pixels in that colour on the sprite remain in the old colour. If I change the screen resolution, the mode change event makes the translation table be regenerated again, and this time it seems to have a correct value for the edited colour.

Can RISC OS still display flashing colours by the way? I can honestly say I can never remember seeing them.

Can RISC OS still display flashing colours by the way?

They are available in 16-colour modes (BBC BASIC Reference manual, page 481) so will work on the IOMD platform. Some other platforms (e.g. the Raspberry Pi) do not support 16-colour modes, so flashing colours may not be available.

Edit: I just twigged that’s in full-screen mode. You may be out of luck if you want flashing colours on the desktop.

Ah, can anyone tell me, would flashing colours confuse ColourTrans_GenerateTable?

I’m fairly certain it only looks at the primary colours.

Flashing colours are a very niche feature, so I wouldn’t worry too much about trying to support them in Paint. As long as Paint only edits the palette when the user tells it to edit the palette, and you can load in a palette file containing flashing colours and have it applied correctly, I think that’ll be enough. I.e. it’s possible to load & edit sprites containing flashing colours, but if you try editing palette entries using Paint then it reserves the right to reset the flashing colours.

Can RISC OS still display flashing colours by the way?

Flashing colours should work in any paletted mode; all that happens is that the OS updates the relevant palette entries every N vsyncs, swapping between the two colours. But it’s only the (single-tasking) 16 colour palette that has them enabled by default, and use within the Wimp is effectively impossible because the palette is shared between all apps (and things like ColourTrans won’t understand that some colours are flashing)