Display UTF8 with character codes above 0xffff

Hello,

I would like to paint UTF8 string characters above 0xffff. I working on Chat application with integration with Telegram. Telegram uses UTF8 characters for the smiles. I would like to use them too as draw lines with text much easy than put images inline in the text flow.
I looked at the some fonts and found that there seems no font which include all Unicode chars. In the Window/Linux the system used multiple fonts and substitute missing characters. The RUFL library promise to substitute missing chars too from other fonts but it seems not work for characters which not fits in 16 bits.
Emojis codes are at U+1FXXX. I tried to converted the “Symbola” font from my Linux using !TTF2F utility (v.0.04).

In the RUFL source I found code that forbids using chars above 0xFFFF.
In rufl_paint.c, lines around 223:

        rufl_utf8_read(string, length, u);

        if (u <= 0×001f || (0×007f <= u && u <= 0×009f))

                font1 = NOT_AVAILABLE;

        else if (charset && rufl_character_set_test(charset, u))

                font1 = font;

        else if (u < 0×10000)

                font1 = rufl_substitution_table[u];

        else

                font1 = NOT_AVAILABLE;

When I look at converted Symbola font using !Chars utility then after shows some chars (several pages of Unicode chars) it give error “Cannot paint this font; Reverting to System font”

Netsurf browser also don’t show that UTF8 emojis (I tried to make simple test HTML page with that UTF8 Emoji chars).

I tried this on RiscOS 5.24 in RPCEmu.

So I have 2 questions:
1. Is RiscOS is able to draw all Unicode chars, include chars above 0xffff?
2. May be !TTF2F do font conversion wrong? How to check it?
3. What is the reason for RUFL to restrict to 16 bit chars only?

Fonts converted from Truetype by TTF2F do often appear to provoke errors somewhere amongst their thousands of characters when the Font Manager tries to draw them. I’m not sure what the reason is – I suspect there’s a number of possible causes, due to the original files not being designed to work with RISC OS.

The most recent version of Chars handles these ‘bad’ characters differently – it will draw an empty box in their place instead of switching to the System Font. It might be worth downloading this version, so you can at least see which characters are causing the problem.

Fonts converted from Truetype by TTF2F do often appear to provoke errors somewhere amongst their thousands of characters when the Font Manager tries to draw them. I’m not sure what the reason is – I suspect there’s a number of possible causes, due to the original files not being designed to work with RISC OS.

One regular problem, which goes back to 4.02 as I recall¹, is that some of the characters in the original font have a large number of turning points which is exacerbated by the conversion process and the end result is a character/font that requires silly-high memory levels.
I think the recent version of Chars refuses to show the offending character(s) and carries on thus allowing access to all the other ‘cleaner’ characters.

¹ I recall some conversions giving the RPC a headache and investigations showed a stupidly complicated (badly drawn) character or two.

To go back to the original question, my guess (ie. unsubstantiated as I have not looked at source) is that the unicode version of font manager is probably only aware of two-byte (ie. 16bit) fonts. Indeed, “back in the day” (ie. circa 2000), I recall people talking about support for “16bit fonts” (referring to the first two bytes of UTF-8). I suspect that when the font manager was updated with Unicode support, this was all that was implemented, and so it is all that TTF2font outputs to avoid creating fonts that cause the font manager to get confused.

Note – this might be nonsense, so it really needs to be checked against source.

However, I thought that chatcube painted text using linux-libraries (eg. freetype) so wouldn’t be restricted by RISC OS fontmanager restrictions, except perhaps in icons or in username lists etc?

Also, in case you’re not familiar with it, https://www.netsurf-browser.org/projects/rufl/ might be of some use. At least for a look at the source code :)
https://source.netsurf-browser.org/librufl.git/

Uh, Andrew? The original message says that the problem is within RUFL! ;-)

OK. So, now I’m humming the “Friends” refrain – “When it hasn’t been your day, your month…” ;)

I’ll get my coat. Anyway, thanks Rick for spotting my not-reading-his-post-since-yesterday foot in mouth. I feel very silly (smile).