ununited codes...

The program I’m writing (BBC Basic, Wimp) fetches data from internet, geographical names. In the data (html, javascript etc) codings are used. My program shows that data in a window. As an example:
Egypt
Ab\u016b Rudays
Ab\u016b Sunbul
Al \u2018Ar\u012bsh

and:

Abū Rudays
Abū Sunbul
Al ‗Arīsh

My program shows it as:
Abū Rudays
Abū Sunbul
Al ‗Arīsh

Instead of:
Abü Rudays
Abü Sunbul
Al ’Arïsh

What should I do to display the data properly?

Hum. You need an home made UTF converter.
I worked on something closed some time ago, but it’s not (yet) complete.
A tool as MSR could help.

What should I do to display the data properly?

If you are drawing your own window you could use Font_Paint with a UTF8 encoding. It would need a font with the necessary glyphs though.

Welcome to the wonderful world of encodings. In RISC OS world, I would recommend having a look at the two subprojects of NetSurf, iconv and rufl.

For the accented codes that are part of the standard character set, a small lookup table should suffice. For other characters, things are now complicated (see previous posts).

Your clues made me do this:
NetSurf shows the geographical names in a proper way. I drag a name to StrongEd and apparently a transcription takes place.
Here are some examples, the first of them is what I see in the file I got from internet, the second is the result of the drag:

Vietnam:
Hòn Chông
Hòn Chông

Nam Ä‘á»⇑nh
Nam D?nh (I can live with the ?)

Egypt:
Al ‗Arīsh
Al ‘Arish

Å¢ÄŴbÄŴ
Taba

France:
Angoulême
Angoulême

So, it is more than accented codes and in most cases the transcription is all right. It shows that the UTF-code is transcripted in RISCOS printable characters.

Which part of NetSurf does this transcription? Iconv or RUfl. As I can’t find RUfl in my system, it must be Iconv. That leaves the questions:
How do I call it, how do I put a string in and how do I get the result?

Use rufl if you want to paint UTF-8 fonts with the Font Manager. Use iconv to handle the encoding conversion “in memory”. The iconv download contains stubs/headers for C. As usual, that’s all the doc you get as a C developer :-)

NetSurf shows the geographical names in a proper way. I drag a name to StrongEd and apparently a transcription takes place.

More like the other way around. StrongED shows the file as it is and NetSurf interprets it to create the correct characters.

You may be able to use the iconv module to convert the file your program retrieves. Try *help iconv (with the module loaded, of course) for some info on how to do that.

I found some additional information about the params on gnu.org

There seem to be hundreds of codings.

What kind of coding is: Å¢ÄŴbÄŴ (Taba)
What kind of coding is needed for RISCOS, using the configurated Desktop font?

(I never felt so blank about a subject)

@ David Feugey: A tool as MSR could help.

What is a(n) MSR tool?

It looks like UTF-8 to me. As for RISC OS, if you do iconv -l in a taskwindow, the first item in the output is X-ACORN-LATIN1, so that’s probably worth a try. If that doesn’t look good enough, try ISO-8859-1 or ISO-8859-15.

What kind of coding is: Å¢ÄŴbÄŴ (Taba)

UTF8

What kind of coding is needed for RISCOS, using the configurated Desktop font?

If you want to see it correctly in edit you can’t. If you want to display it in your own program where you are drawing a window then you can print to the screen with Font_Paint and add \EUTF8 to the font name when opening it. If you are using C the library mentioned earlier is better than font paint as it handles unknown characters better.
If you are using BASIC font_paint will have to do.

I’m afraid, Colin, it’s not a matter of drawing the text in a window. I have a window with ten buttons in it, each button has a [geographical] name. They are put in with Get_ and SetIconState.

Perhaps you could Font_Paint the text to a sprite in a user sprite area and then plot that sprite as the button?

Ok. You can define the button to use any outline font and encoding – see wimp_createicon. Don’t know if the template editor will allow you do define it there – it’s a while since I’ve used a template editor – so you may have to modify the template in your program.

I should have added that if you are using the toolbox then ActionButton_SetFont will do what you want. Just add \EUTF8 after the font name.

Don’t know if the template editor will allow you do define it there

Don’t think so, at least not with a UTF8 encoding. The following should work though.

Set the icon to use an outline font in the template editor. At runtime open a font with UTF8 encoding using Font_FindFont. Set the icon’s font handle using Wimp_SetIconState to the handle returned by Font_FindFont.

What is a(n) MSR tool?

MSR was a fast (ASM) universal text conversion tool made by Augustin Vidovic.

Not sure why the Japanese penguin on Augustin’s site is carrying the old imperial flag, but no matter.

This. Wins. Completely.
http://augustin.vidovic.org/pages/tama/catscan/index.html

(^_^)

@Rick: haha, copycat.

@ Fred: Set the icon to use an outline font in the template editor. At runtime open a font with UTF8 encoding using Font_FindFont. Set the icon’s font handle using Wimp_SetIconState to the handle returned by Font_FindFont.

I did as you described, the text is in the font chosen, but no transcription, alas. I’ll keep on searching…

It works in the toolbox so it should work using wimp functions. However there is a bug/feature which makes it unusable. Certain bytes are changed to the WimpSymbol font so the € (&80) in Al ‗ArÄ«sh displays as a tick instead of being seen as a part of a unicode character.

So unless someone knows how to disable WimpSymbol font mapping on a per icon basis you are going to have to draw your own buttons and use Font_Paint.

@ Colin As far as I can see, this happens: when RISCOS meets a UTF-character like ê (C3 AA), it is able to convert it to ê because ê is in the character set of RISC OS.
As soon as RISCOS meets a â (E2) (or another escape character not beiing (C3)), it is’nt able to do a conversion anymore. Å¢ÄŴbÄŴ has to be shown as Ţābā: a T with a , at the bottom like the ç and two a’s with an upperscore. And I can assure you, there are a whole lot more of these characteristic characters… I suppose it must be impossible to print them, even using Font_Paint, simply because the are no characters in RISCOS.

But: why can NetSurf display them properly? And how can I?

(To show that it really matters to me: 95 % of my program was ready and worked well. As a next 3 % I added a way of searching any country or place in the world. And it works fine as well. The last 2 % is to show the data as it should be. I’m very motivated! :-))

BTW: It’s also the hyper correct way the BBC presents the data. The first button: Å¢ÄŴbÄŴ, Egypt The second: Taba International Airport, Egypt

Using a Homerton.medium\EUTF8 font Ţābā is printed out correctly. You can’t be doing it properly.

Please tell me how you do it.

What I did is this:
in the Templates,window,icon: I set Outline font to Homerton.Medium
In !RunImage,PROCinit: SYS “Font_FindFont”,,“Homerton.Medium\EUTF8”,320,320 TO fonthandl%
In !RunImage I call somewhere: PROCicontext(hmain%,0,A$):REM where A$=“Å¢ÄŴbÄŴ”
In !RunImage,PROCicontext(window%,icon%,text$)
!block%=window%:block%!4=icon%
SYS “Wimp_GetIconState”,,block%
$block!28=text$
block%!8=&40:block%!12=&40:block%!24=fonthandl%
SYS “Wimp_SetIconState”,,block%

text$ is displayed in the correct font, but no conversion took place.