Keyboard Handler, fonts & UTF-8

Many years ago I produced several RISC OS fonts, mostly for esoteric purposes associated with publishing various academic journals, but also for Hindi (Deva Nagari) and Russian (Cyrillic). I also wrote keyboard handlers (using !IKHG) and a “keyboard select” app for the standard Hindi, Greek & Russian keyboard layouts and to enable access to all the top-bit-set characters straight from the keyboard, using Alt-character, Ctrl-character combinations etc.
The fonts I wrote still work fine, and Draw files using them display normally on Risc OS 5.21 on my Pi – but the keyboard handlers don’t work (not 32-bit clean), and as far as I can see there’s no new !IKHG for 5.21. I could write a new !IKHG myself (assuming the SWIs documented in the PRMs are still kosher) but it’s a fair amount of work and I’d rather not reinvent the wheel if someone else has already done it. Also, I understand there’s been some work UTF-8-ifying the system, and I’d rather make all my work fit with that as best possible. I’d quite like to release my fonts (and keyboard handlers) into the wild, but only if they fit into the wider scheme of standards properly.
The fonts all high quality, producing publishing-quality output (not only Deva Nagari & Cyrillic, but also Greek & various Latins), but the character encodings aren’t entirely standard, because I couldn’t get everything in that I wanted under the old system – obviously I’d like to shift things about to use the bigger character space in a more standard way before releasing anything.
All my fonts used floating accents (Acorn User carried an article of mine about this) so you typed the accent as a character before the letter, and the kerning table in the font then allowed kerning-capable apps (such as Impression Publisher) to automatically position the accent correctly. This is very useful if you’ve got a mixture of Latin-alphabet languages in a single document – no need to change keyboard or font at all. It’s also very useful for Hindi, which otherwise has too many diacritic-character combinations to get into a single (reasonable size) font (Hindi is often badly typeset because of this problem amongst others).
Sorry – I’m rambling!
Does anyone have any documentation about any of this? I know how UTF-8 works in the wider world, but to what extent is it implemented in the Risc OS world?

(Back in 2007), IKHG "was never updated to Unicode support", but I accept that this info is probably of little use because of its date.

Yup, I found that post before. But where are those “human readable master keyboard layouts”? In the source, sure, but I don’t want to have to rebuild RiscOS to make keyboard handlers!
I suppose I’ll end up looking at the keyboard buffer to see what’s in there, and writing a UTF-8 (assuming I find UTF-8 data in the buffer…) version of !IKHG myself…ho hum.

don’t want to have to rebuild RiscOS to make keyboard handlers

Would building !IKHG on its own (to obtain a new module) not be enough? Apologies for my lack of technical knowledge – I’ve only ever built the sources once or twice myself… and am (still) not set up to really investigate doing even the simplest of even BASIC amendments myself yet.

Ah, I see it’s in the attic. Does that mean it needs amending/32-bitting, etc.?

If KHG has been retired, then it’s because you shouldn’t be doing things that way any more. Trying to figure out how to resurrect it is not the right answer! It took me about 30 seconds to hopefully figure out what Clive needs to look for despite knowing nothing about IntKey. Here’s what I did:

• this is a document in the IntKey.Docs directory with a fairly recent commit date – it looks like a specification for everything Chris needs to know
• skimming through that for about 5 seconds, I found the definition of these files includes a line of the form “$Layer”
• I checked out a copy of the IntKey sources “castle/RiscOS/Sources/Internat/IntKey”
• I grepped the sources for “$Layer”
• I found lots of mappings files in “castle/RiscOS/Sources/Internat/IntKey/layout/” – presumably you need to look at these as a template of what to do

Does that help?

I’ve never built the sources at all, and hope I never need to! I’ll take a look at those mapping files. I hope I can find instructions about what to do with any new mapping files I write – and I hope it doesn’t involve rebuilding anything more than an odd module or at most a couple…going in search now…

I see – good luck then! Without reading all that stuff, I can’t guess what your next steps are going to have to be. I did spot there’s already a Russian layout file, for what that’s worth.

I tried the Russian one – and it’s just a copy of the UK one, as far as I can see*. The Greek one seems okay, but there aren’t many…

Reading Ben Avison’s post ( https://www.riscosopen.org/forum/forums/2/topics/61#posts-597 ), it looks to me as though the lack of a !IKHG is because it’s assumed that anyone wanting to make a new handler is involved in the generation of Risc OS itself, rather than making custom handlers – not that there’s any real objection to having custom handlers, just that the “market” for them is small (which it undoubtedly is…)

*That was by issuing *country russia on the command line on my Pi, which left me typing English normally! Whereas typing *greece did indeed make me Greek. I wouldn’t expect Russian characters without a Russian glyph set in the font, but I wouldn’t expect plain English either…

Edit: Ah. That’s what “Layers” is all about (Russian keyboard). Understood.

That document would help me to write mapping files okay (finding some actual examples would help with understanding it, of course) – but how one goes about building the module having done that, and whether it can then reside in an app rather than having to be built into RiscOS, I’m still not clear. Further investigations…oh, and this area of stuff seems to be in C…I’m an Assembler (and BBC BASIC) programmer by nature (wrote the ARM Assembler Guide & Instruction Set section of the ARM Architecture Manual for v6, v7A & v7R) & have very little C experience.

INTeresting! The “mapping” files in “castle/RiscOS/Sources/Internat/IntKey/layout/” are Draw files, pictures of the keyboards! Looking further…

“If KHG has been retired, then it’s because you shouldn’t be doing things that way any more.”
That’s all very well, but there doesn’t seem to be any documentation anywhere about how I should be doing it now.
I’ve managed to work out how the Greek and Cyrillic keyboard handlers are behaving now, and that’s fine; I can make my fonts work with those. But there’s no keyboard handler for Hindi, and I can’t see any documentation anywhere about how to make one. I know how to make one by a sledgehammer technique, replacing the entire keyboard driver (assuming the sys calls documented in the PRMs still work as described) – but that clearly wouldn’t be necessary if the documentation existed for how to do it “properly”…

Is anyone actually interested in quality Deva Nagari (Hindi, Nepali) fonts on RiscOS 5.2? If they’re just for me, they’re working on RiscOS 3.7 on my RiscPCs, and I can hack them to work on my Pi with an ugly hack I wouldn’t be prepared to release (all the glyphs in the wrong places in the font, so the Russian keyboard layout can be used…). I can update the Greek and Cyrillic and Modern (based on an 1870s academic journal’s typeface, so no copyright issues) to RiscOS 5.2 no problem, and will do so shortly.

Is anyone actually interested in quality Deva Nagari (Hindi, Nepali) fonts on RiscOS 5.2?

I would be. But only out of curiosity, and a desire to see RISC OS becoming more international. My own interest is higher up the stack in IMEs, but it’s important we have information about how to develop UTF-8 keyboard handlers, so if you can make any headway there, all to the good.

Yup, that’s the area where my problems are. !IKHG gave pretty comprehensive control over keyboard input, and was pretty straightforward to use, but apparently the new system is even easier to use – albeit possibly only at the “building RISC OS” stage? Whether it’s pre- or post-build, it’s not easy to use if it’s not documented.

Is anyone actually interested in…

The answer of course is that anyone actually interested in other scripts, UTF-8, proper typography and quality fonts is not using RISC OS at all, never mind the 90’s Acorn Font Format.

RO ought to support OTF. Til then, from a typography and fontography point of view, there’s nothing to see here, move along, move along.

I don’t agree, sorry. I spent several years of my life publishing academic journals (in the 1990s, true) on Archimedeses and then RiscPCs, because at that time Acorn’s font format was very much the easiest way to achieve high quality typography in multiple languages on modestly-priced equipment. I designed our own fonts (initially to match the fonts already being used in our journals, produced by high-end typesetting systems belonging to Cambridge University Press and costing hundreds of times as much as our Acorn systems – the only way I could convince people that I could match their quality was to match their style as well, so no-one could actually detect when the changeover occurred. Even the CUP people couldn’t, apart from those who knew anyway because they did the setting…)

Acorn Font Format is GOOD – but I agree that it’s no longer relevant in the wider world, which has moved on (albeit not forwards …). But I like Acorn’s FontEd, and the fact that I can easily write my own apps to extend it – and then convert them to other font formats for use on Macs and PCs (again, using a RISCOS app that I wrote myself). I’ve got font editing software on the PC, but FontEd + my own apps is better (at least at the initial glyph design stage).

Don’t discourage folks from using RISC OS. I’m mostly a PC user these days – the only things I use RISC OS for are font design, vector graphics (I like Draw better than anything reasonably priced on the PC, or indeed anything unreasonably priced that I’ve used at work), and the ability to write apps easily myself for doing all sorts of odd jobs (BBC BASIC + ARM assembler + RISCOS wimp is FAR easier and quicker than writing anything for a Mac or PC – for me, anyway, other folks may prefer other systems…)

I liked Impression Publisher (which we used for those academic journals) but lack of compatibility with other systems has killed it for me. Draw file and Acorn font file formats are well documented, so easy to write conversion software for.

If no-one else wants my Acorn font-editing software, so be it. It’s there on my website for anyone who does want it, but it’s no skin off my nose if no-one does. Likewise my fonts when I’ve made them available – the existing state of them is a bit idiosynchratic, very streamlined for producing those academic journals, but a bit specialized and RISCOS 3.70 specific (not many people want to be able to type [superior 2+, inferior o] as a single character by hitting Alt-2, etc. etc., and the keyboard handler that permits this doesn’t work on RISCOS 5 anyway…)

I set my iyonix to utf-8 and changed the desktop font. There were a few issues (hard spaces I think) but everything seems to work reasonably.

Acorn Font Format is GOOD

No, it isn’t. Not even slightly.

It simply cannot be applied in any way to many of the world’s writing systems. It cannot represent the vast majority of professional fonts. It can’t be used to produce quality typesetting.

There’s insufficient space in a forum post to list all of the features it lacks, but the really important ones are:

• No contextual glyph substitution, so Arabic can’t be done
• No vertical metrics, so Japanese can’t be done
• No stylistic variations, so professional fonts can’t be represented
• No pixel fitting when antialiasing, so AA text is blurry
• No gamma correction during AA, so all AA text is wrong

Read up on OpenType, the GSUB and GPOS tables, and the various Scripts and Features they enable. This is a subject in itself but Wikipedia is a good place to start: http://en.wikipedia.org/wiki/List_of_typographic_features

Good luck trying to persuade someone to use RISC OS for Japanese without vertical text support, never mind vertical punctuation substitution. I pity you trying to typeset Arabic without the font smarts to do the glyph shaping and justification for you. As for the slanted baseline of Nastaliq Arabic…

The Acorn font format was never “good” (though the renderer was impressive last century), but as of ISO Standard ISO/IEC 14496-22 the format is totally obsolete, and to suggest otherwise is to betray a profound ignorance of the subject. Sorry to pull rank, but I’m familiar with the three people in the world who know more about fonts than I do, and you’re not one of them.

I realise this reads as grumpy and arrogant, but I won’t put up with romantic drivel and nostalgic blinkers obscuring the facts. RO ought to be using OpenType by now… crikey, even your phone can do that.

I set my iyonix to utf-8 and changed the desktop font. There were a few issues (hard spaces I think)

Yes, sadly there’s no 8bit alphabet fallback at the moment. I have that outside the desktop, and once that’s released it would be sensible to implement it for FontManager too.

In fact it could be implemented for FontManager… I’ll think about that.

Would it be feasible to add support to the Acorn font format to FreeType2 and rewrite FontManager using that? That way support for the old fonts could be retained, and new font formats could also be supported.

crikey, even your phone can do that.

Ouch! I could hear that slap from way over here! (^_^)

It can’t be used to produce quality typesetting.

That’s really nonsense. I used it to produce academic journals (in the 1990s). We took over the typesetting from Cambridge University Press (CUP). Nobody believed I could produce quality typesetting on the Acorns, so to prove that I could, we changed from CUP to Acorn in the middle of an issue of one of our journals. Nobody at CUP except the people who actually did the typesetting could tell where the change occurred. To achieve that, I not only had to match their quality, I had to copy their style EXACTLY – designing all the fonts on FontEd.

There is of course a sense in which you’re right: Acorn fonts aren’t compatible with professional typesetting systems, so you can only produce high quality typesetting if you’re producing camera ready copy (as we were).

the format is totally obsolete

Of course that’s perfectly true, but converting the format to OpenType isn’t hard (I’ve written an app that does it). This is the point so far as I’m concerned; I prefer the Acorn for generating glyphs, but (apart from using them in Draw) I actually use the fonts on PCs or Macs. As OpenType fonts. When designing a font for conversion, you have to be aware that Acorn DOESN’T stroke the outlines, whereas OpenType and PostScript do – this is part of the reason why people think Acorn can’t do quality typesetting: if an Acorn font is naïvely converted, it ends up emboldened by one pixel. (However, this is actually the one point where Acorn got it right, and everyone else has it wrong…but too late now to fix the world…) (Struck out because this is a misinterpretation of what goes wrong in the translation, although the end result is much the same.)

It simply cannot be applied in any way to many of the world’s writing systems.

I can’t comment much on the writing systems I don’t know – notably Japanese, Chinese, Korean and Arabic. I had no trouble applying it to all the Latin alphabet languages, the Indic languages, Cyrillic and Greek. I did notice that FontEd doesn’t provide for vertical character widths, and never checked to see whether they were ever implemented anywhere else – they’re certainly SPECIFIED in the font file specification in the PRMs. Vertical kerning certainly works (or worked up to RISCOS 3.7, anyway). I have to admit I don’t understand how Nasta’līq Arabic manages to avoid line crashes. Even if the vertical character widths were implemented, then the slanted baseline would still present a problem, because of the back-stepping – although since it’s a calligraphic script I presume that’s hand done anyway.

Contextual glyph substitution is easy enough to implement in RISCOS 3.7. I did it for Hindi. You think it’s not needed for Hindi? Take a look at some traditional Hindi typesetting. I don’t know whether PCs and Macs do this properly yet; they certainly didn’t in the 1990s. Okay, I had to implement it myself, and it was part of an add-on tidying up app, not built into Impression Publisher – but is that an OS issue, or an app issue? Certainly it’s a bigger issue for Arabic than for Hindi, that much I do know. (Bad description of what I was doing…)

I realise this reads as grumpy and arrogant […] romantic drivel and nostalgic blinkers obscuring the facts

I’m afraid it REALLY does. You misunderstand me. At no point have I suggested that Acorn font format is anything but obsolete¹, or that it’s good for all writing systems (although with a little hacking it’s capable of things you seem to think it can’t do). What I loved about RISCOS was the ease with which it can be hacked – indeed that’s the only reason I’m still interested in RISCOS.

Sorry to pull rank, but I’m familiar with the three people in the world who know more about fonts than I do, and you’re not one of them.

Arrogant indeed, but I’ll happily defer to your knowledge of the current state of the implementation of typography on computers, and to your knowledge of East Asian and Arabic writing systems. I probably know more about hot metal technology than you or your three experts, and very likely more about 1970s and 1980s high-end filmsetting technology too. Yes, nostalgia rules…

even your phone can do that

Yours probably can. Mine has a rotating dial. I wrote the book that the engineers who designed your phone used to tell them how to program its CPU, but I don’t have such a phone. No, no-one has ever heard of me, which suits me just fine.

¹ Beware. Others might say that of the whole of RISC OS…

I wrote the book that the engineers who designed your phone used to tell them how to program its CPU, but I don’t have such a phone. No, no-one has ever heard of me, which suits me just fine.

There’s a large hospital Trust locally that has had almost zero network problems in the last 10 years.¹
Some people know of me, a few know me ² which “suits me just fine”
I’ve always claimed to be just a cog in the machine, the fact that this particular cog is one that has a go/no go effect on the whole setup might make it more important, but as they say “whatever”

We each do what we do.

¹ Some, like the wife, point out that I’ve done nothing much outside work in that time.
² Of those that do know me even line managers with the same job emphasis don’t understand me (they said so)

We each do what we do.

Indeed. Good post altogether. “Some people know of me, a few know me”… yup, that is of course more accurate for me, too.

Following yesterday evening’s discussion, I’ve just implemented vertical metrics in my !XP1FontEd, following the font file specification in the PRMs. I find that although Acorn’s FontEd can’t generate vertical metrics, it displays them perfectly happily. I’ve not investigated whether any apps take any notice of them yet.

Would it be feasible to add support to the Acorn font format to FreeType2 and rewrite FontManager using that?

That wouldn’t be the way to do it – better to split FontManager in two and have it as the API and caching layer, with multiple back ends… one of which would be the existing Acorn Font Format support, but another could be FreeType or your OpenType library of choice – UFST, FontFusion, etc etc.

We took over the typesetting from Cambridge University Press (CUP). Nobody believed I could produce quality typesetting on the Acorns

I probably know more about hot metal technology than you or your three experts, and very likely more about 1970s and 1980s high-end filmsetting technology too.

I don’t recognise your name, so it’s unsurprising that you don’t know who I am either. [Snip tiresome don’t-you-know-who-I-am rant]

So it’s fair to say that you’ve strayed into my area of expertise.

It can’t be used to produce quality typesetting.

That’s really nonsense.

Oh get a grip. Do you know what dotgain is? When Acorn teamed up with AB Dick to try to sell a RiscPC with Impression as a professional artwork production system to rival a Mac, they didn’t know. Do you have a linearly calibrated monitor and PostScript calibration? Do you produce hinted Type1s for the PostScript printer driver to include or do you rely on the auto-converted and unhinted Type3s to represent your carefully created fonts? Do you use the same long-dropped and unsupported Creo Acrobat recombine plugin that I do to produce composite CMYK artwork for a PDF workflow or do you print stuff in RGB or mono?

These are rhetorical questions. Your assertion is ridiculous without a radically different definition of “quality” than I would find acceptable.

converting the format to OpenType isn’t hard

It also isn’t useful or defensible. Assuming you mean TTF rather than CFF flavour OTFs you won’t be producing hinted glyphs – not even auto hinted. It’s also possible to convert the 8×8 BFont system font to OpenType, but what does that prove in terms of quality? Even if you do produce CFF flavour OTFs (and assuming your glyphs are even hinted in the first place, I don’t have high hopes) how do you convert tangent hints?

Rhetorical.

you have to be aware that Acorn DOESN’T stroke the outlines, whereas OpenType and PostScript do

Look this is utter bollocks. You’re completely wrong.

• PostScript Type3 fonts use the PostScript pixel touching rule which ensures all touched pixels are drawn. The glyphs are filled, not stroked (unless it’s a stroke font, but they are now obsolete… haven’t seen a stroke version of Courier since the 80s).
• PostScript Type1 fonts use a different pixel touching rule but also extensive hinting and dropout control heuristics. The glyphs are filled not stroked.
• OpenType can use CFF or TTF glyph definitions. The CCF case is as Type1 above, whereas TTF has a different pixel touching rule but also has an extensive and complicated hinting language including the ability to move individual pixels even after extensive outline massaging. The outline is filled, not stroked.
• Acorn format fonts fill closed contours with a different pixel touching routine again, but crucially does stroke open contours – this is the font skeleton which, for the record, is Acorn’s only contribution to font technology and not one that has been adopted elsewhere. In fact if you got your fonts from iSV with their risible “No Dropout Technology” then every glyph gets stroked as well as filled.

So I’m sorry, though you’re confident you “know” about fonts I can assure you that you’re very badly informed indeed.

I don’t understand how Nasta’līq Arabic manages to avoid line crashes

This is because you know next to nothing about OpenType. Start here, there’s a lot to learn: http://www.microsoft.com/typography/otspec/

Contextual glyph substitution is easy enough to implement in RISCOS 3.7.

Rubbish, it isn’t supported at all. The whole point about contextual substitution is that your text stays the same but the font chooses the right glyphs to use. Changing your code points to other code points is not contextual substitution. Change font and you’re left with rubbish.

I don’t know whether PCs and Macs do this properly yet

Clearly. Perhaps you should be less definitive about how great the Acorn format is then?

I’m not going to argue with you further on this topic, you have no idea what you’re talking about. And I really do.