Compose mode to imput UTF-8

First post, so hi to everyone!

I’m used to input a variety of unicode characters using the compose system (from old dec terminals) existing in the X server or in the plan9 OS.

For those not familiar with this input system:
-You push the compose key, generally Altgr, to enter compose mode.
-Then input a sequence of (mostly 2) keys.
-The sequence is look up in a table and the correspondent character is input, if founded. if not nothing is input and the compose mode ends.

You can use different types of mnemonics, for example:
/O or es (for empty set) : ∅
<- or la (for left arrow) : ←
You have the idea (I hope that is shown correctly, using netsurf in riscos).

I found AltKeys:
http://www.xat.nl/riscos/sw/altkeys/index.htm
but… no UTF-8 support, no source code.

I found later Rick Murray’s MoreKeys:
https://heyrick.eu/software/morekeys
which in my opinion is going to be very handy if the OS gets a gui for touchscreen devices. I thought I could learn how to catch and replace keyboard input in a module but… no sources.

So, any advice or pointing to resources I can use to achieve this (after learning how to write modules in the first place)?

Thanks in advance.

You might also give !KeyMap a try. Be aware though that Unicode on RISC OS is severely underdeveloped.

Oh, I know it already. Its a 1 key → 1 char mapping application. I could modify the keymap module but… no source!

MoreKeys supports UTF-8. You just need to set the system alphabet to “UTF8” in order for it to work (it’s pointless otherwise as the result would be mojibake).

The module is extremely simple, you can probably figure it out just by looking at a disassembly in Zap.
Basically it waits for the hotkey press and if it is found it’ll push status to a bit of memory that is used as a pollword, in order that the front end application notices, wakes up, and does something.

As for writing a module, here’s how to do it on assembler: https://heyrick.eu/blog/index.php?diary=20120528

And if you’re doing anything more complicated, it’s better to use C so you can concentrate on writing the code and not dealing with tedious rubbish like ensuring the stack is unwound correctly (as RISC OS has exactly zero tolerance for having the SVC stack messed up). Here’s writing modules in C: https://heyrick.eu/blog/index.php?diary=20150323

MoreKeys supports UTF-8. You just need to set the system alphabet to “UTF8” in order for it to work (it’s pointless otherwise as the result would be mojibake).

I want to avoid the use of alphabet. Right now it will make the use of other applications a nightmare, especially in the menus. Would it not be better to adapt the entire system to use only UTF-8, and let the alphabet subsytem as a legacy facility, so old applications can be patched or wrapped to tell the system “I’m too old, please translate my IO text as If we were using alphabet”?

What I want is to push a string of bytes representing a unicode UTF-8 encoded char. I will try to modify the applications I need to
interact with UTF-8 chars to treat text data as UTF-8. Other applications will treat each byte as the corresponding 8bit char from the alphabet in use. It will be rubbish to our eyes, but the next UTF-8 capable program that read that data will see correct UTF-8 chars.

Every text data that flows out there is mostly UTF-8. It is incredible that risc os doesn’t have resolved this issue already.

Thanks for the links, I’ve been there already!

The module is extremely simple, you can probably figure it out just by looking at a disassembly in Zap.

Basically it waits for the hotkey press and if it is found it’ll push status to a bit of memory that is used as a pollword, in order that the front end application notices, wakes up, and does something.

Right now it will make the use of other applications a nightmare

I fully agree. Because there is no sort of automatic fallback to an eight bit character set, it’s currently an all or nothing. One way or another, stuff will end up looking a mess.

Granted, certain purists say that the font manager shouldn’t support fallback if one explicitly specifies UTF-8, however this means either a lot of work in getting the Wimp to do it, or nothing at all gets done. Guess which option has been chosen thus far. ;)

Every text data that flows out there is mostly UTF-8.

Yeah, I’ve already mentioned a few times that my little Creative Zen music player (from circa 2005?) can handle foreign characters without issue and…

It would be easier for a novice to identify the SWIs and friends.

There’s a document that describes all of the SWIs.
The module itself is something like 280 bytes (from memory). It’s not at all complicated.

Don’t you have a copy of the sources?

Would it help if you ran Ricks module through Armalyser?

OK, it’s probably missing a few, slightly zany?¹, comments alongside the assembler², but it’s probably short enough to follow anyway.

¹ It’s Rick, would it be otherwise?

It was written in ObjAsm. The main program (Wimp app) is written in C, but the intercept module is a tiny bit of assembler as it didn’t need to be any more complicated than that.

I can’t comment on the comments. ;)

The module itself is something like 280 bytes (from memory). It’s not at all complicated.

Oh, I opened the module with zap and the assembly mode is even showing me the names of the SWIs.
Now it seems really silly to be asking for the sources…

Yup. Zap is good like that.

To help you along, I’ve also dropped a copy of the source code used to make the module here:

https://heyrick.eu/blog/files/morekeys_module_source_0-03.txt

Yeah… a few quirky comments, as Steve mentioned. ;)

What you’re missing:

• h.SWINames – that’s part of the DDE, look in the Examples.
• h.equx – a copy here: https://heyrick.eu/software/resfinder/equx_h.html

The other two (pushpull, debug) I don’t think are actually used. So try commenting them out.

nemo, if he’s lurking, would tell you that this is a horrible way to intercept keypresses and that one should write a keyboard handler. That there’s practically zero documentation on how to write keyboard handlers and we’re really operating above the level of keyboards and into the realm of associating actions with keypresses in numerous potential languages is why I didn’t do it that way. If intercepting a specific hotkey combination requires one to “write a keyboard handler”, then RISC OS’ keyboard handling is even more broken than I usually give it credit for.
Anyway, just saying. There’s a right way to do things and this probably isn’t it. However, it does work on a regular RISC OS system and that’s the more important thing. We can have perfect, or we can have it today. Pick one. ;)

There used to be !IKHG – International Keyboard Handler Generator – but sadly it’s never been updated since 26-bit days, and the keyboard handlers it generated don’t work since then either 8~( – I used to have proper keyboard handlers I’d written using it, so I could touch type Russian, Greek and Hindi on my RiscPC, with standard keyboard layouts for those languages… could switch keyboard layout at a click on the appropriate icon on the icon bar…

To help you along, I've also dropped a copy of the source code used to make the module here:
<a href="https://heyrick.eu/blog/files/morekeys_module_source_0-03.txt">https://heyrick.eu/blog/files/morekeys_module_source_0-03.txt</a>

Thanks Rick for sharing it!

There used to be !IKHG – International Keyboard Handler Generator – but sadly it’s never been updated

Of course the convenience of this depends on the mnemonics you use and the number of characters the font supports. For example you can prefix a letter of the latin alphabet with a * to input greek letters. *a → α, *b → β, so you just have to remember “* is for Greek”. If you need to input ancient Greek with its orthography, things get more complicated, in that case is better to use a keyboard layout for ancient Greek.

If some Plan 9 user is around here, I’m pretty sure he/she will understand that once you get used to this way of typing, you can’t live
without it anymore!

Would it help if you ran Ricks module through Armalyser?

Thanks Steve for pointing me to this application. I’m amazed by its output.
I can see a lot of work spent there.

All this without having to change the keyboard layout

But I want to change the keyboard layout – I want to be able to change it quickly and easily to the proper layout for the language I’m currently typing in, and change it again and again, every time I switch languages. Otherwise I’m hunting for the right key all the time, which is incredibly slow compared with proper typing.

Obviously if I’ve got a font that has all the characters for all the languages I use then I don’t need to change font – apart from when I want bold or italic or small caps or cursive or whatever. That I didn’t have back in RiscPC days – I had fonts I could switch between Latin and Cyrillic (for Russian), or between Latin and Greek, or between Latin and Devanagari (for Hindi), but in those days fonts couldn’t have enough glyphs for all four languages in the same font*. But now…

• With floating accents – which could be shifted into the correct locations using Impression’s kerning feature & an app I wrote to do it automatically – you could cover all the Latin alphabet languages (except Vietnamese) with a single keyboard layout and a single 224-glyph font, and the whole Devanagari system or Greek or Cyrillic with another keyboard layout and the same font.

But I want to change the keyboard layout

Oh, I know that you want to do that, I just was pointing out that
changing the layout is not a substitution of what the compose key input
offers. For example, I’m used to the US layout. If I have to write in Spanish,
or Portuguese I could do it comfortably using the compose input system, it’s almost
so quickly to type the dead key for acute accent and then a, than to type the compose key
and then a a. The advantage is that I don’t need to think “where the … is now the at sign?” as when I have to switch layouts.
For typing Russian I’m trying to accustom myself to the ЙЦУКЕН layout, instead of a phonetic one. But
if I have to enter small portions like a word, or use Cyrillic characters as symbols the
compose input is handier. I wouldn’t write Japanese using compose key input, but using a specialized IME,
at least to be able to select among the so many homophonic kanji, although kana could be input
with a kana layout.

What I mean is that the compose key input system doesn’t substitute the need of different layouts.

In Xorg I use the command setxkbmap. You can specify the layouts you want to use, the compose key, the key to change the layouts, &c, although I prefer configuring key bindings in the windows manager to change the layout to the one I want. So don’t worry, I will be asking for advice to do something similar soon…

Didn’t you take a look at the code for creating new layouts? IIRC it was well documented.

I don’t know the keyboard layouts for any Latin alphabet language apart from English; I’m very happy typing French or German (the only other Latin alphabet languages I really know much at all) on an English keyboard layout and using floating accents on Alt-Number. But for Greek, Russian or Hindi trying to type on an English language keyboard is hopeless – especially Hindi, where phonetic just doesn’t work at all, with no 1:1 correspondence between English and Hindi phonemes at all.

If I have to write in Spanish,
or Portuguese I could do it comfortably using the compose input system

For western (Latin) languages, the “international” keyboard layout is your friend. There’s US International, and a British version. It’s not so different to the Alt-: method used by RISC OS…

…and given that RISC OS did this in 1992 and Windows in 1995, it p<bleep>es me off enormously that in 2023 Android is utterly hopeless at managing this with a British layout Bluetooth keyboard. I’m in the process of trying to figure out enough of Keyboard Helper to do what RISC OS does…

Of course, this isn’t much help for non Latin languages which are a trauma all their own, but I’ve not run into difficulty (Android aside) ripping in French and Spanish using a British layout.

Here’s a proper Hindi keyboard, laid out exactly the way old Hindi manual typewriters were, so I could touch-type on it: https://clive.semmens.org.uk/024_ROUGOL/PO.html – it’s not remotely phonetically similar to QWERTY.

Using !IKHG I was able to write a keyboard driver that generated the top-bit-set character codes for where I’d put those characters in the old 224 character fonts – it would be better of course to generate UTF8 codes for them, but in those days you couldn’t have more than 224 characters in a font. (Well – 223 really, since #128 is delete.)

For western (Latin) languages, the “international” keyboard layout is your friend

Not mine. You have to look when typing a char like the caret symbol that corresponds now to a dead key if the
next char could be modified, so you should have to push space before. And when you are in other computer with the normal layout, you’ll end up inserting spaces everywhere without been aware.

Are your keyboard 26bit modules downloadable?

Are your keyboard 26bit modules downloadable?

Mine, you mean? Not currently. I’m not sure whether I’ve still got copies. If I can find them I can let you have them if you want them.

I’m not at the Pi at the moment – I’ll probably be able to take a look later this afternoon.

To help you along, I’ve also dropped a copy of the source code used to make the module here:

I’m glad your v0.08 is using a jump table in the SWI handler rather than the compare and branch chain of v0.03.

next char could be modified, so you should have to push space before.

Yeah, I’ve just looked it up on Google. Has the US international layout always been like that, or has Microsoft broken it on recent years?

Using the UK Extended, it’s a matter of holding AltGr and something like ‘^’ followed by a keeps like ‘a’ to arrive at ‘â’. Similar to how RISC OS does it, except certain common keys are easier (AltGr and e → é).

compare and branch chain of v0.03.

It could be better, yes… ;)

I’ve found my little 26-bit era apps that change the keyboards for Russian or Hindi, but not Greek or our special Journal of Physiology keyboard. Sadly I’ve not found a copy of !IKHG that I made the modules inside them with. They certainly don’t work on the Pi, and I’ve nothing else to check them on. If anyone really wants to have a look at them, speak up and I’ll put them in a quiet corner of my website and give you a link. Other than looking what’s going on inside them I’m not sure what use they’d be – they don’t generate UTF-8, they generate 8-bit codes that correspond to non-standard character encodings that allowed me to cram the Latin alphabet, with a comprehensive set of non-spacing diacritics for the Latin alphabet, plus either Cyrillic or Devanagari and all their diacritics, into 223 character fonts…

(Edited: found the (very simple) app to return keyboard to default.)