Suggestion: API modules

I’ve been thinking about a future version of RISC OS, and I’d like to suggest a mechanism to allow modules to provide a consistent interface to clients through a single set of SWIs.

At the moment, as far as I can see (which, admittedly, isn’t far), there are multiple inconsistent ways to provide services like physical or virtual screens, serial ports, mass storage interfaces, etc.

I would suggest:

• APIs are defined requiring that r0 contains a handle for the particular item/device/object to be affected (like file handles, etc., but with a type-specific interface). The interface can contain up to 63 different SWIs. Chunk+63 would be reserved for API modules’ use (the “API SWI”).

• Define a new module flag to indicate that a module is a service provider.

• On initialisation of a module that has that flag set, if the chunk isn’t yet known, an API module is automatically created to handle that chunk of SWIs when called.

• Service provider modules can use the API SWI to register themselves with the API module, to get a mapping between a unique name and an identifier. (Probably triggering a service call or a vector associated with the chunk number.)

• Clients can list registered objects and claim them for their exclusive use. When they’re no longer needed, they can be released again.

• Clients calling a SWI are checked for ownership by the API module, and the SWI passed on to the provider module. That can include (most?) API SWI calls, for information’s sake

So, for example:

Mass storage (block size, number of blocks, read/write block):

Provider modules could include: SCSI, USB, SDHC, Partition manager, file systems supporting image files, etc.
Client modules could include: Partition manager, file systems of different types, maybe more. (Cache module?)

Example use:

1. someone plugs in a memory stick
2. USB provider registers it, triggering service call
3. ext3 filesystem module claims it, reads a block or two, finds it isn’t an ext3 file system, releases it, passes on service call
4. Partition manager module claims it, reads a block or two, it’s partitioned! Claim service call. Register mass storage for each partition (which triggers a service call each time…)

When a file system wants to read a block from an image file in a file system on a partition, the request is passed through two file systems and the partition manager to finally be filled by the USB mass storage module, but the device driver (or cache module) sends the data directly to the desired location.

Serial ports (rate, control (reset), etc.):

Providers: USB, RS483, I2C, etc.
Clients: dunno!

Screens (resolution, depth, monitor capacities, acceleration abilities, etc):

Providers: bcm2836, VNC, etc.
Clients: Screen manager (tells the Wimp where the screens are in relation to each other, whether to start a new session, etc.), movie players, etc.

Would be nice if a module flag could be added to indicate module running in user mode.

through a single set of SWIs.

Would that not suffer a penalty due to the overheads of SWI calling? Maybe a service provider should register entry points, in a manner akin to filing systems with FileSwitch?

there are multiple inconsistent ways to provide services like physical or virtual screens, serial ports, mass storage interfaces, etc.

That’s not really a surprise, as there are various things that are nonsensical when applied to others. For instance, “enumerate contents of root directory” when applied to a display, or “set background colour and clear to it” when applied to a serial port.

Personally, I think it is better to keep different sorts of devices separate, rather than trying to bodge some sort of interface to try to treat them all the same.

The filing system example use isn’t that different to how things currently behave. Remember that one can access the FileCore part and the FAT part (or, in the traditional case, the FAT part from within the FileCore part). You’re looking at the Filer that is talking to SCSIFS or DOSFS/FAT32FS and underneath all of this, they’re both talking to the SCSI driver. Just like how, back in the day, you could read DOS floppies thanks to DOSFS talking to ADFS in the background.

Serial ports (rate, control (reset), etc.)

There already exists a unified “blockdriver” protocol for doing this for many serial devices (though I’m not sure if one exists for SerialUSB?). But, again, there’s a lot of stuff in setting up serial devices (baud rate, flow control, word format…) that has no relevance at all to other devices.

Providers: USB, RS483, I2C, etc.
Clients: dunno!

Because the only similarity is that they are serial in hardware. IIC isn’t like RSxxx which really isn’t like USB.

Besides, RISC OS already provides a mechanism where you can do something like this – the device driver protocol in DeviceFS (kbd:, vdu:, printer:, serial:, etc) with some specific control provided using special fields in the filename, and other control provided using ioctl calls.

@Colin I don’t think that needs a flag; with the approach I’m working on, a module can spawn a task in a new memory slot and do whatever it likes with that application memory. It gets mapped in whenever that task (or any other tasks created by it) are running.

@Rick I think you missed the point a bit. Different types of devices would be kept separate. Similar devices, like file systems, will have a shared set of SWIs to do things like “enumerate contents of directory”. A Filer would just use that SWI for all file systems. Or am I missing the reasoning behind having NetFiler, RAMFSFiler, ResourceFiler, SDFSFiler, and CDFSFiler instead of just one Filer application?

I don’t know the overhead of a couple of SWI instructions, but I don’t think they’re the limiting bottleneck on anything.

Yes, the serial example (just an example!), isn’t great, but they do have significant similarities. The protocols that use them may well be different, but “Send a read command to device 16” still ends up with “transmit these bits” at the end of the day.

_{Sorry for the delay in replying – bicycle trip}

A Filer would just use that SWI for all file systems. Or am I missing the reasoning behind having NetFiler, RAMFSFiler, ResourceFiler, SDFSFiler, and CDFSFiler instead of just one Filer application?

I don’t think there’s much to be shared here that isn’t already – displaying directories is handled by the main Filer module, while the individual ADFSFiler/CDFSFiler/etc modules are mainly responsible for the iconbar icon and the options available from the iconbar menu, which vary between file systems. There are enough similarities between ADFSFiler, SCSIFiler, SDFSFiler and NVMeFiler that all four of them are built from the same component with different defines set, but for the others there’s not enough in common to make extending the Filer to share code worthwhile in my opinion.

Quick link to git project for the BlockDevices system developed for our NVMe (etc) project. It is an open source implementation of some of what Simon touches upon on his post. It is designed to allow an extensible mass storage solution.

https://github.com/c-jo/BlockDevices

FYI Andrew, the API documentation is missing from there. It looks like the fan control API has been uploaded by mistake!

Is this not it?

https://raw.githubusercontent.com/c-jo/BlockDevices/main/BlockDevices/API.txt

I can’t see fan control anywhere :)

That’s the sort of thing, Andrew, but it’s a general pattern that the OS could manage for us.

Sound output would be another example of a service that multiple different modules could provide for the user to choose from. Human input devices (keyboards, mice, touch screens, joysticks, etc.), too.

Human input devices (keyboards, mice, touch screens, joysticks, etc.), too.

A problem is that the existant API does not provide the sorts of facilities that modern devices may require. For example, touch pressure with a graphics tablet. Or multiple touch areas.

That being said, there is probably a need to have a layer in between the user and the hardware. I was informed a while ago that the touchpad facilities of one of the machines (PiTop?) was very basic, probably like my Android portable, whereas I am used to things like tap-drag and for it to auto-expand when it reaches the end, or two finger drag up/down to scroll, that sort of thing.

That being said…

it’s a general pattern that the OS could manage for us.

It is arguably things the OS should be managing for us. But looking at the proposal, what sort of level(s) are you aiming at here? You said earlier:

still ends up with “transmit these bits” at the end of the day.

But the problem with this is that all of the junk like word formats and baud rates are a part and parcel of the standard use case of serial ports; but none of that is ever a consideration for hardware like IIC from the point of view of the user or the programmer. I honestly don’t care if IIC transfer is performed by super-fancy hardware or a little girl with pompoms that knows how to signal zero and one in semaphore. It’s not my concern how.

Therefore, the requirements at driver/hardware level are going to be quite different to the requirements from user/application level. Take, for example, the two-finger scroll on a touchpad. The application should never see this. The driver should receive the information from the device (two points), and make inferences based upon what those points do (remember, pinching in and out does a zoom) and translate the actual points touched into what the user is wanting to do. More sophisticated drivers could support pinching to zoom in, and then moving both fingers at the same time to scroll around whatever was zoomed into. Again, it’s not the hardware that does it, it’s not the application that does it, it’s the bit in between.

Sound output would be another example of a service that multiple different modules could provide for the user to choose from.

Sound is a complex subject. You have five primary parts:

Input → Protocol conversion → Mixer → Output (→ protocol conversion)

Input is where the audio is coming from (microphone samples, internet stream, file…). Mixer is a way of munging different inputs together so that you can – for instance – hear an error beep at the same time as the music you’re listening to. Output is where the sound goes to, be it a speaker or a file or whatever.

And protocol conversion? This is the hard part. There are many types of audio (M4A, MP3, WAV, OGG, FLAC, etc etc) and once you have decoded that into a set of values that represent the voltage level on the wire, there are also many formats (how many bits? how fast? how many channels?).

Oh, and you may want to have the option of special marshalling so that if the input and output are using the same type and format and there’s no other input, to pass directly through without the conversions; essentially plugging the output directly into the input and bypassing the rest.

The funny thing is, sound is very hard to get right but it’s one of our most sensitive senses. Often we can put up with lower quality visuals, be it a budget horror movie or a 320×240 video on YouTube, but if the sound is bad then it can make for an unwatchable experience. Those of you who have ever had to suffer a “camrip” will know what I mean.

I believe there is work ongoing regarding the RISC OS sound system.

I don’t know the overhead of a couple of SWI instructions,

It’s not the number of SWIs, it’s how often they are called. If the code is doing something like byte reading at 115kps, it’s going to add up in a hurry.

but I don’t think they’re the limiting bottleneck on anything.

The original Acorn MIDI module had a SWI that would hand you a pointer to the module’s SWI dispatch code so you could directly call into it yourself without going through the SWI handler. Granted, this was in the days of 8MHz processors, but it would seem that for some applications the SWI mechanism takes too much time.

Note, also, that RISC OS is pretty much the only ARM OS that makes such intensive use of SWIs. Between the overheads of the handler, and the cache pollution (avoided on Linux, I think it passes the SWI number in a register), it’s clear that while it’s a nice idea for making easy OS calls, it’s not particularly practicable in this day and age.

Similar devices, like file systems, will have a shared set of SWIs to do things like “enumerate contents of directory”. A Filer would just use that SWI for all file systems.

This already happens. OS_GBPB which calls FileSwitch which talks to the individual filesystems. That there exist multiple calls to return more or less the same information is mostly a legacy issue.

Or am I missing the reasoning behind having NetFiler, RAMFSFiler, ResourceFiler, SDFSFiler, and CDFSFiler instead of just one Filer application?

Customisation. As pointed out above, there is a Filer that deals with the actual “here are the icons in a window”.
The ones you mention pretty much just plonk an icon on the iconbar, and make a menu to handle system specific things (like “Quit” for RAMFS and whether or not “Format” does anything). That your SDFS even has a Format option is because they’re mostly built from the same source rather than everybody reinventing the wheel over and over again. The RAMFS Filer is different because it has different requirements (such as warning upon PreShutdown and checking whether or not it should even appear).

I never said we should use existing APIs. A major advantage of a new approach is that you can create a new API. If necessary, someone can provide mapping modules for old APIs or legacy software.

what sort of level(s) are you aiming at here?

Any level where there’s a service that can be provided by multiple mechanisms.

Please don’t get hung up on the serial example, I’m hand waving!

Maybe it was a bad idea to lump in IIC with serial, but the service it provides could come from dedicated hardware, or a module using GPIO to set voltage levels directly, or something that beeps to tell the little girl which hand to raise. Like you say, we don’t care how it does it, we just want modules to be able to say “Hello! I’m an IIC interface, if you need me!”.

That’s precisely ensuring that:

• the application doesn’t have to worry about how it’s done, and

• module writers don’t have to understand a new mechanism for every new type of device.

What I’m suggesting doesn’t constrain any API in any way, it just means that there’s a simple, consistent mechanism to manage provided services.

Just out of interest, I just had BASIC call Wimp_ReadSysInfo 7 a million times on a Pi 3; by name it took: 4.55s, by number: 0.66s, and from assembler 0.33, so up to 3 million a second. [Edit: 0.30, if you don’t run it in a TaskWindow]

By the way, I’m not expecting people to send information a bit (or even a byte) at a time. Buffers exist for a reason (and CKernel pipes are better).

Just out of interest, I just had BASIC call Wimp_ReadSysInfo 7 a million times on a Pi 3; by name it took: 4.55s, by number: 0.66s, and from assembler 0.33, so up to 3 million a second.

The SWI is very quick when only only calling it from a tight loop, as there is no cache pressure. It’s what effect calling a SWI has on a memory bound process which is the important measure.

Is this not it?
https://raw.githubusercontent.com/c-jo/BlockDevices/main/BlockDevices/API.txt

How did you find that? I was going by the instructions in the link Andrew posted which says the API documentation is in BlockDevices/prminxml/blockdevices.xml

@Druck The point about millions a second is that the overhead of a SWI call, even hundreds or thousands a second will not tax modern hardware at all.

@Bryan I found it, too, I was working my way down from the root. It was in the parent directory. https://github.com/c-jo/BlockDevices/tree/main/BlockDevices

My mind view of a hardware abstraction layer

Mine, too. The kernel can understand the specific processor core type (they don’t change that often) and the platform specific part that knows (or can find out) what hardware is available can start up appropriate modules to provide access.

The point about millions a second is that the overhead of a SWI call, even hundreds or thousands a second will not tax modern hardware at all.

I can’t help but think that this sort of thinking is why my router takes three and a half minutes to start up.

the user really doesn’t give a monkeys about the bit underneath, “just make it work for me”

This. But, alas, we run into the problems of lack of finance and/or developer time. So I can’t, for example, plug a USBvideo device (documented protocol used by loads of things), read its parameters (size, colour depth, etc) and pull JPEGs from it, like can even be done on Android these days.

Does a USB parallel port work? I don’t think I have one to try.

Or WiFi printers. Actually, thanks to Dave’s excellent work, this does actually work with both old and new (that is to say, URF/AirPrint and PWG/IPP) but it’s a tad more complicated to set up than just pressing the Print button because, understandably, there was no appetite to rewrite huge swathes of Printers in order to support the things that it needs. Printers, actually, is so utterly braindead that selecting different print qualities (draft vs best) requires you to create two different printers with the necessary options. So you can understand why a potentially moveable device (DHCP) on a network would be… yeah…
I hope this will get a wider release soon <nudge><nudge> because it works well. Now that my little laser has kicked the bucket (PSU fault, it seems), I’m left with only having the HP inkjet and, thanks to Dave, I can print to it directly from within RISC OS.
Oh – and Dave – if you think setting up !IPPTrnspt is complicated, I’m more than happy to delete my current setup and recreate it, making a blog tutorial for every step and my previous method was to print to a PDF, FTP the PDF to my Android phone, and print it from that. So, there are degrees of complicated. ;)

module writers don’t have to understand a new mechanism for every new type of device

As I pointed out before (with the infamous IIC vs Serial discussion), there’s definitely something to be said for having unified protocols… but one shouldn’t be afraid of having different protocols for fundamentally different devices if there is no commonality.
Now, clearly IIC doesn’t look or feel or behave like serial. Well, how about MIDI? ;)

what hardware is available can start up appropriate modules to provide access.

This, to a degree. The ARM ecosystem developed using pre-baked chips with “this processor and this video unit and this I/O and it’s at these addresses”, which is different to the PC world where the very design of the machine led to a vast amount of mixing and matching. Those of you who remember (E)ISA cards will remember the amounts of fiddling with miniscule jumpers to set IRQs and such. But even so, hardware tended to be located at fixed addresses and/or provide some degree of basic compatibility unless told otherwise. Which is why my XP box initially boots as an 808x with a 640×480 VGA display.
This led to the idea that you can have one OS on x86 machines and as it loaded it would start probing to find out what was available, either asking the BIOS or poking around known locations.
On the ARM devices? Well, the embedded nature of things means that anything could be anywhere and it isn’t feasible to start poking random bits of memory in order to try to determine what hardware is available. I understand that some ARM devices now provide a “device tree” in order that the OS can parse to know what is where, but for the most part, it looks like for now we’ll still need platform-specific builds of RISC OS. Sure, modules can look and activate/deactive as necessary according to hardware availability (an example here is EtherUSB and EtherGENET in the Pi OS), but we’re still looking at hardware largely being board-specific.

Something that might want to be looked at, some day, is recognising more types of USB device and being able to automatically start up a driver to handle it. In the aforementioned case of USBvideo, there’s not a lot of purpose in having this baked into the ROM or otherwise loaded by default at boot. Far better for it to initialise itself (if not already loaded) once a compatible device has been plugged in.

But, alas, that’s “yet another protocol” on top of all of the rest. The history and legacy of the OS means that there’s a lot of work to do here. Though, clearly, more useful and fulfilling than six-plus-four (shh! don’t wake the troll).

As I pointed out before (with the infamous IIC vs Serial discussion), there’s definitely something to be said for having unified protocols… but one shouldn’t be afraid of having different protocols for fundamentally different devices if there is no commonality.

I really think you’re looking at my proposal backwards! I must not have explained it properly.

The APIs are the “different protocols for fundamentally different devices”; one for UART-like serial ports that do 9600-8N1 things, another for IIC, another for block storage devices, another for human input devices, another for screens, another for networking, another for USB ports, and so on.

APIs should be about how something is used, not how it’s implemented.

All I’m proposing is simply a common way for module writers to interact with the OS that lets the module tell the world “I’m an X, and this is what I call myself”, so that things that can use X things can claim and use them.

The APIs hide the fact that the serial port is on-board or attached to a USB port, the display is a monitor or a VNC frame buffer, the block device is an SDHC card, a USB flash drive, or a USB/SCSI/IDE/PATA/SATA/PCI attached hard disc, whatever.

So, for example, the client of a mass storage device should be able to see its block size and count, whether it’s removable or not, writeable or not, and that’s about it.

As to the hardware detection part, I’m not talking about poking around in memory, I’m talking about Pi versions of the OS being able to check what kind of Pi they’re running on and starting the appropriate modules for that one of the dozen variants.

USB drivers, independent of what manufacturer designed the chip the board uses, would register devices as they’re detected (at startup or on insertion) with their bus, device, and vendor/product ID numbers. There are online databases about what the devices are, all 20,000+ of them.

All I’m proposing is simply a common way for module writers to interact with the OS that lets the module tell the world “I’m an X, and this is what I call myself”, so that things that can use X things can claim and use them.

Ah, yes. That makes sense. :)

Various methods of a similar nature already exist, such as the serial blockdrivers that I mentioned earlier. Also how networking is able to plug in the various EtherXX modules.

I fear you may run into “the typical inertial problem”. That being the case of “I don’t want to use this because it won’t run on some random thirty year old machine that’s running an entirely different OS”.

Seems to me that, too often, the past is holding back the future.

Pi versions of the OS being able to check what kind of Pi they’re running on and starting the appropriate modules for that one of the dozen variants.

The modules check for themselves. Like EtherUSB and EtherGENET. There’s also DWCDriver (running) and XHCIDriver (dormant), but I’m not clued enough to know what that actually means. Something to do with USB…

with their bus, device, and vendor/product ID numbers

And their class information. It doesn’t matter much what the IDs are if it says it’s Steaming Audio → MIDI. Because you know Yamaha and Roland and (insert other brands) are going to have different IDs even though they look and feel pretty much the same. So it’s not necessarily necessary to know the exact IDs for generic devices. And if it is, well, that’s the domain of the driver, surely?

But, yes. It’s certainly an interesting idea worth persuing.

The modules check for themselves. Like EtherUSB and EtherGENET.

There go, you found one that bugs me.
One script for each different NIC so that you either have to ensure only the one you need is present or run them all sequentially and let them sort it out themselves – eventually.
Maybe.
Lots of little BASIC files. Code bloat the hard way, we’ll match the PC eventually.

Various methods of a similar nature already exist, such as the serial blockdrivers that I mentioned earlier.

Precisely, it’s re-inventing the wheel every time, usually in incompatible ways. The GraphicsV vector for displays is another example.

I fear you may run into “the typical inertial problem”. That being the case of “I don’t want to use this because it won’t run on some random thirty year old machine that’s running an entirely different OS”.

Probably, but old machines have all their hardware already supported, what’s the point in running a new OS on old hardware? It will only break things.

Perhaps we can go back to preferring forward compatibility (think anticipating 1280 × 1024 resolution in 1982!) over backwards (while still being able to run old applications).

Precisely, it’s re-inventing the wheel every time, usually in incompatible ways.

https://xkcd.com/927/

Probably, but old machines have all their hardware already supported

That’s not what I meant. I meant that if the older machines don’t support it, there may be resistance to supporting it on newer machines due to the desire to support older ones. We’re already seeing a bit of this in discussions regarding VFP maths. People don’t want to have to maintain/release multiple executables which means they either need to try to square a circle, or support the lowest common denominator.

Now, I have made the choice to no longer support/test with older machines because I’m not trying to make any sort of money from my software. If I was, then maybe I’d have to think about support for the ancient if that would represent a large portion of my potential user base (potentially more of them than us, so it’s not insignificant – I’ll leave it to somebody who actually sells stuff to give a better idea of the old/new percentages).

what’s the point in running a new OS on old hardware?

Emulation.

How did you find that? I was going by the instructions in the link Andrew posted

I didn’t even notice the instructions¹, so I just started looking at the files :)

¹ I’ve probably been ruined by far too many GitHub repositories putting absolutely nothing of value there.

Regarding the issue of calling lots of SWIs;

Linux leaves the data part empty, and you just call ‘SWI 0’ with the syscall number in R0 – so it does leave the ‘data’ field unused; I hadn ‘t really thought about the cache pollution issue, but I can day that RISC OS uses SWIs a LOT during startup (from watching instruction by instruction in an emulator). The overhead (number of instructions overhead) is actually very low!

However, Harvard architecture is another issue.

Regarding possible solutions; perhaps we could invent a replacement SYSCALL type API, using a specific SWI. That SWI could then access the new API using values in X0, etc

The problem then is old software that we don’t have control of. A stopgap measure could be to use a ‘window’ within which the SWI code would assume that any SWI calls made are the ‘new’ format, and so the SWI handler could avoid reading the ‘data’ at that particular SWI instruction address. Such a window could be hardcoded, if the ROM remains high. An example would be, if the ROM is at &FC000000:

SUBS temp_register,LR,#&FC000000
BMI old_RISCOS_SWI_handler
SUBS temp_register,temp_register,#(number of bytes above &FC000000 using the new handler)
BMI old_RISCOS_SWI_handler
(new handler code starts here)

In reality you could use less instructions to do this. Old RISC OS code, and most of the ROM, could be left using the old handler, and a small window at the start of the ROM could use the new handler, and the size of the window can be moved ‘upwards’ as more of the kernel, then rest of the OS, is moved to the new code. Something along these lines may help a painless transition to a new SWI API?

Regarding possible solutions; perhaps we could invent a replacement SYSCALL type API, using a specific SWI. That SWI could then access the new API using values in X0, etc

Like https://www.riscosopen.org/wiki/documentation/show/CallASWI ?