Suggestion: API modules
Simon Willcocks (1499) 513 posts |
I’ve been thinking about a future version of RISC OS, and I’d like to suggest a mechanism to allow modules to provide a consistent interface to clients through a single set of SWIs. At the moment, as far as I can see (which, admittedly, isn’t far), there are multiple inconsistent ways to provide services like physical or virtual screens, serial ports, mass storage interfaces, etc. I would suggest:
So, for example: Mass storage (block size, number of blocks, read/write block): Provider modules could include: SCSI, USB, SDHC, Partition manager, file systems supporting image files, etc. Example use:
When a file system wants to read a block from an image file in a file system on a partition, the request is passed through two file systems and the partition manager to finally be filled by the USB mass storage module, but the device driver (or cache module) sends the data directly to the desired location. Serial ports (rate, control (reset), etc.): Providers: USB, RS483, I2C, etc. Screens (resolution, depth, monitor capacities, acceleration abilities, etc): Providers: bcm2836, VNC, etc. |
Colin Ferris (399) 1814 posts |
Would be nice if a module flag could be added to indicate module running in user mode. |
Rick Murray (539) 13840 posts |
Would that not suffer a penalty due to the overheads of SWI calling? Maybe a service provider should register entry points, in a manner akin to filing systems with FileSwitch?
That’s not really a surprise, as there are various things that are nonsensical when applied to others. For instance, “enumerate contents of root directory” when applied to a display, or “set background colour and clear to it” when applied to a serial port. Personally, I think it is better to keep different sorts of devices separate, rather than trying to bodge some sort of interface to try to treat them all the same. The filing system example use isn’t that different to how things currently behave. Remember that one can access the FileCore part and the FAT part (or, in the traditional case, the FAT part from within the FileCore part). You’re looking at the Filer that is talking to SCSIFS or DOSFS/FAT32FS and underneath all of this, they’re both talking to the SCSI driver. Just like how, back in the day, you could read DOS floppies thanks to DOSFS talking to ADFS in the background.
There already exists a unified “blockdriver” protocol for doing this for many serial devices (though I’m not sure if one exists for SerialUSB?). But, again, there’s a lot of stuff in setting up serial devices (baud rate, flow control, word format…) that has no relevance at all to other devices.
Because the only similarity is that they are serial in hardware. IIC isn’t like RSxxx which really isn’t like USB. Besides, RISC OS already provides a mechanism where you can do something like this – the device driver protocol in DeviceFS (kbd:, vdu:, printer:, serial:, etc) with some specific control provided using special fields in the filename, and other control provided using ioctl calls. |
Simon Willcocks (1499) 513 posts |
@Colin I don’t think that needs a flag; with the approach I’m working on, a module can spawn a task in a new memory slot and do whatever it likes with that application memory. It gets mapped in whenever that task (or any other tasks created by it) are running. @Rick I think you missed the point a bit. Different types of devices would be kept separate. Similar devices, like file systems, will have a shared set of SWIs to do things like “enumerate contents of directory”. A Filer would just use that SWI for all file systems. Or am I missing the reasoning behind having NetFiler, RAMFSFiler, ResourceFiler, SDFSFiler, and CDFSFiler instead of just one Filer application? I don’t know the overhead of a couple of SWI instructions, but I don’t think they’re the limiting bottleneck on anything. Yes, the serial example (just an example!), isn’t great, but they do have significant similarities. The protocols that use them may well be different, but “Send a read command to device 16” still ends up with “transmit these bits” at the end of the day. Sorry for the delay in replying – bicycle trip |
Cameron Cawley (3514) 157 posts |
I don’t think there’s much to be shared here that isn’t already – displaying directories is handled by the main Filer module, while the individual ADFSFiler/CDFSFiler/etc modules are mainly responsible for the iconbar icon and the options available from the iconbar menu, which vary between file systems. There are enough similarities between ADFSFiler, SCSIFiler, SDFSFiler and NVMeFiler that all four of them are built from the same component with different defines set, but for the others there’s not enough in common to make extending the Filer to share code worthwhile in my opinion. |
Andrew Rawnsley (492) 1445 posts |
Quick link to git project for the BlockDevices system developed for our NVMe (etc) project. It is an open source implementation of some of what Simon touches upon on his post. It is designed to allow an extensible mass storage solution. |
Bryan Hogan (339) 592 posts |
FYI Andrew, the API documentation is missing from there. It looks like the fan control API has been uploaded by mistake! |
Chris Mahoney (1684) 2165 posts |
Is this not it? https://raw.githubusercontent.com/c-jo/BlockDevices/main/BlockDevices/API.txt I can’t see fan control anywhere :) |
Simon Willcocks (1499) 513 posts |
That’s the sort of thing, Andrew, but it’s a general pattern that the OS could manage for us. Sound output would be another example of a service that multiple different modules could provide for the user to choose from. Human input devices (keyboards, mice, touch screens, joysticks, etc.), too. |
Rick Murray (539) 13840 posts |
A problem is that the existant API does not provide the sorts of facilities that modern devices may require. For example, touch pressure with a graphics tablet. Or multiple touch areas. That being said, there is probably a need to have a layer in between the user and the hardware. I was informed a while ago that the touchpad facilities of one of the machines (PiTop?) was very basic, probably like my Android portable, whereas I am used to things like tap-drag and for it to auto-expand when it reaches the end, or two finger drag up/down to scroll, that sort of thing. That being said…
It is arguably things the OS should be managing for us. But looking at the proposal, what sort of level(s) are you aiming at here? You said earlier:
But the problem with this is that all of the junk like word formats and baud rates are a part and parcel of the standard use case of serial ports; but none of that is ever a consideration for hardware like IIC from the point of view of the user or the programmer. I honestly don’t care if IIC transfer is performed by super-fancy hardware or a little girl with pompoms that knows how to signal zero and one in semaphore. It’s not my concern how. Therefore, the requirements at driver/hardware level are going to be quite different to the requirements from user/application level. Take, for example, the two-finger scroll on a touchpad. The application should never see this. The driver should receive the information from the device (two points), and make inferences based upon what those points do (remember, pinching in and out does a zoom) and translate the actual points touched into what the user is wanting to do. More sophisticated drivers could support pinching to zoom in, and then moving both fingers at the same time to scroll around whatever was zoomed into. Again, it’s not the hardware that does it, it’s not the application that does it, it’s the bit in between.
Sound is a complex subject. You have five primary parts: Input → Protocol conversion → Mixer → Output (→ protocol conversion) Input is where the audio is coming from (microphone samples, internet stream, file…). Mixer is a way of munging different inputs together so that you can – for instance – hear an error beep at the same time as the music you’re listening to. Output is where the sound goes to, be it a speaker or a file or whatever. And protocol conversion? This is the hard part. There are many types of audio (M4A, MP3, WAV, OGG, FLAC, etc etc) and once you have decoded that into a set of values that represent the voltage level on the wire, there are also many formats (how many bits? how fast? how many channels?). Oh, and you may want to have the option of special marshalling so that if the input and output are using the same type and format and there’s no other input, to pass directly through without the conversions; essentially plugging the output directly into the input and bypassing the rest. The funny thing is, sound is very hard to get right but it’s one of our most sensitive senses. Often we can put up with lower quality visuals, be it a budget horror movie or a 320×240 video on YouTube, but if the sound is bad then it can make for an unwatchable experience. Those of you who have ever had to suffer a “camrip” will know what I mean. I believe there is work ongoing regarding the RISC OS sound system.
It’s not the number of SWIs, it’s how often they are called. If the code is doing something like byte reading at 115kps, it’s going to add up in a hurry.
The original Acorn MIDI module had a SWI that would hand you a pointer to the module’s SWI dispatch code so you could directly call into it yourself without going through the SWI handler. Granted, this was in the days of 8MHz processors, but it would seem that for some applications the SWI mechanism takes too much time. Note, also, that RISC OS is pretty much the only ARM OS that makes such intensive use of SWIs. Between the overheads of the handler, and the cache pollution (avoided on Linux, I think it passes the SWI number in a register), it’s clear that while it’s a nice idea for making easy OS calls, it’s not particularly practicable in this day and age.
This already happens. OS_GBPB which calls FileSwitch which talks to the individual filesystems. That there exist multiple calls to return more or less the same information is mostly a legacy issue.
Customisation. As pointed out above, there is a Filer that deals with the actual “here are the icons in a window”. |
Simon Willcocks (1499) 513 posts |
I never said we should use existing APIs. A major advantage of a new approach is that you can create a new API. If necessary, someone can provide mapping modules for old APIs or legacy software.
Any level where there’s a service that can be provided by multiple mechanisms. Please don’t get hung up on the serial example, I’m hand waving! Maybe it was a bad idea to lump in IIC with serial, but the service it provides could come from dedicated hardware, or a module using GPIO to set voltage levels directly, or something that beeps to tell the little girl which hand to raise. Like you say, we don’t care how it does it, we just want modules to be able to say “Hello! I’m an IIC interface, if you need me!”. That’s precisely ensuring that:
What I’m suggesting doesn’t constrain any API in any way, it just means that there’s a simple, consistent mechanism to manage provided services. Just out of interest, I just had BASIC call Wimp_ReadSysInfo 7 a million times on a Pi 3; by name it took: 4.55s, by number: 0.66s, and from assembler 0.33, so up to 3 million a second. [Edit: 0.30, if you don’t run it in a TaskWindow] By the way, I’m not expecting people to send information a bit (or even a byte) at a time. Buffers exist for a reason (and CKernel pipes are better). |
David J. Ruck (33) 1635 posts |
The SWI is very quick when only only calling it from a tight loop, as there is no cache pressure. It’s what effect calling a SWI has on a memory bound process which is the important measure. |
Bryan Hogan (339) 592 posts |
How did you find that? I was going by the instructions in the link Andrew posted which says the API documentation is in BlockDevices/prminxml/blockdevices.xml |
Simon Willcocks (1499) 513 posts |
@Druck The point about millions a second is that the overhead of a SWI call, even hundreds or thousands a second will not tax modern hardware at all. @Bryan I found it, too, I was working my way down from the root. It was in the parent directory. https://github.com/c-jo/BlockDevices/tree/main/BlockDevices |
Steve Pampling (1551) 8170 posts |
the application doesn’t have to worry about how it’s done, and module writers don’t have to understand a new mechanism for every new type of device. My mind view of a hardware abstraction layer – the user really doesn’t give a monkeys about the bit underneath, “just make it work for me”. |
Simon Willcocks (1499) 513 posts |
Mine, too. The kernel can understand the specific processor core type (they don’t change that often) and the platform specific part that knows (or can find out) what hardware is available can start up appropriate modules to provide access. |
Rick Murray (539) 13840 posts |
I can’t help but think that this sort of thinking is why my router takes three and a half minutes to start up.
This. But, alas, we run into the problems of lack of finance and/or developer time. So I can’t, for example, plug a USBvideo device (documented protocol used by loads of things), read its parameters (size, colour depth, etc) and pull JPEGs from it, like can even be done on Android these days. Does a USB parallel port work? I don’t think I have one to try. Or WiFi printers. Actually, thanks to Dave’s excellent work, this does actually work with both old and new (that is to say, URF/AirPrint and PWG/IPP) but it’s a tad more complicated to set up than just pressing the Print button because, understandably, there was no appetite to rewrite huge swathes of Printers in order to support the things that it needs. Printers, actually, is so utterly braindead that selecting different print qualities (draft vs best) requires you to create two different printers with the necessary options. So you can understand why a potentially moveable device (DHCP) on a network would be… yeah…
As I pointed out before (with the infamous IIC vs Serial discussion), there’s definitely something to be said for having unified protocols… but one shouldn’t be afraid of having different protocols for fundamentally different devices if there is no commonality.
This, to a degree. The ARM ecosystem developed using pre-baked chips with “this processor and this video unit and this I/O and it’s at these addresses”, which is different to the PC world where the very design of the machine led to a vast amount of mixing and matching. Those of you who remember (E)ISA cards will remember the amounts of fiddling with miniscule jumpers to set IRQs and such. But even so, hardware tended to be located at fixed addresses and/or provide some degree of basic compatibility unless told otherwise. Which is why my XP box initially boots as an 808x with a 640×480 VGA display. Something that might want to be looked at, some day, is recognising more types of USB device and being able to automatically start up a driver to handle it. In the aforementioned case of USBvideo, there’s not a lot of purpose in having this baked into the ROM or otherwise loaded by default at boot. Far better for it to initialise itself (if not already loaded) once a compatible device has been plugged in. But, alas, that’s “yet another protocol” on top of all of the rest. The history and legacy of the OS means that there’s a lot of work to do here. Though, clearly, more useful and fulfilling than six-plus-four (shh! don’t wake the troll). |
Simon Willcocks (1499) 513 posts |
I really think you’re looking at my proposal backwards! I must not have explained it properly. The APIs are the “different protocols for fundamentally different devices”; one for UART-like serial ports that do 9600-8N1 things, another for IIC, another for block storage devices, another for human input devices, another for screens, another for networking, another for USB ports, and so on. APIs should be about how something is used, not how it’s implemented. All I’m proposing is simply a common way for module writers to interact with the OS that lets the module tell the world “I’m an X, and this is what I call myself”, so that things that can use X things can claim and use them. The APIs hide the fact that the serial port is on-board or attached to a USB port, the display is a monitor or a VNC frame buffer, the block device is an SDHC card, a USB flash drive, or a USB/SCSI/IDE/PATA/SATA/PCI attached hard disc, whatever. So, for example, the client of a mass storage device should be able to see its block size and count, whether it’s removable or not, writeable or not, and that’s about it. As to the hardware detection part, I’m not talking about poking around in memory, I’m talking about Pi versions of the OS being able to check what kind of Pi they’re running on and starting the appropriate modules for that one of the dozen variants. USB drivers, independent of what manufacturer designed the chip the board uses, would register devices as they’re detected (at startup or on insertion) with their bus, device, and vendor/product ID numbers. There are online databases about what the devices are, all 20,000+ of them. |
Rick Murray (539) 13840 posts |
Ah, yes. That makes sense. :) Various methods of a similar nature already exist, such as the serial blockdrivers that I mentioned earlier. Also how networking is able to plug in the various EtherXX modules. I fear you may run into “the typical inertial problem”. That being the case of “I don’t want to use this because it won’t run on some random thirty year old machine that’s running an entirely different OS”. Seems to me that, too often, the past is holding back the future.
The modules check for themselves. Like EtherUSB and EtherGENET. There’s also DWCDriver (running) and XHCIDriver (dormant), but I’m not clued enough to know what that actually means. Something to do with USB…
And their class information. It doesn’t matter much what the IDs are if it says it’s Steaming Audio → MIDI. Because you know Yamaha and Roland and (insert other brands) are going to have different IDs even though they look and feel pretty much the same. So it’s not necessarily necessary to know the exact IDs for generic devices. And if it is, well, that’s the domain of the driver, surely? But, yes. It’s certainly an interesting idea worth persuing. |
Steve Pampling (1551) 8170 posts |
There go, you found one that bugs me. |
Simon Willcocks (1499) 513 posts |
Precisely, it’s re-inventing the wheel every time, usually in incompatible ways. The GraphicsV vector for displays is another example.
Probably, but old machines have all their hardware already supported, what’s the point in running a new OS on old hardware? It will only break things. Perhaps we can go back to preferring forward compatibility (think anticipating 1280 × 1024 resolution in 1982!) over backwards (while still being able to run old applications). |
Rick Murray (539) 13840 posts |
That’s not what I meant. I meant that if the older machines don’t support it, there may be resistance to supporting it on newer machines due to the desire to support older ones. We’re already seeing a bit of this in discussions regarding VFP maths. People don’t want to have to maintain/release multiple executables which means they either need to try to square a circle, or support the lowest common denominator. Now, I have made the choice to no longer support/test with older machines because I’m not trying to make any sort of money from my software. If I was, then maybe I’d have to think about support for the ancient if that would represent a large portion of my potential user base (potentially more of them than us, so it’s not insignificant – I’ll leave it to somebody who actually sells stuff to give a better idea of the old/new percentages).
Emulation. |
Chris Mahoney (1684) 2165 posts |
I didn’t even notice the instructions1, so I just started looking at the files :) 1 I’ve probably been ruined by far too many GitHub repositories putting absolutely nothing of value there. |
tymaja (278) 174 posts |
Regarding the issue of calling lots of SWIs; Linux leaves the data part empty, and you just call ‘SWI 0’ with the syscall number in R0 – so it does leave the ‘data’ field unused; I hadn ‘t really thought about the cache pollution issue, but I can day that RISC OS uses SWIs a LOT during startup (from watching instruction by instruction in an emulator). The overhead (number of instructions overhead) is actually very low! However, Harvard architecture is another issue. Regarding possible solutions; perhaps we could invent a replacement SYSCALL type API, using a specific SWI. That SWI could then access the new API using values in X0, etc The problem then is old software that we don’t have control of. A stopgap measure could be to use a ‘window’ within which the SWI code would assume that any SWI calls made are the ‘new’ format, and so the SWI handler could avoid reading the ‘data’ at that particular SWI instruction address. Such a window could be hardcoded, if the ROM remains high. An example would be, if the ROM is at &FC000000: SUBS temp_register,LR,#&FC000000 In reality you could use less instructions to do this. Old RISC OS code, and most of the ROM, could be left using the old handler, and a small window at the start of the ROM could use the new handler, and the size of the window can be moved ‘upwards’ as more of the kernel, then rest of the OS, is moved to the new code. Something along these lines may help a painless transition to a new SWI API? |
Stuart Swales (8827) 1357 posts |
Like https://www.riscosopen.org/wiki/documentation/show/CallASWI ? |