Multi-core Wimp

However it is clear that we can run RISC OS as a Linux application on both 32 bit and 64 bit ARM Linux platforms

Is it? Why aren’t we all doing that then?

Cooperative multi-tasking with N processors should give the WIMP the task to keep track of up-to N concurrent tasks running, right?

More, if you consider the kernel workspace as a thread context and switch between them while a core is idle. I’d rather a lighter-weight approach, though.

Thanks – updated – I did try but didn’t realise the quote needed to start on the same line …

There’s also the < blockquote > method, like HTML where the end of the block is the tag with a /
That becomes even more useful if you want to quote a quote (nested)

While on RISC OS a user-space process CAN access the kernel. For instance, whenever a SWI returns to your program a pointer to some data, you can access it directly without doing anything special.

But are SWIs part of the kernel? Can you say for example that the FontManager is part of the kernel?

Personally I think we should start small and not care too much about the icing on the cake.

1) Write a little kernel to manage processes, threads, memory and environment handlers, provide the base calls and utility routines (data exchange, mutexes, …).
2) Put a big lock on SWI management to ensure that only a single thread can access them at a time and start modifying the Wimp, OS_CLI, C’s system() to make use of the kernel.

At that point you will basically have a system that behaves like the current RISCOS with the extra ability to use threads.

From there you can start to see how to improve the system:
- display contexts per processes
- thread save SWI lists
- maybe reimplement some modules as proxies exchanging messages with user mode server tasks so that the modules data remains protected.

If you’re running in a privileged mode, you’re in the kernel.

https://github.com/Simon-Willcocks/RISC-OS-Kernel-in-C

What I’m looking forward to is having a secure OS, and RISC OS running in non-secure (less privileged) environment.

If you’re running in a privileged mode, you’re in the kernel.

Is that really true for modern ARMs? I know that there is a new “system” mode.
I presume that it protects you user mode and I presume that in the SWI handler you can check the current mode and reject SWIs calls in system mode to protect your micro-kernel.

I don’t think so. As I understand it, the system mode just shares the user mode stack, but can access anything.

There is PXN, which is memory that cannot be executed in privileged modes.

@ Andre’

But are SWIs part of the kernel? Can you say for example that the FontManager is part of the kernel?

Dear Andre’, regardless of whatever might be the philosophical implications, SWI run in privileged modes (please note privileged) and within the same address space as the kernel. So, even if you would like to implement a micro-kernel and have multiple layers, the layer where you are going to place the SWI, if you want to maintain compatibility with the apps, has to be accessible by the apps. At least for now and the way RISC OS works now.

So, yes right now they are part of the Kernel, no doubts. In the future things may change (ROOL/ROD decide).

@ Andre’

Is that really true for modern ARMs? I know that there is a new “system” mode.

Is not what ARM allows you to use, is what RISC OS and the apps need in order to work properly without having to re-write both ;)

I presume that it protects you user mode and I presume that in the SWI handler you can check the current mode and reject SWIs calls in system mode to protect your micro-kernel.

TBH, ARM (and many others) are migrating towards Virtualised Architectures, where the OS will be split between Hypervisor and EL1. Where applications will run in “containers” where container can mean different things depending on the OS implementation, but that surely will be “aided” by the ARM architecture.

Right now this cannot be used for RISC OS, unless one rewrite the system so that the portion that is considered “grey area” becomes “managed” (we should say Virtualised) and controlled instead of being “wild west” as it is now.

Am I making sense?

However it is clear that we can run RISC OS as a Linux application on both 32 bit and 64 bit ARM Linux
platforms

Is it? Why aren’t we all doing that then?

https://www.riscosopen.org/forum/forums/3/topics/9068
I have been running RISC OS mainly under Linux. Since only one person is writing the Linux layer the progress is way slower than we might want it to be but it works well.
There is plenty stuff that’s not available yet or unfinished and it would be awesome if more people got involved. From the perspective of RISC OS , Linux is used as a HAL and a second Linux process is assignd the task to interface to the Linux Desktop for graphics, keyboard and mouse. (Sound is still missing)

Cooperative multi-tasking with N processors should give the WIMP the task to keep track of up-to N concurrent tasks running, right?

More, if you consider the kernel workspace as a thread context and switch between them while a core is idle. I’d rather a lighter-weight approach, though.

Since we have no hardware threads and use cooperative multitasking a switch will be done using a SWI from the running task. SWIs are handled by the same core that is running that specific task. No context switching or anything. It is up to the SWI handler to decide what to do next..
Any other switch can only be done from an IRQ. Idle cores happen when there is not enough tasks are running. (Some tasks may be sleeping). So from a perspective of supporting the ‘old’ model of cooperative multitasking it could be feasible to have N cooperative tasks running in parallel.

If you’re running in a privileged mode, you’re in the kernel.

Not necessarily, There are several ‘priviledged’ modes in modern ARM CPUs and Hypervisors can even schedule kernels. The models from ARM2, ARM3 and StrongARM have far been superseded by the newer ARMv7 and ARMv8 models. They allow for many more levels of privilege. They do offer new possibilities and challenges if we want to use them in RISC OS.

The way I see it there are a few paths of possible development, of which one, two or all may go forward.

1) Standard 32 bit RISC OS. Continue developing the current source, adding new features but retaining the maximum degree of compatibility with existing applications. It will continue to run on current ARM processors which support 32bit privileged modes, but will have to be run under emulation on newer processors which have 64 bit only privileged modes.

2) Hybrid 32/64 bit RISC OS. Retain the 32 bit userland, but move the kernel to 64 bit with some sort of trickery to allow 3rd party modules to run in 32 bit user mode. This will retain a large degree of compatibility with existing applications, but will only been useful on a small number of transitional ARM chips which support 32 bit in user mode only.

3) Full 64 bit RISC OS, with a new architecture fixing all RISC OS’s many limitations and bugbears, but zero compatibility, requiring applications to be re-written to some degree and existing 32 bit code to run under emulation and with an API transition layer.

(1) Is where we are now, and will undoubtedly continue for some time. (2) Is really stepping stone to (3) and might be more trouble than it’s worth. (3) Is where we want to go, but we probably wont take the user base with us.

I haven’t put RISC OS on Linux in there, as it could be any of (1), (2) or (3). You could emulate RISC OS in it’s entity or with some API translation for (1) allowing it to run on any CPU. You could have a user native + privileged emulation/API translation for (2) on suitable ARM Linux systems. Or you base 64 bit RISC OS on the Linux kernel or hypervisor and driver model for (3), depending on if the advantages of utilising the driver model as is, outweighs the fudges to the kernel to allow RISC OS style working, as has been mentioned by Paolo above.

;) I wonder if arm’s open source Keil project is of any use?

(3) Is where we want to go, but we probably wont take the user base with us.

In that case, who is ‘we’ ?

;) I wonder if arm’s open source Keil project is of any use?

There is an ongoing effort on ROOL gitlab to build RISC OS also using GCC, I have noticed few changes that started to give it away.

GCC comes with ASAM that indeed supports ObjAsm syntax and GCCSDK has successfully build an Iyonix RISC OS ROM image in the past.

https://www.riscos.info/index.php/AsAsm

ARM Keil (in the other hand) is more focused on Microcontrollers (MCU), SO, AFAIR for ARMv8 it supports only the M series (I could be wrong, so better to double check).

Hope this helps.

@ Druck

Yes, that is possible of course, AFTER “we” have solved the problem of user-space apps accessing directly kernel memory.

What is holding down RISC OS the most, in terms of architecture, is the fact that when a WIMP application is in execution state RISC OS creates a bond that is literally like the old Acorn MOS on a BBC Micro.

in other words the apps runs in the same way it would have done on a BBC Micro. Then, when the app has done its portion of work, give back control to the WIMP that does some magic paging shuffle and creates a “new” BBC Micro instance with a different app.

In the “future” (whatever that in RISC OS Time units means) what we want is that the actual RISC OS Kernel (whatever that may end up to be, custom, micro-kernel, linux, Klingon) is no longer being accessed by the applications and the BBC Micro era will finally phase to an end.

Now this can be solved in two ways:

1) “we” just kill it, and start anew, this DOES work :) 

   (all modern operating system have proven that), but it will create an OS without 

   software (read Rick’s previous comments)

2) “we” virtualise (aka “we” add indirection, we create a “buffer of pages”, replace

   with whatever is the perfect Oxford English terminology) a set of pages used by 

   the OS to exchange data with the Apps and (obviously) we rewrite the modules so 

   that they use that area. 

   At the same time when an application passes a pointer to the kernel that data has to:

   2.1) Be copied in the kernel address space, aka the kernel should NO LONGER access 

        data directly in user space

   2.2) remapped (when possible) in the kernel address space. This is for obvious 

        performance issues in case of large data of bullet 2.1

In order words, if “we” do not implement some form of separation between what “we” consider Kernel and what “we” consider user-land, there is no way to replace the RISC OS kernel with a modern one (Acorn has already failed to replace RISC OS kernel with Match, most likely because back in the day it was a) too hard to separate the 2 above or b) impractical in terms of performance on old ARM systems, we all have seen how slow Linux and BSD were on the RiscPC)

So, what you all think and suggest to solve the above Issues? Forget to talk about Linux for a moment, or BSD, or the usual “Multics does it better” ;) – Let’s talk about the actual details that are preventing everyone from successfully trying to replace RISC OS kernel with something else (whatever that could be).

Here is what I am experimenting with:

1) Working on an Hypervisor based Kernel
2) That kernel talks through messages and resources allocation with another that exists in L1 and works as traditional RISC OS has always done.
3) Functions of kernel in L1 should, slowly, be replaced by a functions offered by kernel in Hypervisor mode (using messaging and other forms of communications) until the Kernel in L1 will become a stub of the Hypervisor.

Benefits of this approach (if I’ll get it working and usable):

• Applications work with this approach and the Hypervisor can be secured offering the possibility to have a secure RISC OS.
• RISC OS and the apps do not run in emulation, they run on baremetal.
• It’s an approach that can be incremental
• It can offer opportunities to run 32Bit RISC OS on 64bit ARM that supports both
• It can offer opportunities to add MAMBO dynamic aarch32 to aarch64 translation to run 32bit RISC OS on AArch64 only ARM
• It doesn’t have the overhead of emulation
• It will make it easier to port RISC OS to new boards
• Will offer the possibility to run RISC OS Application in completely isolated VMs allowing full protection of the system

BUT, I am still experimenting so:

• It’s just tests and code modelling stage, I only work on it when I am not fried after a full week of day work
• Should be considered at barely “hello world” stage
• No idea when I’ll get to the point it’s worth sharing code and ask for others to get involved

Hence this is just some experiment, however it’s an idea that, at least, has in mind how RISC OS works.

Am I making sense?

I note the silence from the commercial users of RISC OS in this arena! As people who make (at least part of) their living from selling RISC OS based systems it would be prudent to involve them. It may be that their customers would not be willing to trade their current way of working for something novel.

How would such a kernel (and its modules) return an error block to userland?

I note the silence from the commercial users of RISC OS in this arena! As people who make (at least part of) their living from selling RISC OS based systems it would be prudent to involve them. It may be that their customers would not be willing to trade their current way of working for something novel.

Totally agreed, and in my specific case I am just experimenting, mostly for my personal knowledge and fun. Not pretending for a moment to replace/change/whatever on the official RISC OS.

How would such a kernel (and its modules) return an error block to userland?

Thanks for this question :)

General description (non technical)

At this time, my goal is to make sure RISC OS can actually boot on this kernel. But the way data will be exchanged between the Hypervisor and the stub kernel is either copying the data (pretty much similar approach to copy_to_user etc in Linux) or remapped to the stub memory, which in my case correspond to the VM memory.

it is possible to transfer data between the two (at this time) via a form of shared memory and a stub driver (very much similar to the way XEN Slice Drivers work on a XEN VM).

This very same approach is also used to create “virtual devices” for the RISC OS side. Virtual Devices help porting RISC OS to new boards as well.

To be more precise, imagine a XEN Hypervisor that actually implements the Drivers and the code required to handle resources and so, no longer relies on Dom0 OS.

More detailed description (a bit more technical details, but it’s still experiments only)

If we consider an SWI error block as a message itself, it can be sent to the VM as a data bloc from a shared driver request.

Lets, for example, consider a virtual network driver, to use it to exchange requests between the hypervisor and the VM. Such device can be used to forge a packet with the Service request and wait for a response or handle the response through a Virtual IRQ in an asynchronous way.

RISC OS seems to be able to use such a mechanism, however portions of RISC OS kernel will have to be re-written to use it automatically.

Anyway, when an SWI error block message arrives from the Hypervisor to the stub Kernel, a module can pass the pointer to the block to the user application as it has always done and the user application will access it as it has always done before.

Same will happen for SWI Requests, the stub will copy the user land data pointed by the pointer to a block of data passed by the user-space app and send it as a message to the Hypervisor Kernel. This will make it easier to implement threads as the kernel is not processing user-space data directly.

Not all SWI in L1 will need to talk to the Hypervisor level, and I only started to experiment, so real world performance are still unknown. However a Virtual NIC using a dedicated messaging protocol it’s simply shared memory, so except for the possible overhead of formatting requests between a module and the hypervisor, which most likely should be raw data anyway, latency should be pretty quick. But I am speculating at this stage.

I may be able to implement better mechanisms in the future, right now I am just working to get it starting.

Also, I’ll need a solid strategy to handle exceptions correctly, but this is something that will come later, when I’ll be more familiar with RISC OS Kernel in more deep details (given the usual long list of edge cases in RISC OS, I do not pretend to have the same knowledge that Jeffrey and people like you have).

[edit improved the answer a bit]

There’d no doubt have to be a small circular buffer of slots to fill with error blocks as you don’t know when the client has finished with any of them.

There’d no doubt have to be a small circular buffer of slots to fill with error blocks as you don’t know when the client has finished with any of them.

Yup absolutely, which is why I am starting from experimenting with virtual NIC drivers, I have plenty of experience with circular buffers on that area of the Linux Kernel.

I am also thinking of swapping buffers (A/B fashion) to allow even more error blocks transfers without losing performance, but again long term plan.

For now I need to:

1) Complete RISC OS full boot to Desktop on top of the Hypervisor Kernel
2) make my experimental module is able to fully talk to the Hypervisor without any issue
3) measure the latency of 2 and the stability of RISC OS executed in such a way

If everything checks, then I may have something to propose to the community and ROOL, but until then is just pure experimenting.

However, in answer to Simon requirement:

What I’m looking forward to is having a secure OS, and RISC OS running in non-secure (less privileged) environment.

This experiment does exactly that, and if it works, then the Hypervisor kernel could contain the components that need to host user password hashes and token verification (as well as using the ARM Trust Zone to store them eventually), so that the RISC OS stub can only pass a request and receive an ack to that request. No sensitive data stored in the RISC OS stub and also possible execution of applications in separate and isolated container.

We’ll see how this goes… But I am more than happy to hear other’s ideas and suggestions on either improving this approach or better approaches to solve the substantial problem of passing and returning data to/from SWIs with the assumption that user space can access kernel memory and kernel using user-space data directly, without breaking existing applications (and without being able to recompile apps either).

I should probably upload my experiments in that area to github, too. Currently, it’s good enough to start up and communicate with the frame buffer and read from the sd card. It’s a weird concept, the idea being that “threads” can move between “maps” using an object-oriented interface mechanism. Basically lots of lightweight threads, most of them blocked waiting for some event.

Here’s what my current C kernel can do, with four cores running independently (the cogs rotate, if you try it out on real hardware (Pi3) the shimmering is down to not synchronising the cores – it’s not important, on qemu the timing of the cores is off because they’re emulated). I have to work out what order to initialise modules, and how to provide access to hardware to the appropriate modules.

https://raw.githubusercontent.com/wiki/Simon-Willcocks/RISC-OS-Kernel-in-C/Screenshots/Screenshot%20from%202021-10-19%2018-07-49.png

1) Standard 32 bit RISC OS.

Running out of steam, but the ability to run existing modules and programs is good.

2) Hybrid 32/64 bit RISC OS.

That’s what I’m looking at. Are there only “a small number of transitional ARM chips which support 32 bit in user mode only”? From the point of view of the Pi4, you can run the whole RISC OS and its modules and applications as a Non-Secure virtual machine, with all the 32-bit modes available. Moving from that to emulation isn’t that big an ask, there are many already capable.

3) Full 64 bit RISC OS, with a new architecture fixing all RISC OS’s many limitations and bugbears, but zero compatibility,

Well, something that treats me, the user, as the most important thing in the world is sorely lacking from all the devices that are only meant to serve us advertisements. Why does every app that might send an image need access to all the images? Why not just let me choose, and give it the images I want to share?

In that case, who is ‘_we_’ ?

The children, duh.

Out of the ruins
Out from the wreckage
Can’t make the same mistake this time
We are the children
The last generation
We are the ones they left behind…

Won’t somebody think of the children?

Okay, okay, I’ll go get my coat.

Why not just let me choose, and give it the images I want to share?

Because it’s set up as an all or nothing arrangement. Perhaps it’s easier to code it that way? Because even iOS, that heavily sandboxes apps, treats the photo stream as a “here it all is”.

Now for my two wish list items for the advert-pushing hardware.
Firstly, why isn’t internet access a controllable permission? Who knows what sort of crap an app is randomly spewing to the mothership and a half dozen other sites (actually, there’s a no root firewall that works by faking a VPN and… it’s quite eye opening). Maybe I would prefer to deny that app unrestricted internet access.
And secondly, why isn’t “run at startup” a configurable option? Far too much crap gets to run at startup. Thankfully my Samsung phone has really aggressive power management that will terminate (with a lot of prejudice) apps that don’t play nice. Heh, I had to whitelist my streaming radio app. ;) But generic builds of Android don’t do this, and some versions (like Android Go) make it quite difficult to see what is actually running on the device. So not having a lot of things running themselves at startup would be useful.

Relevance to RISC OS? As I said recently, part of the apparent negativity towards modern ideas in RISC OS isn’t a blanket rejection of anything that is “new”, it’s because they’ve developed far enough on other systems that it is possible to see the bad points of a good idea.

I’ll give an example. Update notification is an expected part of apps these days. Sometimes it’s done by a store, sometimes the app itself.
Evidence – Windows’ update mechanism being abused to try to foist W10 on everybody.
Evidence – those apps that refuse to continue working if there is an update, and that update breaks some functionality, if not the entire app (I had this with my banking app, I detest forced updates).
Evidence – few of us have actually monitored what sorts of information an app sends back when doing the update check, especially given as how the app usually has access to all sorts of unique identifiers, location (even if only WiFi AP), and so on.

What goes down elsewhere ought to make anybody wary of similar arriving here.