Multi-core Wimp

I have no intention of keeping the Wimp as the task manager, but I do want to keep the feel of the existing system, so it will still manage windows, icons, menus and pointer, so that existing programs won’t feel much different, other than, say, having videos running smoothly at the same time.

Here’s a quick precis of how I’ve structured my C kernel, before I read all the things I shouldn’t do!

• Boot
  • Assumes the existence of a moderate amount (16 MB, or thereabouts) of RAM starting at zero, a processor, and little if anything else.
  • Relocates the “ROM” if not reasonably aligned and, using the MMU component, maps it to virtual address 0xFC000000 (which can be changed, withing reason, by a modification of a single linker script).
  • Allocates some of the RAM to core-specific workspaces (including the vectors and translation tables) and shared workspace.
  • Enters the Kernel with the processor in a known state, and the size and location of the remaining known RAM.

• Processor
  • Utility functions to do things like synchronisation barriers, locks, etc.

• Kernel
  • Informs the memory manager of available RAM.
  • Initialises the processor vectors
  • Initialises processor registers that don’t have a reset value
  • Sets up stack pointers for each mode
  • Initialises the system Dynamic Areas
  • Starts the kernel boot sequence which is currently in modules, because I haven’t decided exactly where the routine should go.

• Modules
  • Handles making calls into modules, vectors, service call handlers, etc.

• SWIs
  • Handles SVC instructions, decodes them, calls either C implementations or module code.

• Memory
  • Memory management; parts of the kernel can request or return physical RAM areas to the free pool; it’s up to them to keep track of the memory, then.

• MMU
  • Manages the translation tables, mapping TaskSlots in and out of virtual memory as requested
  • Asks the current TaskSlot what physical memory it should map where

• TaskSlot
  • Manages memory belonging to a task. The WimpSlots.

The interfaces between the components are designed to be quite narrow, a few public routines, that should describe what they do, and not how they do it.

That’s actually an easy one with a well understood solution. Messages are put on a queue which is protected by a mutex, semaphore or some other barrier. Thus any task attempting to push to the queue while another task is already doing so stalls until the queue becomes free and the message can be added.

Ah, but that’s the thing. While a task is stalled trying to send a message, it can’t respond to the other task’s message, can it?

FWIW, I was messing about writing a 64-bit object oriented, multiprocessor, pre-emptive multi-threading kernel, and wanted to see if I could persuade RISC OS to run under it, for the windowing system. I started by trying to emulate hardware for a virtual machine, but I got sick of trying to emulate things that weren’t essential just to get to the point of it starting to boot. Then I looked at the HAL, and thought it was wildly heavy, and that the kernel could start up with hardly any knowledge of the hardware it’s running on, then simply initialise a hardware specific module which would provide a regular heartbeat and start up a boot sequence.

So I thought I’d have a go.

So I thought I’d have a go.

Thumbs up, here :)

So I thought I’d have a go.

Seven¹ words that change the world more than people realise…

¹ Eight if you’re a tedious pedant that considers “I’d” to be two words.

I have no intention of keeping the Wimp as the task manager, but I do want to keep the feel of the existing system, so it will still manage windows, icons, menus and pointer, so that existing programs won’t feel much different, other than, say, having videos running smoothly at the same time.

I just created release 0.28 of RISC OS Pyromaniac to myself at the start of this month – that’s one release made every month since I started on it. So.. 2 years, 4 months on… I’d say not to underestimate the amount of things that need to work in order to get things working. Running C library tools – with emulation – is just a few months work. VDU system in a limited form is easy, but there’s a lot that is relied on to get things working. Graphics system… well, I’m still going. AND I’ve been intentionally saying ’I’ll do enough to hit the minimum needed to work’.

So yeah, it’s fun – but then I’m not setting out to create a desktop look and feel. I just want to create a debugging tool. You’re being a lot more ambitious.

Don’t be put off, but as I say, there’s a lot of things that need to work :-)

At the moment, I’m cheating. I’m taking a fixed RO image, and trying to bend it to my will. If I find a useful component, I’ll use it, even if it means replicating the whole “Zero Page” area. I just tracked down the HardFont location, so I can display simple text, rather than just hex, and I’m wondering if I can use the existing WrchV code. The reason I’m considering problems with the Wimp messaging system, is that there’s no point in working towards something that won’t work in the end. Can the protocols be modified and still work as expected by the user and applications?

Seven words that change the world more than people realise…

I think they sit close by the words “I don’t think you could make that work” which is sort of a red-rag and bull statement in many cases.

“Can the protocols be modified and still work as expected by the user and applications?”

Wimp2 exists, therefore yes.

How does it cope with the problem of two tasks wanting to send a message at the same time?

I don’t remember – that’s stuff from last millenia :-( You’d have to dig out the documentation.

Do they live here or are there better docs, does anyone know? I’m sure I’m not alone in wanting this conversation to continue. :)

In a bit of a haze this evening, I realised that I’d not addressed the second part of your question and that … well it made little sense.

Is this an exhaustive list of calls that potentially swap out the current task? Wimp_Poll(Idle), Wimp_SendMessage, OS_Exit

No. That’s absolultely not an exhaustive list. Off the top of my head the following calls can swap tasks…

OS_Exit
Wimp_Poll
Wimp_PollIdle
Wimp_SetCaretPosition
Wimp_CreateIcon
Wimp_DeleteIcon
Wimp_SetIconState
Wimp_StartTask
Wimp_TransferBlock
Wimp_Initialise

A quick glance down the SWI calls and I think…

Wimp_ResizeIcon

may also cause a task swap to occur. I suspect it’s not exhaustive either.

Notice that Wimp_SendMessage is not in this list. Sending a message does not swap tasks at all.
That you included it in the list, and not any of the other cases that cause tasks to be swapped makes me think that you think that Wimp_SendMessage switches to another task there and then. It does not. The messages is queued.

You may be using the term ‘swap out the current task’ to mean ‘switch context’ – as in, passing control to another task.

The only context switching points are (I think):

Wimp_StartTask
Wimp_Poll
Wimp_PollIdle

OS_Exit (and OS_ExitAndDie) might be considered a switching point as well, but it’s not one that uses the environment callback interface and it is never returned from.

You ask the question:

But what if two independently running tasks call Wimp_SendMessage at the same time?

I’m not sure what you’re getting at, because this question only makes sense if you believe that there’s only one message in flight at a time – and that’s not true. Any task is able to send multiple messages after one another. They may send them to different tasks. They may send them to themselves. They may send them whilst they themselves are receiving just one message of a number of messages that have been sent to them.

So the answer to ‘what if two independantly running tasks call Wimp_SendMessage at the same time?’ is pretty much identical to ‘what happens if one task sends a message to multiple recipients, or as a broadcast’. The answer is that the messages will be queued and delivered to each of the tasks in turn as they call Wimp_Poll, and finally an acknowledgement responded to if the tasks have not responded to it. Any task which does not return from Wimp_Poll is broken in the classic RISC OS (because it never returns control to the Window Manager) and therefore would stall delivery of the messages. However it would only stall delivery of that message (and others that were to reach that task).

You still have to maintain the behaviour that messages are delivered to tasks in priority order, and you still have to deal with the broadcast arriving at each task before being returned to as an acknowledgement to the sender if it was not ack’d or responded to.

I think this question only makes sense if you believe that Wimp_SendMessage blocks whilst the message is delivered, as then it becomes a potential for a deadlock. There are protocol implications that mean that the direct sequencing of messages may be expected by applications, but that’s always been a pretty unreliable behaviour and such applications would have been prone to fail under the classic WindowManager in pathological circumstances.

Maybe I’m misunderstanding what you’re trying to achieve as the questions don’t seem to be relevant/have obvious answers.

I think, and suspect many others also think, that the collective Wimp_ needs splitting. So the task Simon seems to have set himself looks rather useful.

Just for the ones mentioned:

OS_Exit                  Task
Wimp_Poll                Task
Wimp_PollIdle            Task
Wimp_SetCaretPosition    Desktop
Wimp_CreateIcon          Desktop
Wimp_DeleteIcon          Desktop
Wimp_SetIconState        Desktop
Wimp_StartTask           Task
Wimp_TransferBlock       Task
Wimp_Initialise          Task

Whether the split is reflected in the single core implementation is an interesting question, or would it just be a multicore emulation and hardware, leaving behind the old emulator(s)?
Just musing.

In answer to the question about blocking on message send. Yes, the application blocks (stalls, if you like) while waiting to send a message. However “sending a message” is a process of copying the message into the queue. Something which will take a handful of clock cycles. So even if you have a hundred tasks all trying to send a message at the same time it isn’t going to be noticeable.

Going back to Simon’s original post I wonder if the SendMessage confusion we see in this thread is down to familiarity with Windows’ SendMessage() – which does immediately call the target window’s message handler, so the caller can get a result back from that call. Windows’ PostMessage() posts to the message queue similar to Wimp_SendMessage.

It seems my terminology is a little off, but my list wasn’t far off for what I meant. (I’ve not programmed Windows this millennium, but what you say does ring a bell, Stuart.)

So, Wimp_SendMessage puts a message in a queue, to be delivered by the Wimp the next time the sending task calls Wimp_Poll(Idle), right?

Can you send two messages without an intermediate Poll? Does that ever happen in practice?

(I suppose I could search for the queue size in the source, I just found the 128 task limit, which is nicely fewer than ASID values available.)

Do they live here or are there better docs, does anyone know?

Thanks, Mikko. It seems to identify the same problem.

Wimp2 is designed to be broadly compatible with the existing application structure used by RISC-OS. The only difference is that the 17, 18 & 19 codes handler must be placed in a separate non-preempted routine so that Wimp messages requiring an immediate answer are not returned as unanswered by the Wimp (through a code 19). There are facilities to have the message passed into the normal message queue or have preemption halted until the main queue deals with the message.

Can you send two messages without an intermediate Poll? Does that ever happen in practice?

As I stated in my answer:

“Any task is able to send multiple messages after one another. They may send them to different tasks. They may send them to themselves. They may send them whilst they themselves are receiving just one message of a number of messages that have been sent to them.”

And yes, of course it does happen.

My apologies, of course you did. It seems I have some fundamental misunderstandings of how it works, but just to be clear: in the existing OS, there is only ever one Wimp Task that is executing code after a Poll has returned, right? All the rest are waiting for their call to Poll to return.

I’m just a noob here, and I’m sure this will be regarded as heresy, but it seems to me that if you use a Linux (mini-)kernel as the basis for moving forward with RISC OS, you gain a lot of benefits…

Using a Linux-style kernel could bring you:
Multi-Core, Multi-thread processes
Path to 64-bit
Support for Wi-fi
Support for GPUs
Easier porting to more h/w platforms

It’s already been sort-of done here:
https://www.riscosopen.org/forum/forums/3/topics/9068?page=1

But probably going forward you could assign each RISC OS app to it’s own thread
- allowing, at least multi-core, multi-thread operation at the system & app level

Multithreadiing within the app could be supported by pthreads or similar
- but with one WIMP thread still remaining for compatibility and sanity’s sake…

Anyway, I’m just a h/w engineer by trade, so apologies if you think I’m just blowing smoke
- but I do work with embedded Linux daily and have been known to hack a driver or device tree occasionally

Other things to consider:
- how long you guys have been discussing how to move RISC OS forward in the areas listed about (years & decades)
- how little person-power is available in the RISC OS community to actually do the work of moving RISC OS forward

In the existing OS, there is only ever one Wimp Task that is executing code after a Poll has returned, right? All the rest are waiting for their call to Poll to return.

Maybe I’m being picky, but again it’s important to understand what’s going on in the WindowManager…

This depends on what you mean by ‘poll has returned’. Whilst you’re in the USR mode code, yes this is the case. However, non-task code can be executing in a filter.

Consider…

• Task has called Wimp_Poll.
• WindowManager calls on to the filter handlers to ask them to modify the parameters. Filter code is not wimp task code.
• WindowManager does its things.
• WindowManager is now about to deliver the first message in its list to the task.
• Task is paged in by the WindowManager (actually this may have happened previously, because the Wimp will have done lots of things on the application’s behalf and may have needed to access its memory without having to actually poll it).
• Poll code is offered to the filter handlers.
• Filter handlers may do lots of things with it, they may convert it to another call, or they may decide not to pass it to the application at all. This code is sometimes acting on the application’s behalf (eg Toolbox), and sometimes it’s providing augmenting facilities for the system as a whole (eg replacing existing behaviour, just like vectors might). This filter code is not part of the wimp task, and it is not the wimp task that is executing, but it is executing partially in the context of the wimp task – an exception within the filter manager will be delivered to the wimp task’s exception handlers [at least I’m /pretty/ sure that’s the case…].
• Finally the WindowManager returns control to the task with the new reason code (or it doesn’t because the filter handler decided it should not be delivered, and the WindowManager goes back to doing its thing).

Consider…

• Task has called Wimp_StartTask
• WindowManager creates a new environment for the task and starts it up.
• This task may do a lot of things – it can even call Wimp_Initialise, but until it calls Wimp_Poll you’re still in a position where one of the running tasks has not called Wimp_Poll, and yet there’s another task running (or a stub task – whatever you want to call the thing that the wimp starts before that task calls Wimp_Initialise, if it ever does).
• Once the second task calls Wimp_Poll we’re able to return back to the first take with the task handle of that task [I’m pretty sure it happens on the Wimp_Poll and not on the Wimp_Initialise, but I’d be open to being wrong here ‘cos… it’s the Wimp].

That second task, until it calls Wimp_Poll has complete control of the system (well, strictly ALL tasks have complete control of the system whilst they’re running, but it’s less expected that any given task will change mode and run full screen when a redraw happens, but a lot more expected that a task started with Wimp_StartTask would do so – but whilst less expected, it’s still reasonable / normal for an application to respond to a menu selection of, say, ‘Fullscreen’ by changing mode and running in an entirely single tasking manner), and can do anything it likes, such as changing mode, updating the palette, drawing on the screen, making noises, and taking input from the user. Basically you can assume that any invocation of Wimp_StartTask must stop the world whilst the second task starts.

The point for Wimp_StartTask is that the Wimp_Poll has returned, and yet the task that’s running is not that task. There’s only one actively running obviously, but there’s also one (or more, because we could chain all these Wimp_StartTask’s together in each task) that’s waiting for Wimp_StartTask to return, not Wimp_Poll.

I’ve drawn the distinction between the task’s code and filter code, because the latter can perform all the operations that a task can (except calling Wimp_StartTask or Wimp_Closedown). Similarly, any module can be executing which can perform Wimp actions on the behalf of the task (except calling Wimp_Poll[Idle] or Wimp_StartTask – I think they /can/ call Wimp_Closedown). The distinction I’m making here is between the code that’s executing and the ‘task environment and memory’. Why does that matter? Because those modules and filters may be used by multiple applications and won’t be at all thread safe. You will have races with those parts of the system if you are only considering that the application’s state is isolated – it is, but the things it calls are not.

Adopting an existing kernel, be it Linux, one of the BSDs, vxWorks, FreeRTOS, OpenSolaris or a hundred and one other options does bring some potential benefits. A lot of work has been done for you, particularly in the realm of drivers which are a major effort sink. However it would be a grave error to assume that “oh we’ll just use this and it will all work and we will get a free ride.” Because that never happens.

There are interactions between kernels and operating systems which means that some combinations work and others don’t. Most combinations can be made to work but that requires developer effort.

Putting RiscOS on top of one would doubtless be a lot of work. Like most things in life it comes down to a cost/benefit analysis.

Another thing to bare in mind is that it isn’t all down to effort spend vs. effort saved. Every operating system has its own distinct flavour and features. Before any root-and-branch changes are made it would be wise to come to some sort of agreement on what makes RiscOS unique and interesting. What makes it RiscOS. Because it’s important that a choice of kernel not preclude those features.

For instance, if it were decided that what people liked about RiscOS is that there’s basically nothing a program can’t do, it’s basically sitting on the hardware then any modern kernel with memory protection and process management would be out. You can’t have that closeness with those things.

On the other hand, if what makes the operating system special is the WIMP then fill your boots, what’s under it is largely irrelevant. See, for example, ROX.

One final thing to bare in mind is what developers are interested in working on. Without the deep pockets of a major commercial entity paying for development we rely on people doing it for the fun of it. In which case it’s important that whatever solution is chosen is fun for those developers and they are interested in developing it.

So you might find that skinning a Linux kernel doesn’t seem fun while noodling around with kernel programming is. Or you might find that people can’t be bothered with kernel coding and just want to noddle around with the higher levels. These are all things which play into the analysis.

I’m just a noob here, and I’m sure this will be regarded as heresy, but it seems to me that if you use a Linux (mini-)kernel as the basis for moving forward with RISC OS, you gain a lot of benefits…

I tried that before. ROLF, I called it. It didn’t catch on. https://sourceforge.net/projects/ro-lf/

@ Simon

I tried that before, ROLF, I called it. It didn’t catch on. https://sourceforge.net/projects/ro-lf/

To catch on developers to join you in your effort it should present some sources for them to start following you, while for a general user probably a build that they can run somehow somewhere to give it a try and see if they like it or not.

Or even a video (you seems to have started that project in 2014 when youtube was already a big platform to start sharing ideas).

Linux Torvald used the mailing list to share some code, for example. I am not criticising btw, I am just thinking of possible reasons why it may have not gained popularity.

in the existing OS, there is only ever one Wimp Task that is executing code after a Poll has returned, right?

If by Wimp Task you mean User Process/User Task, yes. And also for the sending messages previous question, only one User Task can send a message at a given time, as soon as it calls Wimp_SendMessage, obviously it gets frozen, but things that happen when in Kernel mode can be confusing.