SMP-friendly SWIqs vs. TaskWindow

If a module registers a SWI as safe for SMP to pre-empt, does that straight away mean that the TaskWindow is free to pre-empt it then also? Regardless of whether the task has created any threads.

I guess the SMP module must have a way to keep track of whether all frames on the stack belong to registered SWIs without scanning the whole back-trace every clock tick. So then would it be possible for it to share this with the TaskWindow, to use it as well?

I feel like we could kill two birds with one stone here.

There’s another thing. At the moment if the TaskWindow pre-empts when the OS is threaded (i.e. something on the supervisor stack; in practice when pre-empting on WrchV etc.) it copies the whole of the supervisor stack to some safe place, and then copies it all back when its ready to restart the task.

Does this commit mean that the TaskWindow can just set up a separate SVC stack for the task to run on right from the get go, without copying back and forth? (Again I’m not at all thinking of a task that’s calling SMP_CreateThread, this is just ways to make the existing TaskWindow work better.)

If so … how does the TaskWindow tell the kernel or whoever what it wants the current SVC stack to be?

If a module registers a SWI as safe for SMP to pre-empt, does that straight away mean that the TaskWindow is free to pre-empt it then also? Regardless of whether the task has created any threads.

With the current TaskWindow: No. With a potential future/alternate TaskWindow: Yes.

It’s a question of whether a suspended TaskWindow task/thread is able to pre-empt a non-TaskWindow thread. E.g. imagine an SWI called Example_SWI which is SMP-safe, and does the following:

1. Locks an internal mutex
2. Updates some internal state
3. Unlocks the mutex

If thread A calls Example_SWI and gets suspended while the mutex is locked, then there needs to be a guarantee that (if another thread calls Example_SWI) thread A can be resumed in order to allow the SWI to finish its processing an unlock the mutex. Otherwise the other thread will get stuck and the system could deadlock.

With the current TaskWindow this behaviour isn’t guaranteed, because TaskWindow only runs its threads when the co-operatively multitasked Wimp allows it to. So a TaskWindow thread which calls Example_SWI could get suspended, and then a non-TaskWindow thread/task could try calling Example_SWI, and that’ll cause a deadlock (on a single-core machine) because there’s no way for the OS to pre-empt the non-TaskWindow thread to allow the TaskWindow task/thread to resume.

The current version of my code doesn’t do much to try and avoid these deadlocks.

I guess the SMP module must have a way to keep track of whether all frames on the stack belong to registered SWIs without scanning the whole back-trace every clock tick.

Correct. For the primary CPU core the SMP module hijacks the SWI processor vector, allowing it to detect whether each SWI executed is MP-safe or not. If a non-MP-safe SWI is detected then it forces the thread into “temporary singlethread mode” until the next usermode callback (OS_SetCallBack). Staying in temp singlethread mode until the next callback isn’t a perfect solution, but it was an easy way of dealing with SWIs which bypass the usual return flow, and shared resources like MessageTrans error blocks.

For the aux CPU cores it uses a different SWI dispatcher, but the end result is basically the same.

So then would it be possible for it to share this with the TaskWindow, to use it as well?

Eventually I’d expect some of the lower-level bits of the SMP module like the SWI dispatcher and IRQ handling to be merged into the kernel, at which point it might be possible for TaskWindow to make use of it to check the state.

Does this commit mean that the TaskWindow can just set up a separate SVC stack for the task to run on right from the get go, without copying back and forth? (Again I’m not at all thinking of a task that’s calling SMP_CreateThread, this is just ways to make the existing TaskWindow work better.)

Yes and no.

Yes: I’m hoping it’ll eventually make SVC stack management easier. (currently the SMP-friendly version of TaskWindow hasn’t had any changes made to its SVC stack handling)

No: It’s already possible to set up additional SVC stacks without that code. RTSupport sets up an additional SVC stack which gets shared by all RT threads, with the contents copied in/out of thread-specific buffers in a similar way to TaskWindow. But because it doesn’t tell the kernel/OS about it, there are a few places where it doesn’t work very well, e.g. Debugger exception dumps will refuse to dump the RTSupport SVC stack because it uses OS_Memory 16 to determine the stack limits.

Other problems related this:

• SVC stacks need to be aligned on a MB boundary, so if we create one per thread they could end up consuming a large amount of address space. I believe this is why RTSupport uses the same tactic as TaskWindow.
• If OS_Memory 16, OS_ReadSysInfo 6, etc. were changed to report the addresses of the current stacks instead of the default stacks then that could break software which relies on them returning the same values from one call to the next. So we may want to introduce a new API which is explicitly for reading the details of the current stack (that kernel commit you’ve linked to will cause OS_ReadSysInfo 6 to return different values, but leaves OS_Memory 16 unchanged).

If so … how does the TaskWindow tell the kernel or whoever what it wants the current SVC stack to be?

As mentioned above, currently it doesn’t.

The SMP module directly peeks and pokes instances of the CoreWS struct. TaskWindow could potentially do the same, although I’d imagine we might want a more robust mechanism in order to make it safer so things won’t break horribly if you’re softloading a module which expects a different layout. Plus there’s no API to determine the address of the CoreWS instance that’s used on pre-ARMv7 machines.

This sounds really promising thanks.

Can we say the reverse, that if a SWI is ‘TaskWindow-safe’ then it is SMP-safe?

I wonder if your Example_SWI is really SMP-safe? Say you have a Wimp app with a GUI thread and a work thread, and the work thread calls Example_SWI. The SMP module could then switch during the critical section, (2), back to the GUI thread which can then poll to another task calling Example_SWI, blocking on the mtx_lock, (1), and freezing the machine.

Can we say the reverse, that if a SWI is ‘TaskWindow-safe’ then it is SMP-safe?

No, I don’t think so. TaskWindow can preempt usermode code, but for SWIs it’s basically cooperative multitasking – the SWIs will only yield at defined points (generally when they call specific vectors or upcalls). If those SWIs were suspended at other points, or if two cores were executing the SWIs concurrently, then things would still break horribly.

I wonder if your Example_SWI is really SMP-safe? Say you have a Wimp app with a GUI thread and a work thread, and the work thread calls Example_SWI. The SMP module could then switch during the critical section, (2), back to the GUI thread which can then poll to another task calling Example_SWI, blocking on the mtx_lock, (1), and freezing the machine.

Yes, that’s definitely a problem, and currently there aren’t any protections against it.

Long-term I think the solution will be to allow for pre-emption in the Wimp. Short-term it may require Wimp_Poll to wait until all threads return to the foreground before performing the task switch, or to wait until all the threads have released all the mutexes/synchronisation primitives which have been marked as global. It’s one of the things that’ll need some experimentation as the system matures.