Pyromaniac Update at ROUGOL, Mon 15th Nov 2021

The next meeting of the RISC OS User Group Of London is:

RISC OS Pyromaniac Update, presented by Gerph

Monday 15th November 2021, 7.45pm

The Duke of Sussex (upstairs in the Chichester Room from 6.30pm)
23 Baylis Road, London, SE1 7AY
http://www.rougol.jellybaby.net/venue.html

Also online via Zoom from 7.30pm

http://www.rougol.jellybaby.net/meetings/

At our November 2020 meeting, Gerph talked about the RISC OS build service that he runs, and introduced the technology that drives it – RISC OS Pyromaniac. It’s been a year, and the underlying OS has seen many improvements and bug fixes.

On the anniversary of that meeting Gerph is returning to talk about the things that were planned to do, and what actually got done. Not all of the work has been directly on Pyromaniac in the last year, so he will also talk about how it has been used to test some things, and a little about the documentation project as well.

As is usual there will be demonstrations over Zoom, and if you have a VNC client you’ll be able to try out some of the demonstrated features live yourself. For those attending in the pub there will be props to look at and comment on. As usual, Gerph will take questions at the end until we are falling asleep.

This meeting will also be held online via Zoom (but do join us in the pub if you can!). Please contact us to receive a link to the meeting. If you have attended any meeting this year, it’s the same link.

http://www.rougol.jellybaby.net/contacts/

If you missed last year’s presentation on RISC OS Pyromaniac, or if you just want to remind yourself where things were, the resource site has links to the presentation video, the slides and the speaker notes. There are also a collection of screenshots of it running and a few code examples contemporary to the presentation.

You can find the site at https://pyromaniac.riscos.online/.

Good luck, Charles. Hope the presentation goes well!

Following up on this month’s meeting announcement, I have a short plea for help to ensure the future of ROUGOL meetings.

When our previous venue suddenly closed at the end of 2019, it was a major worry that we would not be able to find another venue with such good facilities (private room, food, drink, wifi, big screen) in such a central location, and that didn’t charge a fortune so that we could keep ROUGOL meetings free.

So it was a huge relief when we found The Duke of Sussex, with its really smart room, in an excellent location with even better public transport connections, closer parking spaces, and free! It seemed to go down well with members too, with our first few meetings there having increased numbers attending.

We therefore very much want to continue at this venue, and would like to book all the 3rd Mondays for next year at this month’s meeting.

However this is not going to be possible unless there is an increased turnout at the meeting this Monday. The pub are understandably reluctant to continue giving us the room, for free remember, if we can’t guarantee to fill it. They have been very understanding since we restarted after lockdown, but we can’t rely on their generosity forever.

I’m therefore asking as many of you as possible to attend Monday’s meeting in person at the pub, rather than online. I realise that this will still not be possible for some, due to medical worries, but if you are able to, please do come along.

If you can’t come this month but feel that you will attend next year (e.g. after booster vaccines, checking covid trends, etc), email to let me know so that I’ll have some idea of numbers and can then use that to reassure the pub. Contact info – https://rougol.jellybaby.net/contacts/index.html

If you don’t want to come by public transport, car parking spaces are available directly outside the pub on Coral Street from 6.30pm. Note that the pub postcode goes to the front out on the main road (Baylis Road), for parking come in via Waterloo Road / Pearman Street, onto Coral Street. See the map on the website – https://www.rougol.jellybaby.net/venue.html

As an added incentive to attend, Gerph has provided us with some resources that you can get your hands on and look through as examples of what he has been working on.

Thanks for your attention, and I hope to see lots of you on Monday!

Bryan.
(Chairman pro-tem)

due to medical worries

Or in at least one case, sheer distance. It’s 437 miles from our house to the Duke of Sussex…

Good luck, and I’m thinking of you!

sheer distance

Well, yes, that is a reasonable excuse! (the original message was sent to the ROUGOL mailing list, which is mostly “local” people. I probably should have updated the text slightly)

Glad that we had the pleasure of your company at ROUGOL when you were still nearby :-)

sheer distance. It’s 437 miles

That’s sheer? According to Google Maps it’s 11490 miles from my place to the Duke of Sussex!

That’s sheer? According to Google Maps it’s 11490 miles from my place to the Duke of Sussex!

Is that walking or driving?

Thank you to everyone came to the meeting and for the questions.

If you want to know more about Pyromaniac the resource site at https://pyromaniac.riscos.online/ has more details and examples (pictures, videos, code, and links to open source).

The full slides for the presentation together with the speaker notes are available at https://presentation.riscos.online/pyromaniac2/index.html

There is a survey about the presentation which it help me if you could complete: https://survey.gerph.org/index.php/199167?lang=en

Another fantastic talk, Pyromaniac is coming on in leaps and bounds with desktop support. Just a shame it had to stop for closing time at the pub.

Is there a video?

Also, will it run Doom?

I did not finish typing “Thanks a lot Gerph, another great presentation” into the chat before it had to end.

So I do it here instead: Thanks a lot Gerph, another great presentation.

I have to admit that I haven’t really thought through all the potentially cool stuff that Pyromaniac could help with. Debugging, testing, profiling, OS prototyping and development, checking the OS docs for completeness when reimplementing stuff “from API”…for a mere user/application developer without deep OS fiddling ambitions, I guess the problem is currently to decide whether a problem you encounter is due to incompleteness of Pyromaniac, the application, or both.

I note you have added a CDFSSoft implementation, so I am now officially part of the target audience :-)

And don’t forget to notify the Unicorn maintainers that you are using their stuff. Going by their published list of things, Pyromaniac is certainly a very notable use case. Although very difficult to describe in one line of text!

Is there a video?

The videos are usually recorded, but they take time to edit – and my pauses and mistakes probably make that worse. So I expect that may arrive in due course.

However, the slides presented can be found on https://presentation.riscos.online/pyromaniac2/index.html – which are viewable within the webpage.
The slides alone can also be downloaded as PDFs or images, and – more usefully – the ‘Combined slides and notes as PDF’ link on the page contains PDFs with the slides at the top of the page and my speaker notes at the bottom.

Also, will it run Doom?

Doom requires a framebuffer to run in fullscreen mode, so not at present. The desktop version might work but I’ve not got around to updating the modules to be 32bit. I had looked at it but there was too much to do before the presentation to make that work. Plus… it’s not that interesting to run Doom really slowly – other than the fact that it proves that more of the interfaces are working as you’d expect.

The fact that The Great Escape works is, I think, sufficient to show that you can run such things… but that you probably shouldn’t.

I did not finish typing “Thanks a lot Gerph, another great presentation” into the chat before it had to end. So I do it here instead: Thanks a lot Gerph, another great presentation.

Thank you; it was fun – I should try to be more focused if I do another presentation so we don’t run out of time.

I did glance at the chat a little as I was going but it was only after we’d all been kicked out that I realised that I hadn’t followed it as much as I wanted. Usually I’d look through it during the Q&A at the end in case there was anything else that I’d missed. So if I missed any other messages from yourself or others, please let me know and I can try to answer any other questions that people might have.

I guess the problem is currently to decide whether a problem you encounter is due to incompleteness of Pyromaniac, the application, or both.

This is true – but it’s sometimes pretty easy to check. Compare to the same code under RPCEmu … if it’s different then you’ve probably hit a discrepency in the implementation. Not necessarily a bug, because there’s a lot of things that Classic does that it shouldn’t. It’s often made easier by the ability to just turn on more debugging or even add some more to the Python and re-run. Whilst you can do that on Classic, it invariably means rebuilding modules with extra debug in, which purturbs the behaviour more.

As I’ve intentionally only implemented the bits that I need (ok, so I’ve gone way beyond that, but I still try not to do things that really are rarely used) it’s quite possible to hit regions that don’t work, or don’t work as expected. However, usually those parts that aren’t implemented are in such a way that applications should cope. Either because they return errors to say that they aren’t implemented, or ‘SWI not known’ or they return values that indicate that the call was not successful. Those should be handled by the applications, so the fact that things aren’t implemented is useful for indicating problems with the application.

I used to use some of this technique with Select – basically kill off parts of the system until things started to break beyond what was reasonable. If you get a crash because some call isn’t there or because it’s returning properly but you don’t cope with the valid response it’s giving you, then it’s a bug. And fixing that makes your code more reliable – it can survive in the face of a changing environment. So Pyromaniac offers that as well. There are also configurations in Pyromaniac it intentionally change the behaviour to provoke problems.

Whilst testing Python with Chris we found that its directory enumeration was broken. In the OS_GBPB call you can return data and say that there
is no more data to come. The FileCore implementation in RISC OS Classic never did this. It means that there’s an extra system call necessary to discover that there is no more data. Pyromaniac’s filesystem returned the data efficiently, signalling the end of the data when the last data was returned. This type of checking is easy to get wrong, especially if the commonly used filesystem does it that way. Pyromaniac now defaults to the classic way of working, but has the option to use the efficient returns. Thus you can find out whether you cope properly with that case very easily.

I note you have added a CDFSSoft implementation, so I am now officially part of the target audience :-)

Kinda… it’s only for ISO mounting at the moment – ie it only does the data transfers. But I have looked at interfacing it with the native CD systems to read from a CD device in the other modes, or to fake different types of operations so that it’s able to look more like an odd but valid device. The implementation is around 1200 lines of code, 160 of which are constants, and the vast bulk of the module class is just SWI stubs that read their parameters and then return ‘nope, not implemented’.

It would actually be a great module to provide configuration options for because there are so many things that aren’t specified well in the documentation, or have been interpreted in a different way. You could then easily tweak the settings for ‘returns track numbers that don’t start at 1’, ‘has addressing that is offset by x sectors’, ’doesn’t support audioparams SWI’ or whatever. And almost certainly I’ve got things wrong – I used the documentation I had, and the implementation, and a ISO driver that Chris had written to work out how to do it, and I’m still certain that it’s not right.

And don’t forget to notify the Unicorn maintainers that you are using their stuff. Going by their published list of things, Pyromaniac is certainly a very notable use case. Although very difficult to describe in one line of text!

Yeah, that would be good, but I really want to do that once I feel confident with it. I’ve reported a few bugs (and had them fixed), and I’ve answered some questions on their issue tracker. However, writing an operating system using it is definitely something that needs more than a line to describe it. The Unicorn guys are currently working towards the updates for Unicorn 2 – a rebase on the more recent QEmu sources, but I expect that it will also mean Python 3 only, so … that’s a lot more work for me before I get there.

I’m intending to do a longer article about that and presentation maybe, in the future (dunno when) to talk to the wider Python community about how you can use it for OS development. But that’s even more scary because the audience is larger.

Another fantastic talk, Pyromaniac is coming on in leaps and bounds with desktop support.

Thank you :-)

During the presentation I made an offhand remark that I’d got support for handling the OS_ChangedBox updates in the graphics system. That’s a really rarely used part of the system that lets you know about redraws efficiently. I said that almost nobody would care about it and really it was only for completeness.

I had completely forgotten (or never known) that you’d used it in GraphTask for its updates – that’s exactly the sort of purpose it’s for. So it’s really nice to know that there is a use for it, and it’s not necessarily just my ‘completeness’ driver that means it’s good that I’d implemented it.

One of the things that I think I said in the presentation (or if I didn’t, that’s because it got cut for length), the Desktop wasn’t a goal, but being able to run it exposes a lot more cases where there are applications to exercise other bits of the system that hadn’t been hit. I’m pretty sure that I can’t run the taskwindow-like applications – just because of how I’ve structured the Python at the moment – but knowing that their core might work… well that is nice :-)

Before the presentation, about 5pm, I was getting nervous and not feeling very comfortable. I needed something to do before I made tea at 6, to take my mind off worrying. So I decided to write something to distract me. I took a few minutes to decide what to do and I made a cup of tea – because everything is easier with tea…

I gave myself 45 minutes to try to implement the CompressJPEG module.

I failed.

It took me 47 minutes.

Strictly it only handles the RISC OS 3.6 CompressJPEG SWIs, and only then the 3 component form (although the code was there to supply luminance values only, it wasn’t working), and there’s no validation on the input parameters. But it did include the time to write the BASIC code to exercise it and check that it was working in RPCEmu.

Whilst it’s an amusing story and all that, my point was that you can provide real functionality and interfaces in a short time frame with Pyromaniac. Admittedly that’s implementing it whilst also knowing how to work Pyromaniac and the interfaces you’re providing, but it makes for much shorter turnaround on trying things out.

It’s taken me longer this afternoon to write a proper test for the new module in assembler, than it did in writing the actual code. And it will take longer still to add in the validation and checking to the module, but that’s entirely understandable too.

I’d also like to echo that Pyromaniac is a seriously impressive piece of software.

Perhaps the most tantalising thing is how it presents itself as RISC OS 7. Is it a viable route to a next generation RISC OS?

I’d also like to echo that Pyromaniac is a seriously impressive piece of software.

Thank you :-)

Perhaps the most tantalising thing is how it presents itself as RISC OS 7. Is it a viable route to a next generation RISC OS?

Well, it labels itself as RISC OS 7 only because I had to give it some version number and 2, 3, 4, 5, and 6 were already taken :-D

As a route to a new generation, maybe, but only in the sense that you can try things out before you put them into practice in the real thing. It’s too slow and has some architectural limitations that make it quite difficult to work with as you start to look at more complex things.

It is ideal for prototyping and testing out ways of working that need to go into real developments. I mentioned the CompressJPEG module above – getting to a working implementation that followed the spec in 45 minutes is pretty unprecedented, even for me, and I used to rattled off modules like they were going out of style. Other larger changes are possible much more easily as well. For example, lobotomising the system by removing a set of SWIs or modules can be tested without too much hassle – or you can put traces on them to see whether they would be called.

Any change that would affect the Kernel would be significantly easy to try out – during Select development I used to rebuild the kernel, run the romedit tool to patch the kernel into an existing ROM, transfer to the test machine over ShareFS and then reboot the test machine. Whilst using romedit to patch it in is still faster than building a whole ROM, this was still a relatively tedious process. The development cycle with Pyromaniac – accepting that it’s actually not ARM code, etc – is much faster and you can try out your changes far more easily for speculative things.

A good developer uses the tools available to them to ensure the best result is produced that’s the most compatible with systems. Pyromaniac could be considered one of those tools.

In the browser world it was generally accepted that having two independant implementations of a given feature before that feature was considered to be generally usable (I’m not sure how that works now that everyone is converging on webkit for everything :-( ). Whilst pyromaniac isn’t fully featured, it’s a much more useful system than I ever expected it to be.

In those senses, I think it’s definitely something that’s useful as part of the development of RISC OS in general (and of course applications and tools if people wanted to do that).

Thanks again for another excellent presentation. One of the best things I can say is that it didn’t feel like a 3 hour talk! Shame we didn’t have more time for questions and to see more demos of it running. Several people said to me that it was only when you got to the system demo near the end and showed how the debug options could be used to trace individual groups of SWIs, check register values, etc, that it finally clicked and they “got it”, and could see how useful it could be for testing and debugging software and OS developments.

I think a separate (shorter!) video showing only how to install/run Pyromaniac, and then how to drop in your own apps/modules and use the debug options to trace their behaviour would be very useful.

As far as next gen RISC OS goes, I think a very useful thing it demonstrates is that it is possible to replace the kernel with something else, include an embedded ARM emulator, and still run the rest of the existing OS and apps. Modules can then be rewritten one at a time for the new system. Pyromaniac wouldn’t be a practical system to use for this, you’d want to use C or Rust or anything else more performant than Python, but the principle and architecture is sound. It offers an insight into how the post 32bit ARM cpu world might look for RISC OS.

Is there a video?

Yes, but it might take a while to appear, as we have a bit of a backlog (I say “we”, but really it all falls on poor Leo’s shoulders, and I imagine the enthusiasm for editing them is waning after 18 months of it!)

will it run Doom?

Is that the RISC OS equivalent of “Will it run Crysis?” that gets posted on every hardware story on The Register? :-)
If so, maybe it should be “Will it run StarFighter?”!

Oh yes, and slightly overshadowed by the Pyromaniac stuff, was the documentation produced using PRM-in-XML.

Having seen the output and the ease of generating it in different formats (PDF, HTML, etc) I can’t understand why ROOL aren’t immediately moving across to using it for all their documentation! Seems like an obvious move, with lots of benefits and few(no?) downsides.

I wanted to write a blog post about Pyromaniac, but it just made me aware of the many things I don’t know or understand about the various details. Maybe we should move this discussion to “General”?

The “system demo” was very interesting from a performance point of view, because performance varied a lot. Could you provide a bit more details which parts are slow and which are fast, and why? I understand that Unicorn emulation performance is not good (but by “how much” compared to e.g. the RPCEmu interpreter and dynarec?). I understand that jumping around between the Python and the Emulate-an-ARM parts is expensive. I understand that all things that run “native” (i.e. in Python, or subsequently hits the OS) are comparatively fast. I understand that the Cairo bridge along with the attached VNC server implementation is “it depends” – might be fast for the Font stuff because of the Font Manager being Python and the font rendering itself is native. Plotting pixels is slow because of the overhead of probably both the Cairo bridge and the subsequent VNC encoding (I guess this is the reason for the glacial performance of the colour picker?). What was the reason for the slowness of FilerAction – ISTR too many callbacks? Could you give an idea how much “overhead” is involved switching from emulated ARM code to a Python-implemented SWI and back? Which reminds me that I don’t really understand how Pyromaniac manages memory between those parts.

Oh, and the wording confuses me no end – especially when I try to ask things like above. Even “native” vs. “emulated” is complicated in a Pyromaniac context. Anybody a good idea about the wording? I like the “RISC OS Classic” vs. “RISC OS Pyromaniac” distinction for the whole thing, but any offers for the other situations? “ARM code” vs. “Python code” possibly, but what about Cairo and VNC?

Yes, the above sounds very confused. Maybe I should write a few paragraphs in German first to get a better grasp of the whole topic.

A small voice inside my head insists on asking “why not move the whole Python stuff to a Graal environments so I can easily contribute a few thousand lines of Java code, and let the C parts run directly from LLVM bitcode”, but I guess this would be much more experimental than it already is. The Graal environment has its Python support labelled as “experimental”, and my other experiences with different parts labelled like that is that this will guarantee a very bumpy ride. And it is Python 3.8 they support, which probably makes it “no-go” in the Pyromaniac context if I understood things correctly. Ignoring other obvious problems like integrating Unicorn.

Could you provide a bit more details which parts are slow and which are fast, and why?

Just saw that there was a lot more info posted in the Iconbar forum. Reading and trying to understand that now, please ignore me for the moment…

Thanks again for another excellent presentation. One of the best things I can say is that it didn’t feel like a 3 hour talk! Shame we didn’t have more time for questions and to see more demos of it running. Several people said to me that it was only when you got to the system demo near the end and showed how the debug options could be used to trace individual groups of SWIs, check register values, etc, that it finally clicked and they “got it”, and could see how useful it could be for testing and debugging software and OS developments.

Thank you :-)

I think that this is a mistake on my part in the assumption that people would be able to see the advantages of not debugging on the same system. However, hopefully with that and maybe more explanation I can improve that. I’ve just posted a little example of how you might debug a failing system, from within it using RISC OS Pyromaniac – not quite the same as what was shown on Monday, but similar. https://www.iconbar.com/forums/viewthread.php?threadid=12875&page=1#125247 – you can play along on the shell server as it’s completely reproducible.

I think a separate (shorter!) video showing only how to install/run Pyromaniac, and then how to drop in your own apps/modules and use the debug options to trace their behaviour would be very useful.

Yeah, I think I’ve been convinced that doing something like this would be useful. It might even be enjoyable to do :-)

As far as next gen RISC OS goes, I think a very useful thing it demonstrates is that it is possible to replace the kernel with something else, include an embedded ARM emulator, and still run the rest of the existing OS and apps. Modules can then be rewritten one at a time for the new system. Pyromaniac wouldn’t be a practical system to use for this, you’d want to use C or Rust or anything else more performant than Python, but the principle and architecture is sound. It offers an insight into how the post 32bit ARM cpu world might look for RISC OS.

Yeah, I really think that that kinda of work is probably a sensible way to go. One thing that I think is valuable, beyond showing the concept, is that many RISC OS concepts and functions are in a more approachable form in Pyromaniac.

I say “we”, but really it all falls on poor Leo’s shoulders, and I imagine the enthusiasm for editing them is waning after 18 months of it!

I don’t envy him the task but it’s really appreciated the work that goes into them :-)

will it run Doom?
bq. Is that the RISC OS equivalent of “Will it run Crysis?”

I had actually been expecting people to ask ‘Will it run Impression?’.
But then I tried Style that was in one of the bundles with RPCEmu and it was only 26bit and expected Aemulor to run it.

Oh yes, and slightly overshadowed by the Pyromaniac stuff, was the documentation produced using PRM-in-XML.

I have thoroughly enjoyed working on that with Alan, as I said in the presentation. I think we’d both like the fruits of that work to be available to others. If anyone else is interested in being involved in that process, please let me know and we can have a see what we can do – there’s a little learning curve, but the results are good.

(I’m not going over the things that related to timings which are mentioned in the Iconbar article, but there were some things you asked which aren’t answered there)

I wanted to write a blog post about Pyromaniac, but it just made me aware of the many things I don’t know or understand about the various details. Maybe we should move this discussion to “General”?

Happy to discuss this whereever people want to :-)

I understand that there may be some things that aren’t clear – I haven’t touched on much of the innards. Largely that’s because I already think I’m giving too much detail. Although in other places not enough. It’s hard to find a good balance.

The “system demo” was very interesting from a performance point of view, because performance varied a lot. Could you provide a bit more details which parts are slow and which are fast, and why?

Well, I’d like to but I really haven’t actually done that much in the way of working out what’s slow and what’s not. Sprites seem to render relatively well. The icons in the filer rendered slowly. It’d be really interesting to get some statistics on that and try to see what they’re doing. At the moment there’s not an easy way to do that at the granularity of the application’s behaviour. It’d be interesting to include timestamps on the log entries, I reckon. That would probably be the most useful thing to report, I think.

I understand that Unicorn emulation performance is not good (but by “how much” compared to e.g. the RPCEmu interpreter and dynarec?).

The difference between RPCEmu’s dynamic recompilation and Unicorn’s compilation is quite marked – RPCEmu is presumably doing a lot more specific optimisations, whilst Unicorn is a generic system that’s able to run ARM code. It also seems to clear its compilation cache quite a bit. I haven’t been able to find where that happens and what happens, but I suspect that it’s actually happening implicitly every time you go in and out of the emulation environment. Which means that it’s having to essentially re-compile each code fragment regularly. IF you have a hand crafted assembler look that’s executing SWIs that really hurts you.

On the other hand, the degree to which you are debugging matters too – there are different types of hooks that the Unicorn system offers you. Most of the time none are used so the system knows that it can just execute the code as it comes up. But you can have block hooks which are called for each block that is executed. I explained what blocks were and there were some examples in the Iconbar comment reply at https://www.iconbar.com/forums/viewthread.php?threadid=12875&page=1#125247. For each of those block hooks that means that it leaves the emulation system (admittedly whilst threaded) and calls back into Python. Python then does its thing, and then returns back into the emulated system. That makes block hooks slow.

Code hooks are not on the block but each instruction. That makes them even slower – if you’re going to call python for every instruction it’s not going to be fast. And then (usually) the python calls capstone to disassemble the instructions, and converts its disassembly into something that is closer to RISC OS mnemonics, plus includes information about registers and memory. That really slows it down.

In the system demo, you saw it with only the SWI debugging. Which isn’t as slow as the others, but includes
working out what the register arguments are using a lookup table and reporting them into the log.

I have a table in the trace.py code that looks like this:

1. trace
   • Use capstone to perform disassembly of each instruction about to be executed. In this mode, the system will run very slowly as it traps every instruction executed and disassembles them.
   • Slowdown: ~60x
2. traceblock
   • For each block of code executed, remember where it was to allow improved reporting of exceptions and errors.
   • Slowdown: ~1.2x
3. traceblockregs
   • For each block of code executed, report the registers which have changed from the prior report.
   • Slowdown: ~5x
4. traceregion
   • Report where the execution is happening as it passes control between areas of memory.
   • Slowdown: ~5x
5. traceregionfunc
   • Like `traceregionfunc`, but also reports each time control passes to a new function, as determined by the function signature.
   • Slowdown: ~10x
6. traceswi
   • Report each SWI that is executed, together with the registers on entry and exit. This does not need to perform instruction tracing, so does not slow the system as much.
   • Slowdown: Only on SWI calls.
7. traceswiargs
   • Use the built in database of register mappings for SWIs to report the arguments on entry and exit from the SWIs. The database is based on information from OSLib, so naming will match it. This is slower than `traceswi`, but provides a lot more meaning to the values.
   • Slowdown: Only on SWI calls, slower than traceswi
8. traceswiregion
   • Report where the execution is happening for each SWI. This is faster than `traceregion`, as it is only called when a SWI is executed.
   • Slowdown: Only on SWI calls, faster than traceswi
9. tracemsr
   • Report the change in the status register effected by MSR instructions. Requires every instruction to be processed, so slows down quite a lot.
   • Slowdown: ~40x

I understand that jumping around between the Python and the Emulate-an-ARM parts is expensive. I understand that all things that run “native” (i.e. in Python, or subsequently hits the OS) are comparatively fast.

They can be, and it does depend on what I’ve done within those calls. For example, many of the operations may look up configuration settings or check for debug. Whilst those probably aren’t amazingly slow they’re non-zero.

I’ll give a quick example from inside the SWI dispatch code in the Kernel:

def call_swi(self, swi, preserve=True, rin=None, rout=None):
    self.ro.check_running()

    try:
        self.swi_recursion_depth += 1
        limit = self.ro.config['kernel.swi_recursion_limit']
        if limit and self.swi_recursion_depth >= limit:
            self.ro.trace.warning(name='SWI recursion',
                                  label="SWI recursion limit ({}) reached".format(limit))
            raise RISCOSSyntheticError(self.ro, errors.ErrorNumber_PyromaniacSWIRecursionLimit,
                                       "SWI call recursion limit ({}) reached".format(limit))
        if preserve:
            if preserve is True:
                preserved_regs = self.ro.regs.preserve(reglist=self.preserve_list, include_cpsr=True)
            else:
                preserve = preserve[:]
                preserve.extend(self.preserve_list)
                preserved_regs = self.ro.regs.preserve(reglist=set(preserve), include_cpsr=True)

        self.ro.regs.cpsr_mode = self.ro.regs.MODE_SVC
        # Push registers as expected by RISC OS (see Kernel.s.SWIDispatch)
        self.ro.regs.push(swi, self.ro.regs[15], self.ro.regs[10], self.ro.regs[11], self.ro.regs[12])

        if rin:
            for n, value in rin.items():
                self.ro.regs[n] = value

        try:
            self.internal_swi_dispatch(swi)
            output = self.ro.regs.preserve(rout)
            self.ro.regs[13] += 20
            if preserve:
                self.ro.regs.restore(preserved_regs)
        except RISCOSError as exc:
            output = self.ro.regs.preserve(rout)
            self.ro.regs[13] += 20
            if preserve:
                self.ro.regs.restore(preserved_regs)
            raise

        if self.ro.regs.cpsr_mode == self.ro.regs.MODE_USR and \
           not self.ro.regs.cpsr_i:
            # We're returning to user mode, and we have interrupts enabled.
            self.returning_to_usermode()

        return output
    finally:
        self.swi_recursion_depth -= 1

This call is used by most SWI dispatch within the system, and it’s not the best by any means – it was some of the earliest code written and has been extended as I need things doing. Yes, there are no comments on the function.

Let’s just skip the recursion checks and register preservation and go straight to the mode change and stack setup.

self.ro.regs.cpsr_mode = self.ro.regs.MODE_SVC

This is a dynamic language so that’s…

1. Look up ‘ro’ in the self variable – that means string comparisons for the variables (probably hashed) to find the ro variable.
2. Look up ‘regs’ in that value.
3. Look up ‘cpsr_mode’ in that value.
4. Discover that it is a property variable, so there’s a setter function to perform the operation.
5. Look up ‘ro’ on the self variable (for the assigned value)
6. Look up ‘regs’ on that.
7. Look up ‘MODE_SVC’ on that.
8. Call the ‘cpsr_mode’ setter function to perform the assignment of the value of MODE_SVC.

And inside cpsr_mode’s setter function set do a number of manipulations which access unicorn to read the CPSR, and update the mode to the right value.

That’s on just one line in the middle of the SWI dispatch code. It’s really impressive to me that it’s running as fast as it does, because I really have focused on the ease of writing the code and not the efficiency.

I realise this won’t have answered your question, but it should give you an idea of the amount of stuff that’s going on behind the scenes that I have chosen to depriorities – it’s not the speed I’m focusing on.

I understand that the Cairo bridge along with the attached VNC server implementation is “it depends” – might be fast for the Font stuff because of the Font Manager being Python and the font rendering itself is native. Plotting pixels is slow because of the overhead of probably both the Cairo bridge and the subsequent VNC encoding (I guess this is the reason for the glacial performance of the colour picker?).

VNC wasn’t in use for that one, so it’s not that. I suspect it’s a combination of things, but it’ll be doing a ColourTrans lookup for each of the squares, then move, plot for the rectangle. That really shouldn’t be all that expensive. Of course it could be doing that with FP and so invoking the FPEmulator through the abort vector – boy does that go through some hoops as it resets all the state to be in UND mode, calls the processor vector handler, and finally gets out of there and back to the instructions in the main code back in SVC or USR mode.

Of course I could just check the code… There’s a couple of divisions in there which won’t be efficient, followed by two calls to xos_plot. Ideally that could be replaced by a prescaled value for the multiplication and a shift down, which would make it much faster. Whether that’s the reason… not sure.

Interestingly it should be trying to use the Use_Diffusion method first and only fall back to rectangles when it cannot handle that it (ie an error is generated). It’s not clear to me why it hasn’t done that.

It’d be really useful if I could toggle tracing on as it entered the Plot function… but that’s not currently implemented.

Interestingly the Plot function is always calling into Use_Rectangles without an attempt to call Use_Diffusion. This implies that Medusa is not set (it should be set on a RPC era system, and I would have thought that would be the case) or it thinks that the depth is lower than it is. Medusa is determined by whether a sprite mode word can be read as a mode using OS_ReadModeVariable.

Ah-ha… there we go… that’s why it’s reverting to the little squares:

>SYS "OS_ReadModeVariable",(6<<27) OR 1, 9 TO ;flags:P.~flags

Invalid DPI value (must not be 0)

I have a check on the values of the DPI supplied to check that they’re not garbage, and I default to returning an error. However, it is configurable with `vdumode.allow_0dpi_spritemodes`. However if I do that we hit a bug in my ColourTrans (doesn’t seem to support palette pointer of -1) and a further bug that trying to display an invalid PC value in the tracing code reports errors itself.

So it’s kinda lucky that it was doing those rectangles because of the bad mode word because it wouldn’t have worked. On the other hand, at least 2 bugs will be fixed because of that investigation.

What was the reason for the slowness of FilerAction – ISTR too many callbacks? Could you give an idea how much “overhead” is involved switching from emulated ARM code to a Python-implemented SWI and back? Which reminds me that I don’t really understand how Pyromaniac manages memory between those parts.

I think I’ve explained a little of what we do above which causes it to go slow. Not only that but any time we do go in and our of the emulated system we will be accessing the registers through the Unicorn interfaces, individually. But I’m really not sure about that one. I suspect it’s partially the font sizing and updates, and the tiling of the background behind each update, but it’s probably doing a bunch of other operations that I’m not supporting well. It’d be good to have some better tracing of that code to see where it’s going, but I’m not sure at the current time :-(

Oh, and the wording confuses me no end – especially when I try to ask things like above. Even “native” vs. “emulated” is complicated in a Pyromaniac context. Anybody a good idea about the wording? I like the “RISC OS Classic” vs. “RISC OS Pyromaniac” distinction for the whole thing, but any offers for the other situations? “ARM code” vs. “Python code” possibly, but what about Cairo and VNC?

I’m using ‘native’ to mean ‘on the system that Pyromaniac is running on’. I also use ‘host’ there sometimes, but I’m trying to stick to one term. The term for the things that are being emulated is generally ‘emulated’ or just ‘the code being run’ – I’ve not been so consistent and that won’t help. The ‘ARM code’ is often what I mean there. ‘Python code’ or ‘in Pyromaniac’ is usually sufficient. In the case of modules, ‘PyModule’ is the python implementation of a module, or just ‘Module’ for the general case. Again, these could be a little better distinguished – in my head they’re mostly clear but it’s hard to use good words :-)

As for Cairo and VNC… Well…

• Cairo is the graphics library I’m using for rendering many things.
• Cairo is also the name of the graphics implementation in Pyromaniac (compared to the default if you don’t specify an implementation, which is called ‘null’, and basically throws away all graphics).
• The Cairo graphics implementation by default just writes to a cairo surface. It may also write that to a file on exit, and potentially every few centiseconds (to make a video later). However, nothing is ever visible. This cairo graphics implementation is used in the RISC OS build service, so you can write to the screen and do ScreenSave and you’ll get out graphics, even though you never saw anything yourself.
• The ‘VNC’ graphics implementation is a subclass of the Cairo implementation, and so it provides all the functionality that cairo provides, plus when someone is connected, the surface from the implementation is fed to the client (on a separate thread, only locking for enough time to read the surface).
• The ‘WxWidgets’ and ‘GTK’ graphics implementations also subclass the Cairo implementation and they feed their data to a UI window that you see on the desktop.

If there were another library that did graphics, or you felt like it, you could provide other implementations. For example you could create an implementation that wrote out WMF files, or which streamed changes to a socket for display elsewhere, or… um… I don’t know what else you’d do with them.

Does that help clear up some of the terms?

Yes, the above sounds very confused. Maybe I should write a few paragraphs in German first to get a better grasp of the whole topic.

If it’s confused, I suspect that’s because I’ve not explained things well. That’s not surprising as the main audience for the product is me at the moment.

A small voice inside my head insists on asking “why not move the whole Python stuff to a Graal environments so I can easily contribute a few thousand lines of Java code, and let the C parts run directly from LLVM bitcode”, but I guess this would be much more experimental than it already is. The Graal environment has its Python support labelled as “experimental”, and my other experiences with different parts labelled like that is that this will guarantee a very bumpy ride. And it is Python 3.8 they support, which probably makes it “no-go” in the Pyromaniac context if I understood things correctly. Ignoring other obvious problems like integrating Unicorn.

Well, it’d be interesting :-) Python 3 is definitely something I’ll look at moving to. Eventually. Unicorn does have a Java binding, but I suspect that won’t help the Graal environment as it’s not the same ABI… although I presume Graal does have a means for connecting to native methods. It’d definitely be interesting. The more people think about interesting things that can be done, the more likely it is that things will be done.

I hope that has answered some of your questions, although it’s probably generated even more.

Would there be any benefit to running it on an AARCH64 system? Would it be possible to reduce emulation to capturing and implementing the defunct 32 bit instructions?