Pyromaniac Update at ROUGOL, Mon 15th Nov 2021

capturing and implementing the defunct 32 bit instructions

Er, that’s most of them!

I hope that has answered some of your questions, although it’s probably generated even more.

Many thanks. I have to process all that input for a while. I re-read the slides from your first (2020 ROUGOL) presentation, and found Slide 59 to be very useful to get a better grasp on the “operation cycle of things” inside Pyromaniac.

Would there be any benefit to running it on an AARCH64 system? Would it be possible to reduce emulation to capturing and implementing the defunct 32 bit instructions?

The similarities of the instruction set are actually kinda superficial. There was some talk a long time ago about making the instructions map more closely to native instructions – eg having the ARM instructions map to ARM instructions when executing on ARM, rather than the generalised – in QEmu, but the difficulties are that you still need to access through the memory system, and your registers might not be exact mapping to the executed code’s registers.

At one point I’d wondered about replacing the Unicorn implementation of the emulation with the engine out of RPCEmu, but it would mean making that emulation in to a library and making it into a python linkable interface as well. Both of which didn’t fill me with joy – it was actually my first plan before I found that Unicorn existed, but Unicorn essentially did all the work for me, so I didn’t have to go and rework RPCEmu.

It would be neat to do because RPCEmu certainly runs faster.

Which reminds me that I don’t really understand how Pyromaniac manages memory between those parts.

Sorry Steffen, I had meant to answer this a little more as I’ve not covered it anywhere – I’m beginning to think that it might be useful to include some of this sort of info in the documentation, or in the source at least… but time and energy are always a problem…

There are a couple of levels that the memory is accessed at and I’m not sure where is relevant, so I’ll try to answer in each of the levels that I can think of. At the lowest level there’s the memory mapped into the emulation. This it mostly provided by the Unicorn system – you say ‘map me x bytes of memory into this logical location’ and it does. You can also give protection levels but they don’t map well to the protection levels in the emulation so most regions are marked as read/write.

This means that any executed code just gets the memory in those pages and writes back to them. When python code needs to access it, the accesses go through the Unicorn system. The entire interface to read and write memory is through the pyromaniac.memory.Memory objects. These objects know how to get data in and out in different forms. They have some optimisations because they’re expected to be used a lot.

All memory accesses are through the ro.memory object, which is an instance of the above Memory class. It has bounds of 0 to 0xFFFFFFFF, which means it can access all of the memory in the system by index. When you request a word from the object, it will ask Unicorn to retrieve 4 bytes from the emulated memory, and then unpack them into a word.

If you have a Memory object called ‘block’, you can access it using methods, eg:

• block.byte
• block.word
• block.string
• block.string_ctrl

and a few others.

This reads the byte, word or string at the start of the memory block. If you need to access data at a different offset, there are a couple of ways to do it. The most common way that I do it is to use the array notation, for example:

• block[4].byte – read the 4th byte
• block[12].string – read the string starting at offset 12 in the block
• block[8].word – read a word at offset 8

The indexed notation creates a new Memory object at the address of the memory block, plus the offset. This obviously is a transient object and will only be used for the next operation that reads the byte. It makes it pretty inefficient, but it’s a really clear syntax.

Here’s some example code in OS_Word:

        block[0].byte = bcd(year)
        block[1].byte = bcd(now.month)
        block[2].byte = bcd(now.day)
        block[3].byte = bcd(((now.weekday() + 1) % 7) + 1)
        block[4].byte = bcd(now.hour)
        block[5].byte = bcd(now.minute)
        block[6].byte = bcd(now.second)

It’s possible to be more efficient, by using the functions in the object to access the offsets without creating a new object. For example:

• block.read_word(8) – read the word at offset 8
• block.read_string(12) – read the string at offset 12
• block.read_bytes(20, offset=15) – read 20 bytes from offset 15

The interface supports getters and setters for the above operations, and write_* functions for the functional interfaces.

When a SWI call is being dispatched one of the common things that you need to do is extract a bunch of parameters from the blocks that were supplied. This is commonly through the registers given, and using the main memory block for the emulation system. For example this is the entry sequence for the SWI JPEG_PlotScaled:

image_data = self.ro.memory.range(regs[0], regs[4])
x = regs[1]
y = regs[2]
scale = Scale(self.ro, structure=regs[3])
flags = regs[5]

This uses a range method which sets the size property (and implicitly the end getter can then read the end address). It’s not actually bounded in what it can access, but it’s a handy way to carry around regions of memory as a start and end. The Scale object can take either a set of parameters for the scale or a memory block, so that it can directly access the data.

The dynamic areas are created by mapping memory using the Unicorn system. As with Select, RISC OS Pyromaniac does not believe in unnamed areas, and every block of memory must exist within a dynamic area – only dynamic areas can map memory.

And dynamic areas themselves descend from the Memory object, and add in methods to resize the area.
The Heap dynamic areas add in allocation, free and resize calls which call into the OS_Heap methods. The RMA is just a Heap dynamic area.

The ROM dynamic areas have special allocation functions which just accumulate at the end of the memory that has already been allocated in 4K multiples.

The application space dynamic area has a whole load of extra handling for resizing that updates the environment handlers.

Dynamic area creation requires an address space search if you requested the address be auto-allocated. This calls into the memory.space.AddressSpace object to find a region of memory that can be used. The regions are updated with each DA created or destroyed.

Many RISC OS interfaces request that a buffer be filled with a number of items, and if they can’t fit the data in the buffer, they stop and report the space that would be needed. This is so common that there’s a class dedicated to it – riscos.bufferdata.BufferData.

For example you might do something like this:

    # Assuming you have an API which takes a ptr and size in regs[1] and regs[2]
    buffer_ptr = regs[1]
    buffer_size = regs[2]

    buf = BufferData(ro, ro.memory.range(buffer_ptr, buffer_size))

    buf.write_word(...)
    ...
    buf.complete()

    if buf.failed:
        # Ran out of space, so report the size required
        regs[2] = -buf.maxoffset
    else:
        # Report the size remaining
        regs[2] = buf.limit - buf.offset

This interface is used by a few of the filing system calls and other places that write such data. Essentially the object will write the data if there’s space, but if there isn’t it’ll just remember how much space it would have required. Then there is only a single code path which handles the processing of data and there should not be a possibility of returning a size answer which is different than would be populated.

When Unicorn is asked to read/write memory that it does not have a mapping for, it will report an exception. These exceptions come out of the memory system as Unicorn errors (they’re not trapped at that point). Actually there are functions like memory.safe_word which can be called which do trap the exception and instead return None – this makes a lot of the trace handling a lot easier to work with. But ignoring these safe functions, the exceptions propagate all the way up to the execution level, at which point they will be turned into a data abort of the right type, usually with a Python backtrace as well. Within the emulation (ie ARM code accessing memory that doesn’t exist) this is reported immediately as a data abort, and the system goes through the process of entering abort mode and calling the handlers.

I mentioned application space was a dynamic area. It is a dynamic area, but it’s a specialised one which is managed by the riscos.memory.appspace AMB system. This is in contrast to RISC OS Classic where the application space DA is owned by the Kernel and the AMB system directly pokes into it. There are a lot of similarities and a lot of differences in the handling of the application space, but the most noticeable one is that there isn’t a free pool in RISC OS Pyromaniac.

In RISC OS Classic the free pool is a notional set of pages which are not allocated to anything. Reducing the size of the free pool moves pages into the application space, and increasing it removes pages from application space. This is an awkward and somewhat historic artifact of the way that RISC OS manages pages.

In RISC OS Pyromaniac the free pool does not exist. If you want the application space to be bigger you issue the call to change the size of the application space to be bigger. That is, you use the application space dynamic area just like any other.

The concept of physical memory doesn’t exist in Pyromaniac. You cannot see physical pages with OS_FindMemMapEntries, or change then with OS_SetMemMapEntries. Dynamic areas cannot request physical pages. Doubly mapped areas cannot be created. OS_ReadMemMapInfo returns some information, but the number of pages present is just derived from whatever you configured – it doesn’t reflect reality with any hard limit. OS_Memory can be configured to return whatever values you like for the memory region sizes, but they don’t really matter because those regions don’t exist in reality.

I mentioned that there are no anonymous memory regions in RISC OS Pyromaniac. This means that all the regions have dynamic areas associated with them. Notable dynamic areas:

• 11: OS ROM
• 13, 14, 15: SVC, IRQ and UND stacks
• 16: Level 2 page tables (not supported in Pyromaniac)
• 48: Zero page (address 0 – 8000)
• 49: High CPU vectors

Hopefully I’ve covered some of how memory is made available, accessed and manipulated within RISC OS Pyromaniac. And hopefully that answers the questions you weren’t sure that you knew how to ask. If I’ve missed anything or there’s something I’ve not made clear, please let me know and I’ll try to answer if I can :-)