Physical memory allocation

So I’ll be doing a bit more profiling around that area (e.g. performance of AMB vs. regular dynamic areas) and if the AMB routines prove to be a winner I’ll probably try adapting them so that the kernel can use them for most of its bulk page remapping operations.

A bit of fiddling today has resulted in some improvements to the dynamic area shrink code, making it about twice as fast as it was before, making it comparable to the performance of the old AMB shrink code.

That’s the good news, so now it’s time for the bad news again.

• Firstly there are quite a few different memory mapping activities to deal with, which is a pain for optimisation because they all have slightly different requirements.
• Secondly I’ve realised that a lot of the optimisation work will be redundant if/when we switch to a tree-based CAM. Although it’s true that a tree-based CAM will work much better with operations that work on address ranges rather than individual pages, all the range-based routines that I’m currently writing will have to be completely rewritten. And for most cases I’d expect the tree-based CAM (and better page tables result from it) to be more efficient than the routines I’m currently writing.
• Thirdly, the new code, by virtue of being more optimised, has fewer error checks. These error checks are mostly there to deal with memory corruption rather than user error, but they are the kinds of checks that could conceivably be left in place if we had a tree-based CAM (both due to the tree being more efficient, and due to the extra complexity warranting some extra safety checks)

So for now I suspect I’ll focus only on optimising the common AMB operations – mapping pages in and out – and leave the other memory mapping operations as-is.

“optimising the common AMB operations” actually turned into “just use the old AMB memory mapping code”. The old code was so well optimised for its purpose that the only way to improve on it would be to pick a better algorithm (e.g. tree-based CAM), or avoid remapping memory altogether (e.g. ASID-based task swapping), two things that are outside the scope of my current changes. And when you consider that (a) a tree-based CAM won’t help much with lazy task swapping and (b) ASID-based task swapping is only possible on ARMv6+, you realise that on ARMv5 and below the current code is pretty much the only option (although a tree-based CAM would be nice for the machines which can’t support lazy task swapping).

Since the AMB work has highlighted a lot of issues, I’m now looking at how best to tidy up s.ChangeDyn so that the memory mapping code it uses can more easily be tweaked/replaced in the future. Mainly this involves spotting duplicate bits of code that can be turned into subroutines, and making sure that all bits which use the PageFlags_Unsafe optimisation are standalone routines.

A couple of other random observations I’ve made while working on the code:

• For the long-descriptor page table format STRD must be used to write the page table entries because STM isn’t atomic from the point of view of other CPUs or the MMU. So the AMB optimisations that use LDM/STM to interact with the page tables might not work so well (plus of course the page table entries are all 64 bits in size instead of 32, so there’ll be a lot less registers to play with)
• Currently we have the page tables mapped as non-cacheable, non-bufferable, which is one of the reasons why the LDM/STM optimised AMB code works so much better than any alternatives. I’m thinking that we should at least make the page tables bufferable (for ARMv6+, at least), especially since on ARMv6+ we’re already required to use DSB to synchronise the page table writes with the lookups by the MMU. In fact for ARMv6+ this would also have the effect of changing the memory type from strongly-ordered to normal, so apart from the write buffering we’d also benefit from some limited read prefetching. Potentially write-through caching would also work quite well, because DSB should still be sufficient for synchronising the writes.

Potentially write-through caching would also work quite well, because DSB should still be sufficient for synchronising the writes.

Actually, some investigation of the ARM ARM reveals that the newer the architecture, the more support there is for having the MMU read from the caches. So there are a few machines where we could easily make the page tables write-back cacheable, with both the CPU and MMU seeing the benefits.

Early testing of cacheable pagetables on Cortex-A8 is promising:

• Lazy task swapping is 2x faster for mapping in pages (measuring the total time for some code to read a word from every page in the slot, triggering the lazy abort handler for each page) and 6x faster for mapping out
• When Wimp_TransferBlock maps memory blocks in and out, it’s 8x faster for mapping in and 4x faster for mapping out
• Dynamic area grow is 5x faster and shrink is 6x faster

The MMU in the A8 can’t read from the L1 cache, so it’s using an L1 write-through and L2 write-back cache policy for the page tables. I think the MMUs in all the other ARMv7/v8 devices we support are capable of reading from the L1 cache, so they should see even greater benefits.

So the next step is to tidy up a couple of loose ends and then move on to testing on other systems. In particular I’ve replaced the ARMv6+ DSB+ISB sequences with a macro, so that I can see if there’s any benefit to having cacheable pagetables with older CPUs. Possibly just XScale, since its data cache supports both write-back and write-through, whereas older CPUs are either much slower (ARMv3) or only support write-back (StrongARM).

Testing results for the full set of CPUs I have at my disposal:

So all the machines show an appreciable performance gain for most operations, but it’s really the high-performance chips (A8, A9, A15) which stand to gain the most from the changes.

However I don’t have any ARMv3 machines (RiscPC 600, 700, A7000, A7000+) to test against, and my StrongARM doesn’t support lazy task swapping, so there are a few datapoints missing. UPDATE ARM6+ARM7+lazy StrongARM results added

So if anyone has such a machine and is willing to help out testing, over here is a zip file containing a couple of ROM images and the ‘AMBTest’ program I’ve been using for testing (modified slightly to reduce memory usage – you should only need about 3MB spare to run it). The ‘nc’ ROM has cacheable pagetables disabled, and the ‘c’ ROM has them enabled. So if someone can load up both of those ROMs and capture the output of AMBTest (via *Spool) and post the resulting files here then that would be appreciated. Note that the ROMs also contain a bunch of the other memory management changes that I’ve been working on, so it’s important that the comparison is between those two rather than between the ‘c’ ROM and the current ROOL download.

Also it’s possible that one or both won’t boot at all on ARMv3 machines, so it’s important testing before I check in the changes!

getting some results from that would be useful too. (SYS “OS_AMBControl”,5,-1 TO ,A : PRINT A will show 1 if lazy task swapping is supported.

One of my two Risc PCs shows 1 (the Kinetic one running RISC OS 4.03). I’ll have a look at this on Monday.

However I don’t have any ARMv3 machines (RiscPC 600, 700, A7000, A7000+) to test against, and my StrongARM doesn’t support lazy task swapping, so there are a few datapoints missing.

We can let you have ARM610, ARM710 and Rev T StrongARM cards. If anyone has a spare A7000 or A7000+ spare would they be useful for you?

Testing results for the full set of CPUs I have at my disposal

Good improvements. Why is there no improvement with Block map in on the A53?

We can let you have ARM610, ARM710 and Rev T StrongARM cards. If anyone has a spare A7000 or A7000+ spare would they be useful for you?

An ARM610 card would be tempting – it’s got a few special requirements compared to the other ARMv3 machines we support. And I can’t imagine ARM610 cards are flying off of your shelves anymore! I’ll have a think about it (I think Sprow has volunteered to do some ARM610 testing for me, and these kind of changes are pretty rare, so it might be another year or so before I need to worry about ARMv3 testing again).

Good improvements. Why is there no improvement with Block map in on the A53?

I’m not sure. The key routine for that operation would be AMB_movepagesin_L2PT. My local version is a bit different (some changes to PageNumToL2PT mean that it’s now only able to do 4 words of L2PT per loop instead of 8), but the key feature of using STM to write the page table entries is still there. A quick check through the A53 TRM suggests that it’s got a pretty wide memory bus (256 bit wide write interface from L1 to L2, 128 bit wide interfaces through the rest of the system), so it should be able to perform 4-word STMs to quadword-aligned addresses without breaking a sweat. But the A15 also has a 128 bit bus width, and that saw a 4x improvement, so there must be something else at play. I guess the A53 just has better write streaming to strongly-ordered memory

ARM610+ARM710 results added to the table, courtesy of Sprow.

Good improvements. Why is there no improvement with Block map in on the A53?

I’ve re-checked the numbers, and with cacheable pagetables the results are actually marginally slower. Mapping in 16MB of RAM 256 times (so 4GB total) takes 11.5ms with non-cacheable pagetables and 12.1ms with cacheable pagetables. Maybe if I do any more testing on the Pi 3 I’ll redo that one but with a larger loop counter – 0.6ms out of 12.1ms sounds like it could easily be the result of a lengthy interrupt or two getting in the way at the wrong time. (The same operation on the Pi 2 took about twice as long, so the x1.2 result there is a bit more trustworthy)

One of my two Risc PCs shows 1 (the Kinetic one running RISC OS 4.03). I’ll have a look at this on Monday.

I get ‘ROM type not supported’ from !SoftLoad (which is already set up to softload Select2i5).

Sounds like ROL’s softload tool doesn’t like ROOL ROMs – try the ROOL softload tool (up to date binary here, or you can grab the app from a recent IOMD ROM download)

Jeffrey -what does the n[odecompress] do with Softload?

Chris – if you run ROL ‘Softload’ util (ie double click on it- it’s inside !Softload)- what options does it give?

I did a StubG version of ‘Softload’ v1.18 – if anyone wants to try – for the RPC/Emulator – no need of loading the CLib – if just to run ‘Softload’.

Using the absolute file ‘Softload’ you provided, with the roms ‘c’ and ‘nc’, and the command:

/ADFS::Kinetic.$.ANewSoft.SoftLoad nc
in file ADFS::Kinetic.$.!Boot.Choices.Users.Single.Boot.PreDesk.Obey

I now get ‘application is not 32 bit compatible’ but it doesn’t tell me which one. However the ROM loads OK and *Desktop still works (but of course no system resources have been seen).

Results with ROM ‘c’:

001E8480 timer granularity
00009002 page flags
4401A104 ID
test_growshrink
0000DE76 shrink time
0000CC24 grow time
test_mapslot_none
00000308 mess time
00000432 mapout time
00000430 mapin time
test_mapslot_readall
000791A7 mess time
00070636 mapout time
00001BEA mapin time
test_mapslot_highlow
00002611 mess time
00009E17 mapout time
000010C2 mapin time
test_mapsome
00011DDB mapsome in time
0006F6AC mapsome out time
test_growshrink_DA
000D668D shrink time
0002CCAB grow time
All done

Results with ROM ‘nc’:

00008032 page flags
4401A104 ID
test_growshrink
0000E677 shrink time
0000BB04 grow time
test_mapslot_none
00000308 mess time
0000046B mapout time
00000429 mapin time
test_mapslot_readall
0007BEE1 mess time
00083E87 mapout time
00001D69 mapin time
test_mapslot_highlow
0000282A mess time
00009BFB mapout time
00001185 mapin time
test_mapsome
0001ACB7 mapsome in time
000823CA mapsome out time
test_growshrink_DA
000F0EAC shrink time
000379E4 grow time
All done

Hope this is helpful.

Jeffrey -what does the n[odecompress] do with Softload?

Prevents pre-decompression of compressed ROMs. With a compressed ROM, normally the softload tool will decompress it before booting it, either to allow it to be patched or to double-check that that ROM isn’t already running. With the nodecompress option that code will be disabled, so the ROM will have to decompress itself. Mainly it’s for testing the self-decompression support within the ROM.

I now get ‘application is not 32 bit compatible’ but it doesn’t tell me which one.

Yeah, that error is a bit crap.

If you haven’t softloaded RISC OS 5 on your machine before then chances are you need to replace the boot sequence with the RISC OS 5 one (plus the “system resources” download) and then merge in anything else that you need from your old boot sequence.

Hope this is helpful.

Yep, that’s great – thanks. I’ve updated the table with the results.

Yeah, that error is a bit crap.

For reference (for others), it’s generated by CLib if you try to start an application using the old APCS call. You can either…

• start each boot app in turn to see which one fails
• search forum for “Harinezumi”
• use Reporter

I would recommend the oddly named one, but then I’m just a wee bit biased ;-)

With ref to Harinezumi (looks a useful prog) – What is BootFX?

*AddtoRMA doesn’t seem to avaliable on RO4.02 on boot – but answers to *help in Desktop.

Can’t find a ref to a util of that name.
Perhaps a loaded module has the call.

Prefixing the call with a X – seems to work.

BootLog – file is type Data – is that supposed to be Text?

Why is PatchApp module not been 32bitted? – version here is dated 2014.

What is BootFX?

It’s a module included with Pi ROMs that makes the “splash screen” appear when booting.

With ref to Harinezumi (looks a useful prog) – What is BootFX?

As Chris said, it’s a Pi module that makes a fancy startup (big colourful screen, the normal init in a tiny text window in the middle, and a sliding-bar style status indicator). Eye candy.

*AddtoRMA doesn’t seem to avaliable on RO4.02 on boot – but answers to *help in Desktop.
Can’t find a ref to a util of that name.
Perhaps a loaded module has the call.

It is provided by a module called “BootCommands”. It’s built into ROM on 5.xx; not having RO4.02 I don’t know what it’ll be called, but take a look in !System to see if anything looks like it might be it.

Prefixing the call with a X – seems to work.

The X just means “if it throws an error, don’t run around screaming panic!”. Something that really irritates me with RISC OS is that if some component of the boot process fails, the entire boot is just abandoned. That’s part of why I wrote Harinezumi – it reads what is supposed to happen, does it, then reports on whether or not it worked. And then it continues.
I was sorely tempted to have errors pop up a box – Abort, Retry, Ignore. ☺ But after some prototyping and playing with ideas, I decided that to just log stuff and put errors in red and then just keep calm and carry on was the best approach.

BootLog – file is type Data – is that supposed to be Text?

Hmm, I don’t have Harinezumi on the Pi. I’d better pull the sources off the old PC and rebuild them with the latest compiler. You know…just in case. :-)

The file, if it doesn’t exist, is opened as a Data file, and once it has been written, it is supposed to be set to be a text file. I can see “%%SetType %s &FFF” in the executable, so I wonder why it wasn’t being called for you?

Hang on, let me install it on my Pi.

…the words we love to hate → “works for me”. $.!Boot.BootLog is a text file.

Why is PatchApp module not been 32bitted? – version here is dated 2014.

It’s supposed to mess around with stuff to be “StrongARM compatible”, which I think involves detecting potentially iffy code and patching in calls to OS_SynchroniseCodeAreas.
Sprow says of it: Currently AppPatcher is only loaded by the boot sequence for RISC OS < 500 and is built 26 bit only and lives in !System/370. (discussion here).

The long and short of it is that it’s a legacy thing – 26 bit code plain won’t work and 32 bit code should be aware of the behaviour of split instruction/data caches.