Jumbo wimpslots

Something that I’ve been mulling over for some time is giving people the option to have wimpslots which are larger than 512MB. My initial thoughts were that this would be a CMOS setting which would be used to configure the memory map during early kernel initialisation. And by necessity it would be an opt-in thing because there’s undoubtedly some software out there which will fall over if the RMA or other specific dynamic areas cross into the upper 2GB of the memory map.

But now that we’re soon to gain the ability to fiddle with the max wimpslot size on-the-fly (within available limits), I’m starting to think about other ways this feature could be handled. So possible options I can think of are:

• “Unlimited” wimpslot size – instead of placing dynamic areas starting at the lowest address, the OS will be changed to place them starting at the highest address. The max application slot size will automatically shrink as new dynamic areas are created. However, this may cause some conflicts with the way IO space is allocated (which starts at the top and grows down towards the dynamic areas). Perhaps an acceptable compromise would be to remove the reserved IO region and intermingle the IO space with the dynamic areas – chances are most of the IO space will be configured during ROM startup, so it shouldn’t result in too much fragmentation (ROM-created DAs and ROM-mapped IO space all clumped together). And since some software will break with the RMA moving to high memory, this mode of operation will have to be controlled via a CMOS setting or similar.
• Move the RMA to just below the 2GB barrier, allowing for a 1.75GB wimpslot. This could allow large wimpslots for everyone, with OS_DynamicArea 26 allowing the limit to be shrunk post-kernel initialisation if more DA space is desirable (or automatic shrinking similar to the “unlimited” option). However the suitability of this option may well depend on how many other DAs suffer from high-address issues (chances are I think the main troublemakers will have been fixed at around the time machines with 1GB and 2GB of RAM became available – because at that time the free pool still had a logical mapping, so most DAs would have been in the upper half of memory anyway)
• My original plan, of having a CMOS setting to select between (e.g.) 512MB, 1GB, 2GB wimpslot sizes, with the RMA placed just after application space, as it is now. (I was originally favouring power-of-two sizes since it would make it easier to implement some future changes, but that’s moot because Aemulor / OS_DynamicArea 26 means I’d need to deal with other sizes anyway)

Any thoughts? How many people are there which would actually use a larger wimpslot? I think I can only recall one person who’s requested the feature, but if I eventually get round to adding support for sparse memory allocation in wimpslots (making application space more attractive than DAs for UnixLib) then I’d imagine that larger wimpslots could start becoming more desirable.

I think it may be relevant with the browser projects, but in all honesty, both have had to come up with other memory solutions. In most cases this seems to be a series of dynamic areas, but it may be worth talking to Lee Noar and Michael Grunditz as to how they’re handling it. I have certainly seen Iris using 600+ MB of RAM, but not necessarily in one lump.

I know that Lee has also had to adopt a very different model of memory handling to make RISC OS behave more like Linux in that regard, in order to keep WebKit happy. Hence the new ARMEABIsupport module and updates to SharedUnixLib and SOmanager.

but in all honesty, both have had to come up with other memory solutions

Only because they’ve been hitting the limits of what’s possible using application space, I think. Some of the issues were discussed here

Since then we’ve gained support for physical memory pool DAs, which Lee Noar has been working on making use of, and I’ve converted application space to use physical memory pools under the hood. But I haven’t yet got round to exposing the PMP functionality to applications (which would allow for the kind of sparse mapping/memory allocation that UnixLib desires).

I would love to have 512MB+ wimpslots.
I use one basic application that can grow up to this limit.

I suspect this would be useful – I noted that the Iris wimpslot was around 400 MB earlier with one window open viewing eurogamer.net
What I haven’t really done is see how this fluctuates with more windows, sites and tabs open. I have certainly had it running with 8+ tabs and extra window(s), so it is probably running OK within the current limits, but knowing how browsers eat (and leak!) memory, more would probably be merrier.

I have now 1,2 GB used by Firefox… and sometimes go much beyond 4 GB.

Nota: I work for a company that will switch to Office365, so I have a real chance to be able – soon – to make all my work on a RISC OS computer, with Iris or OWB (my main company already use RISC OS computers only).

That’s why I’m waiting so much for the laptop and the new web browsers :)

Good luck on finding out how modern browsers manage their memory. We did an extensive analysis at work to make comparisons between a FatClient with traditional UI vs. our WebClient using a browser-based UI (since our customers would like to know how to size their Citrix servers).

It is completely intransparent and unpredictable what happens inside the browser, and the amount of memory consumed is not in any way correlated to the complexity of the UI. At all.

So from a browser perspective, I’d say “we need to allow the WimpSlot being as large as physically possible”.

the amount of memory consumed is not in any way correlated

Do you mean actual memory consumed, or do you mean address space? They’re not the same thing. On Windows I can recommend Microsoft’s RAMMap to show what’s actually going on with memory usage. The Task Manager is a bit mystical about what it reports.

Do you mean actual memory consumed, or do you mean address space?

Both. We had some experience with the mysterious ways in which Windows handles the memory from analysing various JVMs.

Something that I’ve been mulling over for some time is giving people the option to have wimpslots which are larger than 512MB. My initial thoughts were that this would be a CMOS setting which would be used to configure the memory map during early kernel initialisation. And by necessity it would be an opt-in thing because there’s undoubtedly some software out there which will fall over if the RMA or other specific dynamic areas cross into the upper 2GB of the memory map.

I’d be rather cautious about moving the RMA up >= 2GB. I recall when the SVC stack moved up high at Tematic we had to go and swat a surprising number of bugs; dynamic areas in general weren’t so bad because they’d always had the potential to be above the 26b ceiling, though a few more SWIs did come out of the woodwork due to hiding flag bits in pointers which previously happened to be zero.

Even relatively recently I’ve stumbled across and fixed things that use a negative RMA pointer to mean ‘unallocated’ or similar. I see that’s 2016, I’m sure I’ve done something more recently than that but the sand has fallen out of my ears since then and I’ve forgotten what.

There’s also a body I kicked under the carpet which would start to smell in FileCore, so at the very least you’d want to wait until we get a shiny new FileCore which hopefully resolves that practice. I think the current FileCore is a maintenance dead end.

Sweeping those minor issues aside, I wouldn’t hesitate to enlarge the app slot to 1G or 1.5G (at least then the RMA is still <= 2G), and I wouldn’t bother making it CMOS configurable. Anything that can cope with 512M app slots shouldn’t be upset with 1G or 1.5G because it’s not banging up against any 2’s complement boundaries or wrap around issues.

Yeah, I remembered that there were some issues with FileCore that needed resolving. Waiting for the new version certainly seems like the best approach, especially since there are plenty of other memory related tasks I can sink my teeth into while I wait.

Sprow wrote

more SWIs did come out of the woodwork

Good heavens, R7. I’d never noticed that. My back yard too.

I agree that RMA must not have b31 set. Mind you, I have also suggested having more than one “RMA” – one for stuff shared with userspace, and another protected one. That protected one could be high up. Asking for allocations from it acknowledges the address requirement.

Sprow, has any fix for the Draw_ProcessPath API been proposed? It ought to have been picked up way back with HeapSort et al. I’m ashamed I missed it.

There is some merit in it being a new flag in R1 (which is where flags should be), but it could be in R7b0, either way you get an error without compatibility code.

Actually I would press for both – flag in R1 means “I am using the new API” and flag in R7b0 means “return BB”. Otherwise R7b31 is ambiguous on older systems.

Compatibility code on 26b systems is trivial of course. Thanks DrawV.

(Aside: I’m now of the opinion that OS_DynamicArea should have a ‘new API’ bit too, for the same reason)

I have a compatibility module (192 bytes) using R1 b26 to mean “new API”. This swaps R7 b0 with b31 ensuring one gets an “Invalid address” if one tries to use a high-bit address with the new API on 26b systems (well, on anything without the new API). Naturally for a suitably updated Draw module it need do nothing.

Sprow, has any fix for the Draw_ProcessPath API been proposed? It ought to have been picked up way back with HeapSort et al. I’m ashamed I missed it.

My use of past tense was hopefully clear that this was a past issue – Alan’s even added the flags to the wiki page I linked to. Draw 1.17 (05 May 2003) and later include the fix, and there’s a patch available for earlier versions for people who wanted to use the updated API but didn’t like Castle.

Pay no attention to me, I can’t read.

I’d missed that because Draw 1.14 (09 Jan 2004) [RO4.37] does not contain the fix. 1.27 (10 Oct 2008) [RO6.20] does though (though it’s not quite correct).

I have updated my DrawFix32 module to silently kill itself in the presence of Draw 1.17+, so it is always safe to load it if your application is using the newer API.

patch available

Which only covers two versions of the Draw module. There are at least seven other versions that require a fix. This module covers all cases.

Any thoughts? How many people are there which would actually use a larger wimpslot?

RiscOSM is probably the most memory-hungry RISC OS application which is not a web browser. Earlier in its development, it used the wimpslot for all its requirements and could only render very simple maps on a 26 bit OS. We did not think anyone would want to run it on a Risc PC, but we quickly found that folk who had Aemulor running at boot up hit problems because of the restricted wimpslot.

So we added the ability to use dynamic areas. To start with, this only kicked in if the wimpslot maximum was 28MB. In the summer Aemulor was changed to increase the wimpslot limit. We also had a user contact us who had run out of memory because RiscOSM had exhausted the 512MB wimpslot rendering a particularly large map.

So the next release of RiscOSM, being developed at the computer to my right as I sit here typing, switches to using dynamic areas if the wimpslot approaches the limit, rather than testing the limit for an arbitrary value.

I’m in two minds about whether RiscOSM would benefit from a larger wimpslot limit. We use the Norcroft compiler, and just use malloc rather than flex, so any wimpslot memory taken by RiscOSM is only given back when the application quits. Sparse wimpslots with an improved malloc/free would help here.

With dynamic areas, however, we are careful to request sensible-sized areas and split the map data across several if necessary. They can be freed and given back to the operating system very quickly when the user switches to a different map. Logical address space fragmentation or exhaustion could be an issue, I suppose. I know that when we talked about it a few years ago
the consensus seemed to be that we should not be using dynamic areas in preference to a large wimpslot.

For the record, the suggestion I made in my final post on that thread was one which we implemented, in the end:

One could apply the same principle using application space, first allocating a large block with malloc and then filling it as with the DA. When discarding the map data, the pointer to free space could be reset to the start of the block and the block reused to load new data.

RiscOSM now allocates fairly chunky buffers in the wimpslot to load the map data into, using a very simple allocation pointer rather than a chain of blocks. These are then emptied when the user moves to a new map. This saves quite a bit of time messing about with tiny allocations and probably reduces wimpslot fragmentation.

With dynamic areas, however, we are careful to request sensible-sized areas and split the map data across several if necessary. They can be freed and given back to the operating system very quickly when the user switches to a different map.

They don’t seem to be given back if RiscOSM exits with an error, gradually filling up memory. This is particularly prevalent on VRPC – open a map of London and it all falls over if you move it around a bit, even if you close the map or quit RiscOSM, and is only happy again after a power cycle. If you quit and restart RiscOSM it doesn’t spot the dynamic areas from a previous incarnation and zap them. It is not clear to me whether RiscOSM needs to exist as several simultaneously running applications if you keep restarting it. An error ‘RiscOSM is already running – do you want to kill the previous version’ might be sensible. Then any dynamic area left behind from the previous app can be identified by its title and killed.

I suspect this is happening more than it used to as the map data size is now much larger with latest map data.

seemed to be that we should not be using dynamic areas in preference to a large wimpslot.

That’s not due to DAs being inherently bad, it’s due to a shonky implementation with people requesting them to possibly be “maximum size”… because that will never go wrong, right?

In some cases, DAs are preferable. Certainly preferable to stuffing yet more rubbish in the RMA.

If you quit and restart RiscOSM it doesn’t spot the dynamic areas from a previous incarnation and zap them.

Yes – an app should check it’s the only incarnation, then it should check if there are DAs, and if so they should either be recovered (if there’s a way to check integrity) or deleted and recreated.

They don’t seem to be given back if RiscOSM exits with an error, gradually filling up memory. This is particularly prevalent on VRPC – open a map of London and it all falls over if you move it around a bit, even if you close the map or quit RiscOSM, and is only happy again after a power cycle.

The memory use issue (owing to the wimpslot reaching the 512 MB limit) is fixed in the next release of RiscOSM, out soon we hope.

Yes – an app should check it’s the only incarnation, then it should check if there are DAs, and if so they should either be recovered (if there’s a way to check integrity) or deleted and recreated.

That would be worth doing. It’s suggestions for improvements like this that delay the release of the new version!

The memory use issue (owing to the wimpslot reaching the 512 MB limit) is fixed in the next release of RiscOSM, out soon we hope.

The VRPC problem is not (necessarily) related to the 512MB limit (on VRPC the limit is 28MB). It is more to do with falling over and leaving DAs allocated leaving little free memory?