Aemulor just got broken.

Aemulor 2.51 with bigger wimpslots has just been released.

Unfortunately with current ROMs Aemulor 2.51 fails to start with an “Overlapping areas” error. The breakage is recent, the ROM dated 11-Aug-19 is OK.

The issue was first raised on csa.misc on 25th Aug, “Issues with newer Pandaboard-ROMS and Aemulor”.

See also on Iconbar.

Chances are the culprit is this change. It’s the only recent kernel change, and it did touch a few bits of code which Aemulor might be trying to patch.

I am unable to test the latest ROM images (I did try the latest WandBoard image, but it’s not happy on my set up for some reason)…so this comes from analysing the BeagleBoard binary which I’ve just downloaded.

In Kernel.s.ChangeDyn there exists a table called ‘FixedAreasTable’ which the code in OS_DynamicArea Create(0) checks when supplied with a specific address for the dynamic area to be created. Aemulor has to resort to the ugliness of patching this table with the reduced ‘AplWorkMaxSize’, modifying the in-copy RAM image of the ROM, because otherwise you get that dreaded ‘Overlapping Areas’ error when trying to create the 26-bit RMA/SysHeap etc.

This ugliness – far and away the nastiest thing that Aemulor has to do to work its trickery – is also what causes problems with Aemulor on RPCEmu. The patch is applied by following the SWI dispatch table into the kernel image and then pattern-matching within a fixed address range of that entry point. The table contents have recently changed, so the pattern match fails and Aemulor tries to stumble onwards, which it shouldn’t really do.

As for a solution?…Well, I can re-spin the pattern-matching code (unchanged since 2002/3 and only just broken) to overcome this recent breakage, but it won’t address the need with RPCEmu to introduce a ROM patch manually and then rebuild RPCEmu. An alternative would be to introduce an API that avoids the need to change that table entry? Jeffrey/Ben/other: thoughts?

I think that this ugliness is one of just two things that Aemulor has to do which cannot be achieved via documented, portable APIs. The other is manipulating L1PT entries to map a second image of the OS ROM at &03800000 upwards because some old software (!DeskEdit, and quite probably others) assumes its address (Tbf these days I could achieve that by creating a dynamic area holding a copy of the ROM; nobody’s going to notice the few MiB of lost RAM).

I’ll hold off on a re-spin and re-release for now, pending a response and hopefully the ability to test any new build locally.

HTH

An alternative would be to introduce an API that avoids the need to change that table entry? Jeffrey/Ben/other: thoughts?

I have thought for a long while that integration was the best way.
If the functionality of Aemulor was there in the ROM/HardDisc combination then surely that helps move the whole user group (including the current 26bit only users) forward.
Adrian has been good enough to gift the community with this support tool, perhaps the functionality should be easier to access?

Well, I can re-spin the pattern-matching code (unchanged since 2002/3 and only just broken)

Surely the correct initial course of action is to correct the rom so that things work correctly for Aemulor for the time being. That is what beta versions are about?

At present only a few enthusiasts are affected who have tested the latest beta roms.

Aemulor is an important resource for those using !Impression and we don’t want to drive them away to use Virtual RiscPC exclusively.

I have thought for a long while that integration was the best way.

That would be ideal as it would force a solution if anything broke Aemulor – regress or solve.

An alternative would be to introduce an API that avoids the need to change that table entry? Jeffrey/Ben/other: thoughts?

Yes, a SWI to allow the value to be changed seems like the best way of doing things.

nobody’s going to notice the few MiB of lost RAM

What is a text editor doing faffing directly with ROM addresses, and not starting from a module base address?

and not starting from a module base address?

“Assume makes an ass out of you and me”

then surely that helps move the whole user group (including the current 26bit only users) forward.

Shouldn’t be too hard to add a URL file to the harddisc4 things that takes you to Sendiri (pending permission to include it proper)?

Actually, there is an argument for having a directory of links to useful/essential third party stuff – FAT32FS for example. There are a number of resources spread around the place, the two package managing solutions only touch a part of this, so…

perhaps the functionality should be easier to access?

It’s a shame the source isn’t available in some manner.

To give two examples, trying the old ARCbbs server (v1.64ß) sort of mostly works until the little door manager module is loaded (BBS parlance – a “door” is an extension that takes over the connection so the user can do something like play Tetris). When door module is loaded, the machine stiffs. Completely.
The second is that I resurrected my copy of 1stWord+. Oddly it crashes out all the time when running in ARM3+IOC mode but mostly works when running in RiscPC+IOMD mode (say what now?!? It’s a RISC OS 2 era program!). I say mostly works, because using the menu to do stuff like Find comes up with the message “Window not found”, yet it works using the (nonstandard) keypress.

I… dunno… I don’t really think it’s worth bothering Adrian with what’s likely to be other people’s lame ass coding…

What is a text editor doing faffing directly with ROM addresses, and not starting from a module base address?

IIRC it allows you to grab copies of ROM/RAM modules. I don’t recall there being a proper API for finding the base address of the ROM; whenever I’ve written anything similar, I’ve looked up the UtilityModule which I guess is what you’re suggesting.

I’ll probably change the ROM-mapping code since performance likely doesn’t matter anywhere; Aemulor already has to fake read/write accesses to non-extant memory areas. It can do that with the low ROM image too and hopefully sidestep any issues with L1PT changes.

It’s a shame the source isn’t available in some manner.

It may well get tidied and released one day, but it remains my greatest creation and I remain reluctant. Once released any potential revenue from upgrades and new features may be harmed. Coding is never free, remember; it’s just that at the moment fixes and new features are at my expense.

Re changing RISC OS, perhaps a reason code to OS_DynamicArea akin to that which limits the size of the ‘all of physical RAM’ DMA creation attempts would suffice and not be too ugly/bespoke.

The patch code is mismatching on the latest BeagleBoard ROM because some detail of the address map has changed; I didn’t delve into what, but I presume it was an intentional change and I think it should not be reversed even temporarily.

for finding the base address of the ROM;

Surely just read the address of UtilityModule and set the lower bytes to zero? It logically follows the hidden RISC OS module which is the traditional start of the ROM…

It may well get tidied and released one day, but it remains my greatest creation and I remain reluctant.

And what a creation. Getting 26 bit software working nicely on a 32 bit system. Damn.

However, I was very specific in not saying “open source” for exactly the reasons that you mention. I was thinking more in sharing with one or more people that you trust in order to help improve behaviour with obstinate software, basically to help take the load off of you.

Oh, and as much as I’d love to volunteer myself :-) my school motto was “Know Thyself” and I think I know myself enough to know that I probably wouldn’t understand half of what’s going on.
However, if you happen to have a more verbose debug build that reports what’s going on as it does stuff, I’d be happy to give it whirl with the aforementioned pieces of software, see if anything stands out?

which limits the size of the ‘all of physical RAM’ DMA

I presume you mean DA and not DMA… Jeez, are we still having that “allocate all RAM” problem? I thought that get clamped to a “useful value that isn’t going to bugger everything else up” ages ago?

I don’t recall there being a proper API for finding the base address of the ROM

There isn’t one. And more importantly, there wasn’t one.

whenever I’ve written anything similar, I’ve looked up the UtilityModule which I guess is what you’re suggesting.

Bingo. But OS_Module,18 returns the address of a module, and address!-4 is the length (because OS_Heap and therefore also ROM and therefore defacto), so unless it also tries to grant access to the Kernel it was always misdesigned. Somebody slap somebody please.

it remains my greatest creation and I remain reluctant

I am feeling this in my bones.

However, I was very specific in not saying “open source” for exactly the reasons that you mention.

Fair enough.

my school motto was “Know Thyself” and I think I know myself enough to know that I probably wouldn’t understand half of what’s going on.
However, if you happen to have a more verbose debug build that reports what’s going on as it does stuff, I’d be happy to give it whirl with the aforementioned pieces of software, see if anything stands out?

I’m afraid that’s unlikely to help; Aemulor does of course have debug builds/switches but it is the nature of a CPU emulator/compatibility layer that it can churn out a lot of trace information and often requires multiple tweaks and rebuilds to isolate a compatibility issue.

The machine hangs are a nightmare to debug (trust me!). I did most of Aemulor’s development on the IYONIX pc having discovered the godsend that logging information recorded into the NVidia card’s off-screen graphics RAM would be preserved over a machine reset, but !Reporter may provide some clue as to the Window handle error without changing Aemulor.

Failing that, if you’re okay with supplying a copy via URL/email for investigation and telling me which machine/build you’re using, I may be able to find something and then delete the copy.

I presume you mean DA and not DMA… Jeez, are we still having that “allocate all RAM” problem?

Erm, no the problem was resolved; perhaps I wasn’t clear – besides, as you rightly point out, managing to write DMA rather than DA :)

An additional reason code was added a long time ago to mitigate the problem of address space exhaustion. Perhaps something similar could be done to limit the maximum size of the application space which Aemulor (/others) can reduce upon startup and restore upon exit.

However, I was very specific in not saying “open source” for exactly the reasons that you mention.

Note that I specifically didn’t say donate the source or anything like. For rather similar reasons.

I’d certainly agree with the idea of sharing the info on an NDA basis or similar.

my school motto was “Know Thyself”

That was the title of a series of lectures at Aston University in the 1970’s. I suspect the reasons for the naming were entirely different since the head of the medical facility organised them.¹ I suspect yours were along the Clint Eastwood/‘Dirty Harry’ lines – “a man’s got to know his limitations”

¹ He also ensured the residence beds were wide enough for two without being double and always enquired as to your sex life if you visited with any ailment including a knee injury (me) or were a diabetic (a friend)
Guest speaker on the final lecture was Tuppy Owens who always did her bit just after a film.

I’ve had no response from those who have greater knowledge of RISC OS internals, so I’m going to propose that the following be allocated and I shall aim to write, test and release the implementation and documentation thereof, although I’m currently not set up for building ROMs and submitting changes.

OS_DynamicArea 26

Entry R0 = 26 (reason code)
      R1 = Maximum extent of application slot, or 0 to read only
Exit  R0 Preserved
      R1 = Previous extent of application slot

Use

The purpose of this call is to set the maximum size of the application
slot (_including_ the 32K of address space below the start of application
memory). Existing application memory will not be removed; instead, if
the allocated memory already exceeds the new extent an error will be
returned.

This reason code allows a backwards-compatibility layer to provide support
for older system extensions that must be executed in 26-bit PC modes.

Rather satisfyingly reason code 26 is the next available one :) Today/tomorrow I shall push out updates to all versions of Aemulor in which I have modified the patching code such that it works on the latest ROMs (hopefully on all targets, although I’m sure you’ll let me know if not) and they will try the OS_DynamicArea reason code first before resorting to patching. Hopefully that will address the issue going forwards. Objections? Shout now!

.

Sorry – I lost track of time a bit.

Over the weekend I did implement a call which is almost identical to yours, the only difference being that it always returns the new value, not the old value. But I haven’t got round to testing it yet (hopefully tonight).

A couple of questions:

• Should AllocateAreaAddress be affected by the change? Judging by your previous description of patching FixedAreasTable I’m guessing the answer is “yes”, but I wasn’t quite sure since it would result in 32bit apps creating dynamic areas in low memory (which could then prevent the memory limit from being moved back up if Aemulor is RMKilled)
• Likewise, I’m wondering whether out-of-range values passed to OS_DynamicArea 26 (i.e. those which intersect existing appslots or dynamic areas) should result in the address being clamped rather than an error being returned. That way (if there is a DA left lingering in low memory) the limit can at least be automatically bumped up as far as possible once Aemulor is killed

If we decide that clamping isn’t desirable, then I can easily change my version to always return the old limit instead of the new limit.

n.b. I’m just using “Aemulor” as a placeholder for any compatibility layer which uses the API – so if Aemulor generally can’t be killed once started, imagine for a moment that there is a different module which can be killed (ADFFS?)

My changes for this are in this branch. Let me know if you find any problems or would like any changes. Once you’re happy I’ll submit the merge request.

Hi, thanks for the reply, and the work you’ve done. I think it should return the previous value, because this allows Aemulor/ANother to restore rather than assume the previous limit. I was just mimicking OS_DynamicArea 8. My local Aemulor changes now call OS_DynamicArea,26,new_size TO ,prev_size and then – upon exiting – OS_DynamicArea,26,prev_size.

Remember that Aemulor is a 32-bit module and relies upon being able to create dynamic areas between new_size and prev_size, to create the 26-bit RMA, SysHeap etc…these are regular dynamic areas to the OS. What Aemulor does is to reduce the slot size and then immediately create the areas it needs but I carefully set the base addresses and max sizes to fill out the address space up to the 64MiB point and then allocate a dummy area which fills the rest of the application slot up to its previous limit (&20000000). That area “Aemulor DA space” has an allocated size of 0KB.

Aemulor can create areas on behalf of 26-bit code within that address space by playing with OS_Find/SetMemMapEntries if necessary, although with the OS changes since 2002(!) when I wrote it, there’s probably a more elegant solution using sparse DAs.

So…AllocateAreaAddresses needs to exclude DAs according to the current size, whatever that is (28MiB/52MiB when Aemulor has started and is creating its own DAs between 28/52 and 512MiB). When increasing the size (in practice, restoring the previous size) raising as far as possible may be desirable but point-blank refusal is just as acceptable; it should never happen.

OK – I’ve updated it to return the old limit.

Great. Looks good to me, thank you. I’m now going to start writing a specification for multi-core operation, preemptive multitasking and overlapped I/O, to see whether – upon posting that – you again say “I’ve just about finished that…” ;)

Update on site

Note that there’s also an important fix in there for users of 26-bit Impression builds with updated 32-bit support modules GDraw, FontDraw, GSpriteExtend and DitherExtend. The 32-bit versions of these modules have to be used for correct Red/Blue ordering on Titanium and ARMbook platforms, but Impression installs them in the 26-bit RMA and then Aemulor defers responsibility to the OS since they are 32-bit compatible. When Aemulor stops, if they are still running this would cause serious instability/crashes, so it now requests confirmation before then killing any such problematic modules when asked to stop.

Many thanks. Aemulor just got unbroken.

Update on site.

Aemulor 2.52 is working again on the Titanium and RPi3B+ both running OS5.27, 23-Sep-19.