If RISC OS 3.11 had been written in C, then

Oh the fun of 32bitting a 26bit ‘C’ module without source code :-)

I’ve done dozens using ARMalyser, it always surprised me that such hackery actually worked though.

I’ve done dozens using ARMalyser, it always surprised me that such hackery actually worked though.

The real surprise it that method worked with me driving it.

nemo said

“It is typical in RO4 to have two CLibs – the ROM one that the ROM C Modules are linked to, and a newer RMA-loaded one that everything else then links to. eg on my machine:”

Is that a path we could go down with RO5? Given that it is all softloaded now, anyway? So, we can fix bugs in the shared C library by updating the RO5 ROM itself?

There would be an issue for anyone actually using RO5 from ROM of course; a makeshift solution would be to use a spare word in the kernel workspace to signify ‘RAM-loaded SharedCLibrary is present’. ‘ROM builds’ of RO5 could check the value of this word; &00000000 means ‘no RAM-loaded SharedCLibrary’, otherwise it would be a pointer to (a suitable entry address for the RAM SharedCLibrary;

The RAM SharedCLibrary would need to reset that pointer to 0 when removed from memory, or relocated, otherwise all kinds of fun stuff would happen when RMKilling the soft-loaded SharedCLibrary :D

A way around that would be to have a ‘magic’ word prior to the entry point in the RAM SharedCLibrary, which is checked before doing the branch, and the pointer is zeroed if the ‘magic” word is incorrect.

Given 32-bit RO4 exists, it is a shame there couldn’t be some joining of the codebases, as they will have figured this all out already!

The absolute address in C based modules does look rather annoying, though! 😬

So, we can fix bugs in the shared C library by updating the RO5 ROM itself?

Umm…. What.

Typo; I should have said:

“ So, we can fix bugs in the shared C library by updating the RO5 ROM image itself”

Essentially, I thought that we couldn’t write ROM modules in C (or at least, ones that use the Shared C Library), but that was an incorrect understanding;

I then learnt that RO4 keeps ‘two’ copies of the Shared C Library. Because ROM modules are hard-linked into the ROM copy of the shared C library. But that an updated Shared C Library can be loaded, and (it looked like) that made the ROM Shared C Library go dormant, so the ROM modules could use the newer softloaded Shared C Library.

So, what I was thinking is, given that
– we actually can write ROM modules using the Shared C Library,
– why not just ‘update’ the Shared C Library by updating the RISC OS 5 ROM build, if bugs that need to be fixed (i.e. because they are affecting the function of some other part of the ROM) are found? That could make the CLib issue less onerous.

Although the ‘absolute addresses in C modules on a CPU designed for relocatable code from the ground up’ thing is of course much worse than the CLib problem itself :)

Umm…. What.

Or a slightly longer reply:

I then learnt that RO4 keeps ‘two’ copies of the Shared C Library.

This (here) is not RO4, and anyway the behaviour is the same as it’s how the module system works.

Technically two copies aren’t kept, the RAM loaded one replaces the ROM as far as the OS can see; however since the ROM one is still there, in ROM, all of the pointers to it are still valid. So code pointing to those things will still work.

The ROM apps, and things in the ROM written in C, are statically linked to the library in order that they can run from ROM. This would not be possible if a module had to set up the stubs jump table. The module would need to be copied into RAM, then the jump table populated.
Mightn’t be a big deal with 1GB onboard, but when 1MB was a common specification, it mattered, a lot.

Side note: That’s one of the reasons why the Toolbox wasn’t as popular as might have been expected. On pre-RiscPC machines, it claimed a lot of space in a 4MB machine and was a complete non-starter on anything less.

– we actually can write ROM modules using the Shared C Library,

We already do, it’s just the one in the ROM instead of a loaded version.

– why not just ‘update’ the Shared C Library by updating the RISC OS 5 ROM build

I’m going to assume that you mean very literally poking into the ROM image, rather than simply installing a new one, because that’s the only interpretation that makes sense in the context of what follows.

Why not? Because the way CLib links with programs is by way of direct jumps to the appropriate bit of code. These cannot move in an active system with apps running, so any bugfix that requires even a single extra instruction will cause problems because where do you put that instruction?
If you meant just patch the ROM and reboot, same thing, extra instructions make the code larger and the code within the ROM doesn’t have the ability to magically grow.

Plus, it’s a broken solution in search of a problem. It’s not hard to update the ROM itself (entirely changing it for a later version) which means you can obtain one with the latest CLib and any recent bugfixes and use that. Indeed, if your ROM is recent enough you will not be using a softloaded CLib at all because there won’t be a more recent version to load…

That could make the CLib issue less onerous.

For me, the only issue is that a softloaded CLib won’t refuse to quit. However, these days it is likely to be a somewhat artificial state of affairs that would lead to a softloaded CLib replacing a softloaded CLib.
Either way, given the damage that it could cause, it should still refuse to quit.

is of course much worse than the CLib problem itself

Remember that RISC OS was not intended by the Powers That Be as the operating system for the new ARM machines. So there may have been plenty of compromises along the way.
Something that mostly works is better than nothing at all.

Hmm – If the ‘C’ mod could start up first and copy it’s jump table to RMA – then it could be replaced – but probably easier for a softloaded ‘C’ mod to object in being replaced.

I seem to remember that at the beginning – the ‘C’ mod was loaded as required.

If the ‘C’ mod could start up first and copy it’s jump table to RMA

The jump table is in the client, not the module.

Unless you meant the client module would start up and put it’s jump table into RMA? It’s a bit of a catch 22 because the module needs the C runtime to start… I suppose as an icky hack it could be doable, after all, StubsG, but…

but probably easier for a softloaded ‘C’ mod to object in being replaced.

Yes, as far as I can determine, it’s three lines of code, literally a “nope” response wrapped in whatever conditional signifies a softload build.

I seem to remember that at the beginning – the ‘C’ mod was loaded as required.

When was “the beginning”? It wasn’t present in RO2 so needed to be loaded on demand. It was present in 3.1 but extremely common to have updates provided on magazine cover discs. And, yes, sometimes loading a program from one floppy and another from another would crash the system because of differing versions of the library, although this wasn’t immediately obvious as loading the programs in reverse order would work, red herrings, etc etc. Then one reads an article (by Mike Cook?) that explained what CLib was doing and why that happens and to always keep up to date and… my hatred of the library refusing to die has been a slow burn for three and a half decades.

“I’m going to assume that you mean very literally poking into the ROM image”

Only in the sense of poking the entire ROM image into the ‘partition’ on the SD card of a Raspberry Pi, or anything else that softloads RISC OS; ‘poking’ the rom image would be the polar opposite of what I suggested. My reference of ‘updating the Shared C Library by updating RISC OS’ is due to the fact that the (ROM( shared C Library is tied into the ROM with other stuff calling it. Hence it is something that can only be uograded with a recompiled ROM image.

When I was told we couldn’t write ROM modules in C, because they can’t access the ROM Shared C Library, but that we could write relocatable modules in C, as they set up their tables etc, I thought of ways around that. I noted that ShellCLI is in C, and used the Shared ac Library; that’s why I asked if ‘some parts of the C Library can be used in C ROM modules (some can; stdint.h for one, stuff that is just defines can. My idea was to make a very small wrapper module that links to the ROM Shared C Library, and allows C ROM modules to access it without needing to be ‘hard wired’. However that isn’t needed after all.

The problem with C modules beijg not ‘properly’ relocatable is an interesting problem. There must be a solution for that.

I remember CLib being annoying on RO2 with floppies. And also memory constraints (!NewLook on RO3.11 was nice, but that really cut into the memory on a 4MB machine).

Am happy that the C module thing isn’t as bad as it first seemed – my VDU drivers don’t use C Library functions (the exception is the ‘virtual register SWI interface’ I made – I was surprised to find that it was almost identical to one of the two ‘official’ ways of calling RISC OS SWIs in C. A small amount of assembler code could remove the need for CLib to call SWIs anyway (regarding the use of assembler, while it is ‘bad’, a SWI is an actual opcode and a software interrupt thst uses registers and flsgs, so it is one situation where assembler has to be used!).

Also got VMS v6 / v8 up and running today which was good – it isn’t so different from ARMv3 MMU after all!

I remember CLib being annoying on RO2 with floppies

Sadly ideas for a shared language support library for application use didn’t garner support until after RISC OS 2 went off for ROMing, and a useful APCS-R even later. And the ROMs used back then wouldn’t have been big enough to include the SharedCLibrary; RISC OS 3.1 ROMs are four times the size of RISC OS 2.

The problem with C modules beijg not ‘properly’ relocatable is an interesting problem. There must be a solution for that.

Nobody cared back in the early 90s, even fewer people around now to care about it!

“And the ROMs used back then wouldn’t have been big enough to include the SharedCLibrary; RISC OS 3.1 ROMs are four times the size of RISC OS 2”

If I recall correctly (when looking at a RO2 ROM dump a year or so ago), even BASIC was in compressed form in the RO2 ROM, to save space (this may have been a non-standard RO2 ROM though. If this wasn’t normal, it might have been for the A540 or something, where decompressing BASIC wouldn’t matter so much, memory-wise);

I still remember getting the RO3 upgrade for my A3000 (mid-late 90s, maybe earlyish 1997?); but during the ‘RISC OS 3 .11 ROM upgrade shortage’ (could have been early 1995?)

I needed the upgrade to do some work on (I think A-Levels), to use some software (can’t remember what). The ROM shortage (or at least delays for next batch) was such that there were no upgrades available. The end result was, after talking to Acorn, we drove down to the actual Acorn offices in Cambridge, where we met with (I wish I remember who it was now, but they were ‘very high up’ in Acorn). I remember the foyer, and the person we met had a large office upstairs. And just so happened to have a RO3 ROM upgrade in their desk!

Those were the days, before everything became so ‘corporate’ in the world in general 😬

Rick ruminated

Technically two copies aren’t kept, the RAM loaded one replaces the ROM as far as the OS can see

To be clear for those who are still learning about this big pile of stuff:

When a module is soft-loaded (typically but not necessarily in the RMA) the Kernel takes a number of steps in a particular order:

• The module header is checked for sanity, if that fails it is discarded without side-effects
• All instances of any existing module of the same name are finalised – any of which can object with an error, which results in the new module NOT being loaded but any instances that did finalise are not reinitialised (it may be that child instances die but the ‘base’ instance refuses to)
• If the module is compressed it is decompressed [3.0+]. This can cause ServiceCalls in some OSes
• If the module provides SWIs it is added to the list of SWI handlers BUT as there is no module instance yet those SWIs won’t work
• The ModulePreInit servicecall is issued to allow patching [3.5+]
• The decompressed and patched module is Synchronised [3.5+]
• The kernel attempts to Initialise the new module – if this fails, the module is discarded with the side-effect of having killed any previous incumbent
• Otherwise, post-initialisation the Base Instance is created (and hence SWIs will start working)
• The module is added to the ServiceCall chains in modern OSes
• ModuleStatus,0 is issued [4.3+]
• ModulePostInit is issued [4.39+]
• ModuleStatus,2 is issued [4.3+]

The pertinent point here is that the incumbent ROM module gets its Finalise called. If that succeeds, the new module gets used. However (and this is why I’m quibbling about ‘replaces’) that ROM module is still present in memory, may still be linked to god-knows how many things, and may in fact have failed to OS_Release, OS_ChangeEnvironment, Filter_DeRegisterXxxFilter, OS_RemoveTickerEvent or countless other OS hooks and, in the case of executable modules (or modules linked into executables which applies here) may still be running Wimp Tasks…

So the “replaced” ROM module continues to be very present… although in a diminished state: The only real impact is that it will no longer receive:

• SWI calls
• *Commands
• ServiceCalls

The ne plus ultra being the humble Basic Interpreter, which will happily go on running 50 Wimp tasks even though it’s been *RMKilled at the command line (though not in my version of the OS).

[Edit: BTW there is no such thing as a ‘Base’ instance, there’s just last-man-standing. However, despite all instances being equal, one of them is more equal than the others, but that’s an implementation detail I’m not going to explain]

[Edit2: My SaferModules module undoes a lot of the dangling links I listed above, which is extremely useful during development as it’s very easy for malfunctioning code to leave vectors claimed – something the Kernel does absolutely nothing about]

tymaja testified

even BASIC was in compressed form in the RO2 ROM

2.01 and 2.08, but not 2.00. Presumably because of the addition of the ARM3Support, IRQUtils and WindowUtils modules, 2 years after the initial release.

Stuart said

Nobody cared back in the early 90s, even fewer people around now to care about it!

It turns out that caring isn’t enough. Some Actual Typing is also required, sadly.

I have spent a long time blaming the nebulous “RO3” for the rot, but I can now see the problem was insufficient Stuart.

To be clear for those who are still learning about this big pile of stuff:

“Tha’ knows; if I were thee, I wouldn’t start from ’ere…”

So the “replaced” ROM module continues to be very present… although in a diminished state:

I was talking about “the module” as far as the OS understands. Like Highlander, There Can Be Only One.
As for all of the tentacles that the module may have spread around the system and maybe or maybe not released ¹, that’s the sort of nightmare fuel we’ll just gloss over, okay? ;)

¹ Is it better for a ‘dormant’ module to still have hooks into the OS, or for a ‘dormant’ module to have released its hooks and still be running tasks? It’s a rhetorical question, looking too closely at this stuff is scary.

There’s no tentacles, it’s just static links between one part of the ROM to another. Nothing worth losing any sleep over.

It is a problem when the ROM build works by accident but the RMA build leaves dangling pointers.

Rick rhetoricalled

Is it better for a ‘dormant’ module to still have hooks into the OS, or for a ‘dormant’ module to have released its hooks and still be running tasks?

My OS won’t kill a module if any CAO¹ is in the module or an instance workspace, but that won’t help with static links like CLib.

If the module’s final instance does get killed then SaferModules detaches any remaining danglies whilst complaining vociferously, like a reluctant proctologist. Again, CLib would defeat this.

It’s certainly a Very Bad Thing if a module that refuses to multiply-instantiate can be effectively multiply-instantiated by having a somewhat dormant ROM presence in addition to an active RMA version. BTW, refusing to multiply-instantiate is something that almost all modules ought to do, but almost none do. Another failing.

¹ Unlike the standard Kernel, mine liaises with WindowManager so that all CAOs are vetted. This does mean that some module tasks that can cope with being RMKilled while they’re running will not get the option. I’ve not solved the “I’m not trying to rob you I’m trying to help you” exercise of power.