question about role of modules in riscos

aaaaa

In fact if i understood clearly, modules are basically supposed to be loaded at boot time, even mostly embbeded in the rom, and then stay resident in memory for the whole life time of the system ?

Modules can be loaded and unloaded as you like in the most part, ADFFS for example loads and unloads SoundTracker modules and shim modules as per each games requirements. Most apps will have a !Run sequence that will check for particular versions of Modules and load them if required. They’re essentially OO, but way before OO was ever thought up, so there’s a few caveats.

In theory all Modules should be able to load, unload, be replaceable or relocated in memory on-the-fly. Where this breaks is with some system wide impacting modules such as CLib where the design wasn’t initially thought through enough. Relocating modules in memory is also now not possible as it was originally intended, due to a few minor design decisions early on. RMTidy for example is now ignored, where in early OS’s it would shuffle all the Modules together to free up slack space in the RMA (Relocatable Module Area – big clue in the name).

If application need to dynamically load ‘modules’, or for handling for example specific media types, or plugins, how is it supposed to be done ? Using modules too ?

If the code needs to service interrupts or be available to many applications, then Modules would be the correct method.

That mean that all plugin that all application can potentially use need to be loaded in memory for the whole life time of the system or linked statically as part of the application ?

There isn’t really a system of dynamic module loading/unloading like plugin that is native on riscos ?

No, they can be loaded and killed as required. Apart from its use in CLib (which is really badly designed) and possibly a few other Acorn language libraries, static linking should be avoided at all costs. Modules are supposed to be dynamic.

I saw elf can be used for this otherwise, but i’d probably use something else if modules are not supposed to be used in that manner and that there is nothing that can have this feature, or maybe it’s fine to just load everything on boot idk, but it just seems a little weird to me compared to how things work on other oses so i wonder if it’s the way it’s supposed to be.

Typically you’d avoid loading everything at boot in RISCOS, it’s designed to be dynamic so apps generally load what they require as they load and unload them when they exit if they’re specific to that app. The OS pretty much pioneered what we now have in Windows / Linux, it’s just not kept up with them.

aaaaa

Is your questioning with regard to a bare metal OS you’re writing on ARM, or specifically related to how you would implement under RISCOS? I’m not totally clear on what you’re trying to code.

In your example case, in the RISCOS world you’d probably want to load all the Modules up front with the program, not load/unload in an OO fashion. Why? 1. Because loading/unloading will fragment the DMA and 2. Memory isn’t an issue under RISCOS like it is in Windows/Linux etc – we don’t require swap files as the apps are so lightweight.

However, I’d say your example probably isn’t a good case for Modules as they’re likely to be bespoke to the app. If the app was a suite of apps, then fair enough, Modules would be applicable.

If you’re writing bare metal, the worlds your oyster as they say! Full OO with dynamic insertion via linked entry points and multiple versioning on an app/app basis?

aaaaa

it’s a system that already work over different platform with different tools.

If I understand correctly, you’ve already written an application that’s essentially a framework that acts like an OS. It sits above the host OS and runs applications within itself and you’re trying to adapt it to RISCOS?

Is it available on the web? Do you have a web page you can point too? It might make things a lot clearer as to what you want to achieve and help us assist in adapting it to RISCOS.

aaaaa

Already that nobody understand the code of the os that is already rather rudimentary, compared to the kind of construction that need to be done for more complex application

A select few rather than nobody, kind of expected on an OS this old though as lots of the developers are no longer with us and documentation went somewhat out the window after Acorn closed up.

I must admit, I’ve struggled with ADFFS in certain areas and had to resort to documenting several aspects of the OS. ADFS scatter lists and the sound stack are two examples where the documentation wasn’t concise enough that I had to resort to picking the OS source apart. Scatter lists alone took me six months to get working, it should have been a few hours work!

Jeffrey is the biggest contributor with regard to knowledge and contribution to RISCOS, if he doesn’t know the only other folk are the ROOL team, but as far as I can tell they’re a secretive bunch and don’t really contribute much here. They come up with new builds in the background, Pi2 for example, but it’s all very secretive and not at all an open process, which is a shame. Maybe they all have day jobs and can’t commit the time on here – who knows.

already the fact that understanding this is such a problem should be a signal that something is not right with the organisation =)

You might think that, I couldn’t possibly comment.

I’d like to see ROOL more involved though, at least more publicly accessible and contributing here on a daily basis.

aaaaa

there can be plenty of them available, but ideally only the one used in the scene at the moment should be loaded in memory, and the one that are not needed anymore should be unloaded. Relocation/fragementation can maybe become a problem if lot of modules are to be loaded/unloaded.

Yes, and you can do it under RISC OS. But it’s done on the application side, not the system one. The application loads the modules (if there are already loaded, they won’t be again), and application unload the modules (if they are still used, RMKill will failed). It’s a bit like multitasking: provided by the OS, but controlled by applications. It’s very RISC like: you move complexity to applications (for a RISC processor, you move complexity to code). Good point here: there are no strict rules. Bad point: OS does not provide a lot of automation, just a framework to automate.

RISC OS is not “do it like that”, but more “do it like you want”.

Maybe they all have day jobs and can’t commit the time on here – who knows.

Yup.
Ben for example largely disappeared in the period before Pi2 appeared, apparently working on a contract with the Foundation.

Jeffrey is the biggest contributor with regard to knowledge and contribution to RISCOS, if he doesn’t know the only other folk are the ROOL team, […]

I’m not sure that’s entirely true: others definitely have knowledge of specific areas, and some of the other ex-Acorn people pop up on the newsgroups from time to time with advice.

The bottom line, though, is as you say: a lot of the knowledge as gone, and there’s no-one being paid to know how everything hangs together.

[…] but as far as I can tell they’re a secretive bunch and don’t really contribute much here. They come up with new builds in the background, Pi2 for example, but it’s all very secretive and not at all an open process, which is a shame. Maybe they all have day jobs and can’t commit the time on here – who knows.

already the fact that understanding this is such a problem should be a signal that something is not right with the organisation =)

You might think that, I couldn’t possibly comment.

I’d like to see ROOL more involved though, at least more publicly accessible and contributing here on a daily basis.

I’m not sure that’s a very helpful comment, TBH.

All of the ROOL people have day jobs, and do the ROOL work in their “spare time” for very little (if any) money. Steve R had some figures that he was presenting at shows a year or so back, which outlined how he and his ROOL colleagues spent the time that they could devote to ROOL: IIRC, most was administrative tasks, and very little was actually OS development. It was actually quite an eye-opener: I suspect that they’re already putting in many more hours than many other folk around here.

More of a visible presence would be nice (I’d have appreciated a speedier response to my forum posts and direct emails over the PrintPDF licensing problems a couple of years back, for example), but if a low profile here is the price we have to pay for allowing them to get the important but boring stuff done behind the scenes without breaking them as people, then I think that’s fine. The actual development, which is probably more interesting, is largely left to the volunteers outside of ROOL.

You could, of course, roll up your own sleeves and offer to help out; or you could just complain about things and help foster the impression that the ROOL team could find better ways to spend their spare time than working on RISC OS at all… How about contributing documentation (for the Wiki or into CVS) on the stuff that you’ve found out about the OS while working on ADFFS, for example?

The application loads the modules (if there are already loaded, they won’t be again), and application unload the modules (if they are still used, RMKill will failed).

Really? How does that work? I can’t think of very many modules which track active clients, so how would your average third-party support module know to refuse to die when asked?

In fact if i understood clearly, modules are basically supposed to be loaded at boot time, even mostly embbeded in the rom, and then stay resident in memory for the whole life time of the system ?

If application need to dynamically load ‘modules’, or for handling for example specific media types, or plugins, how is it supposed to be done ? Using modules too ?

You might be getting confused by the existence of things like the Shared C Library.

Fundamentally, modules on RISC OS are “OS extensions”. Once loaded, they’re code that’s always present (at least until they’re *RMKilled), and which can provide * commands and SWI calls, and operate on low-level events.

A lot of the core OS is implemented as modules (and the work Justin did on RO4/6 broke more of the kernel out into modules to help make the code more, er, modular), but there’s nothing to say that modules should only be in the system ROM. You can have modules on podule ROMs or extension ROMs (from the days of the A5000), and – of course – you can load modules from disc into the RMA at any time you like.

Applications on RISC OS all exist at the same location in memory (&8000 upwards): multitasking is achieved by paging different apps into the memory map in turn, so that only one is ever visible to the system at a given time. This obviously means that an application can’t claim things like events using its own code, as for a lot of the time there will be some completely different code – belonging to another application entirely – paged in at the address that it would give to the OS as its event handler.

If you need to claim events or similar (as per the PRM example that you quote), then you need to put that code into a module. You can then choose to either write a small module which communicates with your application via pollwords, or you can put the whole application into a module. I’d do the former, I think, but that option only came in with RISC OS 3.

Other uses for modules are to provide services that need to exist all of the time and not get paged out: talking to hardware, performing regular tasks outside of the desktop system, etc.

Due to the fact that they’re always present, modules can be shared by multiple clients. Whilst that might be the most visible feature nowadays, it wasn’t always the main point.

You could, of course, roll up your own sleeves and offer to help out;

I’m not sure, but I think he has done just that. In any case, Jon makes massive contributions to RISC OS – I’d be surprised if he has a lot of time for anything more! Your own contributions, Steve, are much appreciated also. We’re all on the same side.

aaaaa

In any case, Jon makes massive contributions to RISC OS – I’d be surprised if he has a lot of time for anything more!

I’m a little surprised that he doesn’t seem to understand how things work around here, then. “It’s all very secretive and not at all an open process” (ie. "it’s all a massive conspiracy) isn’t the kind of helpful comment that I’d expect from someone who’s in touch with what remains of the RISC OS community.

Given that the reason for the lack of input appears to be the age-old one of there only being 24 hours in a day, the only way to resolve that is to help out and free up some time for forum contributions. If Jon doesn’t have time for that, then one would have thought that he’d have a good understanding of the problem facing ROOL!

But yes it’s what i wonder too, it doesn’t seem the system has much tracking, or runtime dependency (which is not the problem of package installation or package dependency, all the module are in the same package, just need to know which one need to be loaded in memory), in the case for example of video plugin, an application could be using it, then another one also using it alongside with other modules, and they both would have their own list of module that are needed, that will not all necessarily will need to be freed when the application is killed.

Precisely (apologies if it wasn’t clear, but that was a rhetorical question that I was ‘asking’ of David).

Given that RISC OS has no runtime dependency tracking, the only way to make such a system work would be for each module to implement its own _Init and _Terminate SWIs that all clients call, and then track those clients (very possibly by giving them handles). When the last active client calls _Terminate, the module knows that it’s free to die.

A lot of modules don’t bother with this kind of interface, so there’s no way that they can be killed after use. The only option is if the application author assumes that it’s an undocumented private module and they then ensure that only one client can ever be running at a time – which doesn’t feel to be that great an idea.

With ref to the CLib – what about having a separate jump table in another module or perhaps a extra RMA claim area – that could be left intact – if a new CLib was loaded?

h0bby1:

In fact if i understood clearly, modules are basically supposed to be loaded at boot time, even mostly embbeded in the rom, and then stay resident in memory for the whole life time of the system ?

TThe difficulty is that “module” encompasses many situations, as the OS itself is practically built out of a large collection of modules.
Some modules are required to be started at boot time and then left alone. Examples of this would be CLib (because of the problems in replacing it “on the fly”) and the nested WindowManager in older versions of RISC OS (because you can’t change the Wimp while it has active clients and not expect things to go wrong). Also most of the OS is expected to run and stay running. If you *RMKill random modules (Filer, TaskManager, Internet, etc…) you will find that all related functionality will stop. Getting rid of Filer will remove all of the filing systems from the desktop. Getting rid of TaskManager will remove the “Switcher” (and also your ability to sanely shut the machine down).

On the other hand, modules are also useful for providing facilities that cannot be sensibly provided in an application. For example the USBeep module that I provided on my blog a few days ago – it does nothing more sophisticated than beep when a USB device is connected, however it demonstrates well a thing that could not be performed in a regular application. There are a lot of “background messages” (Events, Vectors, Service Calls, UpCalls…) that RISC OS provides that are best handled in modules. These modules can either live for the duration of the time the machine is running, or for the time the application that requires them is running.
An example of this would be my MIDI module. It loads when a program wants to talk to a MIDI device. It doesn’t need to remain resident once the program has finished with it; however it is no big deal to just leave it installed. Once the receive buffer has filled up (depends upon data flow, but ~20s or so with ActiveSense messages), it won’t do anything more until it has a program talk to it (via SWIs).

If application need to dynamically load ‘modules’, or for handling for example specific media types, or plugins, how is it supposed to be done ? Using modules too ?

It depends upon how the program is created. It would be possible to allocate a block of RAM and load code directly into it. This is how the serial blockdrivers work.

There isn’t really a system of dynamic module loading/unloading like plugin that is native on riscos ?

Not really, no. But some applications support their own versions.

I wrote an application a long time ago that supported “plugins” to provide different types of output. It worked by having a set of files (written in assembler) that were evaluated as the application started. When one was required, it would be loaded into an allocated block of memory. The first word pointed to the plugin name, the second handled file preamble (the stuff to output before the content), then there were routines for “text” and “text with effects” (bitmap for bold, italic, etc), and so on. It wasn’t elegant but it worked and having all of this stuff in separate files instead of all together helped to keep the memory requirements down (this was in the days when 1MiB was a common amount of memory to have installed, which would often be half that once system/screen/etc had all taken their share).

These days? I’d just write the output stuff in C and embed them all in the executable. You can afford to be wasteful if your program takes less than a megabyte and you have nearly 200 to play with…

I would not support ELF shared objects. Not because I have any problem with ELF, but simply because if the OS (or extension module) does not handle the hard parts by itself, that means that I would have to. Things like ELF will be a lot more flexible and capable than that which I would require, which would (as a side effect) make the code to deal with it more complex. For my plugin, all I needed to do was check the memory size, load it, and then call fixed offsets. That was all I needed. There was nothing fancy like resolving symbols.

Jon:

Apart from its use in CLib (which is really badly designed) and possibly a few other Acorn language libraries, static linking should be avoided at all costs. Modules are supposed to be dynamic.

CLib is not badly designed. Sure, it is biting us on the ass now but it has its origins in the days where 8MHz was a typical processor speed. Even when the ARM got faster, like the ARM3, the memory bus speed still sucked.
Accordingly, you would take quite a hit if calling C functions via SWIs. As you know more about the SWI dispatch than many, I’m sure you will appreciate that two branches is a lot faster than the entire SWI rigamarole. Plus there is no need to deal with changing mode – an application enters and leaves CLib in USR mode the whole time.
It was designed for speed.
However, like passing SWI numbers in the SWI instruction itself, it was designed to be a good solution at the time without necessarily thinking too much about the effects in the future. It is possible that it was always envisaged that CLib would be a fixed part of the OS – which ironically is something we are back to with RISC OS 5. The whole “replacing CLib thing is moot for us as the latest version is built into the ROMs”. Consider the ability to replace (with dozens of versions!) in the RiscPC era a nod towards the flexibility of RISC OS that replacing an important system component was possible at all. ;-) And then they did it again with Toolbox, and again with the nested Wimp. Okay, granted, you can’t replace these components multiple times while the system is actually running, but then…you were never supposed to. Try replacing (or just reiniting) FileCore and see how far that gets you. ;-)

h0bby1:

In the idea for example, for something like 3D or video game editing tools, where lot of plugins can be available for example for animators, or other kind of classes or plugin that can be instanciated dynamically, they are primarily made to be run by one application, but also potentially others who need to use those modules, even if they don’t fit totally in a system components either.

I think a way of dynamically loading code – either into the application or into a dynamic area – would probably be the best way here. The RMA was limited in size on older machines (a limitation not present in RO5) and it has no garbage collection so “gaps” in the RMA will remain as lost space. Accordingly, large amounts of data and code that are specific to one or two apps and don’t need to interact with the deeper parts of RISC OS are probably best placed elsewhere.

but i’ve been making things like streamed video player, 3d engines, things in flash, java gl, activex, on many plateform, so i […]

I am thinking that you will probably be best implementing your own dynamic loading mechanism; and given some of the examples this is perhaps best placed in a dynamic area so that it can run on callbacks or events (as applicable) without being swapped out during the normal polling process. RISC OS doesn’t do threads, so you’ll need to handle this sort of stuff yourself. ;-)
Don’t use the RMA (modules). Large things are RMA unfriendly as – as slack space in the RMA is often “lost”. It isn’t as clear cut as that, but generally speaking it is best to assume that space claimed in the RMA is not returned afterwards. If you use a DA, then once your application has finished, your entire framework can be unloaded and disposed of. Plus, being in a shared area of memory (as opposed to tagged to the end of an application), it could potentially be shared between multiple clients at one time.

I also look forward to seeing a practical live demonstration of what you have in mind. Do let us know when you have something executable available.

I’m a little surprised that he doesn’t seem to understand how things work around here, then.

Lot’s of activity outside the sheltered world of these forums I expect.
Thinking back over the last few years some people have been around here “forever” as the saying goes and others are pretty new to the regular contributions. Jon is in the latter category and keen to do more.
Me in charge, I’d open the door and point at a comfy chair. Maybe move on to discussion of office space later.

As far as i know there is no contract or anything that hold to keep things in a way or another for commercial reason today.

Sadly, there is – even if it isn’t immediately obvious.

Personally, I would make numerous changes to the Wimp so it can support Unicode and act a little more like a more modern OS. I would also make the keyboard handling capable of dealing with Unicode instead of that multiple-keypress hack, plus I would revise a number of the event calls so that Event_Mouse would be capable of reporting scroll positions, plus support for touch panel taps if they need anything extra.

There is only one itty-bitty little problem with this glorious plan (asides from, you know, a lack of understanding of the deeper parts of the Wimp code). There is no point doing it if it means every single older application would break. We are pretty much stuck with a lot of legacy cruft, and drastic changes – even if desired – would ultimately be undesirable if existing software fails.

Still it has been made in a certain manner, and probably with some reason, that can get beyond only scope of one application, in the global logic of the system.

Essentially, RISC OS is a single-task operating system. The rest is smoke and mirrors. ;-)

A lot of modules don’t bother with this kind of interface, so there’s no way that they can be killed after use.

If modules are small and not full of bloat, I’d say to just leave it there would be better than enforced killing. After all, loading it again may claim more RMA.

The only option is if the application author assumes that it’s an undocumented private module and they they ensure that only one client can ever be running at a time – which doesn’t feel to be that great an idea.

No, it doesn’t. But I think this may have been the original idea – SWI OS_ExitAndDie…

With ref to the CLib – what about having a separate jump table in another module or perhaps a extra RMA claim area – that could be left intact – if a new CLib was loaded?

The problem, as I see it, is that you are shifting the problem from the application to a place in RMA. It would be more manageable, certainly, as there is only one jump table to patch up. However this must be done in a way that can disable interrupts as the old CLib unloads, loads the new CLib, patches the jump table, and then turns interrupts back on – while at the same time suppressing all of the usual RISC OS background activity such as ServiceCalls and Events. Because you don’t know if a module in C will be responding to them.

This is mostly a legacy issue – I believe the current versions of RISC OS 5 (stable and nightly) both have the latest version as it was last updated around 17 months ago as far as I can determine. And right now, we’re mostly doing maintenance and bugfixes so it may not actually be necessary to run the very latest version available… Even so, regularly changing CLib in the course of a system running is not a normal thing to need to do. If a program absolutely must have the latest version, it should be in !System and a reboot should make it available safely.
The problem is, softload versions of CLib do not refuse to die, so if a program updates CLib (in !System) and then tries to load it, chances are the system will freeze in a few moments. :-(
Really, refusing to die would be the simplest approach. Let the user sort it out.

Really? How does that work? I can’t think of very many modules which track active clients, so how would your average third-party support module know to refuse to die when asked?

Of course you’re right on this point. My fault.
Not a big deal anyway. Modules are not here to be killed.

But yes it’s what i wonder too, it doesn’t seem the system has much tracking, or runtime dependency

Perhaps because existing applications don’t declare their dependencies to the system. And perhaps too because we almost never unload modules. Modules work a bit like DLL, but there are not made for this.

loading all upfront would not necessarily a big problem

It’s between the two. Loaded when needed, but loaded for all the session.

I am thinking that you will probably be best implementing your own dynamic loading mechanism

Dynamic RMA? :)

Perhaps because existing applications don’t declare their dependencies to the system. And perhaps too because we almost never unload modules. Modules work a bit like DLL, but there are not made for this.

Perhaps if the RMEnsure /RMload of a module added the module to a table of Application <—> module we would have a method keeping track of what was using a module – assuming people think it worthwhile keeping track.

Given that RISC OS has no runtime dependency tracking, the only way to make such a system work would be for each module to implement its own _Init and _Terminate SWIs that all clients call, and then track those clients (very possibly by giving them handles).

I can think of precisely one module that does (or did, in a previous incarnation) something along those lines. Jonathan Duddington’s Speak module, back when it was a purely RISC OS development.

The documentation asked that any software using it called a SWI to register itself with the module, and to similarly call a SWI to deregister when the app was quitting. All the module itself did was keep a count, and it would remain loaded while the count was greater than zero.