Building RISC OS Online at ROUGOL, Mon 16th Nov 2020

The next meeting of the RISC OS User Group Of London (online) is:

Building RISC OS Online, presented by Gerph

Monday 16th November 2020, 7.45pm

Online via Zoom, meeting open from 7.30pm

http://www.rougol.jellybaby.net/meetings/

After many years away, Gerph made a welcome return to the RISC OS world on the 1st April this year with JFPatch-as-a-service. But this was no April Fool, and he plans to develop it further into an online RISC OS build tool.

JFPatch started life in 1995, when he needed a way to build modules easily without writing all that boilerplate. This was in the days when software could be downloaded from BBSs and email was genuinely newfangled. Jump forward 25 years, and cloud systems are everywhere and online applications give you everything you want. Running 25 year old software as a cloud service seems like it’s a bit of a niche area, but it hides some clever technology.

Although the service itself was written in about 20 days, it builds upon a year’s work to get to that point. Gerph will talk about why he started doing RISC OS things again, quite a bit about how the service and front end works, and a little about how other people might use it. There will be a demonstration of the service and of the underlying technologies, and he will take questions from the talk afterwards.

This meeting will be held online via Zoom. Please contact us to receive a link to the meeting on the day.

http://www.rougol.jellybaby.net/contacts/

Email was older than Gerph in 1995!

Ha, yes, but not to most home users! I first used email at university in 1988 but wasn’t online at home until 1994 when I got the RiscPC and a Demon account.

Tonight! Still time to contact us for the Zoom link :-)

Wow, that was a jaw dropping presentation, amazing stuff. Brilliant work by Gerph.

Slides are here – http://pyromaniac.riscos.online/slides.html

Video to follow later…

Wow, that was a jaw dropping presentation, amazing stuff. Brilliant work by Gerph.

You expected less?? :)

If you haven’t read “RISC OS Rambles” do so, if you have read them again, and again.
There’s a lot to see that’s barely hinted.
What you’re seeing in Pyromaniac is a fraction. Pitching it as a test platform is interesting, I’m wondering where he’s going to take that? Beyond a testing suite.

The two recent presentations from Jason with the Audio stuff and now Gerph with the …erm…everything stuff show a good way forward for future RISC OS development – use whatever technology and infrastructure that is available, no matter if it runs natively on RISC OS. Because the only thing that really matters is that the FINAL product runs on RISC OS. And that this final product is of higher quality than previously possible in the given – usually severely restricted – timeframe.

Of course it took someone like Gerph to come up with a solution that provides such an impressive scope of…everything. Building, running, testing, SCM connection, CI/CD stuff…

My only problem with Gerph’s presentation was/is that it will take me weeks to think all this stuff through, process my thoughts and ideas surrounding these topics (I have my own “development environment” using tools I know from day-to-day work to develop stuff for RISC OS, but it is very specific for my needs and not universally useful) and trying to think of quick wins I could achieve with the help of his environment.

So much food for thought, so little time :-(

From what I saw, a really good and professional presentation.

I say what I saw, as those of you who have (or once had) small people will know – they have an innate ability to just know when you have something to do just after they normally go to sleep and decide that tonight is the time that sleep isn’t going to happen just yet. I am looking forward to watching it on catch-up (youtube).

If you haven’t read “RISC OS Rambles” do so, if you have read them again, and again.

This. Very much this.

When you (anyone) have read them (RO Rambles)¹, think about the things he wanted to do then and how the developments in virtualisation etc have made even more possible in the intervening years so now you get a glimpse.

I’m pleased to see Justin back again (with whatever name he wishes to use)
I missed the “entry point” round here when I missed out on reading anything for months on end.

Pyro: I wonder how close the system is to being able to build a full ROM.

¹ Does everyone know the URL?

Err – what can be done with this?

Err – what can be done with this?

Best to read all the info at https://jfpatch.riscos.online/
and https://pyromaniac.riscos.online/ the latter is in part a product of the former.

Presented to a gobsmacked ROUGOL as in http://pyromaniac.riscos.online/slides.html.

I just know you’re going to enjoy some of that Colin.

Loved the bit where he showed using *FX0 output from Pyromanianc and it reported RISC OS 7.16 :-)

As he said he wants to have fun and enjoy it and that made me smile as well.

Pyro: I wonder how close the system is to being able to build a full ROM.

That’s a quite interesting question, and actually not one I have considered. You see, I’d be more interested in resurrecting the cross-compilation system rather than trying to build on Pyro – other than the fact that it would prove that the system is functional at the command line.

Why? Well, it’s SO much faster building on even old x86 natively compared to building in Pyromaniac. I can’t speak for the modern RO systems ‘cos it’s been years and huge improvements in the systems, but back in 2005ish when I was building these things for Select, it was orders of magnitude faster. Even the early versions which would run the build scripts on an SA RPC and then rcp the files to my 500Mhz Linux machine, build and then rcp the results back again.

BUT… The build system was essentially (and I may be misremembering this so forgive me if I’ve forgotten my old system, never mind how things work right now, and I’m looking at scripts I’ve not used in 15 years).

1. Checkout a full source tree
2. Run !EnvUrsula (or whatever)
3. Locate BuildSys. (usually BuildSys.ROM)
4. Run !Config (which builds the makefiles)
– First we use the Ordering, Project..Configure, Groups and Sources file to build a Components file by constructing a dependency tree for the Product that you were targetting. This uses perl, and that’s supported. These Groups and Sources files I think correspond to a single Components file that describes the system that’s built, but mean that the type of system can be cotrolled by conditionals. I probably described this in the Rambles, but it’s also orthogonal to this discussion.
– From the Components file, makefiles for building the ROM, the ‘Build’ and the Logs are created. Looks like those are all Perl too.
5. Run !MkRom (which builds a ROM image)
– runs amu on the MakeBuild makefile that was created and that… looks like it just calls the makefiles for each of the components it wants with the right target dirs and options. So from there on, you’re looking at whatever the makefiles for individual components do.
– triggers the amu MakeROM script, which runs the link tool to perform linkages of files; I think this also calls SetOLength and some symbol extraction tools. I can’t remember if I ported those for the cross-compiling build, but they should work for regular OS.
– Calls bigsplit1 to link the ROM and include metadata – that’s just BASIC. Although I was sure I’d written a replacement tool for that, romlink… or is that BuidlROM?. Maybe that got lost in the interveneing years (huh, no I have the code… maybe I never finished them enough to integrate into the build?)

So then we’re down to what the tools are and what works…

Looking at what I’ve got here, it’s looking like I’ve got most of the tools present in the source tree, but that doesn’t necessarily mean that they work. I haven’t tested Canonical at all – although it’s a trivial tool. I KNOW that the c compiler fails if I turn on -g option with certain structures. I’ve clearly not got the 64bitting quite right in some places.

I don’t have a working ‘squish’, so I suspect that BASIC programs compressed won’t work – my build process just copies files instead of doing anything special with them. I think squish was just a ‘load into BASIC and apply the CRUNCH settings’ or something, so maybe that’s not a problem.

I’ve forgotten, but I /think/ I have some problems building things that go into making ColourPicker (the Support library, I think that is?) and Toolbox libraries ring a bell as being a problem.

Ah, I’ve found what it was now. Toolbox modules require MethodGen to work in command line mode, and MethodGen itself isn’t written to be cross-compiler friendly, so I wrote a perl script that can do the source and header file generation. So that would work in the cross-compiling environment and the native environment, OR in RISC OS you could just use MethodGen directly. Probably MethodGen would be faster, as perl is quite slow. [Aside: The tboxmethods perl script is isolated and could probably be open source; I’d forgotten about it. Ah, yes, I was working on updating it so that it could be used to generate suitable headers and implementation files for python so that it could help with the Python 3 system – the last log entry on that says “It’s not quite right but it’s moving in the right direction”]

I’ve not implemented the redirection yet, though – { > foo } on the command line doesn’t work. It’s parsed, but discarded. This is because it requires a transient environment handler to be installed to replace the redirection, which I wasn’t comfortable with. I think that it’s probably easy to do now that the OS is more mature, but I’ve not looked at that.

One bug/feature of Pyromaniac is that OS_Byte and OS_Word calls that aren’t supported raise errors. That’s a BAD THING for RISC OS really, because they’re pretty much guaranteed not to report errors, but it helps locate things that aren’t implemeted. Come to think of it, that’s actually an easy option to make configurable. But right now if anything hit an operation it didn’t like, bad things would happen.

Oh, the SpritePool component would probably break because it uses a load of OS_SpriteOp calls to construct the sprite pool from the individual files. If it’s just the manipulation of the sprites themselves, it’d actually be quite simple to just implement those calls.

For the environment, DDEUtils_Prefix operations are not currently supported, so anyone messing with those as part of the build will run afoul of that. But I think that should only be done by the caller, and so I expect each component should be fine in its build.

So… maybe it’d work mostly now, given the right tools. Maybe some small changes for the redirection case and sprite operations.

It would be interesting to see what happened.

The biggest issue for practical use is the speed. IF you only execute ARM code – a MOV instruction over and over, you get around 6000MIPS on a MacBookPro from 2013 (2.6 GHz Intel Core i7). However, if you execute even a small SWI like OS_IntOn repeatedly, you get about 0.1MIPS. Basically the transition from ARM<→Python (and Python itself) is slow. Probably the slowest thing about the JFPatch service is the fact that the JFPatch tool itself uses OS_Byte to read each individual character – that means huge slowdown. If it sat in BASIC executing code all the time, reading from a buffer, it’d be hugely faster. So that might also affect other parts of the build’s speed as well.

But if you wanted to just say ‘hey it works’, speed isn’t a factor.

That’s a quite interesting question, and actually not one I have considered. You see, I’d be more interested in resurrecting the cross-compilation system rather than trying to build on Pyro – other than the fact that it would prove that the system is functional at the command line.

Judging by the length and depth of your answer you gave it more a minutes thought.
I didn’t mean to distract you. And then I ramble on again.
As I said elsewhere, welcome back.

The main reason for posing the question, which was more musing than anything else, was that a number of people have complained about a free Open Source OS with expensive development tools. Many of them arguing that the DDE should magically become free.

I can’t speak for the modern RO systems ‘cos it’s been years and huge improvements in the systems, but back in 2005ish when I was building these things for Select, it was orders of magnitude faster.

I believe that would still be the case.
That said Chris Evans recently made a comment about rebasing the benchmark tests as the numbers were getting silly large.
With the benchmarks based on the RPC, with its bus congestion, and the modern system also having a CPU speed that is about 8-10 times faster than the Kinetic it does look “interesting”
So compile speeds have increased, but still not to match the other platforms.

It might help if the OS actually used all the cores in the SoC instead of one

I’ve forgotten, but I /think/ I have some problems building things that go into making ColourPicker (the Support library, I think that is?) and Toolbox libraries ring a bell as being a problem.

Start a conversation about libraries and when there’s two of three people talking about which are any good, and why half of what they really need isn’t there, just shuffle away quietly for a few weeks – the conversation/argument/gripe will still be going when you get back.

But if you wanted to just say ‘hey it works’, speed isn’t a factor

I know about enough to poke, prod, and minutely alter a module or two so building single modules as soft-loadable replacements isn’t going to tax any system in terms of speed.

However, for those wanting a free method of building the OS, setting something running overnight probably won’t offend.

I’m surprised you don’t have a Pi as a novelty interest. (I know, too many systems cluttering up the house already.)
The Pi400 system is a convenient boxed up item and probably the fastest available ready to go. No chance of running Select on that though, so you’d be forced into RO5.29¹ or Linux.

¹ Finalisation of 5.28 stable came before the Pi400 release and a couple of tweaks required to make things work.

Judging by the length and depth of your answer you gave it more a minutes thought.
bq. I didn’t mean to distract you. And then I ramble on again.
bq. As I said elsewhere, welcome back.

An interesting question deserves a worthwhile answer – and honestly I hadn’t considered whether it was possible and how far away it might be.

The main reason for posing the question, which was more musing than anything else, was that a number of people have complained about a free Open Source OS with expensive development tools. Many of them arguing that the DDE should magically become free.

Free tools would benefit things significantly, but remember that those tools have already been made freely available – the RO3.7 image bundled with the Easy Start bundle incorporates those tools. I suspect there might be some performance implications, and you might need to trigger some of the -z options on the compiler to avoid some now unsafe operations, but they would probably work just fine on building 32bit code on most systems.

Certainly the code that I build which uses a compiler which is approximately contemporary seems to work ok (‘seems to’ may not be good enough, but hey, that’s the way things go some times). The only real differences are the need to use -za1 to disable unaligned loads. There might be others that I’ve needed but not used, because I’m not building for system that have different requirements.

I know about enough to poke, prod, and minutely alter a module or two so building single modules as soft-loadable replacements isn’t going to tax any system in terms of speed.

Building individual modules should be really simple. At least that’s what I’d expect. I’m meaning C modules here; objasm modules would be even less of an issue, because objasm hasn’t really changed.

So, I’ve spent about 20 minutes or so trying out a further idea, and this is pretty half-baked because I’m just throwing around thoughts given your comments. Let’s say you want to run a build automatically and you want to do so in the cloud without resorting to the very slow system that is Pyromaniac… The RISC OS on Linux thingy seems like it does the job, but you can use RPCEmu, too.

This is a simple scenario I just put together…

You’re got a Linux system, with Docker installed.
You want to run some commands on RISC OS. So you write them into a file and you put them in a directory along with your source code. You use *Spool to capture all the output that you get.

charles@laputa ~/projects/RO/example-run-rpc-build$ ls
My!Boot,feb

charles@laputa ~/projects/RO/example-run-rpc-build$ cat My\!Boot\,feb 
Spool $.Share.Log
.
spool
Create $.Share.Complete

charles@laputa ~/projects/RO/example-run-rpc-build$ rm -f Complete,ffd ; docker run --name Session$$ -v $PWD:/home/riscos/rpcemu/hostfs/Share -v $PWD/My\!Boot,feb:/home/riscos/rpcemu/hostfs/\!Boot/\!Run,feb --rm -p 5901:5901 gerph/rpcemu-3.7 & while [ ! -f Complete,ffd ] ; do sleep 1 ; done ; docker kill Session$$
[1] 17117
QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-riscos'
PulseAudioService: pa_context_connect() failed
Session16283
[1]+  Exit 137                docker run --name Session$$ -v $PWD:/home/riscos/rpcemu/hostfs/Share -v $PWD/My\!Boot,feb:/home/riscos/rpcemu/hostfs/\!Boot/\!Run,feb --rm -p 5901:5901 gerph/rpcemu-3.7

charles@laputa ~/projects/RO/example-run-rpc-build [2]$ ls
Complete,ffd Log,ffd      My!Boot,feb

charles@laputa ~/projects/RO/example-run-rpc-build$ cat Log,ffd 
Dir. HostFS::HostFS.$ Option 02 (Run) 
CSD  HostFS:"Unset"
Lib. HostFS:"Unset"
URD  HostFS:"Unset"
!Boot        DWR/    AcornC_C++   DWR/    Apps         DWR/   
Diversions   DWR/    Images       DWR/    Library      DWR/   
Manuals      DWR/    Network      DWR/    PC           DWR/   
Printing     DWR/    ReadMe-bundle/txt WR/    
Replay       DWR/    Share        DWR/    SHManuals    DWR/   
Sound        DWR/    Source       DWR/    Tutorials    DWR/   
Utilities    DWR/    Video        DWR/   

What does this do ?

1. I run ‘ls’ to show what files exist in my linux directory – there’s one file, my My!Boot,feb file.
2. I cat that file so that I can see what it does. The ‘Spool $.share.log’ creates my log file. The ‘.’ is the commands I want to run. The ‘Spool’ turns off the log. The ‘Create $.Share.Comlpete’ will create the marker file that we use to spot we’re done.
3. Then I run a sequence of commands to execute RISC OS
   1. delete the marker file if it exists
   2. start a docker which:
      1. (—name …) has the name Session$$ ($$ is a unique value for this shell)
      2. (-v …) mounts the current directory as $.Share inside RISC OS
      3. (-v …) replaces the Run file with the My!Boot file that we just saw
      4. (—rm) deletes the container when it completes
      5. (-p 5901:5901) maps the VNC ports to the local system, if you want to watch it in a terminal (you can skip this usually)
      6. (gerph/rpcemu-3.7) runs the published container that has 3.7 in
      7. (&) runs in the background
   3. (while … done) waits for the marker file to appear, checking once a second
   4. kills the docker container
4. I then ls the directory to show that ‘Complete,ffd’ and ‘Log,ffd’ have appeared.
5. And cat Log,ffd to show that we really have run the commands inside RISC OS.

Problems with this? Lack of error handling – you’ll hang if the Complete file doesn’t appear, and the log won’t be written properly unless you actually close the file with *Spool, so if you kill the docker process before it’s closed, you’ll lose logging data.

This is an entirely distinct solution to Pyromaniac, using the hardware emulator as an example – and the RISC OS on Linux project can (I believe) do all in a better manner. But, it’s demonstrating that you can do some of this sort of running of the system on Linux already, and it doesn’t need any Python code :-D

Actually, I think something like the above is what Chocky’s original automatied build system does. Although probably better.

Sorry, huge digression.

I’m surprised you don’t have a Pi as a novelty interest. (I know, too many systems cluttering up the house already.)

I have a Pi in a box somewhere in a cupboard. I ran XBMC on it for a day or two and haven’t touched it since. It was a vague consideration to use a Pi for my testing purposes early on, and my comments from the talk omit other experiences running iPaqs as systems for testing on.

Those were small systems, intended for running the SUT on a representative target hardware (‘an actual ARM device’), and if things went wrong, the machines had to be restarted automatically. We had a number of them, which if they failed, could be restarted and another would take its place until the failed machine either rebooted and reentered the pool or it never came back and would eventually be looked at by someone on site.

Maintaining physical hardware was not a game I wanted to play. At least, not again :-)

Certainly the code that I build which uses a compiler which is approximately contemporary seems to work ok (‘seems to’ may not be good enough, but hey, that’s the way things go some times).

Ah, things tend to be a bit picky on the newest cores so if the code generated has alignment issues¹ then things randomly go pop. There’s also the deprecation of the SWP instruction – again on ARMv7+ things go pop because of that. UnixLib needed a cleanup.

Building individual modules should be really simple. At least that’s what I’d expect. I’m meaning C modules here

C. I have this mental block I’m trying to find the time to get past…

¹ StrongARM was quite gentle and even the XScale was relatively forgiving, but the OMAP3 bit a little.

@charles@laputa@

Ghibli fan, by any chance? ;-)

but the OMAP3 bit a little.

As did the Pi2 with a much stricter implementation of the ARMv7 core. Shouldn’t be a problem with compiled code, but there’s an awful lot of assembler floating around, and being written by mere mortals, can include some silly things like writeback to the register you’re reading, which the Pi2 rejects as an invalid instruction. Well, technically it is, but it’s still a pain when stuff randomly falls over and one has to rummage around with Debugger to figure out why.

C. I have this mental block I’m trying to find the time to get past…

I got myself a big “teach yourself C in 21 days” style book back in the early ‘90s, and used it alongside TurboC++ with PCEm. Didn’t bother with the DDE or RISC OS complier until things started to make sense.

These days, similar book comes with CD containing a compiler and examples, aimed at the XP era machines. Look for ISBN 978-0672324482 on your bookseller of choice and pick up a second hand copy for yourself.
I’ve just bought myself one for €6,95. There’s nearly a thousand pages (!) so I’m sure there’s plenty of stuff yet to learn. And no, you probably won’t learn C in just 21 days because that’s something ridiculous like 45 pages a day. But it’s a good place to start. Nothing to deal with a mental block like jumping in and just doing it.

And, take it from me, while C and BASIC are both procedural languages, that’s about where the similaires end. It does not help to think about what the C code is doing and how you might express the same thing in BASIC. I never really ‘got’ C until I learned to let go of BASIC and think only in C.

C. I have this mental block I’m trying to find the time to get past…

JFDI. Just Do It.

JFDI. Just Do It.

I’ll get there. Took me years to even twiddle with ARM assembler. I really don’t know why.

Anyway, back on track.

@charles@laputa@

Ghibli fan, by any chance? ;-)

Whilst I like Ghibli, it’s actually a reference to Gulliver’s Travels; same reasoning though.

Back when I worked for Bromium, the Macs we tested on were named after cities. They were all MacMini’s, and were sat in the server room. But we introduced a MacBookPro for testing, because laptops are different to the MacMinis. It needed a name and since it was a laptop, I named it Laputa because it can move – it’s a Flying City :-)

And when they made our team redundant they gave us a generous package which included our laptops. Bromium got bought by HP last year, so the redundancies meant that the company survived till then, and my friends still had jobs, and I had decent package and a laptop. So I ended up with it, and I like the name.

Here’s me thinking it was Dr. Strangelove. Kong’s plane’s primary target is an ICBM complex at Laputa.

Here’s me thinking it was Dr. Strangelove. Kong’s plane’s primary target is an ICBM complex at Laputa

Probably a combination of borrowing from other literature and/or a joke name.

Drop a space in before the “p”

Indeed!

The recording of Gerph’s inspiring talk is now available:

https://youtu.be/YBtVgvJNeNQ

I highly recommend it to everyone!