SparkFS (self-extracting) isn't as great for bootstrapping as it could be

Not for the first time I found myself setting up an install of Arculator v2.2 tonight and tying myself in knots just trying to bootstrap a basic RISC OS environment again to play some games. It wasn’t much fun and I think I know roughly what I’m doing with RISC OS, I’d have had to give up if I was new to the platform.

The root of the problem was that self-extracting SparkFS from ROOL (which is clearly intended to help with bootstrapping hence the self-extraction part) fails because it hasn’t seen !System. Then when you get the RISC OS 3.11 Uniboot uncompressed from the old RISC OS FTP share to show it !System, that’s missing CallASWI, FPEmulator and CLib so you have to go and find those yourself somewhere else.

Maybe I’m doing it the hard way, maybe SparkFS is too complicated a piece of software to package more modules with it but I’d really appreciate if the self-extracting SparkFS was a bit more self-reliant, precisely because time and again it’s the only bootstrap tool I need to be able to install the rest of the zip files of software I can copy across.

Am I the only one who hits this problem and I’m just doing it wrong?

Oddly enough, if you comment the System check out of the !Run file then it seems to work (or at least used to: I haven’t tried the ROOL version).

Edit: Actually, I’m thinking of SparkPlug.

IMO, SparkPlug is a better tool for bootstrapping RISC OS 3.1. There’s a self-extracting version of SparkPlug (with the !System check removed) available here if you haven’t already got a copy.

Though SparkPlug may resolve this issue now, be aware that it has not been updated for many years. Indeed, when I wrote about a SparkPlug problem to David Pilling way back 2015, he replied:
SparkFS is vastly better than SparkPlug, everyone should use it.
I would suggest it would be better to get the self-extracting SparkFS fixed.

In general, I agree that SparkFS is a much more useful piece of software than SparkPlug.

But we are talking about bootstrapping RISC OS 3.1, where Size Matters.

Older Acorn machines didn’t have an HD floppy drive: the largest DOS floppy supported was 640KB. CallASWI, CLib and FPEmulator would add 82KiB to the size of self-extracting SparkFS, taking the total size to 617KiB (which is dangerously close to the 640KB limit).

Sure, everything fits now, but in the not-too-distant future it will exceed the 640KB limit. This will render it unusable on older Acorn machines such as the A310.

Sure, everything fits now, but in the not-too-distant future it will exceed the 640KB limit.

The problem here is that the self-decompression stuff makes use of Squash, which isn’t a particularly good compression algorithm. It was chosen as a trade-off between the need to compress stuff, but also to decompress at an acceptable speed on ancient (8MHz etc) hardware.

I just downloaded the self-decompressing SparkFS (535K) and unpacked it (778K) and recompressed it to Zip Deflate using itself (417K).
Simply by changing algorithm, 118K was saved.

Now the extra modules add up to 85K. Squashing them will do absolutely nothing as they are mostly already squashed, other than some stuff in the module header, but that’s like 80 bytes or somesuch.
Zip, however, shaves off 10K.

The Zip module in SparkFS is ~32K, and we only need to have decompression code. It could, perhaps, simply work with a zip file pasted onto the end of the utility to make it easier to create?

Whatever, let’s just assume 417K for the app, 75K for the extra modules, and a pessimistic 32K for the utility code to do the unpacking. That all adds up to 524K which is smaller than the current method.

Therefore, it may be worth looking for an alternative compressor to use instead of Squash if archive size is likely to be a problem. For these one-off tasks, taking a second or two extra isn’t a hardship.

Note: Zip Deflate is ancient. There are better methods (bzip2, PPMd, LZMA, LZMA2 (7zip), etc) that SparkFS doesn’t currently support that may well offer a superior level of compression – 7za, for example, got the SparkFS install down to a mere 316K.

This will render it unusable on older Acorn machines such as the A310.

As you can see, the correct action is not to drop stuff on the floor and hope, it’s to look for a better solution. ;)

(which is dangerously close to the 640KB limit)

You sound like my phone whinging that it doesn’t have 1GB free of its 16GB storage (~4GB taken by the OS).

The disc has x capacity. The data is of y size. As long as y is less than x, all is good. Clearly it is an advantage to have the data be smaller, as you can fit other random junk on the same disc, but you won’t hear klaxons sounding if your file consumes 630K. You’ll just hear chug-chug-chug-chug-chug for a little bit longer.

As a side note – is there a reason why there are two non-identical SparkFS modules within? One in .Resources and the other in .Resources.UK.

The run file says this:

RMEnsure UtilityModule 3.60 RMEnsure SparkFS 1.36
    {wrapped} Run <SparkFS$Dir>.Resources.UK.SparkFS
RMEnsure SparkFS 1.36 Run <SparkFS$Dir>.Resources.SparkFS

What is the reason for this? Could save ~32K there. ;)

The SparkFS module in Resources.UK is not 32-bit, but the one in Resources is.
Perhaps the 32-bit one will not run if before RO3.60 ?

As you can see, the correct action is not to drop stuff on the floor and hope, it’s to look for a better solution.

I think you may be missing the point.

A solution for bootstrapping a RISC OS 3 disc installation already exists: you use SparkPlug to unpack the ZIP archives of your chosen HardDisc4 image. You have the choice of the old Acorn !Boot structure, or the newer version of !Boot supplied with ROOL HardDisc4 and PlingSystem. Getting these archives onto the machine will of course involve a lot of disc swapping. It’s decidedly clunky, but it does work.

What you are proposing is a substantial alteration to self-extracting SparkFS so that it can take the place of SparkPlug in the bootstrap process. This has two advantages:

• Someone downloading the self-extracting SparkFS doesn’t get a nasty shock when they try to use it to bootstrap a RISC OS 3 disc installation.
• Using a more modern compression algorithm means that the self-extracting archive is smaller, and could also reduce the number of disc swaps needed when performing the bootstrap process.

However there are some disadvantages:

• Do we really need another method of bootstrapping a RISC OS 3 installation? Yes you’ve reduced the number of disc swaps from 23 to 16 (at the expense of having to re-archive the disc image as 7zip), but is that enough of an improvement?
• SparkFS can also be downloaded as a Zip file from the Packages server, so the self-extracting version isn’t the only game in town (if you’re worried about it fitting on a floppy disc, for example).

From where I’m standing, this looks like a lot of work for relatively little gain.

Do we really need another method of bootstrapping a RISC OS 3 installation?

I was suggesting a better compression for the self unpacking SparkFS so that it used less disc space… I was not suggesting to convert everything to 7zip or whatever.

Might be able to put together a “get this going” floppy that has a minimal !Boot, and SparkFS in order to have a easy way to install the things necessary to be able to install everything else. That’s the thinking behind it.

Older Acorn machines didn’t have an HD floppy drive: the largest DOS floppy supported was 640KB.

I am fairly sure that DD DOS format had over 700 KiB free. 704 or 706 IIRC.

Apart from that, I can feel David’s pain – I don’t know how often I bootstrapped RISC OS 3.1 stuff (for MIST, MISTer, A310, A3000, Arculator, Red Squirrel…) and I seem to fall into the same traps every time. Because I only do it every two years or so and forget all the details in the meantime. The !System check in !SparkPlug is my personal foe because I usually have a German keyboard layout but RISC OS assumes UK and then where is that comment bar…

I am fairly sure that DD DOS format had over 700 KiB free.

DOS was 720k (specifically, there were 1440 sectors at 512 bytes each).

The !System check in !SparkPlug is my personal foe because I usually have a German keyboard layout but RISC OS assumes UK and then where is that comment bar…

Indeed. Same problem with an ANSI layout.

What about using ArcFS ?

Hasn’t been touched in about 30 years.

The root of the problem was that self-extracting SparkFS from ROOL (which is clearly intended to help with bootstrapping hence the self-extraction part) fails because it hasn’t seen !System.

First and foremost I think it’s helping with bootstrapping RISC OS 5, where the prerequisites (CallASWI, FPEmulator and CLib) are already new enough. For RISC OS 3 or 4 bootstrapping I just have a minimal !System with those 3 modules in scraped from the PlingSystem download, an Acorn vintage Uniboot won’t help you as that only supports APCS-R.

In fact my notes from last time I bootstrapped a Risc PC in April with RISC OS 3.70 say I threw in the towel and fitted a NIC and shared another (already up and running) RISC OS to copy stuff from via the internet. So I have a floppy with Internet, Freeway, ShareFS and IfConfig on it.

As a side note – is there a reason why there are two non-identical SparkFS modules within?

RISC OS 3.50 and earlier don’t support internationalised *Commands, so the other SparkFS module is loaded in that case with baked in English.

Why use SparkFS rather than the self extracting HardDisc4? As I discovered: because self extracting files are Utilities they need enough space in the RMA to load into, again that’s fine on RISC OS 5 with a 512MB RMA limit, but on an A3010 it’s a no go – hence you start with SparkFS and unzip HardDisc4.zip instead.

Probably the takeaway here is that the check for System$Path being set should only be made if CallASWI/FPEmulator/CLib aren’t new enough, and therefore need loading, and therefore need !System. That would result in a more meaningful error message and, for RISC OS 5, no error message at all.

So I have a floppy with Internet, Freeway, ShareFS and IfConfig on it.

Back in the 90s I was using an x86 version of that kind of build to take PC’s from just delivered, to ready to install at user location.
Mainly because doing it manually, repeatedly, is tedious.

It strikes me that with people having home networks that have file shares and more, there’s a good case for building a boot media, with just enough to make the network connection to take you through to a simple menu-driven installation from a file share.

a good case for building a boot media, with just enough to make the network connection

It’s a very good point and is, indeed, how I got my 3B+ setup. The basics from a copy of harddisc4 on a USB key, and all my apps and stuff copied from ShareFS.
[I would have used Direct instead if it came as a zip file, but as an SD image it’s more trouble than it is worth – but I understand that “special attention” needs to be paid to the FAT part – I used SystemDisc to set that up for me.]

It strikes me that with people having home networks that have file shares and more

The problem is that while network ports are ubiquitous now, it’s less common on older hardware.
My RiscPC ¹ has 10baseT. My A5000 ² has 10baseT and Econet. My A3000 has Econet. And the A310 has Econet, but that’s no use to this discussion as it runs Arthur 1.20. ;)

¹ Probably battery-diminished ³.

² Known to be battery-diminished ³.

³ Euphemism for dead.