RISC OS FAST

I have uploaded a picture of a FAST machine – kit built using RISC OS bits PCIE/SATA board. See my wakefield show page.

Using iozone on RISC OS and Raspberry Pi OS with identical hardware shows RISC OS is maxing out the hardware on reads but slower on writes.
RISC OS also struggles with smaller record sizes.
Comparing to an NVME drive on the same hardware using Raspberry Pi OS there could be slightly faster reads but hopefully much faster reads and writes using smaller record lengths.

iozone -e -I -a -s 128M -r 4k -r 8k -r 16k -r 32k -r 64k -r 128k -r 256k -r 512k -r 1024k -r 2048k -r 4096k -r 8192k -i 0 -i 1 -i 2

Raspberry Pi OS SATA Samsung SSD 870 EVO 500GB

RISC OS 5.29 SATA Samsung SSD 870 EVO 500GB

Raspberry Pi OS NVME Samsung SSD 970 EVO 500GB

Petition for the SharedCLibrary to dynamically set a buffer size to something appropriate for the platform in use! So we don’t all have to concoct our own setvbuf() parameters for buffered I/O… I have tended to use 16KB in the past as initially that helped greatly with SCSI on the A540, and now with SD cards on modern h/w.

[Edit: So I couldn’t resist… PipeDream 4.62.02 now uses a 64KB I/O buffer on ‘modern’ systems (faling back to 16KB as before, or even 4KB on wee systems). If I were feeling braver, I could stick the buffer up in flex.]

Using a larger LFAU size when formatting a disk as filecore helps speed up certain disk access.

Here’s timings for compiling a recent Rpi ROM using larger LFAU sizes.

3mins 35s (8K LFAU)
2mins 52s (32K LFAU)
2mins 20 (64K LFAU)
2mins 1s (Ram Disc)

@Chris: Interesting!

As far as I get the LFAU size, it means that each file, even if it’s less than 64 KByte in size will occupy 64 KBytes on the drive ? Is that visible when you count the space that the ROM source takes on the harddrive ?

No it’s far worse than that. On FileCore the minimum size of a file is the LFAU * IDLen+1 so with say 19 bits of ID and a 64KB LFAU the minimum size of the file is 1.25MB, and larger files are then rounded up to the next nearest 64KB. FileCore does have the ability to share a minimum allocation between a directory and/or files, but in practice it not used much, and large LFAUs give horrendous wastage.

It’s been a reoccurring problem for decades as disc sizes increased from MB to GB to TB and LFAUs had to go up, until changes were made to increase the number of bits in IDLen. But this has drawbacks of massive disc maps and lots of zones, which required machines with more memory and processing power to deal with this overhead.

Sure, 8MB is very small – but by RISC OS standards it is huge and as far as I can tell kind of unimportant for most (of my) real live usage!

Just for fun I compared mini.m (eSATA SSD) and RPCEmu on a powerful Windows PC (2.9 GHz Core i7 with fast M2 SSD). In ROMark both were not far apart (up to about factor 1.5) – just for rectangle copy RPCEmu is well more than 15 times the mini.m speed.
BUT for 8MB HDD access windows really beat the shit our of mini.m – getting more than 80 (eighty!) times the speed – but on the smaller files using byte access oddly mini.m outperformed RPCEmu being 3.5 to 4 times the speed. Yep, I reran that and tried RPCEmu on native windows and inside Windows Subsystem for Linux. 8MB files RPCEmu = 80 times mini.m; byte stream mini.m = 4 times RPCEmu.

Looking at the overall handling and swiftness of mini.m and RPCEmu – both certainly in full resolution of 3840×2160 in full color the thing is that mini.m is significantly better! Loading ArtWorks on mini.m is nearly instantaneous; on RPCEmu I can nearly read the name of all plug-ins since it is a lot slower. Same applies to nearly all other app loading and to booting as well. If NetSurf has to recreate its font cache on mini.m so what, on RPCEmu it takes quite a bit longer.

I dare say that even the rectangle copy is something I can’t really care much about since the processor power is so massive compared my own speed moving the mouse etc. that it can easily cope with moving windows etc. so that a faster rectangle copy might be measurable and if you look close visible, but for me not really significant…

Basically, most files on RISC OS are small – massively smaller than 8MB (and if bigger probably some archive or some ported app or the odd movie and big image perhaps) so that high disc speed for that size is not really significant, but the byte stream access seems to be the one of importance during normal use since…

In my test in mini.m and Windows were not far apart; on rectangle copy Windows did have a clear edge, on 8MB file access it was massively faster (80x) but on stream file access mini.m outperformed Windows by just 3.5-4×.
But using the system to write publisher documents, play with the odd app the odd thingy – or just boot and load apps etc. the mini.m is significantly faster than RPCEmu on windows – and the ONLY ROMark test where the mini.m clearly won is the byte stream file access.

My favourite test at the moment is to use !RingBind to turn the pages of the ROSC OS 5.28 User Guide. This shows rectangle copy acceleration (where present) as well as loading data and rendering Draw Files, including using a transformation matrix to skew them.

Starting with a 256GB filecore partition.

LFAU 8K
Used 156Gbytes
Free 100Gbytes

After copying the contents to a filecore partition of the same size.

LFAU 32K
Used 189Gbytes
Free 67Gbytes

On FileCore the minimum size of a file is the LFAU * IDLen+1

I have an 8GB µSD card.

Is this right?

>DIM rec% 255
>SYS "SDFS_DescribeDisc", ":0", rec%
>PRINT rec%?0   : REM Log2 of sector size
         9
>PRINT 2^9      : REM Sector size in bytes
       512
>PRINT rec%?10  : REM Log2 of allocation unit
        10
>PRINT 2^10     : REM Allocation unit in bytes
      1024
>PRINT rec%?4   : REM idlen
        19
>PRINT (19 + 1) * 1024 : REM Minimum file size
     20480
>

So… each file is consuming at least 20K on the disc? That, frankly, seems smaller than I was expecting. I can’t help but think I’ve got something wrong here. Am I calculating the LFAU correctly?

Probably, try running DiscKnight with verbose on, it will work out the minimum object size without getting the decimal point in the wrong place like I did above (now corrected).

try running DiscKnight with verbose on

  ID Length                : 19
  Bytes per map bit (LFAU) : 1024
  Minumum object size      : 20.0K

as disc sizes increased from MB to GB to TB

Yup. What worked for floppies and the small (20-80MB) harddiscs of the day doesn’t really scale well to modern sized devices.
I’m actually surprised that it’s “only” 20K per file on my device.

Sure, 8MB is very small – but by RISC OS standards it is huge and as far as I can tell kind of unimportant for most (of my) real live usage!

Ah, but I suspect that you are misunderstanding. This is the smallest “amount” that every single file on the drive will consume.

To illustrate this, here are the eleven files within StrongHelp.

The second column are the file sizes in bytes.
The third column are the accumulated sizes, what you’ll be told if you Count them.
The forth column is the actual space used on my filing system with a 20K minimum.
The penultimate column is the actual space used if you have a 64K LFAU, as described above by Druck.
And, finally, just for the sake of it, how much that would be if somebody really did have an 8MB minimum.

So you might be thinking that StrongHelp is tiny, it’s barely over 64K. The Filer said so. Well, with an 8GB device, it’s actually consuming 260K (that 33 byte file has 20,480 bytes all to itself), while if your LFAU is, say, 32K or 64K, you’re possibly looking at seven to fourteen megabytes for that barely-over-64K of data.

Now rinse and repeat for every single application and remember there are little !Boot, !Run, and !Sprites(22) files in all of them.

Whatever happened to the old idea of sticking the entire application into a Sparkive?

Now that David Pilling has released SparkFS for free this might be a solution. Just as long as people aren’t still trying to write back to the application directory instead of saving into <Choices$Write> :)

Stuart Swales wrote…

Petition for the SharedCLibrary to dynamically set a buffer size to something appropriate for the platform in use!

That’s going to cause some applications to fail. And if you’re changing it based on platform, you’re making assumptions a) about the characteristics of the platform, b) that you can recognise the platform in the first place, and c) that the developer will be aware that the behaviour of their application’s memory usage will change on different platforms.

If the SCL changes the buffer size that’s in use, then the requirements of the application will be different and in some cases that may cause them to fail because the wimp slot size is too small for them. They might be able to increase it dynamically, if the increase is late on, but if the allocation happens early (say because of file redirection of input and output) you might find that the application fails fatally. It’s possible that this may not be a big deal, but it’s not an insignificant consideration – changing the behaviour of the SCL will affect such things. Additional caching would help, I agree, but it needs some thought to make sure that it doesn’t introduce more problems in the interests of purely performance.

I believe I added an extra emergency stack for handling ‘out of memory’ cases, in Select, and was quite concerned that the 8K increase (IIRC) was going to cause applications to fail. I tried everything I had and nothing failed… but it was still a concern, and so not something that I did lightly.

It would only be likely to cause grief after the application has taken over manually managing the application space (if it bothers to do so) rather than letting the SCL do it. Larger mallocs during init are going to extend the wimp slot if needed, surely. I was only suggesting doing this for SCLs built for specific platforms, not the platform-agnostic soft-load one.

[Edit: “say because of file redirection of input and output” – just had a look and it would appear that redirection already does explict setvbuf of 256 bytes.]

New RISC OS FAST website now available with:

More information
More progress updates
More product details
More products
More rockets (that’s how FAST it is!)

Brilliant website Andy. I’m drolling over those lovely looking beasts right now. I’ll have to save up my spending money though. Please can you explain the benefits of 4Kn /512e Disc Support to RO. I’m sure there are many but I seem to have missed that discussion on here. Guessing it’s for 2TB enterprise discs with 4K sectors emulating 512 for RO or something?

In a nutshell, access to filecore-formatted bigger (and faster) discs that are cheaper than the enterprise-class ones.

Larger, Cheaper and More (LCM) compatible drives available for RO. Lovely. LCM FAST all in an Acorn nutshell?

At present an application directory inside HD4::0.$.Apps contains lots of small files and the !Boot/!Run/!Help files also appear in Resources:Apps.

Perhaps it is time to move to an application being an image file invisibly containing its components but appearing to the file system like a single file that can be run. With Choices and Scrap external, then this would be a read-only image file. This avoids the overhead (on large discs) of storing all those small files.

It could have a file type defined as ‘application’ (type xxx) with the sprite expected to be file_xxx_appname (or perhaps just !appname) rather than file_xxx. The it would appear just like an existing app directory and SHIFT double-click could show its contents.

Or hope that any new FileCore has an option to support 4K allocation units like any other serious filing system these days.
The amount of wastage in FileCore format not only belies it’s floppy based heritage, but is getting embarrassing.

( https://heyrick.eu/blog/index.php?diary=20230311 )

While we’re talking about block sizes of 512 and 4096, I should mention that I have a Zen Stone 1GB media player that has a block size of 2048.

The amount of wastage in FileCore format not only belies it’s floppy based heritage, but is getting embarrassing.
https://heyrick.eu/blog/index.php?diary=20230311

It’s also embarrassing to write a blog about FileCore while getting the maximum idlen value wrong, and not make any mention of sharing (where FileCore will tuck sector sized files at the end of their containing directory to avoid having to splash a whole LFAU on a tiny file). It’s certainly true that the FileCore format’s creaking on larger discs though.

Rick mentioned DiscKnight’s verbose option to see the LFAU, but didn’t say about stats option which will tell you exactly how much space is wasted, both from rounding the size to nearest LFAU for larger files and from the map minimum allocation of LFAUs for smaller ones.

while getting the maximum idlen value wrong

Yup. Mea culpa. But in my defence, this very site does say “Because that fragment block is 2 bytes long, and must have a terminating 1 bit, idlen cannot be greater than 15”. ;)

The way I arrived at the value I did was to simply look at the disc record for a 8GB µSD and 16GB USB stick, both formatted FileCore.

With an idlen of 21 (instead of 19), doesn’t that pretty much quadruple the sizes involved?

and not make any mention of sharing

Exact quote: For simplicity we’ll ignore files sharing a fragment (as it only happens in specific circumstances) and fragmented files. The reality is much more complex than the diagram, but the basic priciples are correct.

So, it was mentioned but glossed over.

You said:

where FileCore will tuck sector sized files at the end of their containing directory

Is it files “up to a sector” or is it files “exactly” a sector?

but didn’t say about stats option

Thanks for pointing that out. I’ll have to give it a whirl on my media and see how much hurt there is.