Very slow USB loads/save on Pi4 with Filecore format USB discs

I haven’t been able to spend time testing this across multiple platforms, but I encountered this whilst testing Iris on Pi4.

On SDFS, RISC OS format, with a pretty generic SD card, it took 8 seconds to load (give or take).

Curious to see how it fared on a USB drive (theoretically higher performance) and USB-connected-m.2-SSD, I copied it over to a couple of discs.

The results were appalling. 30ish second load time. The SSD was slightly better (!), at about 25 seconds. This was formatted with !HForm

I tried both on the XHCI ports and the USB C.

I then ran RISCOSmark which showed very slow byte stream in/out results – 158%/46% vs 568%/103% (read/write) on SDFS. This would fit, I guess, with the Iris performance.

On a whim, I reformatted FAT32FS. Iris loaded much more quickly – faster even than SDFS. Same drive, same USB connection – 1693%/1352%

I know FAT32FS caches writes, so I have to be a little careful with that, but assuming a reboot in between, caching can’t help reads a great deal. A 10x speed increase seems, erm, considerable.

I can’t help feeling this warrants investigation? (I’m rushed this week, so can’t spend a lot of time on it unfortunately)

caching can’t help reads a great deal

Yes it does. Fat32fs actually reads ahead. I.e. it reads more sectors from the device than actually asked for. This is because there is a massive USB transfer overhead per read/write.
I think that the USB has been speeded up since I first wrote fat32fs, but I found that the time taken to read (say) 2 sectors was pretty much the same as reading 1. So I made it read many sectors ahead as most times the OS then asked for those sectors next.

Jeff, is there any chance of incorporation into the main ROM Image?

It’s obviously streets ahead of DOSFS, and, with a few tweaks with regard to capitalisation (just me being fussy!), be a very useful addition to the pantheon of RISC OS!

Jeff, is there any chance of incorporation

No. The library it is based upon is GPL 2. Which means, well, you know what that means.

It’s obviously streets ahead of DOSFS

Indeed.

I made it read many sectors ahead as most times the OS then asked for those sectors next.

Back in the days of DOS and floppy discs, I had a TSR that would do the same thing, caching in high memory. Because it was often quicker to read X sectors from the disc when it was spinning and wait for the program/OS to request them, than it was to keep on spinning up the disc each time.

The library it is based upon is GPL 2

Microsoft made exfat available to linux over a year ago, there is mention of GPLv2 there as well.
Does this mean you have to be an OIN partner to use it?
https://openinventionnetwork.com/joining-oin/join-now/

I am no licensing expert, but I would think, a GPL 2 based program cannot go into a ROM image, but it could go into HardDisc4, or not?

“mere aggregation of another work not based on the Program with the Program (or with a work based on the Program) on a volume of a storage or distribution medium does not bring the other work under the scope of this License”

I think it could go into a rom so long as the module which contained it was not statically linked to the rest of the rom. But there is contoversy here as RISC OS does not fit the assumed model. It can definitely go into ResourceFS or the HardDisc image. But there has been unresolved discussion on this topic before.

It can definitely go into ResourceFS or the HardDisc image. But there has been unresolved discussion on this topic before.

Avoid the issue – put it as a standalone item on the Miscellaneous page like SparkFS.

With the open source status Jeff can still update the code if required as can others.
Oh, and the requirement of the source being available to all is covered too – so that’s bit less storage for Jeff…

Very slow USB loads/save on Pi4 with Filecore format USB discs

Not all USB drives are created equal. I have a collection of SSD and mSATA USB discs (all formatted with !HForm (mostly 27-Nov-2012)

The speed range is quite staggering (over 3 to 1). And the USB interface board can have a quite dramatic effect as well.

All comparisons done using 5.24 with Ras Pi 2.

Given the poor stream performance shown by RISCOSmark, I’d suspect Iris’s buffers are too small for non-readahead FSs. Try setvbuf(f, NULL, _IOFBF, 64*1024) after opening files.

Given the poor stream performance shown by RISCOSmark, I’d suspect Iris’s buffers are too small for non-readahead FSs. Try setvbuf(f, NULL, _IOFBF, 64*1024) after opening files.

This was also slowing down loading Shared Libraries. I noticed this fix had been made to the SOManager module the other day. Now Otter Browser loads in half the time on a Pi 4.

Try setvbuf(f, NULL, _IOFBF, 64*1024) after opening files.

Now Otter Browser loads in half the time on a Pi 4.

And on the Titanium. That was a very good spot.

Glad that my remaining grey cell is still functioning at 75% or so…

Is there a download available for the updated version of SOManager?