Throughput on USB/SCSIFS

On my Pandaboard I’ve been playing around with clonedisc.
For writes, a USB thumb drive I get around 4MB/sec and on my SD Card I get around 12MB/sec.
The SD Card is a USB device so both are going through SCSIFS.

What are other peoples experiences with write speeds on the pandaboard/pi?
Normally I wouldn’t care, but when I create a copy of a 32GB SD card it takes around an hour.

I’m getting a little worried about storing my data on an SD Card, as they do tend to suddenly die without warning.
My aim would be to try and get better performance and then look at getting an SSD or small sata disc installed, and use that for !Scrap etc. and store my data on discs.
I still have NAS backup and that is backed up onto a BD disc and onto a server (which backs up as well)
So I’ve no end of backups, but it would be nice to have a more reliable desktop to start with.
As of yet I’ve not had an SD card fail on my panda, but I’ve experience with them in the past just dying with no warning.

Anyway back to the question about write speeds on USB on RISC OS…
Obviously the speed of the direct attached SDFS drive is of minor interest, but largely out of scope for me at the moment.
Does anyone else have any other figures etc.?

I’m getting a little worried about storing my data on an SD Card, as they do tend to suddenly die without warning.
My aim would be to try and get better performance and then look at getting an SSD or small sata disc installed, and use that for !Scrap etc. and store my data on discs.

Being slightly nervous about the SD sudden death and the flash memory sticks habit of self destruct¹ I opted for use of Memphis3 and using a RAM based swap. It avoids the Sudden Death and speeds things up.
A sudden loss of power does lose your current swap space content which has pros and cons.

Storage access speed? What we need is a magic wand to conjure up a port that works on a system with SATA.

¹ Swapping to and from another OS doesn’t help but they do it on their own even formatted filecore.

For writes, a USB thumb drive I get around 4MB/sec and on my SD Card I get around 12MB/sec.

Different devices have different speeds. When writing an image to SD card on my PC, it runs from 4MB/sec to about 7MB/sec (there aren’t cutting edge fast cards) so is likely to tie in with the timings you will experience. Luckily I’m not trying to copy 32GB, only 2…

Normally I wouldn’t care, but when I create a copy of a 32GB SD card it takes around an hour.

I plan, soon, to get a 1TB USB harddisc so I can copy the media on the 320GB and 500GB discs. I used to back up stuff to DVD-R but that isn’t really feasible with this sort of data (is there such a thing as a USB tape drive that can handle around 100GB a pop and not cost the earth?), so I’ll just get another harddisc and make a copy of it all.
I’ll let you know how long that takes.
I’m expecting to draggy-droppy and then just go to bed.
[and find it bombed out with some bizarre error five minutes later]

I’m getting a little worried about storing my data on an SD Card, as they do tend to suddenly die without warning.

I think Sprow and I would be able to have long argumentsdiscussions regarding the usefulness of a filing system that isn’t even capable of mounting a damaged device read-only for data recovery (if map #2 is good, use that! duh!).
It seems to me that FileCore is rather fragile, and “recent events” have made this more of a problem.
By recent events, I am referring to the fact that in the past I have only ever suffered “Broken directory” on a few occasions, and never a complete “sod off, your maps don’t make sense”. Why? Well, in the olden days when computers made sense and schematics were interesting to look at, it was generally the case that when you write something to the storage media, it gets written.
Quite unlike what actually happens today – moreso with wear levelling where even if you get clever with the idea of poking the flash chip directly, there is pretty much guaranteed to be zero correlation between memory offset and “disc address”. We are placing our trust in mass-produced parts with minimal profit overheads. And that’s not even discussing the cheap knock-off SD cards.
This is, perhaps, even further amplified by the fact that RISC OS does not seem to know how to properly dismount a modern storage device¹, so for all we know it will be performing “housekeeping” when the power goes off.

That said – mine “died” in between pressing F3 to save a file, with nothing else going on. In other words, it appeared to murder itself. Hmmm…

at getting an SSD or small sata disc installed

Isn’t an SSD just another type of flash drive?

I think I might tweak my boot setup to drop Scrap into RAMdisc, as per Steve’s suggestion. The main thing, I think, is to use a reasonable sized SD card (the 2GB works for me, 4GB would be better, any larger likely impractical) and just take an image of it at intervals (interval depending upon frequency of use). Oh, and in the interim, copy important files to a USB stick so there’s a copy on something else.

¹ When I dismount a USB device on the PC, it dismounts “properly”, in that the activity LED goes off and in the case of harddisc, it is parked and spun down. I have not connected a harddisc to RISC OS (it is NTFS) but I have connected the USB sticks, and the activity LED remains on when the device is dismounted in the filer. I wonder if there is some sort of command to tell these devices to cease internal activity as removal/poweroff is imminent?

An SSD is a flash drive, however SSD != SD.
The main difference is SSD drives have controllers which do a lot of other work on the I/O.
Specifically things like wear levelling etc. so you have much less chance of losing all your data.
When buying an SSD card/drive the quality of the controller is important.
Of course if we could get PCI on RISC OS/Arm we could get a FUSION I/O card ;-) 1GB/sec would suit me!

Filecore maybe a pretty poor filesystem, but there isn’t an alternative at the moment – using FAT is painful on RISC OS for me. I’m thinking through a new filesystem design in my head at the moment, not sure I have the skill or time to do it though.
What we need for starters is a ram disc sync on boot and shutdown for !Scrap, Memphis didn’t seem to be truely 32bit armv7 safe on a pandaboard for me :-(

The read/write speeds are damned awful though on RISC OS for me, at least a quater of what they should be.
My OTG converter should arrive today, so I can see if a dedicated USB port will help at all…I’m not hopeful though.

I have play around with CloneDisc and various board. The best copy speed I have with the GHz Pandora. 15MB/s writing speed from SDFS to SDFS (DD in Angström writes the same with 3MB/s!). The other Bords (xM, RPi, Panda (ES)) only writes with max. 12MB/s (SDFS to SCSI). Cards are class 10.
The OTG Port is not supported from RISC OS on Pandaboard.

Raik: Have you tried a Pi. Ben told me Pi’s SD hardware was about twice the speed of the OMAP’s!
He has also said that when using CloneDisc the fastest operation should be Copying from SCSIFS to SDFS.

Sorry, I have try both directions. I think the bottleneck is USB. Is also with the Pandora. Only SDFS read 17MB/s and write 15MB/s. USB read and write max. 12MB/s. I have try various Cardreader, HD (classic and SSD), Pendrive…

Bang on Raik!
Nice clickty click about pandaboard speeds.
Quite a depressing read… :-(
I was hoping I could blame something other than the board….i.e. something that could in theory be fixed.

Other OS same results. Sorry, I did not want to kill your hopes.
The results were the reason that I now use a 64GB SD. I later read that PandaBoard supports only up to 32GB ;-)

What we need for starters is a ram disc sync on boot and shutdown for !Scrap, Memphis didn’t seem to be truely 32bit armv7 safe on a pandaboard for me :-(

Can’t comment on panda, but it has given me no trouble at all on a beagle xM. Don’t try and enable the make Memphis imitate RamFS – that is a known crash.

A check with Armalyser suggests there are some unguarded non-32 bit elements. Some are gaurded code, some are not. The source is available so a recompile without the offending items is feasible.

Well after dismantling my ArminiX to plug the OTG connector, Raik, again was right (which I assumed)… No support for the OTG usb.
Is the lack of support of 64GB anything to do with the speed as well? lacking SDHC support?
How hard would it be to write say a predesk command to copy !Scrap from !Boot to RAM (or where-ever) and then a wimptask that copies it from RAM (or where-ever) back to !Boot on shutdown? (and perhaps sync it every x configurable minutes)

Am I missing something? If I’m not and it’s easy enough I can do that and release it ….
Suppose I can’t call it !BootRAMSyncUK ;-)

Oh! and “Can’t load base style sheet” in netsurf is annoying, no ntp configured and I reset my cmos…oops time was a wee bit out of date!

How hard would it be to write say a predesk command to copy !Scrap from !Boot to RAM (or where-ever) and then a wimptask that copies it from RAM (or where-ever) back to !Boot on shutdown? (and perhaps sync it every x configurable minutes)

Leaving out the specified sync to non-volatile storage that’s a good description of Memphis with the “Set Memphis as Scrap directory” and “Save/Load disc on Quit/Start” options set. Oh, plus a variable size RAM based disc.

Speaking of NetSurf – that’s faster when the cache is RAM rather than disc based where we are dealing with relatively slow “discs” like USB flash and SD. I suspect where we are dealing with SoC setups memory based cache will be faster than any non-volatile.

I have contact Willi Theiss the maintainer of the OMAP4 port. He has already invested a few hours, but gets the OTG support not to work. This is also the reason why we use the internal USB ports for the ARMiniX.
I use such a card for a month or two (since SystemDisc is aviable) without problems. Noticeably faster than a Scan Disc Ultra extreme… and Verbatim. All 64GB class 10.

Here you can find cmosd. Sorry is a German site.
Cmosd was for the a9home but is also usable for other systems (also RPi). This program monitors in the background the CMOS settings and saves every change to disk. At the start of cmosd the saved settings be loaded.

“How hard would it be to write say a predesk command to copy !Scrap from !Boot to RAM (or where-ever) and then a wimptask that copies it from RAM (or where-ever) back to !Boot on shutdown? (and perhaps sync it every x configurable minutes)”

Scrap is meant to be for temp files, aside from !Netsurf’s useage do you have any reason to copy it back to disc?
What I did on my Raspberry was to edit !Scrap’s !Boot and !Run files to run at RAM instead of it’s Obey$Dir variable. You then find out that RAM hasn’t been set up (from Predesk) yet so the RAM setup line from predesk needs to be moved to these files also.

Regarding !Netsurf, I keep a clone of it’s scrap files inside !Netsurf and move them to !Scrap if they aren’t already there.
From what I have seen this would only need updating if you added fonts, but maybe someone could point out where this could go wrong. This method happens in a blink compared with moving scrap contents in and out like another method I tried once.
Really !Scrap should be for temp stuff and permanent stuff should be in Resources. I think the reading from flash is not as bad as writing in both time and wear.

Regarding Memphis3, under heavy useage it does not perform as well as RAM, I found. I think it has been acknowledged a few times there are some issues.

Thanks Ronald for making me think a little ;-) I need it sometimes.
Yes, it was purely for netsurf – which does that awful font scan and stores the results in !Scrap (I assume)
I like your solution, it’s nice and simple – thanks very much!

@Raik – After some more tests it seems the “internal” usb used on the arminiX for the CD and external SD slot is slightly quicker than the rear one, or that’s my experience. Some good work went on with the ArminiX there!

I also have the feeling that the “internal USB” is a little faster and more compatible (I have any devices thats work in the front port but not in the rear!). It is possible that the built-in activ internal hub minimize some problems. We have several tried, not all are working good.

‘I like your solution, it’s nice and simple – thanks very much!’

Glad you found it works, I’m liking the idea of linking the RAM creation with Scrap more myself now, for example what if other things in the !Boot startup use !Scrap, then already it is going to be done in RAM instead of on the SD card.
It might also be helpful if you have to rescue a damaged SD filesystem, because it avoids any writing to the disc prior to running !DiscKnight or CheckMap.

A 256MB RamDisc would be good, what is the restriction of 128MB caused by?

is there such a thing as a USB tape drive that can handle around 100GB a pop and not cost the earth?

Tape? You really, really, really don’t want to go there. Believe me.

I was just looking at prices for DAT 160 cartridges. They are in the GBP25 region. You would be unlikely to get 80 GB on one. That makes removable USB 2.5" discs look very attractive in cost terms, and one 320 GB drive is even smaller than 4 DAT 160 cartridges. And the disc is random access…

If you want to go up market in removable storage, look at RDX. That’s again something that should “just work” with RISC OS, although I can’t afford a cartridge to try it (the company I work for would lend me an external drive, but I don’t want to prejudice one of their cartridges).

A 256MB RamDisc would be good, what is the restriction of 128MB caused by?

Good question. I’ve been wondering about that in the past few days. For years I’ve used a simple prog to copy !Scrap to ramdisc and use it there. Speeds things up and saved on HD noises. Now my PandaBoard has a GB of ram it seems odd not to be able to easily set the ramdisc to a few hundred MB. When working on new programs which have to write out files I tend to start always writing to ramdisc as it avoids my messing up the main HD, etc.

Like others, I’ve had to have !Scrap always keep a copy of NetSurf’s font list, though. Been hoping they might stop needing that as for me the name “Scrap” should be a hint that the contents are not meant to have to persist between sessions.

Jim

I’m on my phone. Had a quick look at RAMFS code – it makes frequent calls to XOS_ReadRAMFsLimits. As I’m on the phone, I’m not going to poke around the kernel code. Maybe this evening when I get home…?

Increasing the RAM disc size has been discussed before, albeit a while ago.

Personally, rather than changing RAMFS to use the application slot as suggested, I’d rather see support added for dynamic areas which access a small window of a larger physical block. So that way RAMFS could claim as much physical memory as it wants but (e.g.) only take up 1MB of logical address space. That facility would also be useful for some hardware drivers which need to allocate RAM for use by a device but don’t want/need it to be mapped in for CPU access.

Rick – whilst you’re there. Is reinitialising a RAM disc safe? I’m sure there /was/ a nasty bug involving creating a RAM disc, killing it off, then making a new RAM disc. However, memory is hazy on that one.

I give up.

This works on a 256MB Pi:

SYS "OS_DynamicArea",0,-1,(220*1024*1024),-1,0,-1,0,-1,"Test"+CHR$(0) TO ,area%,,base%

Go much beyond that, you run in to “Memory cannot be moved” problems (‘cos, you know, there’s no memory left!).

However, any attempts to set the size of the RAMdisc DA to >128MB results in “Memory cannot be moved”. The maximum size can be read as 128MB, but I have not been able to locate the bit of code that enforces this. I’ve looked at Switcher (no!), the ChangeDynamicArea code (no!), RAMFS (no!), etc etc.

Hmmm… I was going to push it up to something around the 200MB mark, copy the RISC OS build environment on to it (counts as 189MB), then try running a build and see if anything blows up along the way. But I’ve so far failed to find anywhere that says “RAMdisc can only be 128MB”.

..

Hmm…

SYS "OS_DynamicArea", 8, 0, 0, 0 TO ,a%,b%,c%
PRINT b%/1024/1024
         128
>

Looks like the OS’s idea of an area size clamp is 128MB. I wonder where this is being set, as the default appears to be the RAM limit, and 128MB isn’t that.
I’ve fudged the limits to be 200MB, but RAMFS is still stuck at 128MB max.

Also tried destroying the RAMdisc DA and recreating it (easy way to kill Switcher ;-) ) and….still stuck at 128MB. :-/

Ah-ha! I think I’ve found it.

Kernel.s.NewReset – line 1304.

I’ll bodge this to 200 and build a new image, see if this does anything different.

Hopefully this will work, I’ve often wondered if it was a simple change to increase the 128MB limit. (Just never got round to looking into it!)