Throughput on USB/SCSIFS

Bingo!

That worked.

But it is bittersweet. While the files in the source archive add up to <200MB, due to various wastages it requires a lot more space on disc. Indeed, 10,701 files (131,233,330 bytes) fills the 200MB RAMdisc.
So, in short, a RAMFS build on a 256MB Pi is not possible. I’m guessing maybe 350MB would be necessary?

If anybody would like to tweak their OS and try this on a Beagle/xM, let me know how you get on. I would do this myself, but I don’t have the OMAP3 sources, only the Pi ones (as the Pi, at least, gives me a display unless the boot fails).

Shame. I’d have loved to see how fast this would have gone.

Looking at the code, it looks like the decision was “arbitrary”, namely a trade-off between usefulness and claiming ooodles of space in the memory map (I think the DA is “allocated” even if it is currently unused and not mapped in).
I would imagine when that line of code was written, machines with more than that much memory were few and far between, so nobody considered the likelihood of building the OS in a RAMdisc? ;-)

; RAMDisc
        Push    "r0-r12"
        MOV     r0, #RAMDiscCMOS        ; find out how much RAM disc configured
        BL      GetConfiguredSize       ; in: r0 = CMOS address, out: r2 = size

        MOV     r1, #ChangeDyn_RamFS    ; Area number
        MOV     r3, #-1                 ; Base address dynamic
        ARM_read_ID r4
        AND     r4, r4, #&F000
        CMP     r4, #&A000
        MOVEQ   r4, #AP_RAMDisc_SA      ; Area flags, if StrongARM  (introduced for Ursula)
        MOVNE   r4, #AP_RAMDisc         ; Area flags
      [ {FALSE}
        MOV     r5, #16*1024*1024       ; Limit maximum size to 16MB while fiddling with FileCore
      |
        MOV     r5, #128*1024*1024      ; A trade off between nice big disc and complete waste of address space
      ]
        ADRL    r6, DynAreaHandler_RAMDisc ; Pointer to handler
        MOV     r7, #-1                 ; Use base address as workspace ptr
        ADRL    r8, AreaName_RAMDisc    ; Title string - node will have to be reallocated
                                        ; after module init, to internationalise it
        BL      DynArea_Create          ; ignore any error, we're stuffed if we get one!
        Pull    "r0-r12"

I’m building an OMAP4 ROM with the RAMdisc size limit increased to 768MB.
Just hope it builds…

The Pandaboard doesn’t like the ROM with the RAMdisc increased to 768MB, upon entering the desktop, you get a garbled display followed by a hard freeze :(

The worst part is, I now have to take the case apart to replace the ROM!

The Pandaboard doesn’t like the ROM with the RAMdisc increased to 768MB, upon entering the desktop, you get a garbled display followed by a hard freeze

Unfortunately any value apart from the default causes the same crash.

The worst part is, I now have to take the case apart to replace the ROM!

You could try changing to network booting the rom. Boot up is slower but if you want to change rom a lot it’s just a case of renaming a rom to boot the one required. I use a raspberry pi as the tftp server.

The Pandaboard doesn’t like the ROM with the RAMdisc increased to 768MB,

Try around 510MB. If you go over 512MB, you would need support for a “big disc” which would require numerous things to be altered in RAMFS – 64 bit calls, addresses as sectors not bytes, etc etc. Read PRM 5a-172 to get an idea.

If anybody would like to tweak their OS and try this on a Beagle/xM, let me know how you get on. I would do this myself, but I don’t have the OMAP3 sources, only the Pi ones (as the Pi, at least, gives me a display unless the boot fails).

Before I got the Pandaboard I used to use a custom rom with a 384MB ram disc on the Beagleboard Xm. It made building roms much faster but not as fast as a Pandaboard so I haven’t tried it in over a year. I assume it still works.

You could try changing to network booting https://www.riscosopen.org/forum/forums/11/topics/2322?page=2

….that’s this page?

Rob, try this: https://www.riscosopen.org/forum/forums/5/topics/1012

I would do this myself, but I don’t have the OMAP3 sources

[…]

Before I got the Pandaboard I used to use a custom rom with a 384MB ram disc on the Beagleboard Xm.

I’m trying to set up the build environment on the Beagle (the 1280×1024 mode completely failed – a good quarter of the screen was off the left-hand side! it seems to be happy(ish (as in I get a display every third boot)) with 1024×768).
Problem is, the USB stick I have as the main disc is really slow. UnTarBZ2 is managing to unpack about 1MB/minute! I think I might try unpacking this on the Pi instead…

Hehe…

The slow USB stick on the Beagle xM unpacks at around 1MB every 60 seconds.
The same device on the Pi does it around 45 seconds per megabyte.
The SD card in the Pi (a generic class 4) does the megabyte in 4-5 seconds.
[don’t have an SD filesystem on the Beagle so I can’t provide timings there]

Accordingly, I am unpacking to SD and will copy across all the stuff once it has been unpacked. [done 46MB in the time it has taken to write this ;-)]

I’ll have a go at getting network booting setup on the Pandaboard.
Gives me something other than DIY to do this afternoon!!

<keanu> whoaaaaaa! </keanu>

Rebuilt the OMAP3 image with big RAMdisc tweak on the Pi. Copied it over. Created a 384MB RAMdisc. Copied build enviro into it. Fired it off.

http://www.youtube.com/watch?v=0IXjyQ2nswA

It currently bombs out building SuperSample, perhaps a quirk because my board has no battery so thinks it is 1970? I’ll look some other time.
But otherwise… I reckon a full build (inc. clean) would take about 25 minutes. <gulp!>

It would be great if more RAM was avaliable for the OMAP3/4 RAM Discs by default. It is quite difficult to use 1GB of RAM (thankfully) – of course I could boot linux and start firefox ;-)

For writes, a USB thumb drive I get around 4MB/sec and on my SD Card I get around 12MB/sec.
[…]
Normally I wouldn’t care, but when I create a copy of a 32GB SD card it takes around an hour.

Just for what it is worth…

I bought myself a 500GB USB harddisc today. It isn’t really viable to burn stuff to DVD-R when we’re talking drives of these sizes. So what I am doing is taking all of the important animécontent from one drive, and all the important stuff from another drive, and making copies of it on the third drive. Which will sit peacefully on the shelf, only to spin up periodically to back up “important stuff”.

Just copied ~12GB and Windows’ (in)famously inaccurate estimation of time said it would take about 22 minutes. It took only marginally less.

I would like to say it is a little better estimating 57 minutes for 47GB, but this estimation changes as each episodefile is written, so I’m guessing an hour fifteen is more realistic.

Which means I’m perhaps getting something in the order of 750-800MB/minute. Which works out to be about 11-14MB/sec. From USB2 spinning rust, through a PC (1.6GHz Atom, not doing anything more stressful than running Firefox), to USB2 spinning rust. Both drives are, actually, USB3 capable (but the computer isn’t).

So, if your SD card’s MB is megaBYTEs and not megaBITs, I’d say – ain’t so bad. (^_^) What is written on the box doesn’t always translate in to what you experience in reality.

Thanks Rick, when I think about my general experience with USB – it sucks (as far as throughput goes).
But on the new RISC OS platforms all we have at the moment is SD and USB.
So I think the answer is 11-14MB/sec is what to expect for writes.
The more I think about it now, the more I think using a NAS device is the way to go.
If only I could get my RISCPC working on the network (I think the nic podule is knackered)

when I think about my general experience with USB – it sucks (as far as throughput goes).

I wonder how much of this is “implementation defined”.

If you have Windows XP (maybe similar for later versions?) and you go to Device Manager and then double-click on one of the “Host Controller”s in the USB subsection, on the Advanced tab it will say that bandwidth consuming devices are allocated 10%. Each USB controller has a fixed amount of bandwidth that all attached devices must share.
All of my USB controllers (there appear to be five?!) are allocated 10%.
Does this imply I could double my USB transfer speeds if I jiggled this to be 20% per controller? I’m not, actually, certain how. It looks like Windows does this automatically (and damn the user if they want to punt files around quickly!).

But on the new RISC OS platforms all we have at the moment is SD and USB.

If you ignore the built-in SSDs in my eeePC, that’s pretty much what I’m using on everything. That said, the USB drives keep a reasonable pace with the SSDs. Cutting edge this ain’t.

So I think the answer is 11-14MB/sec is what to expect for writes.

You ought to do better with spinning rust (power issues aside); however for flash media (any type) there will be limitations in the speed at which devices can erase and reprogram memory cells. It seems from my experience that the faster SD cards have a few megabytes of RAM cache so they can accept burst loads at high rates (perhaps to qualify for their “Class 10”?) however expecting them to sustain that for a two Gigabyte video file… nope. It tails off and starts to run at a more sedate speed.

The more I think about it now, the more I think using a NAS device is the way to go.

It might help for sharing, but I rather suspect you will just be swapping the smallish cache on a locally attached device for a slightly larger cache on a NAS (depending on how it is designed).

If only I could get my RISCPC working on the network (I think the nic podule is knackered)

No. The system FSB is a laggy 16MHz. I don’t know if the Network card runs at podule speed (8MHz) or slightly faster (12MHz? 16MHz?) as I don’t have a copy of the TRM handy – I suspect it is implemented like the Econet cards used to be – a special sort of podule.
The podule bus, running flat out synchronous, will only manage a shade over 6MB/sec. Wheeee!
The bog-standard 10baseT NICs chug along at a sedate 1.2MB/sec (full duplex; 600KB/sec if half duplex or a half-assed cheap switch). When I used to control the NAS-PC from the RiscPC via VNC, it was often an exercise in pain.
With a 100baseTX NIC instead, this will plateau at the 6MB/sec as that is what is possible from the IOMD podule interface.
This is physical throughtput; knock off about a third for TCP/IP overheads, to give you around 4MB/sec effective, if you are lucky.
100baseT runs at around 12.5MB/sec. Knocking off a third suggests it’ll manage approx. 8MB/sec if set up correctly.

Which implies three things:

1. Local USB/SDs are faster than that.
2. Using a LAN to a NAS will be slower.
3. Twice as slow if you do it from a RiscPC.

Note:

• All speeds quoted are K/M bytes, not bits.
• Podule bandwidth calculated by some horrible (and probably incorrect) maths working out logical data throughput from an 8MHz device.
• Network bandwidths calculated based upon my 2mbit ADSL link runs at 260KB/sec; so 10mbit will be five times faster, etc.
• The network must, by necessity, be slower than the media. If it is not, it will probably appear to be.
• DMA can help; God help you if you have a Kinetic.

My problem with the RISC PC is that for some reason the networking is just not stable.
It worked to some degree OK for a few hours last night, unless I “pushed” the networking.
I even tried limiting the switch port to 2MBit to see if that would help.
It was running at 10Mbit half duplex…which if you can ever call that working.
Then today it just refuses to do anything, the cable tester on the switch doesn’t even recognise it’s connected – which is amusing, but not suprising.
At the moment I’ve gone to the CD → CD transfer, that is painful!

Given the speed of a RISC PC is basically nothing, running via NAS or via local disc makes little difference from my point of view, which ever way you run it, it’s not going to be quick.
Relax and be patient seems to be the way with that beastie!

This is totally different, but I have managed to get a NAS setup transfering at 350MB/sec over two gigabit ethernet cards, when the raw speed was only 300MB/sec [NFS caching].
That did require quite a bit of tweaking, jumbo frames etc. but it was fun :-D

I might try and do some benchmarking with tcp/ip sockets from pandaboard to linux – just to see what kind of theoretical throughput we can get. I’d guess on a pandaboard 8MB/sec if it’s multitasking would be around right as you say.
My internet bandwidth has just gone up to a max of 200Mbit, which makes these calculations all the more amusing!

The reason I’m thinking of NAS is more to do with which ever way I cut it, I’m not going to get above 15MB/sec.
Going to say 8MB/sec on a NAS isn’t great, but the tradeoff of less speed over greater reliability and ease of use between systems seems worth it.

Very interesting info about the RiscPC though, looks like that is going to be an ongoing labour of love!
There is a 100Mbit card I can get for it, which may make my life easier connecting to switches.
It’s amazing how many switches/routers won’t even handle 10Mbit/half any more.
But do I really want to spend that much on a NIC for it, that’s a quite a few bags of nice coffee!

Generally I think I need to do a lot more playing.
Thanks again for the info, a bit to chew on there!

Generally I think I need to do a lot more playing.

Test the power supply voltages on load. Most of these network problems seem to come back to slightly out of spec PSU’s. It is an elderly beast you know…

Ah! Thanks Steve, that would explain the random working/not working/crash on module load variations.
I have 2 HD and 2 CDROMS connected, so that could explain it and of course I’m running the graphics at maximum res. (that’s how it came, well without the max. resolution)
Testing power supplies, lol, that reminds me of fixing a obsolete power supply with WD40, air blasting and patience to get the fans working etc. so I could get the data off the RAID discs. Although I forgot to leave it long enough to discharge and spent an hour or two walking around in pain!

Testing power supplies, lol, that reminds me of fixing a obsolete power supply with WD40, air blasting and patience to get the fans working etc. so I could get the data off the RAID discs.

Hmmm, Fujitsu drives¹ with a degrading controller (the encapsulating resin did nasty things to the chip) chilled overnight in the freezer compartment. Ran long enough to extract the 30-40MB of user data.

Although I forgot to leave it long enough to discharge and spent an hour or two walking around in pain!

Before I acquired a mirror from a medical rep, I reached round the colour tube of certain kit to tweak focus pots.
Expressing annoyance on the 40-50kV up the arm required an interesting selection of words and lots of tolerence or deafness from work colleagues.

Can’t think why pressing keyboard buttons to configure data switches these days appeals to me :)

¹ Hundreds of them, bad batch…

Wow! certainly one way to practise at your more colourful lingustic skills!

I ran some benchmarks and on my ArminiX I can get 11964.75 KB/s when not multi-tasking, and hammering the network.
So that’s quite nice, not sure what that will degrade to when multi-tasking is involved though.