Very slow transfers from NAS to Titanium

Over recent weeks I have been doing considerably more transfers between my NAS and Titanium, and have come to realise that transfers from the NAS to the Titanium are very slow. I have done a number of tests and comparisons with an ARMX6.

Hardware wise, the Titanium, ARMX6 and NAS are all connected to the same 8 port 1Gb/s switch. Cables are no longer than 2m. During these tests, I have used both ethernet ports on the Titanium, changed the cable between Titanium and switch, and shuffled the ports used on the switch. None of this had any noticeable effect.

During actual speed tests I used a range of files, from a single 112 MB file, to a directory containing some jpeg files totalling around 30 MB.

I summarise below some speeds, averaged over transfers. In general, as expected, a single large file transfers faster than several smaller files making up a similar total size.

Here are some data to demonstrate the anomalous behaviour.

Firstly ShareFS (as a sort of control). Transfers between Titanium and ARMX6 in either direction run at about 6 MByte/sec, which is reasonable for ShareFS. No problem.

Now the NAS transfers. Both LanManFS and LanMan98 were used. In the case of the ARMX6, transfers ran at very similar speeds. Download from NAS to ARMX6 ran at around 15 MByte/sec. No problem there.

Transfer from NAS to Titanium ran at around 1.4 to 2 MByte/sec. LanManFS tended to be slower than LanMan98, by up to 30%. Thus the Titanium is the best part of an order of magnitude slower than the ARMX6.

I cannot understand why there should be such a difference. I have tried to eliminate other software interfering, by running only !Omni and !Alarm on the Titanium. The ARMX6 was used in its ‘normal’ state.

The Titanium normally runs the latest nightly build (softloaded). The ARMX6 is currently on the latest OS from R-Comp (Sept 30th).

There is no anomalous behaviour uploading to the NAS. The ARMX6 manages around 10 MByte/sec and the Titanium does about 12 MByte/sec.

If anyone has any thoughts on why the Titanium is so slow transferring files from the NAS I should be very interested. If transferring a lot of large files, it is actually quicker to download the data to the ARMX6 and then shunt it over to the Titanium via ShareFS!

and have come to realise that transfers from the NAS to the Titanium are very slow.

I get the same problem. I don’t know what causes it. As a workaround you can restrict the network to 100mbit.
Gets about 11 MBytes/sec in both directions.

ECPVirtual 0 Off
ECPVirtual 1 Off
ECPLink 0 Auto
ECPLink 1 Auto
ECPAdvertise 0 100 Full
ECPAdvertise 1 100 Full
ECPFlowControl 0 Full
ECPFlowControl 1 Full

Also set the default ‘Next’ size to 640 KB as shown in ‘Tasks’. Its set in !Boot.Choices.Bookt.Desktop. ‘Wimpslot -next 640k’.

Thanks for that, Chris. At least it sets my mind at rest that I wasn’t missing something bleeding obvious!

Download from NAS to ARMX6 ran at around 15 MByte/sec. No problem there.

Roughly speaking that would be 150Mbit/s, which suggests a 1Gb link (150Mb/s on a 100Mb link doesn’t happen) and is in the 15-20% throughput range typical of a duplex mis-matched link¹ (it gets worse if there is any significant other traffic and the system has to do retries on the packet transmissions)

The advice to do this:
ECPAdvertise 0 100 Full
ECPAdvertise 1 100 Full

Would, I believe, tell the device to only advertise a maximum link speed of 100.
If it’s possible to do that without the “Full” it might be interesting to see what duplex the link negotiates and the associated transfer rate.

NB. If you’re pushing through a dumb hub the worst device dictates the operation speed/duplex so try the tests with just the send and receive devices connected.

Transfer from NAS to Titanium ran at around 1.4 to 2 MByte/sec. LanManFS tended to be slower than LanMan98, by up to 30%.

That’s suggestive of a network link back-off normally caused by packet collisions.
I’d be inclined to do some packet capture on the local net and check for collisions/retransmits

For the record a properly working 1Gb link should transfer at 950Mb/s quite happily so you would be looking at 95MB/s transfer – if the send and receive systems can internally handle that volume of data.

The other possibility is that one of the 8 lines involved in a 1Gb link is damaged and the system is flipping between an attempted 1Gb and failing down to 100Mb.

¹ Manually setting a device to full duplex and connecting to an “auto” duplex status port will give a mismatch when the auto port defaults down to half-duplex.

For what it is worth, I’ve also seen quite slow Ti network performance to a decent QNAP NAS (intel based) via Omni, just using one gigabit router/switch between. Noticed it the other day when transferring some test files back and forth, although it wasn’t horrible. Just slower than I was expecting. I didn’t bench it.

However, I know from the past that network performance to NASs can be very variable. When testing imx6 network driver, I could get up to about 40 MB/sec to an Intel QNAP NAS reliably, but to a similar ARM-based QNAP, performance was much more ragged. At its worst, as low as about 4 MB/sec. We optimised settings and things carefully, and this rose to about 20 MB/sec to ARM-based, and the Intel remained high. As a result, if you can afford it, I’d recommend an Intel-based NAS over an ARM one. Note that ARM-based NAS units are also restricted (by manufacturers) on RAM, usually, too, which doesn’t help them.

Just a quick update from me, as time limited: I also have a Titanium, running a very recent ROM, gigabit connected to a Synology NAS. Using LM98 I seem to get about 28MByte/sec for an 8MB save, and only 1.2MB for a 8MB load. Which seem an even larger difference than those Chris reported.
Note that loads always seem slower than saves to any FS, and I want to do some further checks & tests when I get chance.

I tried the config settings suggested by Chris (Gransden), but found found no real difference in behaviour.

This problem does seem specific to the Titanium. I have just fired up the IGEPv5 and done a couple of quick tests. That gave results similar to the ARMX6, i.e. download at about 15MByte/s and upload around 10 MByte/s.

One thing I did notice in previous tests on the Titanium – RISC OS 5.24 ran about 20-30% faster than OS 5.27, although still pretty pedestrian.

I tried the config settings suggested by Chris (Gransden), but found found no real difference in behaviour.

Scrap that – I have just tried again with those settings and the Titanium is now better. It now manages about 10 MByte/s download (which is about 60% the speed the ARMX6 and IGEPv5 manage). Upload it manages around 7MByte/s, not as good as the ARMX6, but very usable. I will keep those settings and see how it goes.

In answer to Andrew, the NAS is a Netgear ReadyNAS (supplied by him). I have no idea whether it is Intel or ARM based.

Depending on the age of the readynas, it could be Sparc, ARM or Intel, but most likely ARM. That would likely explain why none of the platforms are super-fast, but it is probably a red-herring as far as the discrepancies here are concerned. There’s certainly no reason why reads should be faster under 100Mbit rather than gigbit! I know we had to do a fair bit of tuning on ARMX6, though, because our initial results were quite variable (depending on device) as mentioned previously. I spent hours tuning various * commands that were implemented to control buffer sizes, to find what combination worked best for various tricky situations (eg. ARM or Intel NAS, ShareFS to VirtualAcorn etc).

However, we had the opposite problem originally with ARMX6 – it was really bad in 100Mbit mode, and good in gigabit mode. Colin G was very helpful in testing and helping track down the bottlenecks there. I think the bottom line is that high speed networking on RISC OS can be extremely sensitive, and you have to be very careful to control buffering etc to avoid excessive packet retries and so on which can bring speeds right down.

There’s certainly no reason why reads should be faster under 100Mbit rather than gigbit!

Physically/electronically – see my earlier comment regarding damage in one of the extra pairs used for Gb – it happens all over¹ Equally, there could be an iffy connection and then…

there’s the issue of buffering, as you mention, where too little capability means the receive end loses stuff and the TCP stack (not re-written yet) slowly reacts and asks for a re-send (while other stuff has also come and got lost so that’s another resend).
The “soft” element carries all the way through to the disc i/o and the data being stored (possibly slower than expected) – again, with nowhere to go and small buffers there is going to be a re-send or 100.

An interesting test would be to check the transfer to/from a RAM disc to eliminate any slowness from the physical disc.

¹ I had extended discussion argument with the support at Worcester when 2 of their 4 LINACs were running slow and their checks showed 100Mb links. It took 3 months to convince them to change the cables that “have never been moved, and we can’t see any real damage” (note the word “real” implying there was some visible damage but they didn’t consider it important).
Change of cables on two units and hey presto fully working Gb links and fast comms.
Explanation on that mystic remote insight is that I’ve seen it before where damage causes the speed negotiation to flip between 1Gb and 100Mb each time failing a renegotiate for 1Gb and dropping back to 100Mb. While the reneg fails are happening dropped packets abound.

and we can’t see any real damage

Yeah, NO.

That’s like saying the eggs in the box are okay because the carton doesn’t look broken…

see my earlier comment regarding damage in one of the extra pairs used for Gb

I assumed that replacing the cable would rule out cable faults.

An interesting test would be to check the transfer to/from a RAM disc to eliminate any slowness from the physical disc.

I actually did most of my tests using the RAM disc. After trying also to the SSD, I found that the speed of RAM disc and SSD were essentially the same, i.e. the rate determining step was not writing to the local drive.

The RISC OS RAM disc isn’t as fast as other RAM based filing systems such as Memphis. Something to do with the memory being set to cacheable non bufferable (or maybe its non cacheable bufferable).

I assumed that replacing the cable would rule out cable faults.

It does, but there’s also the possibility of a dry joint on the board or in the hub/switch you’re going through.

There’s also the possibility that the Titanium NIC driver isn’t as robust an implementation of the auto-detect as it could be.
As an example of how things can be wrong anywhere, for many years Cisco trained (droids) network staff were convinced auto-detect was bad simply because one model of Cisco switch made a total pigs ear of it.

The RISC OS RAM disc isn’t as fast as other RAM based filing systems such as Memphis.

Useful info. Testing with standard drive, RO RAM disc and Memphis should give an idea of any bottleneck there as opposed to network i/o.
Been a while since I used Memphis – I can’t recall whether the official release was made 32-bit. There’s a few different items on the beagleboard (buried in the junk room) that I ‘modified’

Here some test results in MBytes/sec from my Titanium, recent ROM, using HDSpeed v1.30 with 8MB files run 10 times, saved & loaded as a block, for various FS/discs:

FS        Disc            Save       Load
ADFS      SSD              123        116
ADFS      Backup            37         47
SDFS      TiSystem          13         22
LanMan98  DSriscos          34          1
RAM       RamDisc0         500        170
MemFS     Memphis          889       1600
SCSI      SSD over USB      26         29

LanMan and LanMan98 were used to my NAS, and both have appallingly slow Load speeds.
The other odd thing is the two ADFS discs – both SATA SSDs that I thought were the same!
I also thought that RAMFS had caught up with MemFS after changes, but obviously not the case!

Do others find similar speeds?

ADFS SSD 123 116
ADFS Backup 37 47

The other odd thing is the two ADFS discs – both SATA SSDs that I thought were the same!

The difference may be down to formatting, different LFAUs. DiscKnight will show the current values. A big LFAU is for speed, a small LFAU is for storage efficiency, LFAU wastage is also revealed by DiscKnight.

Are the drives of different sizes?

LanMan98 DSriscos 34 1

Throttling the Titanium to 100Mb/s does get the same speed in both directions.

Some drives have RAM caches and their speed depends on how full they are.

SDFS      TiSystem          13         22    <—

RAM       RamDisc0         500        170

MemFS     Memphis          889       1600    <—

One thing seems pretty clear: Transfers to/from a RAM ‘storage’ is significantly faster so the disk i/o is not a good implementation.
It’s probably a buffering issue.

I wrote this:
Transfer from disk to/from memory should be better than the memory to remote network device so, possibly, a routine to transfer data from disk to memory in 8MB blocks and then transfer memory to remote device plus similar routine for the reverse path will give a different transfer rate – that should pin things down.

…and then I decided to check something:
I just checked a trail-of-crumbs from the RO Roadmap where I was looking for disk or filing system items and following a "background transfer link I ended up at this 2010 discussion involving Jeffrey.
In the thread buffering is mentioned as being disabled and the whole thing seems to date back to the late Archie/early RPC era.

The link taking you to that thread references SCSIFS but the thread actually suggests that ADFS is also affected.

Jeffrey refers to some of the work as a five-minute job which reminds us that things labelled as a five-minute-job always take much longer, especially for normal humans rather than Jeffrey :)

I also thought that RAMFS had caught up with MemFS after changes, but obviously not the case!

The headline speedtest results from Memphis are impressive, but there is a bit of a but ….

The test of choice here is a ROM build. Compared to using the Titanium’s RAMFS Memphis’s unpacking of the source code bz2 and ROM build both took at least five times longer.

Added Memphis 3.03 (28 Apr 2005). Is there anything later?

Memphis 3.03 (28 Apr 2005). Is there anything later?

Not that I know of – 3.03 (28 Apr 2005) with a Help file that says 3.02 no indication of what the 3.02 —> 3.03 changes are.

ADFS SSD 123 116
ADFS Backup 37 47

I now suspect the main difference is because SSD is Crucial, and Backup is Hitachi, although I have not checked specs carefully. Also according to DiscKnight Backup has about twice the LFAU Rounding and Map Wastage than SSD. I cannot see what the LFAU is, but looks like a factor 2 difference.

Anyway, this is irrelevant to the discussion, and I want to concentrate on the network speed.

Regarding the various *Configure ECP… settings for EtherCPSW:
- are they effective immediately, or only when re-plugged?
- are they remembered, or do they have to be done each boot?

set the default ‘Next’ size to 640 KB as shown in ‘Tasks’.

Why does that make a difference?

I was using Memphis 3.03 in my tests above.

Regarding the various *Configure ECP… settings for EtherCPSW:
- are they effective immediately, or only when re-plugged?
- are they remembered, or do they have to be done each boot?

Use *rmreinit EtherCPSW to effect changes, which do persist, *status will confirm that as will Interfaces Status in Network configuration.

Anyway, this is irrelevant to the discussion, and I want to concentrate on the network speed.

I’d say it was pertinent to the discussion¹, although your immediate need for faster transfer can probably be addressed by adjustment of the network settings to something that requires less assistance from buffering and hence generates fewer packet resends.

¹ However, sometimes we need to deal with the needs of the few (or one) and the needs of the many must wait.

The Backup is a HDD. It is generally a good idea to mix disc types / manufacturers so as not to stack all your eggs in one basket. For large drives, HDD is also cheaper, and therefore better suited to backup roles, than SSD. One could also comment that the “trim” situation (or lack thereof on RISC OS) makes HDD more suitable for backup type jobs than SSD, as there will be quite a lot of “remove old data and replace with new backups” going on.

The Backup is a HDD

How silly of me! Of course it is – I was confusing it with a Pi I was running for some time with two SSDs! Thanks Andrew.