Very slow transfers from NAS to Titanium

Transfer from disk to/from memory should be better than the memory to remote network device so, possibly, a routine to transfer data from disk to memory in 8MB blocks and then transfer memory to remote device plus similar routine for the reverse path will give a different transfer rate – that should pin things down.

I have been working on GNUtar for the autobuilder. You can make tar volumes of a selected size and run a script or cmd at end of each volume.

tar Mcf RAMFS::Ramdisc0.$.foo -L8M -Fobey/cmd     [dir or file objects to tar (in the CSD)]

Here’s a screen shot from Wireshark showing a typical packet capture when transferring a file using wget on a titanium. The average download speed was about 2.5MB/s.

The same test using a Rpi4 averaged about 56MB/s and there were no ‘Dup Acks’.

Duplicate Ack count gets to 7 in that one little snippet, also TCP Window Full.

The latter is particularly symptomatic of the receive buffer being too small or that the network buffer can’t offload the data to the receiving application fast enough.
Add this to the transfer stats given earlier where all is well when using RAM disk/Memphis and the finger is currently pointing at disk I/O, unless the Titanium NIC is flapping on negotiated speed.

Slowing the network link down avoids the issue with the disk, but it also stops any 1Gb/s <—> 100Mb/s flap on the link.

The same test using a Rpi4 averaged about 56MB/s and there were no ‘Dup Acks’.

Compare the disk I/O of the Titanium and the RPi4 if they are similar then the Titanium problem may also be a NIC driver issue.

The slow downloads are not due to disk i/o as all the testing I’ve been doing has been from RAM disc.

The slow downloads are not due to disk i/o as all the testing I’ve been doing has been from RAM disc.

Indeed – I found no difference between SSD and RAMdisc. The bottleneck is elsewhere. Normal disc reads and writes are orders of magnitude faster than these download speeds.

Try taking the ‘disk’ I/O out of the equation by just running a *Load LargeFile in a timed loop.

Try taking the ‘disk’ I/O out of the equation

All my tests have been using HDSpeed v1.30, which simply uses OS_File,10 or 16 to Save to or Load from memory. Only the disc being tested has any i/o.

All my tests have been using HDSpeed v1.30, which simply uses OS_File,10 or 16 to Save to or Load from memory. Only the disc being tested has any i/o.

Ah, that is truly awful performance with LanMan98 then!

Anyone tried NFS performance?

The slow downloads are not due to disk i/o as all the testing I’ve been doing has been from RAM disc.

Ah, so…
“if they are similar then the Titanium problem may also be a NIC driver issue.”

If everyone is seeing the same poor throughput on Titanium which is ameliorated by advertising 100Mb/s as the limit then it’s likely the NIC driver / NIC buffering.

Do any of the Titanium users have a fully managed 1Gb switch? One where you can see the negotiated speed and duplex and it has a degree of logging of the status.

I’d like to know whether the Titanium into said switch properly negotiates a 1Gb full duplex response on the switch (and vice versa). Until the stats showing way higher throughput with RAM based source/destination I had in mind a possible duplex negotiation failure which will usually result in a throughput of 20% (or less) of the link speed.
Slow LanMan transfer falls into the “problem with the receiving application” territory.

Possibilities for multiple problems here. Try and pin down one at a time.

Some more test results from me, including RO5.24 and NFS…

With recent RO5.27...
Setting    100 Mb/s      1000 Mb/s
           Save  Load    Save  Load
LanMan       8    10      27     1
LanMan98     8    10      34     1
NFS          1     1       1     V slow - Timed out!
With RO5.24 speeds are essentially the same...
LanMan       5    10      26     1
LanMan98     8    10      33     1

After the tests I ran last night, my ECP status (where # was 0 and 1) was:

ECPVirtual # Off
ECPLink # Auto
ECPAdvertise # 100 Full
ECPFlowControl # Full

but I had difficulty reverting to GB settings.
I used
*Con. ECPAdvertise # 1000
on both interfaces, and turned the machine and router off overnight.

This morning, no network access. The Status panel showed a rubbish (default?) IP address, no gateway, and a cross for OK and Full Duplex.

To get network access, I had to revert to Advertise 100 Full and re-boot.
What am I missing to revert to GB?

*Con. ECPAdvertise # 1000

The same thing happens here if 1000 is advertised.
I noticed when the machine is rebooting the link light on the switch goes off just after the beep but never comes back on. If I pull the network cable and re-insert before the dhcp request fails then it boots up normally. Still with the slow speed.
If I use port ‘1’ the average download speed is about 200KB/s.
Both ports work at full speed if capped at 100mbits.

*Con. ECPAdvertise # 1000

With 1000 advertised my Titanium does reboot to that setting, and networks. The network is manually configured.

Having booted into 1000Mb/s reads from a USB pen in the router or a RPi4 Raspberry Pi OS share both show the slow reads. Oddly a share on a RPi3B+ Raspberry Pi OS share reads and writes at 17MB/s and 24MB/s respectively.

Before I made any changes I took a copy of the existing config settings. I haven’t actually tried resetting the Titanium back to its original settings yet.

ECPVirtual     0 Off
ECPVirtual     1 Off
ECPLink        0 Auto
ECPLink        1 Auto
ECPAdvertise   0 1000
ECPAdvertise   1 10 Full Half 100 Full Half 1000
ECPFlowControl 0 None
ECPFlowControl 1 None

I think I was using port 0 at the time.

Don’t know if this is related but I noticed the my internet transfer speed where extremely slow at around 2kb a sec when I tried to download a beta rom on titanium so I switched to Pi 4 and the download speed speed was around 50kb a second.
On a 42 mbs 4g connection

Looks like flow control problems to me. Does

showstat -v

show any mbuf exhaustion or rx errors?

I did do some a few data gathering things. I have an output from a few days ago that includes the following (hope it is not too long to post (at least none of the lines are overly long).

*inetstat -s

ip:
        150673 total packets received
        0 bad header checksums
        0 with size smaller than minimum
        0 with data size < data length
        0 with header length < data size
        0 with data length < header length
        0 with bad options
        0 with incorrect version number
        0 fragments received
        0 fragments dropped (dup or out of space)
        0 fragments dropped after timeout
        0 packets reassembled ok
        150119 packets for this host
        10 packets for unknown/unsupported protocol
        0 packets forwarded
        0 packets not forwardable
        544 packets received for unknown multicast group
        0 redirects sent
        93698 packets sent from this host
        0 packets sent with fabricated ip header
        0 output packets dropped due to no bufs, etc.
        2 output packets discarded due to no route
        0 output datagrams fragmented
        0 fragments created
        0 datagrams that can't be fragmented
icmp:
        24 calls to icmp_error
        0 errors not generated 'cuz old message was icmp
        Output histogram:
                destination unreachable: 24
        0 messages with bad code fields
        0 messages < minimum length
        0 bad checksums
        0 messages with bad length
        0 message responses generated
        ICMP address mask responses are disabled
igmp:
        10 messages received
        0 messages received with too few bytes
        0 messages received with bad checksum
        10 membership queries received
        0 membership queries received with invalid field(s)
        0 membership reports received
        0 membership reports received with invalid field(s)
        0 membership reports received for groups to which we belong
        0 membership reports sent
tcp:
        93563 packets sent
                8459 data packets (506060 bytes)
                1 data packet (55 bytes) retransmitted
                0 resends initiated by MTU discovery
                51442 ack-only packets (226 delayed)
                0 URG only packets
                0 window probe packets
                33593 window update packets
                68 control packets
        149814 packets received
                8526 acks (for 506090 bytes)
                138 duplicate acks
                0 acks for unsent data
                98527 packets (138239541 bytes) received in-sequence
                129 completely duplicate packets (186260 bytes)
                0 old duplicate packets
                0 packets with some dup. data (0 bytes duped)
                51043 out-of-order packets (72821755 bytes)
                0 packets (0 bytes) of data after window
                0 window probes
                0 window update packets
                0 packets received after close
                0 discarded for bad checksums
                0 discarded for bad header offset fields
                0 discarded because packet too short
        38 connection requests
        0 connection accepts
        0 bad connection attempts
        0 listen queue overflows
        38 connections established (including accepts)
        30 connections closed (including 0 drops)
                0 connections updated cached RTT on close
                0 connections updated cached RTT variance on close
                0 connections updated cached ssthresh on close
        0 embryonic connections dropped
        8498 segments updated rtt (of 8497 attempts)
        0 retransmit timeouts
                0 connections dropped by rexmit timeout
        0 persist timeouts
                0 connections dropped by persist timeout
        0 keepalive timeouts
                0 keepalive probes sent
                0 connections dropped by keepalive
        3 correct ACK header predictions
        90137 correct data packet header predictions
udp:
        305 datagrams received
        0 with incomplete header
        0 with bad data length field
        0 with bad checksum
        26 dropped due to no socket
        24 broadcast/multicast datagrams dropped due to no socket
        0 dropped due to full socket buffers
        255 delivered
        111 datagrams output

*showstat

DCI4 Statistics Display 0.03 (30-Sep-19)
Copyright (C) Element 14 Ltd. 1999.  All rights reserved.

Interface name        : ecp
Unit number           : 1
Hardware address      : 70:b3:d5:03:f0:4b
Location              : Motherboard
Driver module         : EtherCP
Supported features    : Multicast reception is supported
                      : Promiscuous reception is supported
                      : Interface can receive erroneous packets
                      : Interface has a hardware address
                      : Driver can alter interface's hardware address
                      : Driver supplies standard statistics
MTU                   : 1500
Interface type        : 10baseT
Link status           : Interface faulty
Active status         : Interface is active
Receive mode          : Direct
Interface mode        : Half-duplex
Polarity              : Correct
TX frames             : 0
RX frames             : 0

Interface name        : ecp
Unit number           : 0
Hardware address      : 70:b3:d5:03:f0:4a
Location              : Motherboard
Driver module         : EtherCP
Supported features    : Multicast reception is supported
                      : Promiscuous reception is supported
                      : Interface can receive erroneous packets
                      : Interface has a hardware address
                      : Driver can alter interface's hardware address
                      : Driver supplies standard statistics
MTU                   : 1500
Interface type        : 1000baseT
Link status           : Interface OK
Active status         : Interface is active
Receive mode          : Direct, broadcast and multicast
Interface mode        : Full duplex
Polarity              : Correct
TX frames             : 93731
TX bytes              : 5379913
RX unwanted frames    : 9910
RX frames             : 160908
RX bytes              : 232997893

Module MbufManager is an mbuf manager

Mbuf Manager          : System wide memory buffer (mbuf) memory management
Active sessions       : 3
Sessions opened       : 6
Sessions closed       : 3
Memory pool size      : 360448
Small block size      : 128
Large block size      : 1536
Mbuf exhaustions      : 0
Small mbufs in use    : 16
Small mbufs free      : 496
Large mbufs in use    : 0
Large mbufs free      : 192

*ecpinfo

EtherCPSW interface statistics

ecp0: CPSW, motherboard, up

Interface driver    : ecp
Interface unit      : 0
Interface location  : Motherboard
Interface EUI48     : 70:B3:D5:03:F0:4A
Interface controller: CPSW
Interface media     : 1000baseT full duplex
Running time        : 0d, 0h 20m 19s
Packets sent        : 93736
Packets received    : 160913
Bytes sent          : 5380245
Bytes received      : 232998373
Send errors         : 0
Receive errors      : 0
Undelivered packets : 9910

Don’t know if this is related but I noticed the my internet transfer speed where extremely slow at around 2kb a sec when I tried to download a beta rom on titanium so

The RISC OS network stack doesn’t like high latency networks. Downloads can be a lot slower than the maximum bandwidth available. A work around is to use a proxy server.

Hi Chris can you recommend a good proxy server?

There is a difference between the EtherTH and EtherCPSW in how they handle received packets. In EtherTH line 688 I left the packet in the ring buffer if an mbuf can’t be allocated. The theory being that his allows the chips flow control to work as the ring buffer can fill.

EtherCPSW line 1336 discards a packet if the mbuf can’t be allocated so the ringbuffer doesn’t fill. I suspect the original handling of the received packet in netbsd multitasked.

You will also note in etherth I added a small delay to improve wireless performance – it probably avoids returning from the interrupt too often.

I don’t have a titanium or fast nas to test but it may be worth experimenting in that direction.

Edit: Except you are not getting rx errors. hmmm….

I suspect the original handling of the received packet in netbsd multitasked.

Almost certainly, especially since quite a number of the systems using netbsd would probably have a NIC with a good capability processor that could handle the i/o on behalf of the main system. As I recall the offload to that NIC processor (Windows PC’s) used to be a user visible option. These days I think it’s just done by default.

Perhaps, as I’ve mentioned before, time sensitive tasks like this should be handled by another core if available.

Hi Chris can you recommend a good proxy server?

I use apache on Raspberry Pi OS. You can also install pi-hole to block Ads to speed things up a bit.

As an example using wget. Without a proxy downloads max out at about 700KB/s. With a proxy 2.2MB/s. 2.2MB/s is the fastest my broadband goes.

ECPFlowControl 0 None
ECPFlowControl 1 None

Setting ECPFlowControl to None fixed the problem with the link not establishing with ECPAdvertise set to 1000.

Setting ECPFlowControl to None

I had spent some time getting connected this morning, because if set to GB at boot it failed – I had to set to 100, re-boot, let that connect, then change to GB. Changing cables, sockets, interfaces, etc had no permanent effect. The first thing I saw online was the above advice – and it seems to have fixed my boot at GB setting! Many thanks, Colin. Presumably that was the setting before I started changing speeds!

Setting ECPFlowControl to None fixed the problem with the link not establishing with ECPAdvertise set to 1000

There’s an important distinction about gigabit that doesn’t apply to 10/100Mbps – a gigabit link can only be autonegotiated, not forced. Both link partners’ PHYs exchange various little blips (seen on a scope) on the line which are outside the packets that make it to the network driver, also training the line to trim out echos.

By constraining the link (not advertising some capabilities) if they can’t agree a mutual set of settings the result is no link at all. The flow control setting is one example of that – it’s not “more is better” – both ends must be capable and agree this, and there seems to be a great deal of variability in what different PHYs can handle, hence the configure setting to allow None/Generate/Respond/Full. Only set it to ‘Full’ if the manual for the switch says that is supported.