Benchmarks

I tried running the IGEPV5 at 1.5GHz but RISC OS hangs on boot up. There’s no display support at the moment so everything is run from the supervisor prompt via serial port.

Do you have a ROM for IGEPv5, Chris? Are you working on the HAL?

Benchmarks showing Raspberry Pi model B and B+ with Iyonix and Pandaboard ES.

In the table below, the benchmarks are expressed in percentages where 100%=SA202 RiscPC running RISC OS 4.02
These benchmarks have been updated to show the improvement in the latest ROM which contains EtherUSB 0.29. This has significantly improved upload speeds to the network using LanMan on the Pi (but such transfer speeds on the Panda have dropped, which is being investigated) and the problem with reduction in speed on Pandaboard has now been cured.

Thanks for posting these results, Chris.

Why did “Draw Fill” slow down so much between R-Pi RC6 and RC12a? Is there an explanation?

but such transfer speeds on the Panda have dropped, which is being investigated

I think you will find that Colin has greatly improved things in his latest test ROM. He did provide a link for anyone who wished to try it.

RISC OS is now running at 1.5GHz on IGEPv5.

-                                        IGEPV5 1.5GHz    IGEPV5 1GHz  Pandaboard ES 1.5GHz

-                                        cortex-a15       cortex-a15   cortex-a9
povray -w1024 -h768 +a0.3 teapot.pov     9s               12s          12s         time for povray 3.6.1 using hardware vfp

shine test.wav test.mp3                  9s               14s          19s         time to encode 3min track, fixed point mp3 encoder

dhrystones per second                    4650000          3120000      4350000

First two RISCOSMark benchmarks:

Processor – Looped instructions (cache)  2106%            1415%        1310%

Memory – Multiple register transfer      21487%           14382%       7990%

I think you will find that Colin has greatly improved things in his latest test ROM. He did provide a link for anyone who wished to try it.

table of results updated

So do you know if the HD write speed to Lanman 98 on the PandaBoard ES has actually improved with my latest version? I note that your last table shows a write speed of 4.5 and this one 6.5 but I see that you have upclocked your panda. Judging from the read speeds it looks like I’ve improved things.

So do you know if the HD write speed to Lanman 98 on the PandaBoard ES has actually improved with my latest version?

It went from 0.3 to 6.5 when I used your test rom. So Yes. It is using smartreflex to switch between 350MHz and 1500MHz and has done for some time, using Chrs Grandsen’s boot stuff.

Thats not what I mean’t. You said that the latest roms broke writes via lanman98 on the panda so presumably a previous version was faster. An earlier table in this thread shows 4.5MB/s but that was for a 1200Mhz panda according to the column title. I was wondering if you knew what the speed was with the 1500Mhz panda and the old working rom. I’m not asking you to do a test I just wondered if you had the data available.

With which ROM did the change occur? I have a few ROMs going back to the end of May. If that is before the change, I could do a quick test.

With all things identical, except the ROM version:
PandaLand OSupg6 21-Apr-2014 1350% 7124% 5.0 6.6 0.26
Daily ROM 16 Jul 2014 1314% 6943% 5.2 6.7 0.26
PandaLand OSupg7beta 28-Sep-2014 1306% 6929% 5.2 0.3 0.29
Test ROM 24-Oct-2014 1286% 6879% 5.1 6.2 0.29(+)
Daily ROM 1 Nov 2014 1339% 7162% 5.3 0.3 0.29

Hence at 21-Apr-2014 and 16 Jul 2014 with EtherUSB 0.26 speeds were ‘normal’ 5.0Mb/s read and 6.6 Mb/s write to LANMan98/NAS.

As at 28-Sep-2014 with the upgrade to EtherUSB 0.29 write speed had dropped.

The test rom posted by Colin (24 Oct 2014) still with EtherUSB 0.29 (+ but fixed to solve transfer speed bug), write speed was fixed.

Processor (1300%) and memory (7000%) speeds are approx the same (1.5GHz) throughout.

Benchmarks showing Pandaboard ES, Beagleboard XM, Raspberry Pi models B and B+,Iyonix and Virtual Risc PC.

In the table below, the benchmarks are expressed in percentages where 100%=SA202 RiscPC running RISC OS 4.02
These benchmarks have been updated to show the improvement in the latest ROM which contains EtherUSB 0.29, with upload speed on Pandaboard fixed. This has significantly improved upload speeds to the network using LanMan on the Pi. They also now include a test of RAMFS speed (using an 8Mbyte file for HD read and HD write to make sure it is larger than the cache).

The ‘mixed memory and processor’ test is to calculate 24 million possible arrangements of the 2, 3, 5, 7, 50 and 75 tiles in Countdown using all possible combinations of arithmetic operator (+ – x /).

Benchmarks for Pandaboard ES running at 1500MHz have been withdrawn due to the unstability of that platform at that speed.

The RAMFS test (which is a much more ‘real world RISC OS benchmark’ than pure memory copy benchmarks run from assembler – the test is based on repeated loads of an 8Mbyte file using OS_file 16 so the file size should be much bigger than the cache as there is a 1Mbyte L2 cache on the Panda) does not give the expected results:

Risc PC has a 16MHz memory bus and so RAM should be bounded by 64Mbyte/s and achieves 20Mbyte/s. Consistent.

Iyonix has a 200MHz 64 bit DDR memory bus and so bounded at 3200Mbyte/s – achieves 15Mbyte/s. Low (other tests have shown 165Mbyte/s but I can’t get anywhere near this).

Beagle XM has a 166MHz 32bit LPDDR memory bus. Bounds at 1333Mbyte.s. Achieves 36Mbyte/s. Low (other tests have shown 480Mbyte/s).

Panda ES has a 400MHz 32 bit dual channel LPDDR2 memory bus, hence bounded at 6400Mbyte/s. Achieves 275Mbyte/s. Consistent (other tests have shown 760Mbyte/s).

OMAP5 has a 532MHz 32 bit dual channel LPDDR2 memory bus, hence bounded at 8500Mbyte/s.

iMX6 has a 528MHz DDR3 clock, hence bounded at 12600Mbyte/s (other tests have shown 200Mbyte/s to 500Mbyte/s depending on memory location).

Does anyone have an explanation please?

For the Risc PC the processor speed (202MHz) is much faster than the memory bus (16MHz) and so you would expect the memory speed to govern. For the others the processor speed is only moderately faster than the memory bus and so you would expect speeds a bit lower than the memory bus would permit.

But for Iyonix and Beagle XM speeds are much lower than this.

Where I refer to ‘other tests’ these are memory copy benchmarks found ‘on the web’ usually based on Linux or assembler [which do relate consistently to expectations from actual hardware].

4 things could be interesting:
- RPCEmu + ROS 5.20
- Iyonix + ROS 5.20
- RISC PC + ROS 5.20
- Pi + safe overclock: 900 – 250 (GPU) – 250 (Core) – 400 (Mem).

“- Pi + safe overclock: 900 – 250 (GPU) – 250 (Core) – 400 (Mem).”:

Pi @ 900/333/450 [GPU/Core/Mem]
Graphics Resoloution: 1920×1080, 16M colours

Test Benchmark
Processor – Looped instructions (cache) 643259 361%
Memory – Multiple register transfer 1765 1089%
Rectangle Copy – Graphics acceleration test 1408 581%
Icon Plotting – 16 colour sprite with mask 16142 807%
Draw Path – Stroke narrow line 9075 581%
Draw Fill – Plot filled shape 1516 103%
HD Read – Block load 1MB file 18070 605%
HD Write – Block save 1MB file 5437 178%
FS Read – Byte stream file in 528 255%
FS Write – Byte stream file out 184 95%

@ 1000/500/500, otherwise as above:

Graphics Resoloution: 1920×1080, 16M colours

Test Benchmark
Processor – Looped instructions (cache) 694401 390%
Memory – Multiple register transfer 2890 1783%
Rectangle Copy – Graphics acceleration test 1702 703%
Icon Plotting – 16 colour sprite with mask 17743 887%
Draw Path – Stroke narrow line 10258 657%
Draw Fill – Plot filled shape 1653 113%
HD Read – Block load 1MB file 17723 594%
HD Write – Block save 1MB file 5796 190%
FS Read – Byte stream file in 476 229%
FS Write – Byte stream file out 143 74%

My model B Pi has CPU/GPU heatsinks and separate power supply, running the 13-Sept-14 Isochronous ROM in all tests. ‘HD’ is a 32GB Class 10 SD card. Over-volting is required for both the overclocking options above, at #2 and #6 respectively. I currently run at 800/300/400 which does not require over-volting: stats below. This seems to be a ‘sweet spot’ in Pi performance, being measurably better than the default and not much inferior to the 900Mhz option.

Graphics Resoloution: 1920×1080, 16M colours

Test Benchmark
Processor – Looped instructions (cache) 554443 311%
Memory – Multiple register transfer 1819 1122%
Rectangle Copy – Graphics acceleration test 1223 505%
Icon Plotting – 16 colour sprite with mask 13845 692%
Draw Path – Stroke narrow line 7767 497%
Draw Fill – Plot filled shape 1294 88%
HD Read – Block load 1MB file 16221 543%
HD Write – Block save 1MB file 4612 151%
FS Read – Byte stream file in 400 193%
FS Write – Byte stream file out 163 84%

I was doing some tests the other day to get a feel for how fast USB should be and noticed surprising ramdisc results. The tests just use the copy command to copy a 17MB file to and from the same disc and to and from the ramdisc. On the Iyonix it was quicker to copy from ADFS to ramdisc than from ramdisc to ramdisc. I expected the ramdisc to show the ultimate performance of the filing system but apparently it doesn’t.

*copy 17 MB file

Raspberry Pi
============

SD CARD (class  6)
sdfs - sdfs              3.44 MB/s
sdfs - ramdisc          18.19 MB/s
ramdisc to sdfs          4.67 MB/s
ramdisc to ramdisc      55.74 MB/s

rebuilding Tungstendev (make/install/join rom, everything already compiled) 106 secs

USB HDD
scsi - scsi              4.80 MB/s
scsi - ramdisc          12.00 MB/s
ramdisc - scsi           8.86 MB/s

USB SSD
scsi - scsi              6.91`MB/s
scsi - ramdisc          13.50 MB/s
ramdisc - scsi          11.29 MB/s

rebuilding Tungstendev (make/install/join rom, everything already compiled) 143 secs

Iyonix
======

PATA HDD
adfs - adfs              8.03 MB/s
adfs - ramdisc          14.05 MB/s
ramdisc - adfs           7.00 MB/s
ramdisc to ramdisc      10.34 MB/s

rebuilding Tungstendev  (make/install/join rom, everything already compiled) 113 secs

USB HDD
scsi - scsi              2.33 MB/s
scsi - ramdisc           4.32 MB/s
ramdisc - scsi           3.74 MB/s

USB SSD
scsi - scsi              2.50 MB/s
scsi - ramdisc           4.36 MB/s
ramdisc - scsi           3.74 MB/s

rebuilding Tungstendev  (make/install/join rom, everything already compiled) 214 secs

The big thing to bear in mind with RAMFS is that the dynamic area is uncacheable¹. I believe the reasoning is to avoid polluting the cache with data which won’t be used that often (once you’ve read/written something in RAMFS, what’s the chances of wanting to immediately read it back again?). Maybe it’s worth doing some tests with cacheable vs. uncacheable – e.g. see if it has any affect on something like ROM compilation time.

Also chances are that the memcpy code that RAMFS is using isn’t optimal!

¹ The exception being StrongARM, which has it cacheable in order to avoid considerably slower access times due to a CPU bug

On the Iyonix it was quicker to copy from ADFS to ramdisc than from ramdisc to ramdisc

Perhaps not surprising: to/from disc uses DMA and PATA with a 133MHz PCI bus whereas to/from RAMFS uses a clearly non-optimised memcpy code. Also the memory addresses are likely to be well separated (which I think can slow things down).

@Jeffrey
Is the RAMFS dynamic area on the Pandaboard uncacheable as well? I think that what might be happening is that a file load command fetches a sector full of data, copies over a word, then copies over another word etc. If the memory is cached then the word fetches and/or sector fetches will be from the cache otherwise they will not. Another possibility is that RAMFS fetches each word from uncached memory as it is required.

If the dynamic areas on the Pandaboard are not cached, then it may be that the L2 cache (1Mbyte) on the Panda is large enough to give this speed. I think this would repay investigation? Iyonix and Beagle are woefully slow in using RAMfs, particularly in loading a 1Mbyte file, barely reaching the performance of the previous century’s Risc PC StrongARM.

Another interesting snippet is that with the cache switched off ramfs to ramfs is faster than adfs to ramfs on the iyonix. Also any ideas as to why for ramfs to ramfs transfers the pi is 5.5 times faster than an Iyonix – presumably they are using the same code.

Is the RAMFS dynamic area on the Pandaboard uncacheable as well?

Yes.

I think that what might be happening is that a file load command fetches a sector full of data, copies over a word, then copies over another word etc. If the memory is cached then the word fetches and/or sector fetches will be from the cache otherwise they will not.

That’s a good point… it will be going via FIleCore so will probably end up reading/writing entire sectors at a time even if it only needed a few bytes. Of course that should only really affect the end of a file, rather than accesses to the middle (FileCore/FileSwitch should hopefully be caching stuff, so once the full sector is read it shouldn’t be needed again in order to fetch the next few bytes)

If the dynamic areas on the Pandaboard are not cached, then it may be that the L2 cache (1Mbyte) on the Panda is large enough to give this speed. I think this would repay investigation?

I may be wrong, but I don’t think we have the ability to mark pages as L2 cacheable only.

Also any ideas as to why for ramfs to ramfs transfers the pi is 5.5 times faster than an Iyonix – presumably they are using the same code.

The Pi has a system-level L2 cache which is outside of the ARM MMU’s control. So even though we’ve marked the pages as uncacheable they’ll still be being fetched via that system-level cache, giving a nice speed boost compared to true uncacheable accesses.

Is the RAMFS dynamic area on the Pandaboard uncacheable as well?

Yes.

I still don’t understand why RAMFS file loads on the Iyonix and Beagleboard are uncharacteristically slow whereas those on Pandaboard are at the expected high speed. Why is Panda not slow? If each time a few bytes are required, it fetches a whole sector this could slow things down but file systems surely don’t work like this? Unless RAMFS is (in some way) particularly inefficient…

The Pi has a system-level L2 cache which is outside of the ARM MMU’s control.

I do understand why RiscPC (dynamic memory is cached) and Pi (see above) are fast.

I still don’t understand why RAMFS file loads on the Iyonix and Beagleboard are uncharacteristically slow whereas those on Pandaboard are at the expected high speed. Why is Panda not slow?

I’d say it’s the other way around. RAMFS file loads on the Iyonix and Beagleboard are at the expected slow speed, while those on the Panda are uncharacteristically fast.

As for the reasoning – out of order execution, perhaps? And maybe improvements to the load/store unit as well – the memory will be marked as bufferable, so won’t be considered strongly-ordered device memory (i.e. hardware registers where each and every read/write must occur exactly as specified). This gives the CPU a bit of freedom in how it accesses the memory. Normally (i.e. on older architectures) this just means it uses the write buffer to merge any writes to consecutive addresses into as few memory accesses as possible. But on the A9 it might be merging loads as well – transforming a series of consecutive load instructions from consecutive addresses into one memory access, avoiding a lot of wasted CPU time and memory bandwidth.

I find this unconvincing (ducks). On the Iyonix with 200MHz 64 bit DDR memory, a read speed of 14Mbyte/s via RAMFS file loads seems slow, especially where the RiscPC with 16MHz memory bus speed achieves 20Mbyte/s.

RAMFS file loads seems slow

I suspect part of what you are seeing is the general brokenness (in performance terms) of memory operations on the Iyonix. This has been mentioned many times in the past, e.g. in posts such as this one.