Benchmarks

Benchmarks updated for speed switching on Raspberry Pi.

Here’s the results of a fairly old Javascript benchmark. The more modern ones take too long or use too much memory to run on all RISC OS machines.
Higher scores are better.

Otter Browser 0.9.10-dev-20160429 (JIT enabled)

V8 Benchmark Suite.

               BB xM        PB ES       IGEPv5    Titanium     ARMx6    RPi Zero       RPi2       RPi3   i7-4770k
              1.0GHz       1.5GHz       1.5GHz      1.5GHz    1.0GHz      1.0GHz     0.9GHz     1.2GHz     4.3GHz

Richards:       1218         2220         3707        3640      1560         438        833       1394      26427
DeltaBlue:       316         1481         2883        3028       879         234        516       1144      23994
Crypto:          865         2219         4143        4030      1499         667        943       1617      30824
RayTrace:        337         1473         2738        2886      1029         270        845       2088      37517
EarlyBoyer:      535         2519         4631        5066      1719         529       1162       2475      40852
RegExp:         59.2          272          711         667       194        82.0        136        327       6345
Splay:           108          650         1121        1198       432         187        373        729      21334
NavierStokes:    647         2351         4114        3855      1595         425        964       1585      26511

Score:           354         1353         2559        2586       914         299        611       1224      23929

And with JIT?

All the benchmarks are with the JIT enabled.

The iMX6 score seems about half that of the IGEPv5/Titianium (Otter and qupzilla crash after the earlyboyer test)

The iMX6 times seem about half that of the IGEPv5/Titianium (Otter and qupzilla crash after the earlyboyer test)

Do you mean times or score?
Times suggests faster than IGEPv5/Titianium. Score slower!

I would say that, in general real world usage, the IGEPv5 feels much faster than the iMX6.

Do you mean times or score?

The Benchmark Suite site explains the results.

“… higher scores means better performance: Bigger is better!”

While working out which way round the results are I inadvertently measured Safari on a 3.4GHz iMac.

Richards:     32294
DeltaBlue:    42282
Crypto:       31757
RayTrace:     72001
EarleyBoyer:  56254
RegExp:        4342
Splay:        20051
NavierStokes: 30596

Score:        28759

P.S. & OT & OTT Firefox on the same iMac was even faster – Score: 32082

Total score of 10073 under Safari on a 1.4 GHz ARMv8 (iPhone 6). It’d be interesting to see how much performance could be eked out of a RISC OS machine with unlimited time and money!

Otter and qupzilla crash after the earlyboyer test

Same here on a Titanium. Obligingly both show the same debug info.

[alloc:304:/linux4/gccsdkqt5vfp/build/Qt5Base/qtbase-opensource-src-5.5.0/src/corelib/tools/qdynamicarea.cpp]
 - RISC OS Error: Memory cannot be moved
   handle (306), size (126976), da_inc (126976), m_size (134213632)

Fatal signal received: Segmentation fault

Edit. The issue is fixed, diagnostic data truncated.

It’s to do with the Garbage Collector Dynamic area being limited to 128MiB. I’ve upped it it to 320MiB in the updated Webkit libraries available here. Just copy the Resources folder into !Boot.
Next edit the !Boot.Choices.Boot.PreDesk.DALimit file and change ‘128’ to ‘256’ and run it once changed.

It’s to do with the Garbage Collector Dynamic area being limited to 128MiB.

Many thanks, that worked.

On the Titanium the overall score for Otter is 2627 and for QupZilla is 2602.

@Chris, sorry that was very misleading.
What I mean’t to say is the scores are about half that on the mX6.
i.e. as you said the titanium and IGEPv5 are faster (about twice from what I saw)

It’d be interesting to see how much performance could be eked out of a RISC OS machine with unlimited time and money!

Here’s the result running under Linux on the Titanium board. Same version of Otter browser.

Richards:     3922
DeltaBlue:    3121
Crypto:       4956
RayTrace:     4689
EarleyBoyer:  6242
RegExp:        952
Splay:        2440
NavierStokes: 4084

Score:        3375

The browser itself is much more responsive under Linux. Being able to use 2 CPUs helps.

Thanks Chris, the GC/DA changes have fixed it!

MX6 scores are: (Qupzilla)

Richards: 1560
DeltaBlue: 879
Crypto: 1499
RayTrace: 1029
EarleyBoyer: 1719
RegExp: 194
Splay: 432
Navierstokes: 1595

Score: 914

Here’s the result running under Linux on the Titanium board. Same version of Otter browser.

So Otter runs 30% faster under Linux: I’m surprised at such a modest improvement. It rather suggests that the benefit of multi-threading is considerably less than pro rata. Having downloaded and installed the latest Otter, I thought for a brief and heady moment that Dropbox was now accessible* – it /was/, but now seems not to want to play nicely: I can log in and see my home page, but can’t open any of the folders. In general though, increased responsiveness and functionality, so all good. Thanks, Chris!
[*IMHO, ability to access cloud storage will mean that RISC OS can be considered a genuinely useable standalone system – at present I require a WinPC interface via LanMan to achieve this].

The browser itself is much more responsive under Linux. Being able to use 2 CPUs helps.

Off topic. So there is dual core support on Titanium (under Linux).
And dual screen too?

It rather suggests that the benefit of multi-threading is considerably less than pro rata.

I wonder how much pre-emptive multitasking slows down things even when no other main applications are running.

The browser itself is much more responsive under Linux. Being able to use 2 CPUs helps.

It rather suggests that the benefit of multi-threading is considerably less than pro rata.

Hmm. Do these Javascript tests actually trigger much threading? Except for delegating file downloads to other threads I suspect much of the tests run on a single thread.

I wonder how much pre-emptive multitasking slows down things even when no other main applications are running.

I guess you could test that by building the same benchmark, with the same version of GCC, for both RISC OS and Linux and comparing the results (RISC OS running from the command line, Linux running from a GUI terminal app? Or exit the GUI and run from command line? Even though there’s no GUI, the Linux command line will still be using PMT)

An interesting test for RISC OS that I’ve been meaning to write for some time is something for measuring IRQ latency. It’s pretty simple, really – just configure a timer to fire a period interrupt, and then log the timer value in (a) an IRQ handler, (b) an RTSupport thread (which was woken from by the IRQ handler), © a callback (which was registered by the IRQ handler), and (d) a Wimp task (woken by the IRQ handler setting its pollword?). When the timer fires the interrupt it resets to a known value, so by comparing the current value of the timer against the reload value it’s trivial to work out the latency at each ‘thread priority’ within the OS. A variant of the system could also be to have the timer trigger a FIQ (and have the FIQ downgrade to IRQ, triggering the rest of the steps), with the FIQ handler capturing a stack trace – so that if you find a situation where there’s high IRQ latency you can find out what the cause is by looking at the captured stack traces.

Hmm. Do these Javascript tests actually trigger much threading? Except for delegating file downloads to other threads I suspect much of the tests run on a single thread.

There’s an easy way of testing that: Open two instances of the browser and run tests in both at the same time. Or open the resource monitor and look at the CPU usage graphs :)

Nothing to do with Javascript, but the gcc ported network stack (from what I’ve seen) really sucks.
That seems to be across all gcc based apps.

There’s an easy way of testing that: Open two instances of the browser and run tests in both at the same time. Or open the resource monitor and look at the CPU usage graphs :)

I tried with Firefox on this PC, pretty much running at 25%, i.e. on a single of the 4 CPUs. Unless the Otter browser relies far more on multiple threads internally, my guess is that the 30% speed increase on Linux is not directly related to the second CPU.
I wonder if it is possible to deactivate CPUs on Linux, that would be the best way to confirm it.

Unless the Otter browser relies far more on multiple threads internally, my guess is that the 30% speed increase on Linux is not directly related to the second CPU.

Quite possible. They’re all benchmarks for the javascript core – no graphics or IO – so the performance would rely on the javascript engine (same in both cases), the compiler (RISC OS using GCC, Linux might be clang?), compiler settings (RISC OS ARMv6 with VFP2, Linux ARMv7 with VFP3/4 and NEON?), support libraries (heap manager and threading likely to be the main two), and any OS overheads (I’d expect task switching to be a lot faster under Linux, because RISC OS is outdated and likes to flush things from the caches)

So there is dual core support on Titanium (under Linux).
And dual screen too?

Yes, and yes. The benchmark data posted earlier noted there were two CPUs in the system.

I wonder if it is possible to deactivate CPUs on Linux, that would be the best way to confirm it.

It is – add “nosmp” to the boot arguments and it’ll only use 1 core.

Benchmarks updated for Virtual RPC Adjust SA running on an i5 with an M2.SSD drive. Most benchmarks up by 25% to 50%.