Automated testing and CI

However, there’s advantages in testing on fake systems

One of the biggest advantages that I can think of is when delving deep and dirty. Which, sadly on RISC OS, is all too common given how much stuff runs in a privileged mode.

I have a piece of software, a module, that fakes itself as a type of mouse pointer. This fails with a different, third party, piece of software when using the Adjust button behaviour. Now I believe my code to be correct because the behaviour works in other contexts (such as the Filer), however it’s too easy to say “not me, it’s the other guy”. If I’m to pin the blame on somebody else’s code, it’s rather useful to be damn certain that it isn’t me that cocked up. I have the sources for my code, and I have the sources for a slightly older version of the other program. What I don’t have is the inclination to want to do anything about it.
Why?
Because all of this low level stuff is running in either SVC or IRQ mode. So when it blows up, the result is the machine stiffs.
It’s pretty bloody difficult to get meaningful results out of a corpse.

This is where some sort of emulation and/or Pyromaniac could be extremely useful – as the underlying system will still be working so things can be poked and prodded and even if RISC OS has keeled over dead, it may well be possible to rummage around in memory (at the host level) in order to try to determine how things got to that state.
Sadly, it could be something as stupidly simple as one too few or one too many words pulled from the supervisor stack. But you’ll never know running it on real silicon as all you’ll see is a machine suddenly ceasing to respond.

No, it’s very much not.

I think you need to re-read what I wrote:

Whether it answers all the questions you may have is a different issue. It’s still a valid test, but perhaps not a test that answers your specific question.

My point that I expanded on is that you need to know what questions can be answered by the test and what questions cannot. You may need other tests to answer your specific question.

First you need to know the list(s) of possible questions and then what tests will answer them or guide you to new questions. Fault-finding is usually a journey through the layers of an onion, or perhaps a 3D snakes and ladders?

As you say:

However, there’s advantages in testing on fake systems

and:

This is where some sort of emulation and/or Pyromaniac could be extremely useful – as the underlying system will still be working so things can be poked and prodded and even if RISC OS has keeled over dead, it may well be possible to rummage around in memory (at the host level) in order to try to determine how things got to that state.

At this point it might be useful if Pyromaniac offered a simple to use method of dumping all that info to a file for the user to peruse at a later point – possibly in a form that a client util could display in a summary/verbose form?

Andy S take on RPCemu – with it producing output -so you can localize the error in RO.

I think it works with modules as well as Apps.

I think you need to re-read what I wrote:

Ditto. ;-)

Because the main question that I was aiming at was not “how to test” but “by which measure do you trust your test tools”? It is normal that physical devices may require some form of calibration. It is also normal to try to measure the wrong thing with a device, in much the same way that you can put a nail into a piece of wood using a rock.

Also it’s useful to remember that in many cases (physical and software) the act of taking measurements affects that which is being measured.

As to knowing what to test, that’s where the years of experience comes in, and is an entirely different subject. ;-)

or perhaps a 3D snakes and ladders?

Sometimes it feels more like three dimensional chess played in the dark when all you have are pawns and the opponent has nothing but queens.