Testing on RISC OS

Back in May 2021, I gave a presentation to ROUGOL (https://presentations.riscos.online/testing/index.html) and wrote an article for Iconbar (https://www.iconbar.com/Testing_on_RISC_OS/news1688.html), about software testing. It is a subject that is very important to me. I’ve written a lot of software over the years and have worked with companies producing a lot of software. And one thing that holds true is that the amount of testing that you do is never sufficient to eliminate bugs – and that failing to test guarantees that there will be bugs.

When I say testing here, I’m largely talking about automated testing. It is good to manually check that your software works properly by hand – manual, ad-hoc testing – but it’s fundamentally a recurrent cost that you will have to pay every time you make a change. This is in contrast to automated testing, where you write tests that exercise your work, and then you can re-apply quickly after every modification. Automated testing pays off by making it possible for you to concentrate on making changes, with the confidence that when (not if) you break things, you’ll know about it.

The RISC OS Pyromaniac project – now an entire operating system – was originally (and remains, albeit changed in scope) a system to allow testing of RISC OS code. It is the largest project I have created, and it is woefully under-tested. That said, it has ~2800 tests which cover about 70% of the code – making it also one of the best tested pieces of software that I’ve ever written. This matters to me. Producing an operating system is not like writing a small application, module or tool – it has to work, and you must have confidence that it works. Not that you shouldn’t care for applications or modules – because obviously you should – but if you cannot rely on the OS, then you may as well go home.

The RISC OS build service was created to allow some automated testing within the cloud. Ok, so it was a bit of a silly thing at first, but it exists now for this purpose. Many examples exist (https://github.com/topics/riscos-ci?o=desc&s=updated) which use the RISC OS build service to build their code, from simple modules like LineEditor through to full applications like Nettle and filing systems like LanManFS.

I created a repository of tests (https://github.com/gerph/riscos-tests), derived from the RISC OS Pyromaniac tests, to allow others to start using them as the basis for testing the OS. Or at least encouragement to be able to do so.

So, that’s why gave a presentation and wrote an article… I don’t think that RISC OS – the operating system, or the components that run on it – is well tested. Or for that matter that it is considered important enough to happen. I had hoped that, given the multiple strands of operating system development, and people working to improve the OS and writing new applications, these many seeds laid around would produce some fruit.

I don’t know if that’s the case… so tell me… did it have any effect at all? Even if you didn’t know any of the things that I’ve just mentioned, have you been producing tests with your software anyhow?

I rarely automate testing, basically because it means a lot more work. Yes, I’m lazy.

I do however code (when using C) in a modular style and I frequently embed conditional calls to DADebug to aid in debugging.
An example of this was mentioned just the other day, in that my Manga app went through several different types of fetcher (and at one point had both as options) but the calling functions had no idea. They just said “fetch this”.

Another example is the crappy Arthur desktop clone that I threw together in a few days. Because of the abstraction between the window drawing code and the actual code that drew stuff on the screen, I made RISC OS and MS-DOS versions at the same time.

If everything was lumped together, it would have been a mess. But by grouping related functionality in separate code modules, it becomes possible to replace one or more to change how the entire program works without affecting other code, so long as the implied contact of what the function accepts, does, and returns is maintained.

I probably should do more automatic testing, but then I probably should learn a better language than C and move to a platform where I can do things far more quickly… ;)

There’s nothing wrong, and everything right, with what you have been suggesting. The problem is one of inertia. Having done things “in this way” for some forty years, are people going to want to change when the OS itself fundamentally hasn’t ever improved anything in this respect? I mentioned DADebug above. This is just spitting data to a circular buffer for reading what happened along the way. It’s only one step above spewing bytes to the VDU. The debugger is lousy, it’s far too easy to kill either it or the entire system, and it seems most of the people serious about programming use something else and cross compile back to RISC OS.

Yes, ironically these system flaws make testing more necessary, but testing just wasn’t a big thing for those of use brought up on BASIC. Either it worked, or it didn’t. And worse, thanks to a commercial need to support multiple types of computer, the sorts of code that would be published in books was dreadful. I recently looked at the Osborne book about making your own adventure game. I have (or had?) the original but you can now download PDFs for nostalgia. Anyway, since other machines didn’t support procedural coding like BBC BASIC, the program is an awful pile of linguine with GOTO and GOSUB in plentiful use.
This isn’t to excuse bad code. I started like that and improved myself mostly through learning the hard way of coming back to a program after a time away and having to rewrite it all as the messy code and lack of comments made it damn near impossible to figure out what was going on. So I comment, use clear variable names (no specific style other than defines in capitals and variables in lower case, I don’t include the type in the name), and group related functions in modules (also helps with having module level variables to avoid needing lots of globals).
I guess it’s just a shame that testing was not a part of this, but I guess when the 2023 DDE can be bested in that respect by the 1987 TurboC (with a rather nice integrated debugger)…. :/

Keep on saying this. If even one serious coder (I don’t consider myself “serious”) adapts a better testing regime, then you’ve made a positive change. God knows far too much software is really poor to the point of being substandard, due in part to the laziness that things like continual integration bring. There’s no longer a company-ending problem if the software has bugs, nothing is burnt into ROMs these days. If there’s a problem, just patch it and issue an update…. that’s the way it goes nowadays.

A more improved testing (or in some cases any testing!) might help prevent obvious brokenness ever being released in the first place.

Oh, and maybe one day the various lawmakers will get around to preventing software companies that sell a product from using a long and complicated EULA in order to excuse any and all liability, while at the same time failing to provide even basic QA on the product being shipped. Just ask anyone who’s had to deal with a Patch Tuesday erasing the ability of their machine to talk to their printer….again. Or when their antivirus errantly decides that a critical part of the OS is a “must be exterminated on sight” file (leading to the pretty blue screens of abject failure). Things that you’d have thought that even rudimental testing might have found…

Just one point on your Iconbar article:

they repeat the license details in every single file. Resist the urge to do this.

While this bugs the hell out of me (the entire RISC OS codebase is full of verbose legalese crap at the top of every bloody file), this is likely less to do with the mentality of the programmer and more to do with the licence telling them to do that.

In my own case, the top of the files identify the application, which module it is with a one line description, who wrote it, which version/date the module was last modified, a one or two line note about the licence (depending on whether or not I give a URL), and then any specific notes that should be taken to apply to the module as a whole.

I don’t, personally, believe that a wall of licence text has any legal weight over a simple “this code is licensed under Xyzzy” as in both cases you’re stating your wishes as the software author. Therefore the wall of text at the very top (as is often instructed by the licence) is self-aggrandisement on the part of the licence.
Especially given as those who have no intention of respecting a licence will do so whether you use six words or six hundred.

In the specific case of RISC OS, it’s quite a pain in the arse to always have to scan down to locate the end of the blah blah (something like 14 lines) and then discover that the code not only has an ambiguous name, but has an ambiguous explanation. I just looked and there are multiple files that say “Vdu driver code – Sprite stuff” and that’s some of the better intro commenting. ;)
The kernel’s “Middle” doesn’t say anything at all, and the name just implies “random stuff in the middle” (which is about right, but still…).

The artist formally known as Charles Ferguson declared

Producing an operating system is not like writing a small application, module or tool – it has to work

Amen brother. Alleluia.

I don’t think that RISC OS – the operating system, or the components that run on it – is well tested

Nor well documented.
Nor well understood.
Nor well designed.
Nor well defined.

Absolutely agree that Regression Testing is a must in any significant production environment. But I’d go further and say that beyond bedroom-coding-sized-projects (me and thee) any significant project should have dedicated QA.

General programmers are BAD at testing (I go further and maintain that general programmers are bad at interface design, but that’s a different soapbox).

I’d also recommend a developer build of the OS that goes to extreme lengths to check absolutely everything (perhaps that’s what Pyromaniac is), but that the distributed OS is as forgiving and streamlined as possible.

Rick remarked

I don’t, personally, believe that a wall of licence text has any legal weight over a simple “this code is licensed under Xyzzy”

Correct. You will note that you sign an NDA once, it is not repeated on every document you thereafter see. And giving somebody the text of a contract DOES NOT behold them to that contract unless they agree to it. “If you’re reading this you’ve agreed to it” does not stand.

Returning to gerph

it has ~2800 tests which cover about 70% of the code – making it also one of the best tested pieces of software that I’ve ever written

One must always be aware that you writing the test for your code implicitly asserts your understanding of the API that you’ve implemented…

Or to put it another way, there’s a danger of marking your own homework.

This is the value of an adversarial QA team – they try to expose your assumptions and misunderstandings. This is not something that humans are good at doing for themselves.

In a few years (or less!) we’ll have AI QA. That’ll be a game-changer for the industry.

No we wont. It will be even more important to have human generated testing to deal with all the AI generated crap lazy developers will be cutting and pasting.

In a few years (or less!) we’ll have AI QA.

Great. Something that understands the code even less than the humans that wrote it.

And giving somebody the text of a contract DOES NOT behold them to that contract unless they agree to it.

Software licences work by tricking copyright.

You DO NOT have an automatic “right” to use, own, or copy a piece of software.
Except there’s this legal crap that says you does under certain conditions. There is no obligation on you to pay any attention to the terms, but if you don’t, you don’t have any rights to use the software.

Whether or not this is fair or correct is an entire discussion in itself, but for the moment as weird and broken as the analogy may be, nobody has come up with anything better.

Sadly, these days, licences for the use of software are being abused in all sorts of ways – you cannot disparage the product, you cannot benchmark it, you cannot show yourself using it on social media, disputes must go to arbitration, no class action allowed, etc etc etc.

It’s very one sided, where’s the suggestion of recompense if things go south because they screwed up due to basic lack of competence or even concern? Techies fixing things isn’t done for free… Instead, no warranties or liability. Which is acceptable in an open source or free product, but less so in something that costs €$£¥.
After all, if you pick up a fridge from the municipal dump and it doesn’t work, you get what you pay for. But if you buy a new one that fails to keep things cold, well, there are various sorts of consumer guarantees that can be invoked. It’s not to much to ask that something you buy basically do it’s job, right?

The Sale of Goods Act does not exempt software. No contract terms or user agreement can nullify law. These verbal constructions are part of the Art of Dissuasion – don’t bother complaining.

I recently got a letter from my broadband provider (plus TV and phone) telling me it would be going up to £95 a month. So I rang to cancel. A friendly Indian guy did the best he could, but could only reduce it to £80-something. I insisted on cancelling so he put me through to a nice lady in Cardiff. Result? £41 a month.

Boiled frog. Inertia. Politeness. Confrontation-aversion. All part of the Art.

“By opening this package you are agreeing to these terms”

<crosses terms out, opens package>

While this bugs the hell out of me (the entire RISC OS codebase is full of verbose legalese crap at the top of every bloody file)

There is the other end of the scale – last time I looked, the source for the PC card (PC Pro) contained a reference to a licence file, which was not present. Rumour was that it might be GPL, then again it might have been one of the other less irksome ones. Who knows…

“By opening this package you are agreeing to these terms”

Which, due to folding, can not be read in full anyway thus are patently not legally binding, even if the premise that statement is based on is false anyway.
Is a double fail a success?

Which business was it that buried a “and your first-born shall belong to us” in its small print to see if anyone actually read it?

I very much agree about the value of automated tests.

I was introduced to compile time unit tests 20 years ago on a large codebase and that change was by far the biggest contribution to bug reduction I’ve ever seen. New code was written alongside its tests and the vast majority of bugs occured in the older untested code. Tests were added to reproduce reported bugs and then the implementation fixed so the tests pass. Those tests then help to guard against regressions.

I now have very little time for to do much in RISC OS, which is the the biggest blocker which is the only reason I’ve not played with Pyromaniac, it looks like an amazing bit of work.

Regrettably, most of my RISC OS stuff doesn’t have much in the way of automated tests. There is a bit in a C library I wrote to make writing toolbox apps like Avalanche easier, but coverage is poor. I also have something which patched the C stubs to inject some sanity checks around malloc, free, memcpy, etc. at runtime. IIRC there has been some changes to the stubs so it doesn’t currently work.

I don’t think that RISC OS – the operating system, or the components that run on it – is well tested.

I agree with you, but I also know that people at ROOL wish to improve that.

I don’t know if that’s the case… so tell me… did it have any effect at all?

I am a big believer in Software Testing myself and for what concern test automation on RISC OS I also ported two (relatively) popular libraries to help people implementing tests automation on RO:

https://github.com/RISC-OS-Community/mUnit

https://github.com/RISC-OS-Community/GreaTest

Are people downloading them? Yes, GitHub says so, this community is pretty much downloading everything I have made on GitHub (included the template repo?!?!? btw why that one guys? lol ) and the numbers a quite relevant for a community so small as this one.

Are they using them? I do not know. But if people need, I could make some videos on how to use them with your own code, just let me know. (actually we already planned some to be made in the near future).

Even if you didn’t know any of the things that I’ve just mentioned, have you been producing tests with your software anyhow?

Yes, as an example here:

https://github.com/RISC-OS-Community/uCLib/tree/develop/tests

These tests builds with the library code and gets executed at every build, ensuring that every change gets tested immediately. However, they are not even close to the number of tests I need for that library, but given that people are only using what I make and not providing any help in building it, it takes time (time I largely do not have because my work takes already a lot of my time).

IMHO, one of the biggest issues on RISC OS is automating GUI Apps testing. I use an hacked version of !KeyStroke to achieve that, it’s ugly and unpresentable, but it works and for now, given the lack of available time, that’s what I use.

I obviously downladed and started to use also your Pyromaniac which is exceptionaly good for debugging and testing, I wish it would implement 1-to-1 RO 5 API as well (and yes I know I am asking for the moon, your time is limited too, but hey we are talking about what could be done to make things better, it’s not urgent, as not a lot of things are different).

BTW, thanks for the work on Pyromaniac, it’s really an amazing work.

I don’t know if that’s the case… so tell me… did it have any effect at all? Even if you didn’t know any of the things that I’ve just mentioned, have you been producing tests with your software anyhow?

Over the past few years I’ve definitely increased the amount of testing I’ve been doing, partly in response to your posts about the subject. Most of them are designed to be automated – detecting failures automatically instead of requiring the output to be checked manually – which is a big step up over the small scraps of test code I’d typically use in the past. But we still need to add a “Run tests” option to !Builder, and set up a system to allow GitLab to run the tests.

Nemo wrote…

One must always be aware that you writing the test for your code implicitly asserts your understanding of the API that you’ve implemented…
Or to put it another way, there’s a danger of marking your own homework

Absolutely. This is why I absolutely hate the idea of any serious endeavour that focuses on development without independent testers. It’s better to have some tests written by the author than none at all, but having someone else perform testing or at least review and provide input is far more important.

This is another reason for producing the tests repo – people can see what I’ve done. They can see where it’s lacking. They can suggest and it’s open source on github so they can fork and give PRs. Please do this… accumulating a selection of tests is only going to benefit RISC OS over all.

The main reason for posting this was to draw attention to the fact that these resources exist and to hope that people will pick them up. RISC OS Pyromaniac will benefit from bug reports, and from tests that were never thrown at it. RISC OS Classic will, obviously benefit from having any form of tests whatsoever.

Jeffrey Lee wrote:

Over the past few years I’ve definitely increased the amount of testing I’ve been doing, partly in response to your posts about the subject. Most of them are designed to be automated – detecting failures automatically instead of requiring the output to be checked manually – which is a big step up over the small scraps of test code I’d typically use in the past. But we still need to add a “Run tests” option to !Builder, and set up a system to allow GitLab to run the tests.

I’m really glad that it has had some effect in encouraging some testing -

Do remember that the Build service exists and can build and execute code supplied to it. It is a proficient version of RISC OS. Honest :-) Yes, it is not RISC OS Classic, but it follows the APIs defined where possible and provides far more feedback when used correctly than you ever could have on RISC OS Classic. Similarly, RISC OS Pyromaniac can be used within such systems – my own GitLab instance runs tests on many RISC OS modules and programs using Pyromaniac to check that it behaves.

As I suggested in the article and presentation, RISC OS developers tend to consider the tests as an ‘extra’, not a prerequisite to development, so it’s good that it’s being seen by some people as important. It’s good to hear that it’s considered, but do try to make the testing a priority. It isn’t optional – at least in my view.

You asked in a recent thread about how to test RISC OS things on non-RISC OS systems. This is, again, exactly the sort of environment that RISC OS Pyromaniac is absolutely designed to work within.

Paolo wrote:

I wish it would implement 1-to-1 RO 5 API as well.

Where sensible, I’ve implemented some of the RISC OS 5 APIs. I won’t, however, make it incompatible with documented APIs.

I won’t, however, make it incompatible with documented APIs.

Well said that man. <stares menacingly at SetVarVal, DynamicArea,0, GetCaretPosition, SetCaretPosition and CallAVector>

I’ve implemented some of the RISC OS 5 APIs. I won’t, however, make it incompatible with documented APIs.

Problem is, love it or hate it, the RISC OS 5 APIs are where the action is these days.

And as for the “documented”, well most of that is between two and three decades out of date, in places somewhat inadequate, in places just plain wrong (nemo has already pointed out some), and in places simply nonexistent.

It is nice to be able to point to a piece of paper and say “this should happen”, but if that’s not what the OS itself has been doing in the last 30 years, then the documentation is wrong (or the code, or both, I’ll let others argue that).

Then, of course, you have well meaning people coming along later and making changes to existing APIs which may or may not be…troublesome, and fiddling with other stuff for random reasons potentially causing breakages.

So… documentation? A sort of idealised “this is what we’d like to have happen”, which may or may not match what actually does happen. ;)

Rick remarked

It is nice to be able to point to a piece of paper and say “this should happen”, but if that’s not what the OS itself has been doing in the last 30 years, then the documentation is wrong (or the code, or both, I’ll let others argue that).

Exactly this. Sometimes the code is obviously wrong. Usually the documentation is wrong (or certainly incomplete). But sometimes the reality is somewhere in the middle.

When a new API is published and shipped there is a window of opportunity to declare every bit of the implementation wrong for not matching the docs… but once some amount of time has passed and some amount of code has been written to use or accommodate whatever that implementation is doing, then de facto that contributes to whatever the API actually is.

Ultimately that means that after 35 years no one knows what the API of OS_WriteC is. Welcome to RISC OS.

“this is what we’d like to have happen”

The reality of documentation production is “this is what I think the developer told me when I asked him about the scribbled diagram he’d sent to clarify the out-of-date document he’d emailed”.

Problem is, love it or hate it, the RISC OS 5 APIs are where the action is these days.

In this thread I am talking about the testing of RISC OS. My views on the issues of incompatibilities have been made clear repeatedly.

My stated position is that I will not create incompatible interfaces, where such an incompatibility is required. RISC OS Pyromaniac may modify the behaviour of some interfaces to allow them the applications under them easier to test, or remove features if it’s not possible – that’s why we have error returns, but that’s all.

In this thread I am talking about the testing of RISC OS.

Yup, but your definition of “RISC OS” may differ. ;)

I’m with gerph here. The RISC OS API was unambiguous in 2004 when “RISC OS 5” (a hastily rebadged NCOS) turned up. I told Castle then that their version was insufficiently compatible with RISC OS and was told “it was early days”.

It’s still not managed to achieve compatibility with the RISC OS API as it existed when they launched. Missing API is one thing, but having flags and in some cases registers in the wrong place is indefensible.

But I’m beginning to suspect that some people wear incompatibility as a badge of “progress”.