Native RISC OS Unit Tests?

I think I may know the answer to this, but…

Do we have any tools for running unit tests on stuff that requires a RISC OS environment? I know we can test stuff which doesn’t need RISC OS by using other platforms, but are there any tools out there to help with validating code that is interacting closely with parts of the OS and therefore isn’t really a candidate for testing over on Linux?

Google is saying “no”, but I could easily be missing the correct combination of search terms.

I’m not clear whether you’re after a generic solution (eg. is there a framework equivalent to Jenkins to run unit tests on RISC OS), or a specific example of a test that needs to be run on RISC OS because it can’t be hosted on Windows or Linux or whatever.

One thing in the latter category I can think of is FSBash which sits on a FileCore format disc calling lots of RISC OS SWIs. Attempting to host that on a foreign OS would be a little futile!

Unit test frameworks are tied to the language used, so I assume you are after something for C? In which case there are lots of open source ones to choose from. I don’t know any that have been specifically ported to RISC OS, but the simpler ones should be pretty straightforward. A bigger challenge would be to get everyone to agree which one to use.

@ Steve Fryatt

Do we have any tools for running unit tests on stuff that requires a RISC OS environment?

For GCC C++ we have CppUnit on !PackMan

For C UnitTests you can use CUnit it ANSI C, I think I made a build somewhere on my mini.m if you need, not yet packaged in !RiscPkg because I am working on an automation to build my stuff, so if you want I can zip it up and send it via ChatCube or something?

For other programming languages I have no idea… sorry

Apologies for the vagueness of the question…

I’m looking for something that can be used to test C code¹, but with the ability to test stuff which interacts with the OS. For example, I’d like to be able to verify that a Wimp library routine to “read text from an icon into a buffer” does exactly what it claims across all of the edge cases that the Wimp offers. I imagine this means that it will need to be running on RISC OS, and as a Wimp task.

My suspicion is that I’m into “custom Wimp application” territory, and that doing something myself might be the path of least resistance – but I thought I’d ask in case it had already been done.

¹ Doing the same for BASIC might be nice, but is less of a priority… and I’d not expect an off-the-shelf solution for that.

So CUnit is a unit testing framework, example code here:

https://cunit.sourceforge.net/example.html

(An example speaks for thousands words)

For what you seems to be wanting to do, you should combine it with a UI testing Framework, which I don’t believe we have in the modern form, but we used to have something usable for the job back on the Archimedes. Forgot the name… sorry

Basically a tool to record your mouse moves and clicks and keyboard input in time and being able to replay it against the UI…

So CUnit is a unit testing framework

More recently here, it seems.

Thanks for the pointer! It looks similar to what I had in mind anyway, and a very quick play with the Autobuilder seems to result in an ARM ELF library file falling out of the other end. Whether it works or not is a problem for another day, I think.

The only question is how it might coexist with our cooperative multitasking, as part of my back-of-an-envelope scheme seems to rely on being able to run a test on a null event and then assume that when the next null event arrives it should have completed. If the library works, I’ll have a read of the documentation and see.

Basically a tool to record your mouse moves and clicks and keyboard input in time and being able to replay it against the UI

Whilst that would be nice, for now I think I can test what I’m actually interested in by simply wrapping it in a Wimp task and feeding it a template file with some carefully-crafted windows and icons. Full test coverage can wait… :)

we used to have something usable for the job back on the Archimedes. Forgot the name… sorry

Basically a tool to record your mouse moves and clicks and keyboard input in time and being able to replay it against the UI…

From your description, racked my brains, slept, woke remembered and then searched for the package.
Memory said it had a Vince connection and:

Keystroke from Quantum software.
Free download.
Mostly in BASIC, modules are 32 bit and Vinces text at the top of the Quantumsoft page says there might be updates, which implies he might have the source for the modules (although fixing any ARMv7+ issues may not be hard even without)

Steve, there’s no reason why the example you gave can’t be done with any generic unit test framework, it’s just that the test setup is a lot more complex, than you’ll see in the cunit examples.

I would not rely on any form of mouse recording to drive WIMP testing, replaying clicks takes too long for testing and there is a lot of timing variability which can screw with things, you need to ensure the wimp calls are made programmatically for maximum reproducabiliy.

For your example in your test setup you need to initialise the wimp and create a window with the desired test icons on it, then call your routine under test to read the contents, and verify them. Getting the window on the screen will take a few even polls, and the tests can be triggered either on a null poll, or a message action. You probably dont have to worry about playing too nice with the Wimp and do as many tests as you like before yielding. The takedown routine will need to close the WimpTask afterwards, but obviously performing as much testing as possible on one task invocation will be a lot quicker than running the task up each time.

If being driven by an auto-builder, you will need a program to mange the testing, kicking off the test applications, and monitoring when they have quit, I also recommend getting it to kill off anything that is taking too long.

@ Steve

The only question is how it might coexist with our cooperative multitasking, as part of my back-of-an-envelope scheme seems to rely on being able to run a test on a null event and then assume that when the next null event arrives it should have completed. If the library works, I’ll have a read of the documentation and see.

I generally convert unit test libraries to log on a file, this not just because of the cooperative multi-task, but mostly because I can code on RISC OS (or other alternative OSes) mostly at the weekend, so log files help me to keep track of where I have left…

Don’t know if this can be of help to your case…

@ Steve Pampling

Keystroke from Quantum software. Free download. Mostly in BASIC, …

You’re da man!!! hell yes I remember Keystroke, very useful back in the days! Thanks for finding back the name and the download link!

:)

which implies he might have the source for the modules

I should have looked properly before posting – the source is included for the single keystroke module in a directory called “source”¹ so its definitely fixable if there are any problems.

¹ Suggestions for excuses I could use for missing that all gratefully received.

I would not rely on any form of mouse recording to drive WIMP testing, replaying clicks takes too long for testing and there is a lot of timing variability which can screw with things, you need to ensure the wimp calls are made programmatically for maximum reproducabiliy.

Since it’s a scripting system it produces what you set the script to do the same reproducible way every time and the time interval between actions should equally be adjustable to a specific value.

What exactly do you see about a set of commands sent by a programme as being anything other than ensuring that “the wimp calls are made programmatically” ?

In the PC world the equivalent was (still is I think) “AutoIt” – widely used as a fast imitation human (without the errors or comfort breaks)

Suggestions for excuses I could use for missing that all gratefully received.

Easy. It’s the weekend. You aren’t expected to be firing on all cylinders on the weekend as it’s the period of recuperation before the “oh my god it’s Monday…(tell me why) I don’t like Mondays (tell me why) I don’t like Mondays (tell me why) I don’t like Mondays I wanna shoo-oo-oo-OO-oo-oot…the whole day down”.

the same reproducible way every time

With some potential limitations, for example editors tend to open their windows at a fixed position with an offset applied depending upon how many documents have already been opened.

lot of timing variability which can screw with things

There’s a part of me that wonders if having timing variability isn’t a bad thing. In the real world, the only sort of OS that makes guarantees with regards timing is an RTOS. Otherwise, there are many many things happening in the background that could make subtle changes. So the timing of things shouldn’t need to be that sensitive.
[indeed, when Wimp_Poll returns depends upon all the other tasks and what they’re doing]

@ druck

I would not rely on any form of mouse recording to drive WIMP testing, replaying clicks takes too long for testing and there is a lot of timing variability which can screw with things, you need to ensure the wimp calls are made programmatically for maximum reproducabiliy.

It depends on what’s your goal, for example to test forms that are particularly complex (tree-views and 3rd party gadgets) and “creative” approaches, then you probably want to simulate the “real-user” as much as you can. But again in the end it’s a matter of preferences.

For what concern keystrokes I remember it was ok on the Archimedes (could create sequences very reproducible), the only thing I would wanted to add was storing multiple tests and execute them in sequence, but it was more address to create demos than testing software. However if the source is available we can modify it to become a proper UI test tool.

There’s a part of me that wonders if having timing variability isn’t a bad thing

If you want variability, generally speaking you use a human. If you want large numbers of repeated tests but with interval variations then generating a pseudo-random number for the interval and using a programmed/scripted system is the thing.

However if the source is available we can modify it to become a proper UI test tool.

Various BASIC files and a module (with supplied source)
Could be re-written/amended/ported to C and, if Gavin is reading, probably reproduced in Lua.
There’s probably a portion of the existing features that would have no use in a test tool and a few missing features in test tool terms.