Automating Wimp GUI

Are there any apps that will fake interaction with the WIMP GUI? The sort of thing I mean is a way to write a script/program that will do things like:

1. Find the !Edit icon on the iconbar and click Menu over it
2. Read the menu and find ‘New file’
3. Slide right over it, select ‘Command’ from the submenu
4. Switch focus to the empty file window that appears, type some text
5. Click Menu and slide to File→Save as
6. Drag the save box to a specified directory viewer

Obviously this action could be done from the command line, but there are other apps where the only actions are available from the GUI. It’s feasible to write a one-off script with lots of Wimp_SendMessage calls, but I wondered if there was a nicer toolkit to write scripts like this.

There’s a scripting language called AutoIt for Windows – wondered if there’s anything similar like this for RISC OS?

bq There’s a scripting language called AutoIt for Windows – wondered if there’s anything similar like this for RISC OS?

Some years since I played with AutoIt but I know it grew a little, the back then RO item with similar capability is !Playback http://www.apdl.co.uk/freeware.htm (down the bottom)

The PlayRecMd module is not 32 bit. Neither is the click voice module but that’s a lesser issue

That’s interesting. Though it’s based on recording mouse clicks, which are very vulnerable to changes. For example, ‘find the Edit window containing $.! Boot.!Run and press the Close icon’ is a lot more robust than ‘move to 692,575 and click Select’.

Would Task Messenger do what you want? From the !Help file:

This very short program allows you to issue several commands to wimp tasks from obey files: Quit, BarClick, MenuClick and DataLoad.

That’s interesting. Though it’s based on recording mouse clicks, which are very vulnerable to changes. For example, ‘find the Edit window containing $.! Boot.!Run and press the Close icon’ is a lot more robust than ‘move to 692,575 and click Select’.

I’ve not looked at AutoIt in about 8-10 years so I suspect addition features may be in there now.¹ However, even back then, as I recall, both the “identify window and work on features of that” as well as the “move to a specific screen position and work on what you think is there” methods were present.

The latter was of course simpler to implement but more prone to acting on random elements.

The problem with identifying “the Edit window” would be identifying the specific window. In the example you gave it would need to specify the filesystem and disc as you could have $.Run on more than one disc or filesystem. i.e. full path required.

I can certainly see a multitude of uses for a RiscIt utility. e.g. If the playback was embedded in the OS software authors could record tutorials for incorporation in their software. “New users click here”…

¹ I did a number of scripts which took the interest of a colleague who used a variant (he did a lot of work) to automate a database update procedure and save a lot of money.
Option 1 Let outside company employ many people for weeks to manually update a database with missing data (present in another database) at a cost of thousands.
Option 2 Use the script to do the update work by leaving a laptop running the sequence for 45 minutes.
For some reason people opted for option 2 :-)

Would Task Messenger do what you want? From the !Help file

That’s a lot simpler overall, also not 32 bit (but simpler than Playback so less to fix)

TaskMessenger looks the sort of thing, though a bit limited (only iconbar clicks are supported)

‘Identifying the Edit window’ is the kind of process such an app would take care of:

1. Walk down the window stack
2. For each window read the titlebar and find out which task it belongs to
3. Pick the first window that matches two conditions:
   1. Owning task is ‘Edit’ (but not ‘TemplEdit’)
   2. Titlebar contains ‘$.!Boot.Run’

The match can be fuzzy, because in some programs I can’t know if the titlebar is “Zap – ADFS::HardDisc4.$.! Boot.!Run (dwo*) (Obey)” depending on the exact settings of the machine (is it being edited in Obey or Text mode, is !Boot on ADFS or SDFS, what’s the disc name called, has the file already been edited, etc etc).

I briefly tried a Windows app that used the ‘move the pointer about’ method, combined with OCR of what was underneath the pointer. It was amazing that it worked (was able to read out the contents of text fields using OCR) but too flaky to use (window opened at different place or a different layout, OCR tries to compensate but gets it wrong). AutoIt is harder to use (you have to write a script saying what you want rather than just recording a series of clicks) but much more robust.

A couple of possibilities that might do the trick:

Druck’s WindOpen
Quantum Software’s KeyStroke

Hmm… KeyStroke might do the trick – there’s also source with it. Might take me a while to drag out a copy of Impression to read the manual though.

I managed to do enough RPCEmu hoops to convert the Keystroke Impression manual into PDF, which you can download.

Looks like a very handy tool that will probably do the job, thanks for the tip.

BTW, the original KeyStroke page on quantumsoft.co.uk/keystroke seems to have been taken over by a bizarre form of ‘SEO’ – mixes original text, spammy links and legit but unrelated links.

I managed to do enough RPCEmu hoops to convert the Keystroke Impression manual into PDF, which you can download.

Nice.
Interesting – there’s a few bits of corruption in the text that show when reading on a PC, but all is well in RPCEmu on exectly the same file – spotted in the contents pointing to page 27 and 30. The pages themselves have similar display issues (PC reader)