Request for community help - improved CCres tool

One ticket in our internal database that’s been there since, ooo, forever is to create a tool that enables us to diff Wimp Templates (and Toolbox resources) files. This would help in processing code submissions. Ideally, we’d have all the Templates and Resources files in CVS stored as a textual representation and the build process would run these through a tool to convert them back into the binary format. The same tool could be used to reverse the process, for example after you make some changes to the Templates in a template editor.

Now, there’s already a tool out there called CCres but this failed our evaluation for a few reasons:

It doesn’t produce a binary identical Template file when converting to text and back → fail. Is doing evil things like changing the string terminator characters and not storing any unused/padding byte values. It also manages to bloat the files quite a bit for some reason.

I can sort of see why: the text file format is describing windows, not Template files, so it doesn’t attempt to describe how the file is laid out, only how the window should end up (assuming nothing clever is being done – I expect it probably screws up the Wimp Templates, for example, which get corrupted by many Template editor apps). They also missed a trick when basing their syntax on OSLib, in that they could at least have made it close to being C99 compound-literal syntax: a missed opportunity.

What we need initially is a tool that will map to and from the binary and textual representations, maybe as an extension/update to CCres. Then, we could add that as a standard tool in the build – licence permitting – and finally replace all the Templates and Resources files with textual versions in the source tree, as well as adding a build step to convert them back to binary.

I’d like to open this one for discussion before we actually nominate anyone to get cracking on actually doing this…

Does it really need to produce an identical binary template? I would have thought it would be enough to just be consistent with unused/padding bytes, so after the first time it’s converted, any future conversions will be consistent. Again I’m not so sure that making the string terminator characters consistent is that bad a feature either.

It there is a better syntax, I’m all for creating a new tool that uses a new syntax.

I don’t think I’d be able to get it done fast enough, but if a detailed specification of the Text format and what the fields are was created I’d be willing to at least attempt a program. As I don’t have much time it would also need someone else who would be willing to test it.

Does it really need to produce an identical binary template?

I’m putting words in Steve’s mouth here, but I suspect it all comes down to the time required to test things properly. If translating every template to text and back again produces binary-identical results then it’s easy to prove that the translator is working correctly. If it doesn’t produce binary identical results then you’ve got to go through and look at each file individually to work out why that’s the case (random junk padding bytes in the original file? Bug in CCres? Does the difference between the two files actually cause any problems at runtime? etc.) otherwise you run the risk of breaking the OS.

Note: Ben got part of the way through writing an in-house tool for this but I think he only got one side of the translation working (but at least in a exact, to-the-byte mode of representation). They only issue with starting from that would be he can’t locate his sources (!) they are likely to be on his Iyonix that currently won’t boot… I’ll dig through old emails when I next get a chance to see if his work-in-progress did the rounds within ROOL.

Our ticket was raised in 2009 and Ben’s first comment was “finish off a tool I started writing last year”, which at least gives me a timeframe to search emails in (or after).

Does it really need to produce an identical binary template?

Would this not imply a necessity of knowing not only what the contents of the windows are, but also the internal structure of the template file?
For instance – should files be saved in the order that the windows were created? Should they be saved alphabetically? Should they be saved with the most used window first?

Unless I have missed a hard and fast rule buried somewhere in the PRMs, there is no particular definition of the layout of templates files, other than they be valid window/icon descriptions.

The solution? Surely a template passed through the tool once will subsequently be consistent through further passes?¹ [if not! the tool would need to be considered “broken”, even if it was just doing something lazy like allocating memory without clearing it before use]

Another (harder!) solution might be for a source system that can better cope with non-text files, perhaps with an ability to work with plugins to permit making sense of such files based upon their file extension…

¹ Assuming there is no conflict between it and a template editor that may rearrange the templates in the file.

Would this not imply a necessity of knowing not only what the contents of the windows are, but also the internal structure of the template file?

That’s correct.

Unless I have missed a hard and fast rule buried somewhere in the PRMs, there is no particular definition of the layout of templates files, other than they be valid window/icon descriptions.

There’s your problem… PRM4-459 (Appendix E – File Formats).

Steve: I suspect I know (some) of the answers below so some of it is framing the question or discussion, but I suspect it’s helpful to do nonetheless:

What would the proposed benefits of such things be to ROOL / the developers?

Does that fit part of a larger ‘roadmap’ in the build system?

Only asking as clearly it has some uses, and if they interest someone and save ROOL volunteer time then all well and good. But as the development is on the build system and not the actual ROM (thus developments which the end users don’t immediately see), and potentially it would end up with an extra step in the build process thereby adding complexity, it’s useful to be crystal clear where that fits the process.

(NB I’m playing devil’s advocate to some extent, not actually disagreeing with you that it’s useful!)

The benefits are mainly to ROOL and (indirectly) to developers. Having templates and resource files stored in CVS in a textual format will make it far, far easier to see what changes in each revision of the file. This in turn makes it considerably easier for ROOL to check any submitted UI changes to published applications (like Paint or configuration plug-ins, etc).

The cost to the developer is the added build step, which would add next to no time to a typical build and would happen automatically so should effectively be transparent. Clearly, if your intention is to edit one (or more) of these files, then you’ll need to manually run the textual representation through the converter before editing it and back again before submitting the sources back to ROOL. That will be pretty insignificant when compared to the size of any actual development work, of course.

If the converter tool and textual representation is designed well enough, some types of edit might be just as easy to make by hand directly to the textual version of the file.

In the very long run, you could conceivably leave the textual version in the application and have it transparently translated when the application is loaded…

This is not a change which will have any real impact or demonstrate any improvement to the end user; it is entirely about process and making things a bit better for anyone who uses the ROOL CVS repository.

There’s your problem… PRM4-459 (Appendix E – File Formats).

Okay, sloppy wording on my part.

When I say “layout”, I meant specifically the ordering of data. Of course the file has a specific format it must adhere to!
What I meant was… The windows could be provided in any order, so a conversion tool cannot rearrange them in any way if to have a chance of generating an output binary that matches the input binary… That said, template editors would also need to zero out unused parts of Indirected data. Do they? Do they all ?

The windows could be provided in any order, so a conversion tool cannot rearrange them in any way if to have a chance of generating an output binary that matches the input binary…

Correct. It is a completely unambiguous format and thus the translation should be likewise. It’s not rocket science; just parse the template file and output the results in a human-readable format that itself can be parsed back into a binary-identical file. It really isn’t hard! :)

That said, template editors would also need to zero out unused parts of Indirected data. Do they? Do they all ?

Yes. We will still get spurious diffs as a result of people editing the templates/resource files because some editors have an evil habit of changing the string terminator character to something they like. Others might well zero-initialise any unused space after strings or similar.

But the two critically important bits here are:

1. once the application is demonstrably working, the process of → to text → to binary will result in an identical binary file which will work just as well as the one we tested and…
2. it should be fairly easy to spot (and maybe also reject or fix-up) things that are unintended changes, e.g. stuff the editor did rather than stuff the developer intended when they were using the editor

In the very long run, you could conceivably leave the textual version in the application and have it transparently translated when the application is loaded…

The thing that concerns me with plans like this is you are throwing a lot of parse issues to the app-doing-stuff stage. At the moment, either a template file is valid or it is not. Generally, they are valid (because broken files are no good…).
However, if it is a textual description – how do you handle broken descriptions? Too much data? Too little data? Missing quotes or semicolons? It is acceptable to guess what was intended, or is it acceptable to immediately fail? Should you fail the icon? The window? The entire thing?

This is one of the reasons why I prefer a custom binary data format to stuff like XML. A data file has a very specific format. XML can be quite vague and in places rather broken. In fact, one only needs to look at the HTML¹ source of a small number of web pages to begin to try to imagine the vast number of kludges that must be present within the parser in order to get sane results from some of the crazy input.
No, I’m not advocating making HTML binary. I am suggesting that you consider that if the templates file is a readable editable document, it will need a pretty flexible and competent parser if it is not to choke on any little inconsistency. And note that the RISC OS GUI, like most, has a pile of built in quirks – icon border types, ESGs, whether or not something is Indirected, old vs new format, blah blah. I can see people making some interesting files, for lulz mostly.

I’m not saying don’t make use of textual template descriptions. I’m just saying it ought to be a part of the build process, not the runtime process.

¹ Yes, I know XML and HTML are not the same, but they share a common “feel” (with XHML closing the gap even more). I could point you towards some malformed RSS feeds, but better to skip over to the world of HTML where you will find endless examples of horrible markup. All stuff a capable browser needs to be able to cope with. God help it.

The thing that concerns me with plans like this is…

That’s why I very specifically said “In the very long run, you could conceivably leave the textual version in the application”. Translation, in case I wasn’t clear enough: everything else I said is what matters, this bit is one possible, maybe one day, pie-in-the-sky, could happen if you think about it, future. Just that. ;)

So don’t get concerned about the bit that doesn’t matter. Because that’s not what I’m asking for.

Ah, just dug out the email from Ben where he sent me his program. Sadly it was the ELF executable and not the sources. More digging required… :(

Thanks for the reply Steve. The other benefit I had thought of was: once you look at template files in this way and work your way through a two-way parsing process, it isn’t much extra work to add a template file style parser to it: i.e. checking that groups of icons are correctly aligned, making sure style guidance is adhered to to some extents etc, and flagging warnings for them.

this bit is one possible, maybe one day, pie-in-the-sky, could happen if you think about it, future.

Oh, yes, I know that future. That’s about when I’ll get my various projects finished… ;-)

So don’t get concerned about the bit that doesn’t matter.

Awww… That’s my specialty.

Awww… That’s my specialty.

:-D

Do bear in mind that it is perfectly possible for numerous indirected icons to point to the same indirected content. One of my template editors did that to keep the file size down. How quaint.

Also one can stick any old data in a template file – I’ve seen ones with menus in for example (though that was a LONG time ago).

Caveat factorem.

I’ve not got an overwhelming impression that anyone is planning on taking up this one. I’ll leave this thread as-is in case somebody fancies grabbing this task.

I do have some experience of writing a program to show the differences in Template files in ‘english’, not as a byte-by-byte comparison ‘byte x was y now z’. My initial requirement was to find what changes there were in two such files with over 60 templates which had the same origin, but had diverged considerably over many years. I did look at reading the actual Templates files, but decided to compare the output from CCres instead. Currently it is a simple single-tasking Basic program (and only suitable for my personal use!).

A snippet of example output is…

---------------  Template "BoxBar"  ----------  Icon 0  ---------------
extent l,b,r,t  was -12,-82,28,38           now 0,-80,40,-6  diff 12,2,12,-44
icon_flags      del ICON_BORDER             add ICON_FILLED
icon_bg         was COLOUR_VERY_LIGHT_GREY  now COLOUR_MID_LIGHT_GREY

1. Different template editors can change the physical sequence of templates in a file, and it is not necessarily the same order as the editor displays. If the first physical template suddenly becomes the last, then any comparison which uses the physical sequence shows that everything has changed. Any diff should show sequence changes, but separate from changes in templates with matching names.

2. Different template editors can round co-ordinates differently, I suspect. I saw many that had changed by a very small amount (<4 pixels) which may, or may not be, important. My program includes a threashold for these differences.

3. Some Window Flags vary for no real reason that I found, for example WINDOW_BOUNDED_ONCE

4. Because Icons are numbered by their location in a window definition, if any icons have been re-numbered they are physically moved. This can make it very difficult to see what changes have really been made to existing icons!

5. I did experiment with re-creating a Templates file from CCres output, and then doing byte compares of the new and original files. It certainly confirmed that currently CCres does not produce a binary identical file, as previous posters have commented!

6. If a new tool (say CCresPlus) did produce an output file that was able to be converted by CCresPlus back to a binary equivalent, The output file would have to contain additional information to CCres output. This may make it easier to produce a mechanistic diff, but that diff would be less useful for us humans to see what the real logical changes are!

7. I agree that if CCresPlus did not produce binary exact files, it would be difficult and dangerous to test. But if a Templates file was changed by three different editors, the diffs for each change could be very large, even though the logical changes could be very small! I regard the ability to see what has logically changed is very important, and I suspect a compromise would have to be made.

8. Rick said

Surely a template passed through the tool once will subsequently be consistent through further passes?

This would only be the case if there had been no changes: surely any change by a Template Editor could make it inconsistent again?

9. Steve said

Having templates and resource files stored in CVS in a textual format will make it far, far easier to see what changes in each revision of the file.

This depends whether you want to see every change, or just the logical changes.

10. Steve said

once the application is demonstrably working, the process of → to text → to binary will result in an identical binary file which will work just as well as the one we tested and…

I agree this is critically important, and he then added…

it should be fairly easy to spot (and maybe also reject or fix-up) things that are unintended changes, e.g. stuff the editor did rather than stuff the developer intended when they were using the editor

… but this may be more difficult than it seems.

11. I have no idea what extra challenges Toolbox Resources files add!

I’ve not got an overwhelming impression that anyone is planning on taking up this one. I’ll leave this thread as-is in case somebody fancies grabbing this task.

Any chance of you finding Ben’s code and putting it somewhere we can look at before abandoning this one? If he’s done half the work in a conversion one way it should make it a much easier task.

To cope with some of the issues Martin’s raised, perhaps we should go with a two-stage approach?

Stage 1: Convert all binary template files to a text form which can be converted back to a binary-identical version of the original file.

Stage 2: Whenever someone edits a template file, they run the converter in a mode which approximates CCres’s current behaviour. I.e. all the unnecessary crap gets stripped out, and the output is formatted in a way that it’s easy to see what the logical changes were to the file.

This means that the first time someone edits a text template file there’ll be a big mess of changes due to it switching from the full “stage 1” output to the reduced “stage 2” output. But those changes will be relatively safe, as we can assume that the author has given the new templates file a reasonable amount of testing. On subsequent edits the changes as viewed in CVS will be pretty minimal, as it’ll be a “stage 2” format file for both the old and new version.

Of course it’s questionable how useful the “stage 1” output will be, if we’re only planning on keeping the file around until the next change is made. So it might make more sense to just do a phased switch over to the new format – i.e. keep all the existing binary files as they are, but make it a rule that when you’re editing a component which uses a binary template your first task should be to convert it to the “stage 2” text form (and give it a fair bit of testing).

AIUI Ben’s Iyonix is not booting and the hard drive might be fine. I would have thought it worth trying the hard drive in another computer or in a USB enclosure.

Any chance of you finding Ben’s code and putting it somewhere we can look at before abandoning this one? If he’s done half the work in a conversion one way it should make it a much easier task.

and:

AIUI Ben’s Iyonix is not booting and the hard drive might be fine. I would have thought it worth trying the hard drive in another computer or in a USB enclosure.

Yes – it’s more a case of Ben finding some time to extract his Iyonix from some packing boxes (he’s been in a complex house-moving process) which is the problem.

For the time being, I suggest we pause any thoughts of development work until Ben gets a chance to obtain the sources to his program.

I missed this thread when it was still current, but if help is still wanted, I might be able to offer it. My principle questions are:

1. What environment is it expected to operate in? I’m most comfortable with BBC BASIC, but is this expected to run non-natively or in whatever environment the CVS system runs? Does it have to be compilable with GCC?
2. Are you serious about the requirement to reproduce the current binaries exactly following conversion to text and back again? Does this include any uninitialised data and other gremlins which may be lurking in them? If so, how would these be stored in text form?