ZLib module

As requested by a couple of people in the ROM compression thread, I’ve started work on a ROOL version of ROL’s ZLib module. Although most of the SWIs do map directly to zlib functions, there are a few things that I’ll need a bit of help with:

• ZLib_GZOpen is able to work with gzip files which have RISC OS file attribute metadata stored in them. Does anyone know how the metadata is stored, or can they send me an example file I can examine?

• ROL supply a C zlib library that interfaces with their ZLib module. What’s it like? I’m guessing it just uses standard versions of zlib.h and zconf.h so that it can be used as a drop-in replacement for a staticly linked version of zlib. But what happens if you try using the library on a system without the ZLib module? And is there any control over whether streams are task associated? Is the library prebuilt or in source form? (If it’s prebuilt then there’d typically be two versions – one for modules and one for applications. Which means the different versions could be hardcoded to use the right task associativity setting)

• Does anyone know how the ZLib module detects when a task dies? And is it really the Wimp task that the data is associated with, or the program that created the task? (i.e. whether the streams should be freed on Wimp_CloseDown or on OS_Exit) I’m guessing it would be the latter, to allow command line tools to make best use of the module. In which case, does anyone have a handy list of termination events that I need to watch out for? I have a feeling I’ll need to use OS_ChangeEnvironment to trap the error & exit handlers, but detecting when programs start/stop isn’t something I’ve had to deal with much. I’m hoping there’s a better way of doing it that I’ve missed.

Regarding gzopen, didn’t !Squash have some gzip support in it? Maybe that’s what is being referred to? Or perhaps the !Gzip port from a while ago. I assume the “filetype” extra field described at http://www.gzip.org/format.txt is perhaps what it is on about? (search for “Acorn” in that file)? In fact, it looks like filetype stuff appears twice in that list!

C library – AFAIK that’s the problem with the ROL implementation, and hence why we can’t really use it in live software.

Regarding gzopen, didn’t !Squash have some gzip support in it?

Yes – it looks like ROL added gzip support to their version. Something else to go on the todo list I guess.

I assume the “filetype” extra field described at http://www.gzip.org/format.txt is perhaps what it is on about?

That looks like it’s what I’m after – thanks! I guess I should have been more thorough in my searching.

Out of the two Acorn/RISC OS metadata formats that that doc mentions, I have a feeling it’ll be the ‘AC’ one that ROL are using. It looks like it’s the only one that’s been documented (I can’t spot any trace of Adam Goodfellow’s gzip on the ‘net), it was registered by an Acorn employee, and there are actively maintained gzip programs which make use of it (e.g. Duncan Moore’s gzip port)

Did you make any progress on the Zlib module in the end?

It’s about half finished; I’ll see if I can find the time to finish it off in the next couple of weeks.

For detecting when programs terminate, I think I need to be listening out for Service_WimpCloseDown. ROL’s documentation suggests that the task association only works with Wimp tasks, so I’m guessing that that’s the approach they’re using as well. Certainly, none of the other methods currently available (e.g. installing error/exit/upcall handlers via OS_ChangeEnvironment) seem to be capable of providing programs with a 100% reliable way of determining when another program dies. E.g. if something installs an error handler then it can work out when a program’s probably going to die, but unless the handler gets installed before the program is able it install any handlers of its own it can’t be certain that a handler further up the chain won’t allow the program to recover from the error and resume execution.

I’d say the only reason they’re detecting at all, is to cater for tasks being killed by the user in the Wimp. As you say, there’s no way of detecting the rest.

I look forward to getting my hands on the completed module.

The module is now “finished”. The Compress/Decompress SWIs, gzip SWIs and task association have been tested, but the rest is largely untested (although since the other SWIs are all basic wrappers around the zlib C functions there’s not much that could go wrong). I’ve added the module to !System in the disc image, and also uploaded a copy of the module here so any interested parties can have a play with it without having to download the full disc image.

What are peoples thoughts on distributing the companion C library? I guess it would make sense to put together a product that builds the module + library and packages it up so it can be hosted as a download on the ROOL site.

The logical thing would be a download of the whole lot on the ROOL download site, and also inclusion in future C tools releases as a standard library?

Indeed, one aspect of the C tools which might be a worthwhile update would be to include some common cross-platform libraries pre-built in object form. IIRC all the libraries supplied with Norcroft are RISC OS specific libraries (although you could argue that Clib and the TCP/IP stuff are common ground). I suppose this is pie in the sky stuff, but it might be nice to have ports of some other common libraries in the distribution (eg. libjpeg, glib and so on).

Seems to work perfectly for my little project, thanks.

Geez is GZSeek slow though! Back to the drawing board for me I think, need to recode and avoid seeks.

Couple of points, now I’ve had a more in depth play.

1. What version of CLib is it dependent on? So I can add the pre-req test before loading.
2. It doesn’t seem to report an error, or set V if you try to GZOpen a file that doesn’t exist. Subsequent calls to SWI’s based on the rogue returned file handle seem to also do things – what I’m not entirely sure.
3. Can you please include the legal notice which needs distributing with it, to comply with the open license.
4. Am I correct in assuming the Task Association is only relevant for WIMP tasks? Do I need to unassociate when calling from a module? The original docs hint at this, but don’t say how, they also aren’t clear on what happens if you’re not using a control block.

Speed of decompression is impressive, loading APD’s under ADFFS is the same speed as loading their uncompressed versions – a lot quicker than I expected.

Geez is GZSeek slow though! Back to the drawing board for me I think, need to recode and avoid seeks.

Yeah, if you seek backwards then it has to go back to the start and decompress everything again until it reaches the seek point. Seeking forwards will be better (it’ll just throw the data away), but still not as fast as seeking through an uncompressed file would be. The same is true with most compression algorithms.

1. What version of CLib is it dependent on? So I can add the pre-req test before loading.

It looks like the current requirements are:

• CallASWI 0.02 (or RISC OS 3.7+)
• FPEmulator 4.03
• CLib 5.17

2. It doesn’t seem to report an error, or set V if you try to GZOpen a file that doesn’t exist. Subsequent calls to SWI’s based on the rogue returned file handle seem to also do things – what I’m not entirely sure.

When writing the module I’ve assumed that ROL’s version doesn’t translate zlib errors to RISC OS errors. In the case of gzopen(), NULL (i.e. zero) should be returned if the file can’t be opened. If you tried using a null gzip file handle then strange things certainly would happen! (although zlib does check for null pointers on each function call, so it shouldn’t run the risk of trashing memory)

3. Can you please include the legal notice which needs distributing with it, to comply with the open license.

Sure, I’ll upload a new archive tonight.

4. Am I correct in assuming the Task Association is only relevant for WIMP tasks? Do I need to unassociate when calling from a module?

Yes and yes.

they also aren’t clear on what happens if you’re not using a control block.

The ZLib_Compress and ZLib_Decompress SWIs have flags to control task association of the control block that the module implicitly creates to handle compression/decompression (this is what the R1 workspace pointer gets used for). At the moment gzip file handles (and the control blocks contained within) don’t support task association, so are always in an un-associated state. However I can’t think of any reason why the ZLib_TaskAssociate SWI couldn’t be updated to support them (although I guess they’d have to be un-associated by default, for compatability with ROL’s version – assuming their version does behave in the same way)

When I get a chance I’ll have a go at wikifying the SWI calls and documenting which RISC OS errors will be returned when.

3. Can you please include the legal notice which needs distributing with it, to comply with the open license.

Sure, I’ll upload a new archive tonight.

Done.

Might be worth a stubsg compile to make it CLib neutral if being distributed with apps?

Just downloaded the standalone ZIP, but the license isn’t in it. Thanks for the other answers though, I’ll update my code accordingly.

Whoops – looke like I uploaded it without the .zip extension. Try again, it’s definitely there now.

Justin Fletcher has covered some aspects of the design of the RISC OS Ltd ZLib module here in his rambles — I don’t know whether that helps to add to the documentation already available. The ROL module seems to be pretty neutral as to what environment it might be running in.

I see that the licence in the distribution is the Castle shared source licence which applies to RISC OS itself. It is not very clear to me whether this allows me to distribute the module with a commercial application (either in a !System directory or inside the application directory). It would be useful to be able to do this because RISC OS 4.02 does not have the ZLib module, and that version of the OS is still quite widely used.

Is the ZLib module being included in RISC OS 5 these days? It’s not on my BeagleBoard, but that’s rather out of date, being RISC OS 5.18!
Thanks!

I don’t want to be “the ass”, but what stood out for me was “Porting the ZLib library”.
The page linked mentions ZLib, GZip, and PNG. What is the heritage of the code? Original work? Or this http://en.wikipedia.org/wiki/Zlib_License ?

I doubt either Justin Fletcher’s ZLib module for RISC OS Select, or Jeffrey’s ZLib module for RISC OS 5, have reimplemented the ZLib library in any way. In each case, they have embedded it in a RISC OS module, which means they have implemented SWI interfaces and done a few other bits and pieces. The ZLib licence which you point to allows developers to incorporate the ZLib code in products (such as a module) but does not require that any product be distributed under the same terms, nor that the source code be made available. Thus RISC OS Ltd are free to restrict Justin’s ZLib module so that it may only be distributed with RISC OS Select, and Jeffrey is free to impose the Castle shared source licence on his work (though equally he might have chosen to use the ZLib licence, unless he had to incorporate code from elsewhere in RISC OS of course). The liberal nature of the licence means it’s practically certain that the only original work will have been turning the code into a module. In fact, Justin implies this in his ramble when he refers to the lack of testing at one point, as the ZLib code itself would be thoroughly tested already.

Jeffrey had speculated about some aspects of the ROL ZLib module and I hoped Justin’s article might provide some clues. I’d still be interested to know whether ZLib is standard with RISC OS 5.19 or not, and whether it is OK to distribute it with an application.

Just read the text of the ZLib licence. It is….refreshingly simple, isn’t it? ;-)

My installation of RISC OS in RPCEmu is recent (UtilityModule is dated 23-Jan; the info box in the TaskManager says 14-Mar – shouldn’t these be the same?) and there is nothing in the standard ROM modules that looks like ZLib. There’s also nothing in the !System→Modules directories that I could see.

and whether it is OK to distribute it with an application.

Yes and No.

The base licence says this:
1.1 Subject to the restrictions set out in the remainder of this Licence Castle hereby grants to You a non-exclusive royalty free worldwide licence to use, copy and distribute the RISC OS source code and object code in any medium PROVIDED THAT you ensure that each copy You distribute incorporates the text of, or an Internet link to, this Licence.

Furthermore:
The phrase “Derivative Work” means any derivative work which is based on, or derived from, RISC OS or any part of RISC OS (whether by modification or translation) and any work subsequently derived from such Derivative Work or any part of it PROVIDED THAT such work is only intended to be used in conjunction with an embodiment (whether physical or emulated) of one or more versions of the ARM processor architecture.

So it looks like you can distribute the module if it is Castle licenced.

The problem is, you want to distribute it with your program.

More quoteage:
If any part of any Derivative Work distributed by You can reasonably be considered an independent and separate work then this Licence shall not apply to any such part where You distribute such part as an independent and separate work;

Therein lies the problem. “distribute such part as an independent and separate work”.

Would a non-linked support module within an archive (carrying the licence text, of course) be sufficient to consider your app and that module as not the same thing; or would including it in the same archive be too close?

Could be worth getting in touch with Castle to ask if a part of the RISC OS sources and something licenced under the Castle licence can be included in within an application intended for use under RISC OS.

I wouldn’t imagine there would be an objection if it is to support an application for RISC OS (note: IANAL and I’m not Castle nor able to speak on their behalf) however it is better to get clarification first rather than regret it later.

OK. I’m trying to compile some C using the ZLib SWIs direct, using ZLib 0.02 as supplied from Jeffrey’s web site. (I get the impression from the Select documentation that their version was supplied with some C stubs, but as I am writing from scratch rather than porting something it does not much matter.)

I’m assuming the SWI interface is as described in the RISC OS Select documentation

However, if I try to initialise a stream for inflating stuff, I get Z_VERSION_ERROR back straight away, which is very odd. Here’s the code:

    int *strm = M_ClaimMemory( 52 );
    int regs[10];
//    char *ver = "1.1.4";
    char *ver = "1.2.5.f";

    regs[0] = (int) strm;
    regs[1] = (int) ver;
    regs[2] = 52;
    if ( !Swi(0x53AC9,regs) )  E_SwiReport();
    ret = regs[0];

There is no error from the SWI, but at this point ret is equal to Z_VERSION_ERROR. The “Swi” function is just a wrapper round _kernel_swi — in effect

e = _kernel_swi( 0×53AC9 | 1<<17, regs, regs );
return !e

The M_ClaimMemory gets a block of 52 bytes and clears them to zero, so the stream block should be correctly set up.

As you can see, I have tried a couple of possibilities for “ver” though my reading of the zlib source code (the library, not the module) suggests that zlib itself only checks the first character of the version string. I am not sure whether that is right. Perhaps I need to pass it in another format?

Can anyone tell me where I am going wrong?

Thanks!

Jeffrey said

Yeah, if you seek backwards then it has to go back to the start and decompress everything again until it reaches the seek point. Seeking forwards will be better (it’ll just throw the data away), but still not as fast as seeking through an uncompressed file would be. The same is true with most compression algorithms.

I faced exactly this problem. I solved it with snapshots – store a small number of copies of the state of the decompressor at various input/output offsets. Seeking then reduces to restoring to the nearest previous state and discarding a much smaller number of bytes.

Much better than buffering the entire decompressed contents, which the system in question had been doing. This is especially true when the system features a number of overlaid ‘decompression’ filters, all with their own buffering.

It turns out my problem with the ZLib module was a simple mistake. I had misunderstood the documentation about the stream control block, and 56 bytes are required. The size of this block, passed in R2, has to be exactly right.

It would still be good to have some clarification about whether Jeffrey’s ZLib module may be distributed with applications (e.g. for use on RISC OS 4.02, RISC OS 5.16 etc.), and what conditions there might be. Obviously I could compile the zlib library myself and statically link it to the application, but that rather defeats the purpose of the module!

Thanks!

I don’t suppose anyone has a copy of the ROL ZLib Module documentation, the site is now blank and it appears it was excluded from WayBack.

It’s still on the site, they just moved everything around without setting up any redirects.

http://riscos.com/support/developers/riscos6/programmer/zlib.html

I don’t suppose anyone has a copy of the ROL ZLib Module documentation, the site is now blank and it appears it was excluded from WayBack.

More accurately the content of the site was covered by Wayback, but then someone had a a hissy fit and told Wayback not to publish the material – which is in the Wayback terms if you look.

The re-arrangements with broken links don’t help and any material that goes is just gone unless someone did a wget of the whole site before the broken links.