Android application to manipulate sd image from the site

aaaaa

aaaa

I think tar.bzip is the Unix “standard”. When I get archives of my website they are the same and it is quite a pain to need to extract something like 2GiB of stuff just to expose the underlying tar file just to extract a single file. Yes, a regular zip archive would be a lot more convenient.

In technical terms, GZIP offers better compression but at the expense of having no concept of an underlying file structure within. There’s the difference.

aaaa

Maybe compressing all file together can give slightly better compression ratio than compressing them separately though, but it must not be drastic.

I think that is the point. You aren’t compressing all the files together, you are essentially compressing one large file so the compression can work its best across all of the data.

Don’t know about LZW – but it may be possible to use it now?
http://web.archive.org/web/20080113182757/http://www.unisys.com/about__unisys/lzw/
This doesn’t exist on the live site, and a search for LZW returns nothing. It does explain, as a side effect, why GIF dropped out of favour extremely quickly. GIF uses LZW and… it was “encumbered”.

In a zip file, all file are compressed with the zlib inflate, and there is just a central directory and file header to say where each compressed file begin in the archive.

Yes. It maintains a file structure allowing easy access to a single file, either the nice SparkFS method or the simpler WinZip method.

then it can be mapped to a file system more easily than with a tar ball.

There is no difficulty with a plain uncompressed tar ball. The problem is the compression, but in Unix tradition, this isn’t tar…

Another consideration is that there is no mechanism to preserve file type metadata in a tar archive. There is when using a zip file, which means should a source archive be available as a zip, an update could be a drag and drop.
The reason my source is always out of date is that the CVS changes file cannot currently be merged into the existing source (without some sort of CVS running). Unpacking the full archive takes forever and it must all be unpacked. Then the older copy must be deleted which isn’t something that RISC OS can do quickly. Indeed I feel like it is pretty stressful on the filing system (and the SD card!).
How much nicer it would be to open a zip and drag the contents out to copy with the “Newer” option enabled!

aaaa

For typical source data, bz2 compresses much much better than GZIP or plain ZIP deflate. The typical difference in my experience is 20-50%. Plain old LZW is typically much worse than ZIP deflate. The original PKZIP had three different algorithms for compressing that were slightly modified LZW. So I think it is safe to say that deflate, being the younger algorithm, is likely to compress better.

A compressed filecore image holding the source tree could make sense if riscos supported either partition or somekind of loopback mounting system.

RISC OS has supported loopback filesystems for a long time (since RISC OS 3? 2? Arthur? Not sure exactly when). We call them “image filing systems”. E.g. SparkFS and ArcFS provide access to common archive/compression formats as image filing systems. DOSFS is a builtin image filesystem for accessing FAT partitions (I’m not sure if the PC card used to work the same way, but the old software PC emulator used to use a FAT disc image located on a RISC OS filesystem. Being able to easily and transparently access the contents of that image from within RISC OS was a useful feature. Nowadays the image filing system capabilities of DOSFS are mainly used to allow access to the FAT partition on the Raspberry Pi, i.e. the magic !Boot.Loader file).

For some reason FileCore itself isn’t natively supported as an image filing system, despite filecore disc images having a filetype assigned. But there are some third-party apps for it, e.g. FCFS

aaaa

aaaa

RISC OS has supported loopback filesystems…

Perfect summary :)
I hope that FCFS will be 32bit & ARMv7 compatible soon.

Maybe a compressed filecore image could replace the archive format.

FCFS can compress filecore images on the fly. A very powerful tool, but read only (for now), and 26bit only (for now).
It would be cool to have a free read only version and a commercial read/write one (subliminal message…).
And of course a tool to write back images to a real disc.

aaaa

aaaa

LZMA is obviously much more advanced than LZW. And it is not “somewhat similar to LZ77”, but uses an advanced version of LZ77 as one of its compression stages. Saying that LZMA is somewhat similar to LZ77 is the same as saying that bzip2 is somewhat similar to RLE.

aaaa

Is there a easy way of finding out which form of compression used – ie by referring to the start bytes of the file.
Bit like RISC OS programs putting a name at the beginning of their files.

aaaa

RISC OS has supported loopback filesystems for a long time (since RISC OS 3? 2? Arthur? Not sure exactly when).

RISC OS 3. But then RISC OS 3 was as big a change over RISC OS 2 as RISC OS 2 was over Arthur. Two skinny PRMs (Arthur) became four rather thicker PRMs (RISC OS 3).

(I’m not sure if the PC card used to work the same way, but the old software PC emulator used to use a FAT disc image located on a RISC OS filesystem.

Yes, I have somewhere an image file set up ready to roll Windows 95. All I need to do really is wire the x86 card to the Pi. Piece of cake, eh? ;-)

For some reason FileCore itself isn’t natively supported as an image filing system, despite filecore disc images having a filetype assigned. But there are some third-party apps for it, e.g. FCFS

FCFS isn’t a native FileCore image. There is a header of some description; and anybody who doesn’t want to waste space will probably have used the version that skips unused blocks (maybe even with compression applied). Sure, a FileCore image can be reconstructed from it, that’s the whole point, but chances are it in itself isn’t a straight FileCore image.

Yes, I’d like to see FCFS as well – I think it has application with the newer hardware, not least as a method of quickly backing up SD cards and such while also allowing full access to the contents.

As far as the RISC OS build process is concerned – I would like to believe that somewhere in ROOL is a hub with a bunch of Pis and Beagles and Pandas hanging off of it, but to be honest I would be more inclined to believe that they have a private build of the DDE that runs on something else (Linux?) that builds everything and packages up the relevant files more or less automatically. Whether or not we can ask for a Zip or FileCore image or anything else depends not so much on epic grovelling, but more the capabilities of the build platform. It probably isn’t hard to add support for Zip files, it’s a fairly standard format. However support for zip files with filetypes on a non RISC OS platform may be a fair bit harder…
[it’s this in reverse as well, a basic PC can extract the entire Pi code in maybe five minutes tops; which would be great if the result wasn’t a load of gibberish…]

Something like crincklyer can easily reach a /16 size ratio on most exe.

Windows EXEs? T0o0 0b0e0 0f0a0i0r0,0 0t0h0e0r0e0 0i0s0 0a0 0L0O0T0 0o0f0 0r0u0b0b0i0s0h0 0i0n0s0i0d0e0 0t0h0e0 0a0v0e0r0a0g0e0 0E0X0E0 0f0i0l0e0, such as strings in UTF-16 as indicated by the previous line. ;-)

Is there a easy way of finding out which form of compression used

Of course. Bytes six to nine of the per-file header indicates this. https://users.cs.jmu.edu/buchhofp/forensics/formats/pkzip.html

I can’t find any info regarding how the RISC OS filetype is stored, maybe the InfoZip source would shed more light on this…if knowing such a thing is necessary – though I would look at the Extra Field data as a first guess.

aaaaa

aaaa

As far as the RISC OS build process is concerned – I would like to believe that somewhere in ROOL is a hub with a bunch of Pis and Beagles and Pandas hanging off of it, but to be honest I would be more inclined to believe that they have a private build of the DDE that runs on something else (Linux?) that builds everything and packages up the relevant files more or less automatically.

It’s a bit less extravagant than that. They’ve got a Unixy server which runs some autobuilder scripts I put together. All the code compiling is done using the standard build pipeline, running in a slightly tweaked version of RPCEmu. That handles the standard ROM + hard disc image download (which are mainly just zipfiles created from within RPCEmu). For the Pi RC images, I believe it’s very much a manual process, mainly due to the lack of automated (and cross-platform) tools to manipulate FileCore images.

However support for zip files with filetypes on a non RISC OS platform may be a fair bit harder…

GCCSDK builds a copy of zip which supports filetypes (via the standard ,xxx suffix)

I can’t find any info regarding how the RISC OS filetype is stored, maybe the InfoZip source would shed more light on this…if knowing such a thing is necessary – though I would look at the Extra Field data as a first guess.

Currently i don’t know if sparkfs support storing riscos attributes, i asked previously in this thread, but apparently no.

SparkFS would be pretty useless if it didn’t support storing RISC OS attributes.

As Rick guessed, the extra attributes are stored in the extra field data. See the set_extra_field() function in acorn/acornzip.c, and the extra_block struct definition in acorn/riscos.h, in the Zip sources.

aaa

Yeah, that should work.

aaaa