Building RISC OS
108 posts, 15 voices
Steve Revill (20)
1361 posts
|
LOL. SparkFS incorrectly used the RISC OS filetype &DDC for both its own ‘SparkFile’ file format and for zipfiles. Zipfiles have had an official allocation of &A91 for a long, long time (since before the turn of the century?). IIRC, most software was updated to reflect this back in 2005/2006. Certanly, newer SparkFS versions reflect it. Our MimeMap update is somewhat behind the curve; most people who’ve been tracking the various unofficial MimeMap distributions have had this change in place for many years…
Wrong. The ‘SparkFile’ format is very different indeed to the ‘Zip’ file format. |
Frank de Bruijn (160)
228 posts
|
Indeed. I’ve maintained my own MimeMap file for years. It’s based on versions created by Joseph Heenan, Richard Chiswell, Gary Locock and Tim Hill. There’s a note in its history section, dated 21/09/2005 and written by Gary Locock, about separating zip/a91 from the other archive types. SparkFS will actually change the file type of a &DDC typed zip to &A91 if you drop it on the icon bar icon. |
Chris Hall (132)
3554 posts
|
I will compare what !UnTarBZ2 1.05 produces, with the fixed tar, as downloaded here The 7Zip method uses some juggery pokery (mimemap?) within VRPC to attribute file types to popular types of files without the ,hex suffix. One strange effect is that directories generated a different result when obtained as an object by: Under 7Zip the object file type inferred from the load address reports as &FFF It is not normally visible under RISC OS to see such information about directories and so it probably doesn’t matter. I have corrected this (so that the comparisions will match) but don’t know whether this matters. Both methods produce 17816 files (115921549 bytes) so that’s a good start. Comparison: One file appears to be wrong (it has a different CRC, same file length, and the filetype is now correct): [Also two filenames are mungled: so testing complete. Result PASS. Hooray! Hope this helps. |
Rick Murray (539)
13840 posts
|
It’s all there in the !Run file, but…
Yup, mine does – the ‘spark’ on the directory is a ‘Z’ for zips, though to be honest the two look remarkably similar. I guess the problem isn’t helped by SparkFS being somewhat agnostic of the format – if it can open it, it will, and nobody is supposed to care much beyond that. :-)
Not that Zip files don’t themselves have plenty of variation! |
nemo (145)
2546 posts
|
Not 1.37 – it does the exact opposite! I don’t even have a proper icon for A91. Mind you that’s dated February 2000, which I gather is prior to the Great Zip Flip. Sigh. So I’m not confused, I’m just very, very old. I shudder to think how many DDC zips I have scattered among the seven hard discs on my iconbar and the hundreds of CDs, PDs, Jaz discs and opticals backing up the gigabytes of artwork I’ve produced over the decades. |
|
nemo (145)
2546 posts
|
VRPC has an extensions file inside the HostFS2 plugin directory – that does the equivalent of MimeMap at the HostFS end. Note that VRPC’s HostFS is buggy and slow – shift-clicking on any Windows file in the emulator causes HostFS to change the file extension of the file to |
Colin (478)
2433 posts
|
Chris. Thanks – I appreciate the effort. It looks like 7zip has problems with filenames with commas in them. Don’t know why 7zip hasn’t typed the doc file unless the comma in the name confuses it. The filenames in untarbz2 are not munged 7Zip has munged its version so thats ok – phew.
These things are sent to try us. Was it any faster? I wasn’t sure if the speed was because you were on a slow machine |
|
Chris Hall (132)
3554 posts
|
Don’t know why 7zip hasn’t typed the doc file unless the comma in the name confuses it. It can be either text or a Word file. It is VRPC that uses comma for filetypes and access permission. Was it any faster? Yes. I wanted to catalogue the 7zip result on VRPC but the 28M wimpslot is not enough to catalogue 18000 files (by about a factor of 1.5). All the time is taken copying stuff across the network to catalog it on an Iyonix. I may tweak !Cat to avoid images and use memory more efficiently to save this bit of the task. |
|
Colin (478)
2433 posts
|
Steve
I’m making it all optional.
in The beginner’s guide to ROM builds it states that you should use EraseCVS. Am I right in thinking that when downloading CVS repository archive as opposed to a source code archive you don’t want to use EraseCVS but for the source code archives you do? Also do the CVS repository archives want any filetyping/,hex removal at all. If this is the case I can add a button to switch between the 2 defaults. |
|
Chris Hall (132)
3554 posts
|
Another test using the fixed !UnTarBZ2, compared against 7Zip. This time I tested ‘DiscDev’: unzip time (untarbz2): 15m Files with names containing commas but where the text following the comma is not a valid hex value Some filenames are different between 7zip and untar but the contents match – the name is therefore unchecked: Some files are left as text by untar but are filetyped by VRPC: These failures are all defined as acceptable. There are no other failures and the test is a PASS (again). It does identify some sloppy checking of the CVS where filenames containing control characters, commas etc. have been allowed through. If you are building roms on the VRPC this may cause difficulty? The VRPC platform was one of the fastest for programme development until the Pandaboard came along. I think that is sufficient testing of the new !UnTarBZ2 unless anyone thinks differently? Detail below: |
|
Steve Revill (20)
1361 posts
|
Yes, historically we always did an EraseCVS step on our checked-out sources before doing a build because there have been components who would end up with bits of CVS clutter inside their built target (for example, you might have some CVS directory and files within the resources for a module or application in ResourceFS), I’ve no idea whether this is still a problem but I felt cautious when creating UnTarBZ2. The full sequence would normally be:
Our source code tarballs essentially do the first three steps. When you decompress the archive, I’m assuming it’s going to be used there and then for builds, hence having EraseCVS pass as default. However, I can see it’d be more useful to make it optional so you have the ability to keep the original checked-out sources with all the CVS stuff in it – this is especially helpful to ROOL if you want to submit any changes back because then we know exactly what revision of everything your work is based upon.
EraseCVS shouldn’t do anything to the actual CVS repository because the stuff it removes is the stuff that CVS adds to the checked-out sources, not the type of things you’d find in the repository itself. Although I suppose there’s a minute danger it could cause problems if there were a directory called CVS within the repository, but I’m not even sure you’d be allowed to commit/import that. |
|
Colin (478)
2433 posts
|
Blimey it took me longer to rewite the frontend to add options than it did to write the untar program in the first place. Anyway here’s UnTarBz2_106a.zip I need to rewrite the help file but you can let me know if it does what you want. I’ve set the defaults for a first time user who has downloaded the source code and wants to compile it. I haven’t added a ‘CVS archive’ default button but could if you wanted. |
|
Chris Hall (132)
3554 posts
|
I’ll do some more testing then … |
Chris Gransden (337)
1207 posts
|
Would it be possible to add an option to the ‘untar’ command to exclude the ‘CVS’ folder from the extract. This would mean much less files to extract which should also speed up the extract. I did a diff between the ‘DiscDev’ folder extracted with ‘cvs’ on linux over sunfish and the extracted contents of the downloaded archive on RISC OS. All files were present and correct. |
|
Colin (478)
2433 posts
|
Obviously possible and I had thought about it but without the erasecvs sources I wasn’t exactly sure what was being erased. CVS folders will be erased but is there a check to confirm that they belong to CVS if you see what I mean. So it was a case of when in doubt don’t touch it :-) |
|
Chris Hall (132)
3554 posts
|
Possible problem with 1.06a. With the options set as under Note that any remaining commas in filenames are shown below as character B8 which looks much like a comma (as extraneous commas in a CSV file are unhelpful). I tried decompressing on the PC side using 7zip to a NAS but (apart from taking 19min rather than 3min to HDD) found that reading the NAS on my Iyonix meant that the filenames had a ,hex,v or ,v suffix. I therefore went back to using the tried and tested method of decompressing on the PC side using 7zip to the HDD and looking at the files under VRPC (or from my Iyonix over the network in the Public folder under VRPC) but the files still had the unwanted suffix of ‘,v’ or ‘,hex,v’. Puzzled – what is this ‘,v’ that is being added to every file? It prevents me testing the decompression of the ‘full cvs repository’ and it causes a decompression of the full cvs repository on a VRPC (a valid platform for software work) to be access-prevented under VRPC as filenames ending in ‘,v’ are seen as zero length, read access prevented. Decompression using untarbz2 still leaves the filenames with an unwanted suffix. It looks like the source code tarball for the whole CVS repository has not processed correctly and has left the ‘,v’ suffix on the files today. |
|
Chris Gransden (337)
1207 posts
|
How likely is it that there will be a folder called CVS in the source tree that isn’t a ‘cvs’ folder. I imagine it would confuse ‘cvs’ anyway. |
|
Chris Hall (132)
3554 posts
|
@Jeffrey: Archives to test: Any and all source archives from here You did mean including the ‘Whole CVS Repository’ source archive didn’t you? This one seems to have filenames ending in ‘,v’ which makes the files zero length and read-access-prohibited under VRPC making testing impossible. |
|
Chris Gransden (337)
1207 posts
|
That archive is a copy of the actual ‘CVS Repository’. You’d have to check the files out first to do anything useful with them. The filenames you’re seeing are how files are stored in the repository when something is checked in. |
|
Chris Hall (132)
3554 posts
|
Anyway I have done another test using !UnTarBZ2 version 1.06a, this time with the IOMDHALDev archive. download time: 20s unzip time (untarbz2): 12m Analysis is using !Cat version 0.19 Files with names that contain a comma (after processing the ,hex suffix) can have their One filename is different between 7zip and untar but the contents match One filename is different between 7zip and untar but the contents match – the name is therefore unchecked (this file is of zero length so has no content): These failures are all defined as acceptable. There are no other failures and the test is a PASS (again). |
|
Colin (478)
2433 posts
|
I’ve done a quick test and skipped “.CVS.” in the path and “.CVS” leaf name. Also skipped “./cvstag” and “./cvsignore” leaves and that reduces the untar/erasecvs phases to 2min for tungstendev on an iyonix. I also discovered that eraseCVS doesn’t erase “/cvsignore” So I’ll tidy untarbz2 up a bit and post an update. |
|
Chris Hall (132)
3554 posts
|
Snap! |
|
Colin (478)
2433 posts
|
Ok here’s an updated version UnTarBz2_106a1.zip this now does from decompress to files in 5 mins with cvs file removal. Chris. regarding the ,v files. They are cvs working files and the ,v shouldn’t be removed. There are 2 types of archive on the rool site ‘source code archives’ and ‘cvs repository archives’. Hopefully confirm what I’m saying (then I can add an button to set it) but I think the repository downloads want downloading without any of the options ticked. ie everything is just typed as text, and ,hex files don’t have the ,hex removed. |
|
Chris Hall (132)
3554 posts
|
Ah! That explains it. However the ,v suffix was not well conceived. If you are using a VRPC platform for your software development, then all the files with a ‘,v’ suffix (and that is all of them in the CVS repository!) will appear to all RISC OS software on VRPC as files of zero length that have no read access. If the checking out process can still function (on the Windows side the files have the correct size and read access, so checking out is still possible if the software runs under Windows rather than RISC OS) under these circumstances, I would be quite surprised. I’ll do some checking now for version 1.06a1 … |
Jeffrey Lee (213)
6048 posts
|
No, the method of using ,xxx extensions for representing RISC OS filetypes on Unix/Windows machines (and in particular VRPC’s implementation of it) was not well conceived. Like a lot of other Unix stuff, CVS predates RISC OS – wikipedia reckons the first implementation was devised in 1986 (although the first official release was in 1990). |
