Building RISC OS

Clearly, RISC OS is overburdened with resource, given we can spend man hours having this discussion… At least there’s nothing important we could all be getting on with. :(

Copyright is not a credits roll. […] Acknowledging contributors is unrelated to declaring the copyright owner.

That’s true, but you usually find the two together. There probably ought to be a bit that says “This software uses <random FOSS licence> software, blah blah” as well; though all that stuff is probably better printed under the RISC OS banner (at startup) given the variety of FOSS licences used, as opposed to polluting the Switcher information (it would, like, triple its size!).

In that case it is 70 years from the end of the calendar year in which the work was created.

Which is why a date range is useful – this version of RISC OS is not eleven years old.

Copyright owner != contributors.

It would be nice for a roll call of contributors – some versions of RISC OS have that as an easter egg. However it is exactly that – an easter egg, and not a necessity. I am aware that if I write any code intended for RISC OS, then it will be “absorbed” into the ROOL codebase and become covered by the Castle licence. In effect, I am voluntarily giving my rights over my code to Castle. As such modifications are intended for the betterment of RISC OS, as far as I’m concerned it’s a Win-Win situation. They get the code, we get the binary, and it’s all a lot simpler to administer.

Steve:

given we can spend man hours having this discussion… At least there’s nothing important we could all be getting on with. :(

Hours? I don’t think I’ve spent one, yet, never mind a plural word. ;-) I’m actually unpacking parts of !Configure, this time using SparkFS. It’s still kinda slow, but at least it stands a chance of not choking on long paths. So I can write this on the eeePC while waiting on the Pi doing its stuff. ;-)

BTW – is src-disc-dev/5/21 supposed to have the entire RISC OS source code within it? I was under the (possibly misguided) belief that it was just the non-ROM stuff present in the Harddisc4 archive…?

Did anyone volunteer to give an updated UnTarBZ2 some testing? I’ve not noticed anything.

Did anyone volunteer to give an updated UnTarBZ2 some testing? I’ve not noticed anything.

Yes. I will. And I can produce a diff between the results using two different versions (using !Cat). Point me to the bugged and ‘clean’ untar utility and some tarbz2’s to check. I will then also compare it to the results using 7Zip.

Just download untarbz2 and tar.zip put the tar app file from tar.zip in !Untarbz2.Tools

BTW – is src-disc-dev/5/21 supposed to have the entire RISC OS source code within it? I was under the (possibly misguided) belief that it was just the non-ROM stuff present in the Harddisc4 archive…?

That’s not (or shouldn’t be) the entire OS source code. However, it will be most of it, as a lot of components export headers (SWI names/numbers, error numbers, flag/option settings, etc.) which are required in order to build the disc-based components.

Did anyone volunteer to give an updated UnTarBZ2 some testing? I’ve not noticed anything.

Yes. I will. And I can produce a diff between the results using two different versions. Point me to the bugged and ‘clean’ untar utility and some tarbz2’s to check. I will then also compare it to the results using 7Zip.

Broken untarbz2: https://www.riscosopen.org/tarballs/untarbz2.zip

Fixed untarbz2: Probably best to let Colin point you at that, I’m not sure where his latest version is

Archives to test: Any and all source archives from here

I’m guessing but 7Zip may not remove the CVS components

I’m guessing but 7Zip may not remove the CVS components

SparkFS didn’t, and it also made a bit of a mess with filetypes. Luckily I was only extracting a few things to build manually so it wasn’t a big issue.

Can’t say for what parts of the OS code were present as I’m out in the garden taking a break (7up has released a new cherry flavour drink, it’s nice, but then I’m a sucker for Sakura ;) ). What I can say is that I saw the kernel files in kernel (you know, Arthur2, MOSdict, OSthis-and-that…). I wouldn’t imagine they are empty files; and given the BCM source tar file is about 128Mb while the disc tar file is closer to 153Mb…

“CVS components” (i.e. CVS folders) aren’t removed by untarbz2 either – they’re removed by !EraseCVS

Filetype extensions on the other hand, are handled by untarbz2. And 7Zip most certainly won’t remove them, as there’s no way of storing the relevant filetype metadata on the files on Windows (or at least no way that anyone’s bothered to implement within 7Zip)

I have used 7Zip successfully to untar source files ready for building a rom. My recollection is that I unzipped them into the public directory of Virtual RiscPC and looked at them from the Iyonix via the network and the file types were there.

“CVS components” (i.e. CVS folders) aren’t removed by untarbz2 either – they’re removed by !EraseCVS

< pantomime dame mode>
Oh yes it does.

It includes eraseCVS and runs it at the end.:-)

“CVS components” (i.e. CVS folders) aren’t removed by untarbz2 either – they’re removed by !EraseCVS

< pantomime dame mode>
Oh yes it does.

It includes eraseCVS and runs it at the end.:-)

Ah, fair enough.

“CVS components” (i.e. CVS folders) aren’t removed by untarbz2 either – they’re removed by !EraseCVS

UnTarBZ2 does say:

EraseCVS (0.08, 11 Aug 2000)
Erasing CVS dirs from 'ADFS::FastHard4.$.Test.Test1.Result1'

The first test is with UntarBZ2-bugged (1) against UntarBZ2-fixed (2) against 7Zip (W). To compare the results I have used !Cat 0.18 with the option ‘ShowAll’ , ‘SaveAsCSV’, ‘CalcCRC’ and ‘Entries undated’ (and no image filing systems on the icon bar). This ignores the file and directory datestamp but checks file contents (via a CRC of the first few megabytes of each file) and file type. I have tested it under RISC OS 5.20 (running Shared C Library 5.59 in case that matters) on an Iyonix.

It is a little quicker to use 7Zip to unzip the files and then copy them over the network to the Iyonix rather than use UnTarBZ2. Much much quicker to use 7Zip (a few minutes) and then compile them on Virtual RiscPC (no copying required and 7Zip is much faster than UnTarBZ2 (about 40 mins)).

BCM2835Dev CVS tested: 22M (tar/bz2) → 130M (tar) → 18284 files
Results:
Comparision 1:W (we know 1 is buggy so don’t need to repeat this comparision)

Only gets 8309 files (114666216 bytes) 
vice 18284 files (115344672 bytes) 
(because of EraseCVS, so I've removed 
that from the bug-fixed version [so that I can compare with 7Zip 
which does not do EraseCVS], may do a comparison 
between 1 & 2 later with EraseCSV on both)

Comparision 2:W (testing the fixed version)

First check 18284 files (115444000 bytes)
vice 18284 files (115344672 bytes) so the detail may matter:
Well it fails - some files are not set to allow read 
access, for example
castle.RISCOS.Sources.HWSupport.NVRAM.
  Utils.Lazarus.MakeImage.Resources.2208,896TS/doc
which is a file of zero length set to prevent read access (as
seen via the network and by VRPC and thus copied as such) but a file of 97k 
with normal access (as generated by UnTarBZ2 and 7Zip (viewed 
on the Windows side)
Manually copying that single file does get to 18284 files (115444000 bytes)
[from 7Zip] but a lot of directory entries have an incorrect filetype of FFD
for directories (should be FFF) but correcting that as well:
 Comparing files FxdUnTar.csv and 7ZIP.CSV
..... FxdUnTar.csv
    1:  Directory of ADFS::FastHard4.$.Test.Test1.Result2.BCM2835Dev
    2:  .bsd,FFFFFF00,0,Directory
..... 7ZIP.CSV
    1:  Directory of ADFS::FastHard4.$.Test.Test1.ResultW.BCM2835Dev
    2:  .bsd,FFFFFF00,0,Directory
..... FxdUnTar.csv
 4318:  ------------------------------------------,0,0
 4319:  .castle.RiscOS.Sources.Desktop.Wimp.TestO.[ˆ‰£Š‹],FFFFFF00,0
 4320:  .castle.RiscOS.Sources.Desktop.Wimp.UrsulaLog,FFFFFF00,0,CRC:932F
..... 7ZIP.CSV
 4318:  ------------------------------------------,0,0
 4319:  .castle.RiscOS.Sources.Desktop.Wimp.TestO.[êëúèï],FFFFFF00,0
 4320:  .castle.RiscOS.Sources.Desktop.Wimp.UrsulaLog,FFFFFF00,0,CRC:932F
..... FxdUnTar.csv
 7022:  .castle.RiscOS.Sources.HWSupport.DualSerial.Docs.HALCalls,FFFFFF00,0,CRC:689A
 7023:  .castle.RiscOS.Sources.HWSupport.DualSerial.Docs.serial/htm,FFFFFF00,0,CRC:7C59
 7024:  -------------------------------------------------,0,0
..... 7ZIP.CSV
 7022:  .castle.RiscOS.Sources.HWSupport.DualSerial.Docs.HALCalls,FFFFFF00,0,CRC:689A
 7023:  .castle.RiscOS.Sources.HWSupport.DualSerial.Docs.serial/htm,FFFFAF00,0,CRC:7C59
 7024:  -------------------------------------------------,0,0
..... FxdUnTar.csv
 7381:  .castle.RiscOS.Sources.HWSupport.NVRAM.Docs.STB400,FFFFFF00,0,Directory
 7382:  .castle.RiscOS.Sources.HWSupport.NVRAM.Docs.STB400.2501822/html,FFFFFF00,0,CRC:A428
 7383:  .castle.RiscOS.Sources.HWSupport.NVRAM.Docs.STB400.CVS,FFFFFF00,0,Directory
..... 7ZIP.CSV
 7381:  .castle.RiscOS.Sources.HWSupport.NVRAM.Docs.STB400,FFFFFF00,0,Directory
 7382:  .castle.RiscOS.Sources.HWSupport.NVRAM.Docs.STB400.2501822/html,FFFFAF00,0,CRC:A428
 7383:  .castle.RiscOS.Sources.HWSupport.NVRAM.Docs.STB400.CVS,FFFFFF00,0,Directory
..... FxdUnTar.csv
11583:  .castle.RiscOS.Sources.Lib.RISC_OSLib.Doc.fwrite,FFFFFF00,0,CRC:05EC
11584:  .castle.RiscOS.Sources.Lib.RISC_OSLib.Doc.LFSv1¸5,FFFFFF00,0,CRC:7511
11585:  ------------------------------------------,0,0
..... 7ZIP.CSV
11583:  .castle.RiscOS.Sources.Lib.RISC_OSLib.Doc.fwrite,FFFFFF00,0,CRC:05EC
11584:  .castle.RiscOS.Sources.Lib.RISC_OSLib.Doc.LFSv1,FFF00500,0,CRC:7511
11585:  ------------------------------------------,0,0
..... FxdUnTar.csv
20250:  .castle.RiscOS.Sources.Video.Render.Fonts.ROMFonts.Doc.GlyphNames.glyphlist/txt,FFFFFF00,0,CRC:C485
20251:  .castle.RiscOS.Sources.Video.Render.Fonts.ROMFonts.Doc.GlyphNames.unicodegn/html,FFFFFF00,0,CRC:37E1
20252:  .castle.RiscOS.Sources.Video.Render.Fonts.ROMFonts.Doc.GlyphNames.zapfdingbats/txt,FFFFFF00,0,CRC:25C2
..... 7ZIP.CSV
20250:  .castle.RiscOS.Sources.Video.Render.Fonts.ROMFonts.Doc.GlyphNames.glyphlist/txt,FFFFFF00,0,CRC:C485
20251:  .castle.RiscOS.Sources.Video.Render.Fonts.ROMFonts.Doc.GlyphNames.unicodegn/html,FFFFAF00,0,CRC:37E1
20252:  .castle.RiscOS.Sources.Video.Render.Fonts.ROMFonts.Doc.GlyphNames.zapfdingbats/txt,FFFFFF00,0,CRC:25C2
..... FxdUnTar.csv
21802:  -------------------------------------------------------------------,0,0
21803:  .castle.RiscOS.Utilities.Autobuild.ABRelease.Resources.Generic.Licence_v1_1/pdf,FFFFFF00,0,CRC:7197
21804:  .castle.RiscOS.Utilities.Autobuild.ABRelease.Resources.Generic.ReadMe/txt,FFFFFF00,0,CRC:576E
..... 7ZIP.CSV
21802:  -------------------------------------------------------------------,0,0
21803:  .castle.RiscOS.Utilities.Autobuild.ABRelease.Resources.Generic.Licence_v1_1/pdf,FFFADF00,0,CRC:7197
21804:  .castle.RiscOS.Utilities.Autobuild.ABRelease.Resources.Generic.ReadMe/txt,FFFFFF00,0,CRC:576E
..... FxdUnTar.csv
22094:  ---------------------------------------------------------------------------,0,0
22095:  .castle.RiscOS.Utilities.Autobuild.ABRelease.Resources.Tungsten.soft.Changes/html,FFFFFF00,0,CRC:9D4F
22096:  .castle.RiscOS.Utilities.Autobuild.ABRelease.Resources.Tungsten.soft.CVS,FFFFFF00,0,Directory
..... 7ZIP.CSV
22094:  ---------------------------------------------------------------------------,0,0
22095:  .castle.RiscOS.Utilities.Autobuild.ABRelease.Resources.Tungsten.soft.Changes/html,FFFFAF00,0,CRC:9D4F
22096:  .castle.RiscOS.Utilities.Autobuild.ABRelease.Resources.Tungsten.soft.CVS,FFFFFF00,0,Directory
..... FxdUnTar.csv
23790:  .mixed.RiscOS.Sources.HWSupport.FPASC.doc.DynRound,FFFFFF00,0,CRC:497C
23791:  .mixed.RiscOS.Sources.HWSupport.FPASC.doc.fpe400/zip,FFFFFF00,0,CRC:9F58
23792:  ------------------------------------------,0,0
..... 7ZIP.CSV
23790:  .mixed.RiscOS.Sources.HWSupport.FPASC.doc.DynRound,FFFFFF00,0,CRC:497C
23791:  .mixed.RiscOS.Sources.HWSupport.FPASC.doc.fpe400/zip,FFFDDC00,0,CRC:9F58
23792:  ------------------------------------------,0,0
..... FxdUnTar.csv
24054:  ----------------------------------------------------------------------,0,0
24055:  .mixed.RiscOS.Sources.HWSupport.USB.Controllers.DWCDriver.dwc.doc.DWC_otg_linux_sw_relnotes/pdf,FFFFFF00,0,CRC:9EFE
24056:  .mixed.RiscOS.Sources.HWSupport.USB.Controllers.DWCDriver.dwc.doc.DWC_otg_linux_sw_user/pdf,FFFFFF00,0,CRC:2C31
24057:  ------------------------------------------------------------------,0,0
..... 7ZIP.CSV
24054:  ----------------------------------------------------------------------,0,0
24055:  .mixed.RiscOS.Sources.HWSupport.USB.Controllers.DWCDriver.dwc.doc.DWC_otg_linux_sw_relnotes/pdf,FFFADF00,0,CRC:9EFE
24056:  .mixed.RiscOS.Sources.HWSupport.USB.Controllers.DWCDriver.dwc.doc.DWC_otg_linux_sw_user/pdf,FFFADF00,0,CRC:2C31
24057:  ------------------------------------------------------------------,0,0

So probably not a good idea to have filenames with commas in them:
castle.RISCOS.Sources.HWSupport.NVRAM.Utils.Lazarus.MakeImage.Resources.2208,896TS/doc

Also the ‘fixed’ UnTarBZ2 gets some file names wrong (see above) but these seem to have odd characters in the filename as well:
.castle.RiscOS.Sources.Desktop.Wimp.TestO.[êëúèï]
.castle.RiscOS.Sources.Lib.RISC_OSLib.Doc.LFSv1¸5

Also some /html files are given a text file type:
.castle.RiscOS.Sources.HWSupport.DualSerial.Docs.serial/htm
.castle.RiscOS.Sources.HWSupport.NVRAM.Docs.STB400.2501822/html
.castle.RiscOS.Sources.Video.Render.Fonts.ROMFonts.Doc.GlyphNames.unicodegn/html
.castle.RiscOS.Utilities.Autobuild.ABRelease.Resources.Tungsten.soft.Changes/html

and a zip file:
.mixed.RiscOS.Sources.HWSupport.FPASC.doc.fpe400/zip (should be DDC)

and pdf files:
.castle.RiscOS.Utilities.Autobuild.ABRelease.Resources.Generic.Licence_v1_1/pdf (should be ADF)
.mixed.RiscOS.Sources.HWSupport.USB.Controllers.DWCDriver.dwc.doc.DWC_otg_linux_sw_relnotes/pdf
.mixed.RiscOS.Sources.HWSupport.USB.Controllers.DWCDriver.dwc.doc.DWC_otg_linux_sw_user/pdf

Hope this helps .. there is more work to do on UnTarBZ2.

Testing complete. Result FAIL

Hope this helps .. there is more work to do on UnTarBZ2.
Testing complete. Result FAIL

Looks like a 100% pass to me. 7Zip has changed characters truncated names (files with commas in them) and added filetypes which are not specified in the file. eg obey files etc have ,feb attached to the end so is supposed to be typed. any file that isn’t typed with ,abc is a text file. Anything typed ffd is typed that way in the file.

That file in Wimp.TestO is certainly a tricky one. The sources on my Iyonix (which are fetched from CVS using Linux, then copied over using Sunfish) list it as being [<>£v^], except with the Wimp arrow characters instead of ASCII (characters 136, 137, 138, 139). I think it’s safe to assume this is correct.

For the HTML/PDF/zip files, those do actually end up as the relevant types for me. I think this is intended, so perhaps that is a bug in untarbz2. I believe the reason they’re showing for me is because Sunfish is using the mimemap file to assign filetypes to files which have standard Unix/Windows extensions. I’ve always thought this was the correct thing to happen, as otherwise we’d have ,xxx style extensions on the files in CVS. And without mimemapping being applied, you’ll end up with things like the Castle License PDF appearing as a text file in ROM download archives. However, a quick test with John Tytgat’s native CVS client (which is the only one I know of that works, at least on 32bit machines, and is the version recommended by ROOL) reveals that (at least by default) it doesn’t use the mimemap file. Maybe ROOL can provide some information here, on what the CVS/NFS implementations that were used in Acorn/Castle/etc. used to do in this situation.

For filenames containing commas, that’s definitely a case of whatever software you’re using to copy the files to RISC OS messing up and removing stuff that it shouldn’t – perhaps falsely interpreting it as a “,xxx” style extension (although, yes, we should probably avoid adding files to CVS with names which could be misinterpreted as filetypes!)

That file in Wimp.TestO is certainly a tricky one. The sources on my Iyonix (which are fetched from CVS using Linux, then copied over using Sunfish) list it as being [<>£v^], except with the Wimp arrow characters instead of ASCII (characters 136, 137, 138, 139). I think it’s safe to assume this is correct.

That is how my tar downloads it – the display on this forum mucks it up.

.castle.RiscOS.Sources.Lib.RISC_OSLib.Doc.LFSv1¸5

is correct

HTML/PDF/zip files

Well the tar file doesn’t have the ,xxx extension on these files so presumably its a mapping problem creating the file if they are supposed to be typed. The only way to know if the types are correct is for every file to have the ,xxx extension.

Obviously if the nice Mr rool says the files should be typed and doesn’t want to/can’t change the file types in the archive then I can add mime mapping to the program easily enough – not my prefered option though, not explicit enough.

If someone can define to me the errors that are acceptable – like not correctly generating filetypes and how I handle files containing commas followed by a slash (which under VRPC sets the access permission to list but not read) and filenames which contain top bit set or control characters (which are undefined) – then I could say that the errors it produced were within acceptable bounds. Alternatively I can test further and just report the errors and leave others to judge whether they are acceptable.

tar.zip (now contains the modified !untarbz2)

Ok I’ve added mimemap mapping for .ext files. I’ve also speeded it up so that the whole process – checksum to erasecvs – now takes about 8.5 mins to dearchive the TungstenDev archive on an iyonix.

What is acceptable – my view.

The filenames used in the archive originated for use on RISC OS machines and as such should be converted from the archive unmodified. ,hex files should have ,hex removed and filetype set. Converting .ext files is debatable as its dependant on the mimemap file having a conversion and there may be a good reason for a .html file to be a text type. This isn’t a problem with ,hex files – but I’ve added it anyway. I don’t think access rights should be added when dearchiving. To include metadata it would be better to have a file name format which included the load and exec addresses.

Don’t use VRPC so don’t know about the ,/ convention but it’s unlikely to appear in the ROOL archives.

The most important is the correct filenames and types but as I say whether or not index/html should be typed as html or text is ambiguous index/html,fff is not.

So as I see it and errors due to filenames is a bug in 7zip. I would expect filetypes to now be the same and permissions are not supported by UntarBZ2

Like to give it another try?

Thanks

.castle.RiscOS.Sources.HWSupport.DualSerial.Docs.serial/htm
.castle.RiscOS.Sources.HWSupport.NVRAM.Docs.STB400.2501822/html
.castle.RiscOS.Sources.Video.Render.Fonts.ROMFonts.Doc.GlyphNames.unicodegn/html
.castle.RiscOS.Utilities.Autobuild.ABRelease.Resources.Tungsten.soft.Changes/html
.castle.RiscOS.Utilities.Autobuild.ABRelease.Resources.Generic.Licence_v1_1/pdf (should be ADF)
.mixed.RiscOS.Sources.HWSupport.FPASC.doc.fpe400/zip (should be DDC)
.mixed.RiscOS.Sources.HWSupport.USB.Controllers.DWCDriver.dwc.doc.DWC_otg_linux_sw_relnotes/pdf
.mixed.RiscOS.Sources.HWSupport.USB.Controllers.DWCDriver.dwc.doc.DWC_otg_linux_sw_user/pdf

These files don’t have a comma suffix in CVS so they won’t get one in the tar file. This means they’ve probably never had a RISC OS filetype and if you’re seeing it, it’s because something else is using MIMEMap (or similar) to add it.

As a side note, zipfiles should have type &A91, not &DDC.

filenames which contain top bit set or control characters (which are undefined)

I think you just have to try to extract the files with the same set of 8 bit characters as used in the original leafname (as stored in the tar file). It may not look the same on other systems, but it should at least look as intended on RISC OS.

Rick said:

There probably ought to be a bit that says “This software uses software, blah blah” as well; though all that stuff is probably better printed under the RISC OS banner (at startup) given the variety of FOSS licences used, as opposed to polluting the Switcher information (it would, like, triple its size!).

Actually, RO4’s licence button in the Switcher’s info window is a very good compromise, especially with my ModCredits module that adds the copyright and licence information from existing modules to whatever licensing information is output in response to service call &42680.

Steve

So when downloading the archives should the files without ,hex be typed?

As a side note, zipfiles should have type &A91, not &DDC.

A good example of the ambiguity caused by using mimemap, DDC is the type chosen by mimemap

So when downloading the archives should the files without ,hex be typed?

A good question. I’m not sure there’s a right or wrong answer – if you want to strictly replicate what was put into CVS then the answer is ‘no’. However, in the more general sense, it’s usually helpful to set the RISC OS filetype using any .ext via the MIME map where a comma suffix wasn’t present. Certainly, if you checkout from CVS on a Linux machine and then move stuff across with (say) Sunfish, that’s what’ll happen anyway.

You could make it optional and have a button on the UnTarBZ2 GUI to control it (default to off).

A good example of the ambiguity caused by using mimemap, DDC is the type chosen by mimemap

Only if your MimeMap file is out of date (only by a few weeks though!) – the latest one in CVS has the correct allocation.

You could make it optional and have a button on the UnTarBZ2 GUI to control it (default to off).

Come to think of it, the EraseCVS step should probably also be optional, defaulting to off.

Rightho! I’ll do some more testing.

As a side note, zipfiles should have type &A91, not &DDC.

What fresh hell is this?

I’m rather confused, as I’ve been using DDC for zips for an extremely long time. DDC certainly is a zip file. However, I now see that A91 is too…

So what is the distinction between DDC and A91? It’s not the file format, I can tell you that for nothing.