More slick installation of major updates

It seems like ROOL assumes that everyone is a techno-guru with regards to distribution of major updates, that’s not meant as a criticism, but an opportunity:

The 2012 clock fix thing is a good example, perhaps not offering a soft-load would have reduced the amount of radio chatter? Wouldn’t it have been nice to automatically delete any clock fix patches at the same time, and to install dependant disc components too?

I remember the useless “Iyonix Update Watcher” and how it did half a job not very well. We could learn a lot from Microsoft here, Windows update works.

So – can I suggest one for the wish list?

• Ability to upgrade disc components regardless of location. This can’t be that hard, first policy would be to look for a system variable for its location, if that fails try the ‘standard’ locations for a default disc image, if that fails launch a full search of Boot$Dir.^ On finding a candidate, do a file CRC of the thing before overwriting.
• Ability to remove (or move to some backup directory) known conflicting components, like old patches no longer needed. Similar search policy to above.
• Some fuzzy logic to trim stuff out of PreDesk obey files.
• Probably script driven so it doesn’t need to itself be updated each time a new update is available. Or maybe have some BASIC functions which get called at various points in the process.
• An ‘agenda’ button like the system merge has.
• Some kind of simplistic dependancy checker, like RMEnsure on steroids. For example, don’t install 5.16 unless disc component XXX is version YYY. That’d be a fun admin task to maintain, no I’m not volunteering!

There are always going to be machines in the field that this falls over for, and that’s OK, presumably those people know why they messed about with something, and are able to ask for help on mailing lists etc. But, if dealing with 50% of the low hanging problems reduces 90% of the support effort, those machines that need hand tweaking should be in the minority.

I remember the useless “Iyonix Update Watcher” and how it did half a job not very well. We could learn a lot from Microsoft here, Windows update works.

Yes, it does, but it’s also very limited. I don’t believe that WU update solves very much that IUW didn’t. That is, still a half-arsed job. Note that in Windows, every other 3rd party application has its own special updater. That’s partly because many Windows apps are proprietary, which is much less true on RISC OS.

Unless you’ve been hiding for many years ;-), you should know system you’re after is !RiscPkg, which is heavily modeled on Linux update systems. Yes, still needs lots of polish, and there is still mumbling about packaging ROOL components, which would have indeed solved many of the update problems (although a lot of those were also user-inflicted, with people trying to apply things they didn’t understand and didn’t work anyway).

Most of the riscos.info software is now under RiscPkg, as is the NetSurf stuff. There is however, still a lot of work to be done in terms of management/administration.

The case of ROOL stuff is mostly easy, since things in !Boot rarely have non-standard locations, so for those it doesn’t have to worry about tracking apps that are moving.

Anyway, there’s been lots of discussion about this in the past. I refer you to the RiscPkg mailing list, GCCSDK mailing list, ROOL forums and Google.

It seems like ROOL assumes that everyone is a techno-guru with regards to distribution of major updates

That is kind of true. Here’s the full truth: it seems like ROOL don’t have the time to move from where we started from to where we need to be in terms of releasing code in any way other than via slow incremental changes.

For example:

• First of all, there was the ROOL site with nothing to download.
• Then we started putting stuff in a cvs repository.
• Then we started to document how to build things (and at first it mostly wouldn’t build).
• Then we got more sources released.
• Due to enormous pressure, we started putting some pre-built bits on the web site (a few apps and modules)
• Seeing how much trouble people got themselves into, we’re starting to direct people away from bits and bobs downloads now and more towards working setups
• We started dealing with ROM releases more formally so tried to invent some mechanisms and website back-ends for that
• They we get beaten up for not sufficiently documenting the changes… so spend lots of time improving that
• We also worked hard on making it obvious what the version number is of everything on our site
• Next would be moving everything over to RiscPkg – where possible

You see it’s a problem of whatever we do, it will never be enough, never be well documented enough, simple enough, clear enough, etc. And our time is severely limited. My day job is well over 50 hours a week this year!

So yes, we appreciate that only the brave and techno-competent will be making the most from what we’re doing right now and ideas for improving that are still welcome. But don’t expect a sea change over night – especially if you want us at ROOL to do all the work!

Don’t beat yourself up too much Steve. I’m no techno-guru but managed to download and install successively the 5.16 softload and flashrom on top of 5.13 without any problems whatsoever, followed by the SCSIFS upgrade, and I daresay most other upgraders had an equally straightforward experience. Sure, the process no doubt could be made more automated and foolproof, and probably needs to be if RISC OS is ever to break out of its niche status, but the progress that has been made since the foundation of ROOL has been truly remarkable, and a credit to all concerned. Without ROOL I would likely be an ex-RISC OS user by now, and I think all us users need to remind ourselves that you and your colleagues are working on RISC OS without remuneration and in what is otherwise good pub time… from me anyway, a big ‘thank you’.

Without ROOL I would likely be an ex-RISC OS user by now…

Hear, hear!

Thanks. At 2am after working through the weekend, it’s easy to feel a bit like all you get is complaints about what you could have done… I’m feeling much more positive again now you’ll be pleased to hear.

And Rob – just because I had a bit of a rant doesn’t mean I don’t think you’ve got a point. :)

Rob – just because I had a bit of a rant doesn’t mean I don’t think you’ve got a point.

No problem, I was wearing rant-proof-under-garments anyway. 50 hour day job? Not sure there were 50 hours of daylight last week…

I don’t believe that WU update solves very much that IUW didn’t. That is, still a half-arsed job. Note that in Windows, every other 3rd party application has its own special updater. That’s partly because many Windows apps are proprietary, which is much less true on RISC OS.

I hadn’t initially thought about upgrading 3rd party apps, but WU is at least smart enough to find my copy of Office even though it’s not installed in “c:\Program Files”. I assume it finds it using some registry key, or for RISC OS its system variable. WU is also good at dependancies: it wont let me uninstall SP3 without first removing IE8 for example.

I can certainly be a step-wise addition of functionality. Starting by just considering !Boot would be good, then other ROOL apps like ChangeFSI (scattered whereever they are) later.

you should know system you’re after is !RiscPkg

I was certainly expecting that to crop up in conversation, and using !RiscPkg as the underlying format or engine is fine. That would just leave the Configure plugin and adminitration to do.

I hadn’t initially thought about upgrading 3rd party apps, but WU is at least smart enough to find my copy of Office

I’m not sure that’s a very clever response. Office, IE8, VisualStudio, etc, etc. Are all MS apps, and not 3rd party at all. And for all the appeal of comparing Windows to RISC OS, the similarities to RISC OS are superficial, and the distribution of updates and software is quite different. And still, for all the obvious differences, the Linux model is much closer to RISC OS, and the RiscPkg model is a very obvious fit (heavily based upon the Debian setup).

I was certainly expecting that to crop up in conversation, and using !RiscPkg as the underlying format or engine is fine.

I find it rather odd, then, that you didn’t mention it initially. No one suggests that RiscPkg is perfect (it has many problems), but discounting it out of hand is no good.

That would just leave the Configure plugin and adminitration to do.

It’s unclear what you are proposing. Any functionality here has to go into !RiscPkg (or alternate front end, like Alan Buckley has attempted). The thing is, any incomplete effort here just will not do. It was clear to me (alas, few others) that IUW was going to be far from satisfactory even when Iyonix was released more than 7 years ago. The traditional way (which is, despite endless arguments to the contrary) whilst easy for seasoned users, remains confusing for new. Many of the questions which arise on RISC OS forums are exactly about where to get the latest version of something.

Without ROOL I would likely be an ex-RISC OS user by now…

Hear, hear!

I know you don’t want 100 “me too” comments but I’d just like to point out that not only is ROOL keeping RISC OS users in the fold, but they’re also bringing back old RISC OS users (like me).

Back on topic, I’d disagree that Windows Update works – it’s a mess in a number of ways. OS X has a much cleaner way of performing updates.

Next would be moving everything over to RiscPkg – where possible

Since this seems to be back on the table again, I’d like to pose another question (hopefully without killing the thread the moment I speak this time!) – What needs to be done in order to move in this direction? Seriously, I’m willing, but I don’t know what you want me to do or where you want me to start doing it. If necessary I could just start going through the applications section of the downloads page and sending you packaged versions of each one, but somebody’s still got to integrate the stuff into the autobuilder…

To be really useful, it needs to be completely automated. But don’t worry about right now. What you might want to do is make suggestions about what packages might be made, and what is put in each one.

In particular, !Boot has several sub parts, and bear it mind we might want to split it further in future. Also that some parts are really “configuration” as per packaging concepts. So, just to start with, some suggestions would be good to get things moving.

Having another look at the policy manual, there doesn’t seem to be any space in the specification for installing items which live in !Boot (other than in Resources or !System). Unfortunately, this is a deficiency in RiscPkg, and the only person likely to be able to help there is Alan Buckley (hello! Are you reading this thread?), Graham Shaw apparently having been abducted by aliens. Therefore I suggest we don’t focus on the !Boot components for now.

(As there is provision for the inclusion of a dummy !System directory, I presume this is designed to be merged with !SysMerge, so surely a similar arrangement could be rigged up for Boot Merge?)

The most likely candidates are therefore the desktop applications (which are, after all, basically the kind of thing RiscPkg was designed for). As I recall, the big sticking points were the need to write long descriptions for each package, and sort them by use and importance.

The short description field can be sourced from what’s already on the download list, and the best way to implement a longer description would be to use the already-extant information system. So that might be the first thing I do – filling out all the blanks in there.

Anyway, I’ll make a start on producing a set of ‘dummy’ packages and an index file for a subset of applications in that list (tomorrow – it’s 22:15 on this side of the pond!) to get a feel for what needs to be done, and hopefully write up some documentation on the wiki as to the basic process required. I’m afraid I have no access to the autobuilder (nor any knowledge of the language it’s written in) so ROOL will have to do the hard stuff themselves. :-(

Alan Robertson is the guy doing wiki docs. You want Alan Buckley, and yes he reads this. alan_baa at hotmail dot com if need be.

I dread to try and contribute as it usually seems to stop a thread dead:-(

but… I had got a prototype setup for packaging and I hoped the ROOL guys could have a look at the files I sent to Steve to see if they were any use. However it doesn’t look like they’ve got the time.

My main problems were due to the fact I wasn’t sure where the built applications etc were put on the disc after the current build, where I should put the control files and obey scripts to build the packages and where to copy the results to.

My system had various helper programs and scripts to do some of the packaging automatically. However the packages are put together, it will always need the description for the control file manually edited. So any efforts to get the descriptions done would help any future packaging efforts. The Section for each may also be required.

I’m happy to put more time into this if it will help, but from feedback so far (i.e. very little) it’s hard to see what I can do.

I could probably move forward a little if I had the following information:

1. How can I create a test environment that contains the built programs and modules that need packaging? I don’t have the castle tools so I can’t build it myself, but maybe I could simulate it by unpacking one of the download files to a specific location on my hard disc.

2. Is there a variable that is set to point to the above directory, which I could use in my obey files?

3. How will the result get transfer to the web? Do I just create a directory that someone can then just use rsync or something similar on?

4. Where should I put the obey files/control files and utilities on the ROOL site?

There were other questions that arose from my last experiments, but they will come out as more items are packaged.

Unfortuately I haven’t the time, the setup or the knowledge of the ROOL build processes to be able to be completely self-sufficient on this and just produce something that could just be updated and used.

As far as I can tell it looks like the whole autobuild process will need to be modified to properly support RiscPkg-ing of files. So I’d say the best thing to assume is that your script will be called on a per-package basis, with the following parameters passed on the command line:

• The path to the built component – either an individual file or a folder/application full of files
• The path to the directory containing the component source code (which is where your new file containing the package description and other metadata would be stored. It may also be possible for your tool to pull some of the data from the existing VersionNum/VersionASM files)
• The destination path/filename for what the archive should be created as

If you can do that, then you should be able to produce a simple testbed which will let you create the metadata for all the components that ROOL currently offer for download – just download everything from the site, unpack each one in its own folder, create all the metadata files, and then have your testbed run through everything and see if it produces working RiscPkg archives.

Anything beyond that I think will need the involvement of ROOL, or at the least someone who has the C compiler and enough time to dig around in the autobuilder/srcbuild/makefiles to work out what needs changing to get everything packaged properly. It may turn out to be easier than I think, but since I’m not sat at home looking at everything it’s a bit hard for me to tell!

Also, one particular caveat: sometimes the same makefile is used to build multiple components (e.g. USBDriver, OHCIDriver, EHCIDriver all come from the same makefile. HForm and SCSIForm also.) So your tool would also have a fourth parameter for specifying the component name; you could then use this as a prefix for your metadata filename.

We have progress!

Possibly the most useful thing I’ve managed to do so far is produce a drastically cut-down and more human-readable version of the RiscPkg policy manual (downloadable from riscpkg.org), which is very healthy, fibrous and completely indigestible. Hopefully this will get things moving a bit quicker. I’ll put it on the wiki after I finish typing this.

Jeffrey, what sort of script did you have in mind? If the only requirement is that it can be called from RISC OS, then I could probably knock something up from BASIC, but I seriously need to know a lot more about what format the parameters are going to be and, above all, where I’m going to find all this stuff. I was proposing to use the existing short descriptions on the download page to provide the RiscPkg ‘short descriptions’ – any idea where I could grab these from?

I just noticed the filetypes in the ChangeFSI archive are horribly broken. One for the bug tracker. :-(

Well, I can’t find a proper place to upload it (a temporary or miscellaneous section would be good) so I’ll email it to the info address.

Jeffrey, what sort of script did you have in mind? If the only requirement is that it can be called from RISC OS, then I could probably knock something up from BASIC

I think “anything that can be called from RISC OS” is a safe assumption, so BASIC should be fine. Perl or C would probably be better from a portability standpoint; after all, we should try and be aiming for a situation where everything can be cross-compiled using GCCSDK.

but I seriously need to know a lot more about what format the parameters are going to be

I think it’s fairly safe to say that you can just make it up as you go along for the parameter formatting. However to make the script more expandable/foolproof it would probably be a good idea to use ‘named’ parameters, e.g.:

riscpkgscript -src=path.to.source.folder -input=path.to.input.file.or.folder -output=output.filename -component=foocomponent

and, above all, where I’m going to find all this stuff

Can you be a bit more specific about what stuff you need to find? Otherwise I’ll can’t do much more than point you to the root of CVS ;)

One file you may find particularly useful is the ModuleDB which contains all the “internal” component names, their paths in the source tree, the name of the module produced (if it’s a module), etc. However your tool won’t need to parse this file itself; srcbuild will do it for you and pass the relevant paths on the command line. Also note that although ModuleDB lists an installation directory, that may not necessarily be where the files get installed to – sometimes the components file for a particular product will override the install location (e.g. the Binaries components file). Also note that installation, in this context, is purely talking about where the build system places them after they’ve been built – not necessarily the location a user would install them to. The installation location (in the srcbuild sense) would be what’s passed to your tool as the “input” option.

If you’re unsure what I’m on about when I talk about products/components/etc. then reading up about the build system on the wiki is probably a good idea :)

I was proposing to use the existing short descriptions on the download page to provide the RiscPkg ‘short descriptions’ – any idea where I could grab these from?

If you’re lucky you might be able to find them in the SVN repository for the website source code

If you’re unlucky, you’ll have to copy and paste them from the web site directly :)

I was proposing to use the existing short descriptions on the download page to provide the RiscPkg ‘short descriptions’ – any idea where I could grab these from?

The downloads pages are generated dynamically from configuration file in YAML format. The short descriptions provided therein often contain HTML data. Presently, download archives are structured internally in five groups with five corresponding configuration files covering all of the fully indexed downloads, most (all?) of which are in .zip or .tar.bz2 format.

To decide upon the best way to expose this information to third parties, I would need a clear idea of what was being packaged and how much of the packaging process was automated – e.g. if the lists themselves needed to be accessible automatically, tracking updates, or just as a one-off. It sounds like proceedings are some distance away from this position at present.

I must admit I’ve always found the RiscPkg documentation rather impenetrable and wonder if the burden the policy document appears to place upon the software developer might be excessive for all but the most demanding applications. So often, RISC OS applications don’t even need installing – just run from the archive – or at worst drag ”!Foo” to your HDD. Sure, there are more complex cases and packaging is very helpful here, but for a user to be convinced of the need for some mysterious pseudo-magic package manager, they’re going to have to be convinced of the distinct advantage this provides over their current understood approach. That’s not a technical argument – I’m well aware of that – it’s a user-facing, user-friendly argument, and one I’ve yet to see stated (e.g. right now, all I see are reams of text, no diagrams – not even a screenshot in the user guide – and no examples).

It doesn’t help that http://www.riscpkg.org/ appears to contain lots of broken links and often somewhat bizarre instructions (justifying myself as a RISC OS developer just to be able to send packages? – http://www.riscpkg.org/documents/new-developer.xhtml – that gives far too strong an impression of ‘arrogant ivory tower’ for my blood, even though I’m sure that wasn’t the intention). I’m supposed to contact Graham Shaw with this, but the link is broken. Is RiscPkg even maintained anymore? Doesn’t this kind of thing just help justify fears that a central repository would immediately stagnate if its maintainer left? Are we absolutely sure that this particular packaging approach is correct? IMHO someone needs to play devil’s advocate to make sure we don’t go down a blind alley.

Andrew, did you get the file I sent to info@riscosopen.org? This is a considerably condensed version of the impenetrable documentation. You will probably find it helpful.

As far as I can tell, this program will eventually need to be run in the same place as the current location of all this information (i.e. the ROOL server), so its current location isn’t that relevant, unless I make my program fetch it from the website (impractical at best). I’ve decided to assume that all relevant information (that required in order to build a package – see the file I sent you) will be passed as a command-line parameter, or at the least can be determined from the location of the application being packaged.

There’s still Alan Buckley and his alternative package manager, so I don’t think RiscPkg is dead yet. Unfortunately, for obvious reasons, I can’t subscribe to the mailing list, so can’t answer for how much activity there is.

I don’t think registering yourself as a developer is necessary – all you basically need to do is put an index file in an obvious place on the ROOL website and tell people to alert !RiscPkg to its existance. The magic technology will do the rest for you.

As building the package list is a more-or-less one-off process, it’s not that important that the process can be automated, until of course new packages are added to the list!

EDIT: Oh, one more thing: do you think !MakeModes can be justifiably placed under the ‘Device’ (“Device drivers and control software”) section?

Unfortunately I can’t look at what I did exactly at the moment, but I believe I had scripts that passed parameters. As there seems to some interest in this I’ll dig them out tomorrow and have a look.

Martin – I’ve written some C programs and obey files to do some of the parsing – do you want me to send them to you so you can have a look at them as well. If so let me know your email address and I’ll send them to you. My address is in Peter’s email above.

I’m still working away at my alternative package manager, but as with all RISC OS development progress is slow. I’m hoping I can make my next release more generally available.

The gccsdk autobuilder packaging site is updated and run entirely independently from the riscpkg site. I can’t see why the ROOL site can not be the same, so ROOL itself will be able to look after the packages.

There isn’t much activity on the riscpkg mailing list. But the GCCSDK autobuilder is steadily increasing the number of packages that are available from it.

For all it’s faults the riscpkg package manager is available now and it works. The code for it is freely available so it is not dependent on one person.

I’m hoping my alternative front end will make it friendly to use for the end user, but like everything else that will depend on the useful feedback I get.

Alan -

What we really need is an update to the RiscPkg specification that allows inclusion of dummy !Boot as well as dummy !System structures if we are going to get anywhere.

OK, last post before I fall asleep.

I’ve knocked up something which more or less works (except it won’t zip the output – I might deal with that tomorrow). It relies on a certain number of things and isn’t yet complete. Principally, it makes it necessary that package names of modules are the same as their title strings, and it requires access to a file which gives all the RiscPkg-specific information (the location of which is given as a command parameter) – once I hear back I’ll start rolling these files out for everything else on this site. It didn’t crash for the selection of desktop applications I tried it on. I’ll finish it (well, slightly more finish it) tomorrow and email my results.