Packaging applications

Those who have been reading the Raspberry Pi thread will have heard me mention packaging. I thought I’d elaborate a bit here, and see if we can refine the detail a bit.

Firstly, let’s define packaging. These days RISC OS programs come in Zip files which you download from the net. These tend to be structured in a haphazard way. At its simplest, packaging is about making programs automatically installable.

A common objection from RISC OS users is that packaging forces you to install applications in a specific place (which is not actually true, just a limitation of the current tools). But consider a disc image – they have to put the applications somewhere in the first place. Images may be built with particular purposes in mind – for example an education image, or a non-network image, each of which have a particular subset of the full pool of applications. Manually copying around files to make images gets tedious, as does upgrading programs.

So I see packaging as an automated way of installing applications, and the PackMan/RiscPkg system as the only one that currently exists for RISC OS. They have their limitations, but better to improve what you have than re-invent the wheel, especially given time constraints.

This provides a couple of other benefits for free, both of which are familiar to app store users. App stores enable you to find new software that you haven’t heard of before. This is especially important when you’re new to a platform and don’t have a clue what any of the apps are called. Also it enables authors to push out updates. Security patches go out faster, and bugs can be fixed more efficiently. In the case of a disc image, it enables new and updated software to be pushed out to network-connected users after the image has been shipped.

I don’t intend to force users to use packaging: the apps on their machine were installed by a package manager and can be updated using that tool if they wish, but they’re otherwise normal RISC OS apps. They’re free to move them around, delete them, etc, etc – they will just stop being managed by the packaging system if they do this. Metadata about management is simply hidden away so that the packaging tool will work if desired. Think of the package manager as a slightly clever ‘copy’ tool.

Personally I’ve been looking at using the GCCSDK autobuilder to handle some of the work of packaging, as it already has tools to support many of the operations. But I wouldn’t force other packagers to use the same system… a package generated by any means is as good as any other.

Now, the areas I’m less clear on. A package manager as a simple copy tool doesn’t need an online connection, but it’s necessary for new app discovery and update. In the Debian/RiscPkg paradigm, a repository consists of a collection of packages (.zips in the RiscPkg case) and a Control file that says what they are. The location for the repository is a URL, from which the URL of the control file is calculated.

An installation can reference one or more repositories, but the advantage of keeping them relatively few is that dependencies (eg NetSurf requires Iconv, Tinct and SharedUnixLib to be installed first) are easier to manage within the same repository.

My basic idea for how to manage the repository for the RPi release is as follows. I’d appreciate comments (hopefully not derailing us too much about the general packaging flamewar, which has been done to death elsewhere):

1. Add links to the Control file and the package .zip for that software’s
   entry on the RPi wiki page: (adding another two columns to the table)
   https://www.riscosopen.org/wiki/documentation/show/Raspberry%20Pi%20disc%20image%20proposals
   You’d have one Control file for all your programs, and one zip per package.
   The link to the Zip is so people can download it manually.
2. The repository maintainer(s) run a script that fetches these sections of Control file and sticks them together, fetches the packages and puts them in the repository. This needs a bit of manual care (checking dependencies are met, avoiding duplication)
3. The repository is hosted somewhere stable-ish (suggest a DNS CNAME for starters).
4. Further care is used in pushing out distributions – a ‘stable’ distribution, and one for ‘testing’. Pushing out software to the ‘stable’ distribution should have a certain amount of care to check it doesn’t break anything else. (But much less care than Debian’s ‘wait 2 years’ policy)
5. A plan needs to be arranged so this doesn’t depend on a single point of failure (eg the repository maintainer(s) going AWOL).

So, comments please?

That’s a very helpful explanation. You’re right that the AWOL issue needs addressing in the medium term, with maybe the usual suspects being approached.¹

In the longer term, perhaps a successor/rebirth of AAUG would be worth considering. There should be enough breadth of membership within the user groups for any “throwing toys from the pram” to be diluted, offering some stability of hosting. (Such an association would need outright ownership and management of the domain, rather than this being handed out to a single group to look after on behalf of the association.)

¹ For any approach to riscos.info, we’d need to confirm that the Creative Commons licence relates only to wiki contributions and not somehow to hosted software as well.

Could the package manager not “manage” the users end as well, in that if the user has a wish to install the app in a location of their choice, the package manager would maintain a record to be used for any updates ETC.
The user would have to tell it where and if they didn’t, the default location would be used.

Could the package manager not “manage” the users end as well, in that if the user has a wish to install the app in a location of their choice, the package manager would maintain a record to be used for any updates ETC.
The user would have to tell it where and if they didn’t, the default location would be used.

I would support this totally.

Conversely, personally I will not use a package manager (as either a user or developer) until the user can override the default location for an application. I want to organise my disc(s) to suit my way of working, not what someone else has thought a good idea. I would not want to impose my design on users of my software.

Nevertheless, I support the concept of a package manager, it is just the lack of any user choice of location I object to!

I think libpkg (the engine behind PackMan and RiscPkg) already supports overriding the default location. PackMan provides a file you can edit to say where particular applications go… it just needs pulling out to a GUI. It gets slightly more complex because you might end up installing dozens of applications at once and you don’t want to be presented with dozens of ‘Save as’ boxes. OTOH I don’t think PackMan will install more than one app at once (RiscPkg will) so perhaps that’s not an issue in the short term.

Some packages, eg modules, probably shouldn’t offer the user any choice of install location. So there’d have to be a flag in the package set by the maintainer or something.

Ideally the packages would be installed by dragging from the packager window to the location required.

(Perhaps using a modifier to choose between actual install or just a stub.)

I originally had reservations about the current package managers forcing apps to be installed in certain places. But with the ability to easily create a ‘stub’ wherever you want, I’ve not found this to be any problem at all in practice.

And as I said, rather tersely above (was typing on my phone while in bed :D ), sometimes you want to force files to be installed in a certain place. For example, stuff in !System or !Boot.Resources. Of course we have existing tools to do this, eg SysMerge whether the stand alone version or the one in !Boot.

An installation can reference one or more repositories, but the advantage of keeping them relatively few is that dependencies (eg NetSurf requires Iconv, Tinct and SharedUnixLib to be installed first) are easier to manage within the same repository.

One thing that LibPkg does do to help this a little is that the same package can
be placed in multiple repositories and it will install the latest one from any
of the repositories.

My basic idea for how to manage the repository for the RPi release is as follows.

I was thinking we should use the standard riscpkg repository, as most, if not all the software should be common to all RISC OS machines.

A plan needs to be arranged so this doesn’t depend on a single point of failure (eg the repository maintainer(s) going AWOL).

Would it be possible/better if the riscpkg repository was kept as the master and some kind person with plenty of webspace provided a mirror of it?

Could the package manager not “manage” the users end as well, in that if the user has a wish to install the app in a location of their choice, the package manager would maintain a record to be used for any updates ETC.

I am currently considering how to implement this in PackMan. I believe LibPkg will allow this, it just needs extra code in PackMan to implement it.

The interface will be of the form install it first time to it’s given position and then move it within PackMan to where you want it. After that PackMan will have a record of it’s position so it can update it.

It will probably be a few months at least before I can get this in a release though as I don’t expect to get much time on it before Christmas.

Would it be possible/better if the riscpkg repository was kept as the master and some kind person with plenty of webspace provided a mirror of it?

What sort of traffic would there be? And would it be possible to use bitorrent, falling back to the repos when there are no seeds?

I don’t have a problem about using the RiscPkg repository, but is Graham contactable? Also, some of the software on there isn’t ARMv7 safe (how can this be indicated to stop ARMv7 machines downloading software that doesn’t work?) – I found one app that isn’t even 32 bit safe. And I would probably suggest a more refined list of apps to be advertised: PackMan doesn’t present any categories (AFAICS) so the list is a bit overwhelming – most of it is libraries or command line apps which are just confusing to new users.

I don’t have a problem about using the RiscPkg repository, but is Graham contactable?

Yes he is – though it some times takes over a week to get a reply.

Also, some of the software on there isn’t ARMv7 safe

This is a good point. From this I can see why you want a separate repository/list.

I don’t know if Graham would allow it, but maybe a separate managed package index could be set up for the ARM7 compatible stable packages on the riscpkg site. This way the packages themselves would still be shared amongst everyone and we would have one master version, but the new users could be presented with an ARM7 safe list.

PackMan doesn’t present any categories (AFAICS) so the list is a bit overwhelming

I think you are thinking of RiscPkg, one of the reasons I created PackMan was so you could see a list of packages by category.

most of it is libraries or command line apps which are just confusing to new users.

This point makes me think a separate controlled package index may be a good idea.
Though I did think one of the points of the Raspberry Pi was to encourage programming, so the libraries and command line apps could be useful.

I don’t know if Graham would allow it, but maybe a separate managed package index could be set up for the ARM7 compatible stable packages on the riscpkg site.

I’ve now asked Graham about this and he said he would be OK to host and such a package index.