SysMerge equivalent for SharedLibs

The python / youtube threads have shown that people are getting into a right muddle with SharedLibs, and whilst I appreciate the general advise is to use Packman to re-install everything, anyone wishing to supply commerical software using shared libraries, or otherwise provide standalone distributions of software has no easy way to avoid messing up !SharedLibs in the process.

I’d like to suggest that a !SysMerge equivalent be written for !SharedLibs, that will ensure that libraries are merged in without damaging things, and stopping other programs working.

I think the need for this is fairly urgent, and lack of easy maintenance could make releasing SharedLib software later in the year rather, erm, “fraught”.

Thoughts/comments welcomed.

I’d like to suggest that a !SysMerge equivalent be written for !SharedLibs, that will ensure that libraries are merged in without damaging things, and stopping other programs working.

It must co-exist with PackMan, however, otherwise you’ve effectively killed off package management for anything which uses shared libraries — at least as soon as a user wants to install some commercial software.

Would a better solution not be to make an installer which uses the same package database as PackMan, and which can ensure that the necessary dependencies are installed as part of the process? It can still install the actual application in whatever way it wishes, but installs the shared bits in the same way that PackMan would.

This would seem to have the benefit that installing commercial software wouldn’t trample on anything already installed “properly”, and vice versa. You would have the problem of sorting out any unsuitable pre-installed dependencies when the commercial software was installed, but any more “brute force” approach would simply break the existing install and any packaged software at the same time.

ETA: Also, given the issues that we’ve recently seen with broken shared libs getting out into the wild for a short time, the idea of commercial software shipping with shared libraries that aren’t under easy control seems a bad one. If there’s a problem with a library in the package management system, you update the package and the problem is fixed. A broken library in a commercial software distro could continue to cause carnage to other software every time a user with the install files tries to use them on their system.

Since !SharedLibs is a creation of the RISC OS GCC team, this seems like something that should be discussed on the GCC mailing list.

Yes, Steve, I agree. Indeed, such a tool might sensibly include some packman library code to check for any updated libraries online.

But, don’t get hung up on the word “commercial”, it was just an example of a situation where you might want to distribute something outside of PackMan.

My point is, right now, even “properly” installed stuff seems to be a bit messy, and Chris G is having to release some bits via his website, and some via packman etc. Meanwhile there’s betas of Python 3 trying to co-exist with older 2.7 Python and so on. It’s something that we need to take seriously.

Jeffrey – yes, you’re probably right, but that’s not a list that is read by many on here (I know I’m not a member). ROOL forums has rather become the central hub of things.

don’t get hung up on the word “commercial”

I’m not. My concern is simply that as soon as you have two ways to control things, users can only install stuff which uses the one that they’ve plumped for.

Chris G is having to release some bits via his website

Do we know why? If there’s something that stops libs being pushed out via package management, it would be good to know what so that it can be fixed.

If it’s just that the libraries aren’t compatible with what is in the packaging system, then that’s your problem right there.

This is an interesting thread to me. Since I have encountered this issue with the various browsers I have been trying. For instance there seems to be a different set of !SharedLibs files required for Obrowser as against Otter itself.

In another case where the browser has its own !SharedLibs inside itself, and to be able to run that browser I have to stop Sargasso running to be able to run that browsers SharedLibs and then I can rerun Sargasso without issue.

An installer would I suspect help, but I note the comments by Steve about Packman and package management as well, personally I have never been exactly that keen on Packman – I suspect that might partly be with its bad habit of wanting to create folders in the Apps folder and installing things there – but that for another thread I think.

I do agree its fairly urgent this gets sorted before it gets worse and become a support nightmare.

ROOL forums has rather become the central hub of things.

Maybe it just belongs in Aldershot? ;-)

before it gets worse and become a support nightmare.

I’m JAFO, but I’m following this thread too.

The primary distinction between a package manager and an app store (or downloading archives from websites) is that it is supposed to resolve dependencies and ensure that all the right bits are in the right place. This didn’t used to be a big deal with RISC OS where “make sure CLib is in !System and it’s up to date” was about as complicated as it got for most software. But now, with ported software using various ported libraries, we have the choice of either running massive executables with all the stuff statically linked, or smaller executables using library code. The latter is more logical for the obvious reasons, and it also allows libraries to be updated/fixed independently of the applications using them.

If this isn’t working, or if something is causing that to malfunction, or if we have incompatible versions of the same libraries…then we have a problem.

Agreed Rick, I’m basically trying to raise the issue and get people talking / thinking / coding sooner rather than later. Idea being to sort it before it becomes too much to handle, and also to try and resolve the “perpetual beta” situation which came up at the last London usergroup meeting zoom session.

Basically, what was happening was that with everything “in flux”, people were treating beta stuff as release versions, and complaining when it didn’t all work perfectly together. I can understand the frustration, and whilst it is an inevitablity with public beta processes, the sooner we think / talk / tackle this the better.

In what way does the current situation (tons of alpha/beta quality code in the form of shared libs gets shipped to people willing to test things) differ from that before (tons of alpha/beta quality code in the form of modules…)?

Remember the Toolbox situation and how much confusion that produced with inconsistent versioning et al? We’ve been there before, and with modules it is a lot worse because of the Highlander principle, while shared libs can coexist in various versions at the same time.

I think we need to find out first what went wrong in the past, what the best practice is to provide maximum isolation for distributed patch versions, and look how other OSes solved the problem. And map that to our situation of a severe lack of manpower for even the simplest things.

I fear that there is no foolproof way to do those things – see Windows DLL hell, Linux stable/unstable/experimental release trains, Java semantic versioning/SNAPSHOT approach…at the end of the day, it is combinatorics killing you unless you invest a lot of manpower.

I can understand the frustration, and whilst it is an inevitablity with public beta processes, the sooner we think / talk / tackle this the better.

Much of the beta in normal use problem with the OS stems, I think, from a rather long cycle time for the beta to stable transition. v5.24 was December 2018 so we’re looking at nearly two years for the gestation of the new stable(v5.26, or 5.28 in new numbers).
In that time quite a few new features have come in and the only way people can use those new features is to go beta.

Another disadvantage of the log cycle time is that the completion of the beta-to-stable transition involves checking more changes for previously unnoticed flaws and ensuring all features are properly documented. Documentation can be a pain – that’s among the reasons for the iMX6 image only having a beta status is it not? (There are other reasons of course)

There should be an optimum cycle time that deals with the additional features needing move to stable and the workload on the ROOL manpower ensuring all is actually “stable” and documented.

Much of the beta in normal use problem with the OS stems, I think, from a rather long cycle time for the beta to stable transition. v5.24 was December 2018 so we’re looking at nearly two years for the gestation of the new stable(v5.26, or 5.28 in new numbers).
In that time quite a few new features have come in and the only way people can use those new features is to go beta.

This thread is as I understood it about various beta versions of the SharedLibs for applications ported to RISC OS not the OS itself.

Documentation can be a pain – that’s among the reasons for the iMX6 image only having a beta status is it not? (There are other reasons of course)

Not really, its because as the iMx6 port status says the R-Comp versious have some runtime customisations, – the two I can think of are the USB work, that ROOL will only add to there version once it accepted into the BSD build, the other is to do with HDMI sound.

This thread is as I understood it about various beta versions of the SharedLibs for applications ported to RISC OS not the OS itself.

I was referencing the beta on beta elements. How much of this is application and library beta status alone and how much is also beta OS is all open to debate.

I am struggling to see the problem here. If the end user uses PackMan, then everything will be OK, won’t it?

I have never used the Store app, but I would assume that it is a shop-front interface around a package manager, or provides its own package manager.

Think back to the old days, and the joys of merging System or Fonts. Pain. And then things like CCShared came along and we had even more complexity.

Me, I’m in favour of the old ways!

But – there it is …

Am I the only one who thinks that package management, though undoubtedly well-meaning, seems to cause as many problems as it solves?

I have never used the Store app, but I would assume that it is a shop-front interface around a package manager, or provides its own package manager.

Alas, no. It just gives you a zip file and leaves you to your own devices.

seems to cause as many problems as it solves?

I think the main problem here is stuff being released/installed/modified out of the realm of the package manager.

Remember back in the old days when swapping floppy discs without the system noticing would lead to everything getting messed up as the right data would be written to the wrong disc? I forget, was that Arthur or one of the MOS? At any rate, this is the modern day equivalent.

It just gives you a zip file and leaves you to your own devices.

Which, still, for the vast majority of things means draggy-droppy, done.

Most of the stuff available on riscosports.co.uk is a holding area until the packages get added to PackMan. That’s the most time consuming bit. Plus the holding area is getting larger.

Very few of the VFP packages exist yet for the Shared Libraries. There are quite a lot to do. Once they are done and available from PackMan it should speed up getting everythng else available from PackMan.

Currently my !SharedLibs folder on a Pi4 is 1.2GB. Most of these don’t have packages in PackMan.

The shared library dependencies for Python 3 and Pygame should be appearing in PackMan soon. Just waiting for them to be built and uploaded.

An installer would I suspect help, but I note the comments by Steve about Packman and package management as well, personally I have never been exactly that keen on Packman

That’s irrelevant to this issue, as my suggestion is that just the non-PackMan installer uses the package repositories/databases for the shared libraries only. They “have” to go in a fixed place, anyway, so it doesn’t make any difference.

The reason is simple. If you don’t, then as soon as a user installs a piece of software which uses Andrew’s hypothesised stand-alone library installer, that prevents that user from using PackMan for anything involving shared libraries. Not unreasonably, PackMan can’t handle other things putting files in locations where it wants to install stuff, so anything else installing shared libraries will block PackMan from managing shared libraries. And that means that you can’t install anything using PackMan if it requires shared libraries.

This would be somewhat anti-social, to put it mildly. Given the confusion that pre-existing files in the locations of packaged components already causes, a tool which makes that happen more often isn’t a great idea.

I think the main problem here is stuff being released/installed/modified out of the realm of the package manager.

This. Very much this. Mostly stuff that’s already used by stuff inside the realm of the package manager. Like shared libraries…

Most of the stuff available on riscosports.co.uk is a holding area until the packages get added to PackMan. That’s the most time consuming bit.

If by this you mean the packaging, then yes, I agree. PackIt is a pain to use if one already has a near-automated release process, as it adds another step to the list of things to be done. PlingStore is also a nuisance in this regard, but at least it uses the same archives that will go on my website and so doesn’t change the build process; package management needs a different file building, and so does.

Is there any demand for a command-line packaging tool that can be used from a Makefile? On RISC OS, or on both RISC OS and within a cross-compiler environment? I’ve now got a few hacky Perl scripts which do most of the job within my own build process under the GCCSDK; I guess I could add improving them for more general use to my to-do list.

If the problem is getting the finished packages into a repository, then that’s pretty trivial with the ROOL setup (which I hadn’t realised until a couple of weeks ago).

@ Steve Fryatt

Is there any demand for a command-line packaging tool that can be used from a Makefile? On RISC OS, or on both RISC OS and within a cross-compiler environment?

Yes, and I already started to create mine given that the RiscPkg works very similarly to deb packaging, but (AFAIK) it has no automation like RPM/DEB etc…

I’ve now got a few hacky Perl scripts which do most of the job within my own build process under the GCCSDK; I guess I could add improving them for more general use to my to-do list.

Good to know, I also started to write my whole automation on linux after extracting the info I needed from PackIt and documentation. I started to write some model in bash and planning to write a tool in C, but if there is already available code that does a full automation, then I am happy to use it :)

About the issues with !SharedLibs and !PackMan:

- did anyone noticed or have already reported that when Packman goes a little funky on a !SharedLibs update and requests the user to “Backup and Retry” well the final result is just !SharedLibs WITHOUT the 3rd parties shared libraries installed previously? (and this happens also with shared libs installed properly and via PackMan). So it seems that the “Backup and Retry” may be considering !SharedLibs like any other package and not a special one…

In the case of the problem above well I let you imaging what happens next, but pretty much any application that relies on shared libs that are not part of the full !SharedLibs will stop working and the reporting of this to a user is not so “user-friendly” and obviously PackMan will also report that such 3rd party lib is installed, which makes things even more confusing.

If the package manager is addressed to a non-technically skilled audience then it may be worth considering adding an equivalent of “updatedb” or a “user-transparent” process to at least refresh the DB without having to uninstall everything using PackMan and then reinstall everything again.

Last question: I am setting up my own repositories (PackMan sources I guess is the right term), will I be forced to add my own Shared Libraries to the ROOL main repo anyway?

I started to write some model in bash and planning to write a tool in C, but if there is already available code that does a full automation, then I am happy to use it :)

Not full automation by any means. :)

My process, roughly, is to lay the package out in the output folder of the project, with a template Control file created from PackIt. There’s a Perl script to take that template file and fill in correct values (the package version, at present), writing the result into the correct place. I then use the GCCSDK Zip to turn it all into a package file.

There’s a separate script to scan the existing packages, looking for the most recent version of the package being built. If there’s already an x.xx-1, it creates x.xx-2, for example.

Finally there’s a script to build a repository index, and a package list for ROOL’s system.

They’re on my site, in the PackTools archive – beware, as they’re very specific to my setup and lashed together to more or less work.

The process is driven by the Shared Makefiles that are also on my site, if a package name is supplied in the appropriate Make variable.