SysMerge equivalent for SharedLibs

If by this you mean the packaging, then yes, I agree. PackIt is a pain to use if one already has a near-automated release

I’ve never used PackIt. The GCCSDK autobuilder already creates them automatically . The hard part is getting to that point. Creating the recipe and making sure all the inter dependencies get built and then testing locally to make sure nothing breaks. I’m gradually working my through fixing the recipes in the autobuilder that don’t currently build and also creating an equivalent VFP package.

The GCCSDK autobuilder already creates them automatically.

Interesting, thanks… I’d better add looking into what that offers to my to-do list ahead of spending more time on improving my system, then, just in case I’m reinventing an existing wheel.

@ Steve Fryatt

They’re on my site

Ok, thanks a lot! I’ll have a look. :)

Personally I was working on a post build process (using a template file for the project), it takes the output from make and packages it into a zip with the same structure as !PackIt would do. It also creates the md5sum for the zip and adds it to the repository index. So all I have to do from my linux box is run an rsync with the public repo after I am completed a build and it sync everything included the repo index.

Still debugging few bugs mostly on handling properly the repository index.

Store sounds a bit odd. So it is just a front-end for finding and downloading zip files? It doesn’t manage any dependencies or automatically upgrading applications?

Good to hear about the intentions with your ports, Chris. I have not really spent any time looking, but it always seemed odd how it was about downloading zips and ad-hoc updates. Sounds like the building blocks are there for it to ‘just work’ with PackMan.

If we want big serious applications, then use and re-use of existing libraries is the only way to go. There are always people asking for browsers, for example… So just live with it. You can download a browser with PackMan and the libraries will be pulled in. No mucking about.

If someone wants to muck about, or try getting stuff working on their Risc PC, then that’s fine, but surely you are on your own?!

Store sounds a bit odd. So it is just a front-end for finding and downloading zip files? It doesn’t manage any dependencies or automatically upgrading applications?

Not quite.

For free software it does this and lets you know (by highlighting the relevant items) of any titles that have been uploaded since you downloaded them.

For paid for software you can register payment or make payment and then it knows you have purchased that item and allows you to download it as well as highlighting updated titles. Dependencies will be packaged with the application either included or with instructions, exactly if you had purchased a disc with the software on it.

The correct way forward is to standardise on a distribution method and for all distribution points to use it, irrespective of them being open source or commercial.

Dependencies should never be bundled with packages, that’s very bad practice, they should be linked as a dependency from the source repository and in the case of SharedLib, each sublibrary should be independently packaged.

Currently PackMan is the only viable package management solution available, although it does have its issues, such as it’s oversized Absolute which prevents its used across all versions of RISCOS and its inability to check directories on a file by file basis as a true package manager would do.

The lack of a generic scriptable packaging tool is also an issue. Alan very kindly coded one for JASPP so I could maintain the 300+ games packages and all their dependencies with zero effort – but this is very bespoke. It’s open source, so I suppose folk could take it and create their own mass packaging tools, but its not something most people would have the time to do themselves.

RISCOS itself is a good example of how not to deploy software. It includes lots of apps, Modules etc, which should all be individually packaged and synced from a central repository. Its fine to release software with the OS, but the software should always have an update source, to allow the software to be maintained.

I’d forget about PackIt, its only any use for the odd bit of software that’s updated very infrequently, it should really needs replacing with a scriptable packaging tool that can be used by ROOL etc during build sequences.

The correct way forward is to standardise on a distribution method and for all distribution points to use it, irrespective of them being open source or commercial.

I agree with you, but to reach that point I think RiscPkg needs a bit more improvements. I am not an expert on RiscPkg, but I do use heavily RPM, DEB and other packaging formats and I am not sure why a packaging system like RiscPkg is/was so much focused on the GUI side while it should have started from a CLI/scriptable point of view, this to improve sustainability of the packaging process itself via automation.

Maybe it does have automation already somewhere in some hidden corner of the GCCSDK, if so any pointers to the right place would be useful and I can write a tutorial on how to use the automation after I’ve learned how the existing one works. But if it works only on GCCSDK then we do have a problem, it should work on RISC OS itself too.

Dependencies should never be bundled with packages, that’s very bad practice, they should be linked as a dependency from the source repository and in the case of SharedLib, each sublibrary should be independently packaged.

Agreed and given that RiscPkg seems to support multiple versions of the same sharedlib it should be feasible if the mechanism of requesting a different release works fine. However please read about the potential issue I have found while testing PackMan, if it goes funky with a !SharedLibs update it seems to ask the user for a “Backup and retry” which only re-install the standard !SharedLibs, not the 3rd parties that where installed in the previous !SharedLibs. If this issue is a bug (and not just something that has happened to me) then we have a serious problem relaying on PackMan…

The lack of a generic scriptable packaging tool is also an issue.

Yup totally agree on this one, as far as I can tell we all are basically solving this issue in different ways, so why not to solve the problem once and for all?

RISCOS itself is a good example of how not to deploy software.

Very true, but then instead of a pre-installed image we would move to an installer. This still requires CLI/Scripting support (given that an installation may start from absolute nothing, so more convenient to use CLI, at least until the installer has detected all the features offered by a platform). But yes I’d love RISC OS to be packaged, so that it could be also easier to update (or re-package on specialised distributions) and updated also more frequently than just every 2 years (but this also requires a possible different testing process, so not sure how feasible it is for RISC OS).

I’d forget about PackIt, its only any use for the odd bit of software that’s updated very infrequently, it should really needs replacing with a scriptable packaging tool that can be used by ROOL etc during build sequences.

+1 :)

RISCOS itself is a good example of how not to deploy software. It includes lots of apps, Modules etc, which should all be individually packaged and synced from a central repository. Its fine to release software with the OS, but the software should always have an update source, to allow the software to be maintained.

Fortunately all the apps supplied with RISC OS are individually packaged, synced from a central repository, with an update source which is in the list for PackMan in the latest RC16 Pi release.

This previously wasn’t possible because the bits and pieces were scattered on the wrong servers, and it wasn’t until a key upgrade completed that it became possible. However, that news article dates from August 2018, 2 years ago – sounds like your package lists are out of date if you can’t see them.

Taming the monster that is the Boot application remains to be packaged, if anyone’s at a loose end?

Taming the monster that is the Boot application remains to be packaged, if anyone’s at a loose end?

What sort of person would be able to do this? It is easy or hard? What are the requirements for tackling this work?

What sort of person would be able to do this?

Somebody with a lot of patience?

You understand, I hope, that if you ask the twenty of us here for opinions of how Boot should behave and what ought to go where, you’ll get at least a hundred answers, most of which will conflict with each other.

It is easy or hard?

Given that RISC OS has been made 32 bit, but Boot is pretty much the same as it was since the RiscPC arrived…

What are the requirements for tackling this work?

Given it’s the boot sequence we’re talking about, probably access to functional machines running every major release since 3.10; plus an understanding about how the choices stuff works, as well as scrap, how modules are loaded (all those subfolders), the many hooks, and knowing the major applications that dump stuff into Boot Resources (Zap and anything formerly by Computer Concepts come to mind).

You may also need to have some programming skills, as one of the major problems with the Boot is that without additions (such as my Harinezumi), it’ll choke and wave the white flag the instant there’s an error, usually leaving your machine in an indeterminate state (everything before the error gets loaded, everything after never gets a look in).

Then, once it’s been tamed a little, working out how the hell to package it given that there may be multiple versions of the same module that end up in different numbered directories; and of course the fun that is AcornSSL that doesn’t necessarily bump the version number each time it gets updated.

In other words, lots of fun to be had if you enjoy a bit of digital S&M.

plus an understanding about how the choices stuff works, as well as scrap, how modules are loaded (all those subfolders), the many hooks,

Ever sat and followed the sequence through as diagrammed in the old ROL 4.02 boot diagram?
Hmmm, Harinezumi… you probably have the sequence burned into your brain.

and of course the fun that is AcornSSL that doesn’t necessarily bump the version number each time it gets updated.

Is this the point that we point out exactly why that (non)versioning is not only terminally stupid but an impediment to certain development work – like this.

Chris Hughes

An installer would I suspect help, but I note the comments by Steve about Packman and package management as well, personally I have never been exactly that keen on Packman – I suspect that might partly be with its bad habit of wanting to create folders in the Apps folder and installing things there – but that for another thread I think.

This probably is for a different thread, but Packman just uses these locations by default. You can configure the defaults to be somewhere else and most application packages allow you to place them wherever you want at install time.

Dave Highton

Am I the only one who thinks that package management, though undoubtedly well-meaning, seems to cause as many problems as it solves?

Your probably not the only one, but there are a few of us who think it solves more problems than it causes:-)

Chris Gransden

Currently my !SharedLibs folder on a Pi4 is 1.2GB. Most of these don’t have packages in PackMan.

And I for one appreciate what you are doing to make them available in PackMan.

The shared library dependencies for Python 3 and Pygame should be appearing in PackMan soon. Just waiting for them to be built and uploaded.

I think we’ve got the Python 3 dependencies, but not the Pygame ones yet. Hopefully they should appear soon, but time is short.

Steve Fryatt

If by this you mean the packaging, then yes, I agree. PackIt is a pain to use if one already has a near-automated release process, as it adds another step to the list of things to be done.

PackIt isn’t really designed for that workflow. It’s useful for non-built things where just dragging a new version into PackIt and updating the version number is all that’s needed.

For a built package you can use PackIt to create the initial Control/Copyright files then just extract them from the zip file it creates to use as part of your build process. That’s usually how I start a new package.

Is there any demand for a command-line packaging tool that can be used from a Makefile? On RISC OS, or on both RISC OS and within a cross-compiler environment? I’ve now got a few hacky Perl scripts which do most of the job within my own build process under the GCCSDK; I guess I could add improving them for more general use to my to-do list.

I’m not sure what extra tools are needed. The packages are just a zip file with a particular layout and some text Control files.

I do my packaging from Makefiles. I just create the folder layout in a temporary directory. Copy in the Control/Copyright files and just run zip program to zip up the result. Some programs I use something like sed to update the version number. In another I use lua to update versions everywhere first.

I have also create packages using ordinary RISC OS Obey files and I believe ROOL are using BASIC. So you should be able to use anything you like.

The difficulty of course is know what goes in the Control files. It is documented though and PackIt is a reasonable way to create the initial one that you then use in your automated builds.

Paolo Fabio Zaino

About the issues with !SharedLibs and !PackMan:

- did anyone noticed or have already reported that when Packman goes a little funky on a !SharedLibs update and requests the user to “Backup and Retry” well the final result is just !SharedLibs WITHOUT the 3rd parties shared libraries installed previously? (and this happens also with shared libs installed properly and via PackMan). So it seems that the “Backup and Retry” may be considering !SharedLibs like any other package and not a special one…

You are correct PackMan isn’t treating !SharedLibs as something special and it really should be for the Backup and Retry. I think I may have seen this mentioned before, but if it’s in the middle of a forum I might have only skim read it. This is a bug. Did you try to report it to the PackMan author (me)?

Last question: I am setting up my own repositories (PackMan sources I guess is the right term), will I be forced to add my own Shared Libraries to the ROOL main repo anyway?

You are not forced to add your Shared Libraries to the main ROOL repo. But if they are libraries that other people will use it would be nice if you did. Obviously if you are putting a Shared Library that already exists on the ROOL repo on your repo you should use an existing package or ensure that you help get your version in the repo if it’s better.

Jon Abbott

Currently PackMan is the only viable package management solution available, although it does have its issues, such as it’s oversized Absolute which prevents its used across all versions of RISCOS

PackMan and the back end LibPkg are open source software on GitHub. I don’t have the time to look at shrinking it’s size, but if anyone has any suggestions they are always welcome. I will bear this criticism in mind though.

Also the package format is open and documented and if someone has the time to create a better packaging program from scratch, please do – I won’t be offended:-)

and its inability to check directories on a file by file basis as a true package manager would do.

Not sure what this means. If it’s off topic can you email me with details.

The lack of a generic scriptable packaging tool is also an issue.

As I said before, I’m not sure what’s needed from a scriptable packaging tool. You can use any language that can create text files or zip files (or run external tools to do it) to create the packages.

Alan very kindly coded one for JASPP so I could maintain the 300+ games packages and all their dependencies with zero effort – but this is very bespoke. It’s open source, so I suppose folk could take it and create their own mass packaging tools, but its not something most people would have the time to do themselves.

Not sure that there are many cases of people packaging multiple programs like this, but it will always need bespoke logic to read information from whatever database of apps you are using to figure out what’s needed in the package.

I’d forget about PackIt, its only any use for the odd bit of software that’s updated very infrequently, it should really needs replacing with a scriptable packaging tool that can be used by ROOL etc during build sequences.

See my responses about PackIt and scripting above. ROOL already have a BASIC program that runs during build sequences to create packages.

I’m not sure what extra tools are needed. The packages are just a zip file with a particular layout and some text Control files.

Very few: the main stumbling block here is the version.

Ignoring packaging, I build a test release of any of my software by typing make release and watching appropriately-named application and source zip files fall out. I might add VERSION=1.23 to make a “numbered” release; otherwise, it gets the current Svn revision of the source folder. If packaging isn’t to be a chore, I really need a valid package to fall out as well – with the correct version in the control file. That has to include the package version, taking into account any pre-existing package for the same build version that has been published. Any manual involvement from me is a backwards step.

It looks from this thread as if we’ve all largely got to the same place via different means: the package assembled in a build folder, and copied into a zip file by Make. PackIt is a handy tool for creating that initial package content. Everyone has rolled their own solution for the version number issue.

Since Jeffrey seems to have come up with a very similar approach to mine¹, I suspect that I’m not the only developer to be thinking this way. I only wonder if, were it less “DIY”, more people might be packaging stuff up now.

¹ Even down to my apparently creating skopt independently in all but name, implementation language and finesse.

Steve, having seen the comments here and elsewhere I can see why some tools could be useful. Just because no special software is needed, it doesn’t mean it shouldn’t be created if it makes things easier.

Back to Andrew’s original question.

Some sort of merge tool to let people who just don’t want to use PackMan be able to install shared libraries seems like a reasonable idea.

If someone wants to do it the format of the shared libraries is something like:

SharedLibs.lib.abi-2/0
SharedLibs.lib.abi-2/0/vfp
SharedLibs.lib.??? – for the new GCC8 stuff

libxx/MAJOR/MINOR/PATCH/so Actual library (ELF format)
libxx/so/MAJOR Symbolic link used by running programs
libxx/so Symbolic link used when compiling programs

MAJOR version changed when you make incompatible API changes,
MINOR version changed when you add functionality in a backwards compatible manner
PATCH version changed when you make backwards compatible bug fixes.

So you would want to read the libxx/so/MAJOR link to find the version for the API version you are installing and react appropriately.
You may want to update the libxx/so version so compiling uses this version as well.
You can remove and replace the previous libxx/MAJOR/MINOR/PATH/so version with a new one. In which case you must update the libxx/so/MAJOR link. If the MAJOR in the version is updated it should be treated as a new shared library.

Hopefully someone from the GCCSDK developers will correct me if I’m wrong.

PackMan is an add on, so I’m not sure the system should do things to keep PackMan informed, rather it should adapt to the System.

SharedLibraries came after PackMan so it doesn’t treat them as something special. I think it should. I’ll probably look into getting PackMan to treat them more like modules so it doesn’t overwrite existing versions unless it’s version is newer. The logic I specified above should apply to PackMan as well.

@ Alan Buckley

Thanks a lot for your reply Alan, very helpful indeed :)

This is a bug. Did you try to report it to the PackMan author (me)?

Not yet, I have been playing with RiscPkg/Packman just by 4 weeks which actually means (4 weekends in actually testing/development/use time).

Will address to you all my findings/questions then, any preferred channel or is Bugs section here at ROOL Site good enough?

You are not forced to add your Shared Libraries to the main ROOL repo. But if they are libraries that other people will use it would be nice if you did.

Totally agree and not trying to be “anti-social” lol, but I am also porting a lot of libraries that are specifically for Developer’s use only like AI/ML, Unit Testing, Programming languages (specifically Prolog and Forth on my queue which are not as popular as let’s say Python), so I do not want to overload standard !SharedLibs of stuff that may not be of general interest (or potentially even interest to others at all).

Obviously if you are putting a Shared Library that already exists on the ROOL repo on your repo you should use an existing package or ensure that you help get your version in the repo if it’s better.

Sure, thanks for the info. Right now my Shared Libraries are just produced by me (AFAIK) and definitely not already present in the standard !SahredLibs. They sometimes depends on stuff that is already on !SharedLibs but that seems to be working fine with PackMan and all the test-installations I have done so far worked fine (tested on a lot of different RISC OS boards I own), so it appears to be pretty stable right now.

I also think I am the only one having a !PackMan source that uses HTTPS instead of HTTP also for the index and the downloads and it works fine when a user has installed !CaCertificates package in !Boot.Resources, posted instructions on my Blog on how to make that work because I haven’t found them elsewhere…

Before someone ask me why I did the above: Simple, the index reports the URL from where !PackMan will have to pull the Package, so an attacker and/or MiTM can simply sed/replace that tag on the fly (when packman is requesting an index update) and re-direct !PackMan to download a package from his compromised server instead of the original place intended by the package’s author. So this appear to be a security issue in the !PackMan design? (but as mentioned it’s easily addressable by adopting HTTPS for the index as well)

I also think I am the only one having a !PackMan source that uses HTTPS instead of HTTP also for the index and the downloads and it works fine when a user has installed !CaCertificates package in !Boot.Resources

The JASPP source has been available since day 1 via both HTTP and HTTPS, personally I think HTTP sources need to be dropped, but I’ve stuck with advertising HTTP as the preferred source until AcornSSL is officially part of the OS.

I did mention to Alan about rewriting PackMan to use AcornSSL in preference to embedded libraries as it might substantially reduce the size of the Absolute and hopefully allow it to work on legacy RISCOS. In parallel to this, I did start coding a package installer written in BASIC last year which will eventually become the JASPP game distribution frontend – I do have a rather bespoke requirement though, that needs to work on 3.x

I also need to distribute some large software packages that are over 1gb, which I don’t think PackMan will currently cope with in a user friendly manor. Again this is a rather bespoke requirement and not something I expect Alan to look at.

For general package distribution, it would be very helpful if we had a package installer that covered all OS versions, ideally this needs to be in the form of a CLI driven Module that PackMan and other frontends can hook into.

@ Jon Abbott

The JASPP source has been available since day 1 via both HTTP and HTTPS

cool, will switch to HTTPS for the JASPP source then, thanks a lot! :)

For general package distribution, it would be very helpful if we had a package installer that covered all OS versions

Totally agreed on this, I am still a big fan/user of the older OS versions too (as well as a big fan of the JASPP project and gaming on RISC OS), so defo a +1 on this.

ideally this needs to be in the form of a CLI driven Module that PackMan and other frontends can hook into.

Agreed again, also the PICO project could benefit from this and what I am working on as a non Desktop deployment.

@ Jon Abbott

I did mention to Alan about rewriting PackMan to use AcornSSL in preference to embedded libraries as it might substantially reduce the size of the Absolute and hopefully allow it to work on legacy RISCOS.

I’d missed or forgotten you said that. I have been thinking about switching to AcornSSL to remove dependencies on the ported linux libraries as I’ve had some issues with them in the past. If it reduces the code size as well that would be great.

Do you know if AcornSSL is supported on all relevant RISC OS versions?
Also do you, or anyone else, know which modules would need to be installed on older versions of RISC OS?

In parallel to this, I did start coding a package installer written in BASIC last year which will eventually become the JASPP game distribution frontend – I do have a rather bespoke requirement though, that needs to work on 3.x

It would be nice if you still kept the PackMan packages as well.

I have been thinking about switching to AcornSSL […] If it reduces the code size as well that would be great.

A grab of the one loaded here shows it’s ~340kB at present. How does that compare with the libraries you link into PackMan?

Do you know if AcornSSL is supported on all relevant RISC OS versions?

It uses a dynamic area hence is RISC OS 3.50 or later, the area is ~220kB on my Titanium here.

In theory there’s no particular reason why that couldn’t go in the RMA for RISC OS 3.10. However, given there’s no DCI4 network drivers for RISC OS 3.10, and you’d be softloading the Internet module, Resolver, MbufManager, MimeMap maybe, plus 560kB for the above 2 bits, I think HTTPS on an Archi is probably a bit of a non-starter. Risc PC and later, sure.