PackMan package building & publishing tools

In this other thread a few people have mentioned a desire for an easier way of building packages, which is suitable for use in scripted build systems (makefiles, etc.). Basically I agree with this, and have a few issues of my own that I’d like to share.

My story: A couple of years ago I started to look at converting the various bits of software on my website to be packman packages. This resulted in an article moaning about the sorry state of the RiscPkg website (Sorry Alan!), and a couple of more constructive articles about the RISC OS-based package build system I built, and a C/C++ tool to help with publishing packages. There’s more work that I need to do on both the packaging tools and my website (so far it’s just SunEd and skopt which are packaged, and neither of them are available via PackMan’s default distributions yet), but since work on RISC OS generally takes priority over work on my website, I haven’t found the time for it yet.

A summary of things:

• Package building is performed via extra rules added to makefiles
• I’ve been using sed to inject values into the Control file
• I’ve been using command-line zip to build the package archive
• Because I want things to work automagically, I want zip to run any time the contents of the input folder changes. But make’s dependency tracking only works with files, not folders, and I don’t want to have the makefile contain a list of all the files in the archive (especially since it won’t work properly if I need to delete a file from the archive). So I ended up writhing a BASIC script, ‘manigen’, which essentially does a recursive *Ex to list all the files which are going to be included in the package. The output file manigen produces will only be updated if its contents needs changing, allowing the file to be used as a dependency in the makefile. This is all covered in the second article linked above. On non-RISC OS platforms, I suspect that the function of manigen could be performed by some clever combination of shell commands.
• I haven’t yet worked out how to build source code archives in a way which conforms to the RiscPkg specification.
• Building packages is only half the battle; you also need to rename the package file to its canonical name and update the index file(s) that PackMan / ROOL use to discover which packages are available. Hence the development of ‘skopt’ (third article linked above).
• But I also want the packages to be findable via my website, so one of the future abilities I was planning on adding to skopt was the ability to spit out a JSON file (or similar) containing the package list & metadata. I can then either upload the JSON file as-is to my site and use a PHP page to generate the HTML on the fly, or it could be fed into a text templating tool to generate static HTML. Why JSON and not have the website parse the packman index file directly? Because the packman index file is missing key metadata like the date the package was built, or the ability to specify an icon/image to show for the package, or screenshots, or the link to the source code package, or the ability to list previous versions of the packages, etc.

So I don’t think we necessarily need someone to create a tool to build PackMan packages. But since building packages is a non-trivial task, it would be nice if there was a “PackMan/RiscPkg-approved” standard set of makefile rules / scripts available which people can easily integrate into their projects. And obviously I think things are a bit lacking with regard to publishing packages (but hopefully skopt will fix that, once it’s gained a few extra features).

Predictably, may I suggest Lua as an appropriate minimal packaging tool. Whether in RISC OS, Linux or Windows there are libraries for:

1. compression/decompression (zlib)
2. iterating over directories
3. reading/setting time stamps
4. file operations

and manipulating databases using labelled records.

@ Jeffrey Lee

Thanks for posting this and +1 on your summary of requests/requirements for the packaging tool :)

So I don’t think we necessarily need someone to create a tool to build PackMan packages. But since building packages is a non-trivial task, it would be nice if there was a “PackMan/RiscPkg-approved” standard set of makefile rules / scripts available which people can easily integrate into their projects.

I actually have wrote few packaging automation tools in the past for companies I have worked for, so I have gained some experience on the matter.

To reduce timing costs and optimize sustainability over time we shoul dhave a tool to build Packages I’m afraid…

If we decide for a full Packaging automation tool then it should:

PRE-REQUISITES

- Determine (via template/playbook/recipe) build dependencies (for example requires static libraries, building environments etc.)
- Determine (if required) build memory requirements etc.

BUILD PROCESS

- Fetch the source code from versioning repositories (github/gitlab/svn/cvs etc.) (and be capable of specifying the tag we want to build (git can do that) and/or the branch we want to build (maintenance purposes)
- Interface (control) with the patching infrastructure (for example for dynamic code porting, I have worked on this as well, a bit painful but can be done, we do have diff and patch on RISCOS AFAIR) (this means apply specific patches where needed for example to port a C source to compile on DDE or to add specific fixes for a given architecture etc.)
- Interface (control) the binary building process (Makefile/CMake/scripts)

PACKAGING PROCESS

- Interface (control) the packaging of the binary and separately the source
- Optionally (at least this is generally so on Linux/BSD) update the repository index
- Deploy on the testing repository (scp most of the times does the job via RSA keys or also rsync)

Another relatively new way of doing this is via DevOps, which in the most simplistic form that will lead DevOps people to hate me for presenting it so (lol apologies for that to the DevOps community) means rebuild everything every time and make sure your build process and results (artifacts) are repeatable.

Not sure if my experience on the matter is useful and/or required, but if so I can offer my help once I have finished to understand RiscPkg, which I am using by like 4 weeks (I left RISC OS scene when RisckPkg was being designed) and for which I already managed to create my own repos and packages and everything seems to be working fine although if the full automation is still missing.

So I would define myself a newbie with RiscPkg.

@ Gavin Wraith

I personally have no issues with using Lua, which I kind of prefer given that I can expand it in C/C++ relatively easily.

Generally what’s required from a programming language to be a good tool for Build/Packaging automation are:
- a good relationship with RegEx/strings manipoulation
- good interfacing with the OS commands/external tools
- fast runtime execution given that the process itself can be time consuming
- at least support for Structured/Procedural programming paradigm to organise the source in a way that is easy to maintain

Lua is above and beyond the requirements above, so if it’s stable enough on RISC OS I think it’s a good choice

Jeffrey

This resulted in an article moaning about the sorry state of the RiscPkg website (Sorry Alan!)

No need to apologies to me, I don’t control the RiscPkg web site. I don’t think people realise that PackMan is just the client side of the Packaging system. I have little or no control over any of the websites that host packages. All PackMan knows about is where the index is and how to get it and what to do with the information it’s given.

I haven’t yet worked out how to build source code archives in a way which conforms to the RiscPkg specification.

I haven’t either and PackMan has nothing in it to handle them. With the prevalence of the internet now I’m thinking of adding a few control fields to the package to specify where the source is and how to get it. I’ll probably open a discussion on this at some stage in the future.

So I don’t think we necessarily need someone to create a tool to build PackMan packages. But since building packages is a non-trivial task, it would be nice if there was a “PackMan/RiscPkg-approved” standard set of makefile rules / scripts available which people can easily integrate into their projects. And obviously I think things are a bit lacking with regard to publishing packages (but hopefully skopt will fix that, once it’s gained a few extra features).

From the various discussion and your comments above I think this is a very good idea. I don’t have any real websites myself, so I can’t host it though I’m happy to look at/comment on anything that is created.

Paolo, the GCCSDK autobuilder supports almost everything you mention. It’s also hooked into a Jenkins CI (currently broken as I need some time to fix networking on the backends). Source URLs go in, packages come out. The one thing we don’t have is automated testing – something I looked into but didn’t make a huge amount of progress with).

However it’s not a panacea for the kind of RISC OS programs that are written in BASIC or similar – there’s nothing to build, so why a complicated build system that you have to run on Linux? A relatively lightweight automatic package building framework as Jeffrey suggests would complement the autobuilder just fine.

Jeffrey, the ROOL package index file just goes into the GCCSDK build-website scripts, that take a pile of .zips and make a website and index files. If anyone wants to rework that using JSON or whatever, using extra metadata and nice icons and stuff, that’s completely up for grabs. It is simply a very minimal bit of python on the server to fetch the packages into a tree, then run build-website.

Paolo, the GCCSDK autobuilder supports almost everything you mention.

Theo, thanks for the info and saw some of it on my copy of the GCCSDK, but that’s GCCSDK (that runs on Linux). I mean’t a build process to run on RISC OS itself and to use to package everything included ROOL DDE based programs, Lua, Python, BASIC and also a couple of new (old) programming languages/compilers I am porting (Prolog and Forth).

But ok, not a biggie, if the community doesn’t want it I’ll just build it for myself.

But ok, not a biggie, if the community doesn’t want it I’ll just build it for myself.

I think you’ve assumed something there.

The Linux based stuff people have done is basically utilising something on another platform which those specific people have experience of.

“the community” is largely inexperienced with Linux and would probably appreciate something that runs on a platform they do know – RISC OS.

GCCSDK was originally conceived to work around several problems. The slowness of RISC OS hardware (in the RPC era compared with GHz multicore PCs), and the dependencies of existing libraries on build systems that wouldn’t run in RISC OS (bash, autoconf, etc). Both of those are still true to an extent. The other thing it buys is repeatable scriptable CI-able builds. A lot of RISC OS native builds are being hand driven and often difficult to build on anything but the author’s machine. It’s a good idea to improve the tooling for those builds, especially to remove some of the quirks so that software can be built natively simply and more repeatably. But until such time as the RO CLI is up to the point where it can handle projects of scale (elsewhere I regularly build LLVM with 3000 C++ source files – having 36 cores helps a lot!) there will still be a place for cross building. Making life easier for native builds helps in a complimentary way.

@ Steve Pampling

“the community” is largely inexperienced with Linux and would probably appreciate something that runs on a platform they do know – RISC OS.

That is my understanding too. My answer was to Theo only, I simply didn’t want to start a debate between “full build process” vs “small tool” on RISC OS, to me everyone is entitled to their own opinion so I’ll just try to build what I personally need and share it with others via my PackMan sources, whoever will want to try to use it is welcome :)

@ Theo

GCCSDK was originally conceived to work around several problems.

Yes and it does solves a lot of problems as you’ve mentioned, for who have such a need. But as Steve mentioned there are also people that simply want to use only RISC OS, which is ok too IMHO and that also do not want to spend too much time playing with the technical details of RiscPkg.

IMHO this doesn’t conflicts or reduces at all the value of the GCCSDK.

A lot of RISC OS native builds are being hand driven and often difficult to build on anything but the author’s machine.

Finding it takes longer to get the damn thing to build than make the changes you want, isn’t fun, and isn’t condusive to encouraging developers to retrun to RISC OS.

A lot of RISC OS native builds are being hand driven and often difficult to build on anything but the author’s machine.

I’ve given up on trying to get WebJames built using the original (DDE) method. Never managed to track down whatever RegEx library it wanted (I found…two, I think, both have various errors about missing definitions and such).

For my software, I try to make it build with nothing more than the DDE and my incarnation of DeskLib (which is available as both a fully installable thing and a library file), because I got sick (back around Y2K) of trying to track down obscure stuff that was required that “might be on hensa, or arcade, or maybe some random Geocities site that no longer exists…”. This is cyclical, GOTO previous paragraph.

Finding it takes longer to get the damn thing to build than make the changes you want,

You want pain, use the Arduino IDE to build something for the ESP32. On my PC (a 2.4GHz P4 with ~3GiB running XP), it takes longer to build my NetRadio code (what, about 10K of C?) than to build the entirety of RISC OS from scratch on a Pi2 using a class 10 SD card.
I shit you not.

There’s a lot of stuff I’d like to do, such as caching (at the moment it fetches 32 bytes and chucks them directly at the audio chip) which is maybe a 100th of a second of data (if that) which makes it sensitive to other WiFi users. If I hit refresh on this page, the audio would probably hiccup.
I’d also like to add NVRAM settings, and a serial port setup mechanism so that one doesn’t have to burn the WiFi identity and channel list into the code, you can instead plug it in as a serial device and set it up that way.
But it would do my head in to wait some twenty minutes just to have the thing tell me I missed a single sodding comma.
I have no idea how they managed to make a compiler so unbelievably slow.

[it’s not antivirus, I already disable that prior to building, the difference was negligible]

It’s not the slowness of the build I’m complaining about, it’s your first point about finding all the stuff you need to build it, then recreating the highly customised development environment people have forgotten they have spent the last 30 years setting up.

Not that my software will be any easier to build when I finally get around to open sourcing it.