Alternative build environment

Over on the ‘Let’s get started with a Pandora port’ there was a comment about a freely available build environment. I’ve thought about this a little… rather than polluting that thread I’ll put down some notes here.

I think there are two potential goals… a free build environment, that doesn’t involve spending money on tools, and a cross-build environment so you don’t have to own RISC OS hardware to drive it.

I believe Peter Naulls has done some looking at GCC to see how it might be involved in building the ROOL code. It might be worth asking him what he thinks.

So far as I understand, I think the elements of the build system are:

• The frontend GUI programs (!Builder and others)
• Obey file scripts
• Makefiles
• BASIC/perl/etc other scripts
• C and assembler compilation

!Builder can easily be replaced with just a simple command line, as can !EraseCVS. I haven’t looked at the other programs, but I think those could also be fairly easily replaced.

The perl is probably easy too. Makefiles can be tweaked to be cross-platform. BASIC can be run under Brandy. The bigger problem is the need for a CLI environment – there are a lot of system variables flying around and these need to be preserved. It might be easiest to rewrite the Obey files in Perl, if the RISC OS perl environment is up to scratch. That will make them cross-platform, with a bit of care.

The Norcroft compilers themselves could be run on Linux using QEMU’s user mode emulation (see QEMU on riscos.info). Or with more work the GCC compiler slotted in. I’m not sure if ‘as’ (the GCC AOF assembler) is up to the complex macroing that happens in the ROOL code, but it may be these days.

But the biggest problem so far is the web of build scripts that would need converting, which is a fairly tedious job.

I would like to suggest Lua for the scripting. I find Perl impenetrable. However if nobody is using Lua the suggestion becomes pointless.

There was a whole lot of work put in by Pace into cross-compiling RISC OS. This was about the same time as the 32-bitting was done: as part of this, we had to switch to using objasm rather than aasm, since the latter assembled instructions in a form that malfunctioned on 32-bit-only processors and the sources to aasm had been lost.

The result was that the makefile to every assembler module had to be rewritten, so to kill two birds with one stone, they were rewritten in a form suitable for cross-compilation (or cross-assembly to be technically correct). Look in the makefile for any assembler component, and you’ll see it includes the shared makefile fragment AAsmModule – all the OS-specific details are localised here. If you’re familiar with makefiles, you’ll notice that these use GNU makefile extensions: the Acorn make utility AMU was updated to include many GNU extensions around this time specifically to enable this.

I believe Pace got as far as building complete ROMs under Unix, as long as they contained only assembler components. However, the work stopped before the equivalent makefile fragments were mature enough to support C and mixed C/assembler components being built into ROM images. It had to wait until Tematic for this: look at these makefile fragments in RiscOS/BuildSys/Makefiles:

• CApp
• CLibrary
• CModule

Using these, the makefile for a component can be reduced in most cases to little more than a list of source and header files required to build the component in a way that’s independent of the host OS and toolchain. Sadly, Tematic never had a need to retrofit these to older makefiles and most new components that were written from scratch to use them haven’t been open-sourced, so there are few examples for you to look at, but try these makefiles

• SoftLoad
• DThreads library
• DThreads test module (although you’ll notice I’ve been lazy and specified the library as a RISC OS path, where for cross-compilation you’d want to define a symbol for it in Makefiles:ModuleLibs so that it can be referenced in an OS-agnostic way

A good first step towards cross-compilation and support for alternative toolchains such as GCC would be to rewrite the makefiles one by one to use these makefile fragments. This would probably become a fairly mechanical process once you’ve written one or two. Any volunteers?

I wouldn’t worry too much about all the (!)Mk* files – they’re there as a convenience for developers, and are rarely very complicated. !Builder is really just a pretty front-end to allow you to run an Env file and execute the srcbuild tool, which does the real magic of the build.

Now, srcbuild and romlinker are two bespoke tools that you’re not going to find anywhere else because they’re very RISC OS specific. srcbuild is surprisingly complex – it parses the Components and ModuleDB files to determine which components to build in which order and using which build phases and using which build switches. romlinker is responsible for doing the final assembly of the ROM image. While of course you could attempt to do a clean rewrite of these, it seems rather a waste of effort, and there is no firm specification to work to to ensure compatibility.

Although these tools can be cross-compiled (Pace used them for their cross-compilation builds), ROOL has been holding off releasing the sources because they depend in part upon a library licensed from ARM and Alcatel, that it’s looking unlikely that we’ll be able to release in source form. Possible ways forward might include releasing a binary DLL of this library for Windows/Linux/Mac, or alternatively writing a cleanroom reimplementation of the library. Any thoughts? Volunteers?

There would of course be many other minor details to address. Norcroft-style inline assembler is fortunately quite rare in the released C sources, and could easily have GCC alternatives added. I assume GCC (the mainline branch if not the RISC OS port) supports the ATPCS calling standard used by the HAL. Some bits of the build don’t properly distinguish between building tools for the build host and for the build target and use the same compiler for both – but these can all be ironed out in time, I’m sure.

Although these tools [srcbuild and romlinker] can be cross-compiled (Pace used them for their cross-compilation builds), ROOL has been holding off releasing the sources because they depend in part upon a library licensed from ARM and Alcatel, that it’s looking unlikely that we’ll be able to release in source form.

What’s so special or specific about that library ? Is romlinker just an AOF/ALF linker or something more ? And is romlinker also responsible to strip off & apply the RelocData data of the built modules and link all SCL calls to SCL itself ?

I assume GCC (the mainline branch if not the RISC OS port) supports the ATPCS calling standard used by the HAL.

I don’t know when GCC got ATPCS support but the first release GCCSDK 4 (based on GCC 4.1.1) seems to have it, although I’m fairly sure it requires some minor changes as some of our APCS-32 code changes are also triggered by the other calling standards. It just needs some developer time to address this.

The Norcroft compilers themselves could be run on Linux using QEMU’s user mode emulation (see QEMU on riscos.info). Or with more work the GCC compiler slotted in. I’m not sure if ‘as’ (the GCC AOF assembler) is up to the complex macroing that happens in the ROOL code, but it may be these days.

‘as’ predates GCC port (see as README) and was indeed used as back-end assembler for all RISC OS GCC ports until GCCSDK 4 where we started to use binutils (and ELF object format). We recently re-included ‘as’ as ‘asasm’ (to avoid a binary name conflict) in GCCSDK 4 tree (no releases of this so far) and John-Mark Bell and I significantly brushed it up by teaching it more objasm features needed to build RISC OS assembler sources.

We’re not there yet but with a little bit of persistence or extra developer hands, it should be a nice objasm clone. And it outputs AOF or ELF.

C modules are already linked as binary images at their final ROM addresses before romlinker is called (assembler modules are position-independent so don’t need linking in that sense). What romlinker does is is to take a list of linked modules and assemble them (either contiguously or at fixed addresses, as specified) into a binary file of specified size, ensuring no overlap or overflow, padding with zeroes, adding a CRC at the end and poking a couple of magic values into the Kernel or HAL. It also has options for building expansion ROM headers. As I said, it’s quite specialist.

  Error handling, (c) ARM:
    err_fail
    err_init
    err_report
  Extract program name from argv[0] OS-independently, (c) ARM:
    program_name
  OS-independent file handling, (c) ARM:
    wf_filesize
    wf_load
  Hash table, (c) Acorn, so could be released:
    hash_assoc
    hash_cinew
    hash_enter
    hash_lookup
    hash_value

  Error handling, (c) ARM:
    err_init
  Host abstraction, (c) ARM:
    host_dir_sep_char
    host_init
    host_system
  Extract program name from argv[0] OS-independently, (c) ARM:
    program_name
  OS-independent file handling, (c) ARM:
    wf_filesize
    wf_rename
  Hash table, (c) Acorn, so could be released:
    hash_assoc
    hash_enter
    hash_key
    hash_new
    hash_value

I think you can see that it wouldn’t be rocket science to do a cleanroom rewrite of ARM’s parts of these lists, it just needs (a) someone to carefully specify the requirements – I’ve started this by drawing up the above list – and (b) someone else to write code to those specifications.

Here’s a list of tools that are built using CLX. Some of them can’t be open-sourced because the tools themselves contain third-party code, but I won’t bother analysing that here. How many others (apart from srcbuild and romlinker) don’t already have open-source clones?

amu
binaof
c++
cc
cfront
chmod
cmhg
ddt
decaof
deccf
diff
find
libfile
link
modgen
modsqz
objasm
objsize
ResGen
romlinker
rpcgen
squeeze
srcbuild
unmodsqz
xpand

This is something of a repeat of what’s already been said here.

I’ve built parts of RISC OS some time ago with GCC - mainly Draw and some of the libraries. The challenges then were mainly updating the assembler to understand more of the objasm stuff, but that’s had substantial further improvements since I poked at it.

There are various other funnies – notably correct use of case in filenames vs #include statements, and perhaps some other path statements.

As noted, the big task is writing the actual build system – fortunately this is something I’ve had substantial experience with – which I would try to automate as much as possible, or at least make very easy to add to manually.

I really don’t think it’s worth pursing a native GCC compile – except perhaps for some of the standalone stuff like !Paint and such mentioned above – a cross compile will be many times faster, and will benefit from various scripting that is not really practical under RISC OS.

One tool that we’re missing is an OSS squeeze, but this isn’t critical. Also from the list above, I’m not sure what everything does, but I’d note ‘romlinker’, ‘rpcgen’ and ‘ResGen’.

My guess is that 90% of the ROOL sources can be cross compiled with GCCSDK without much drama. I do actually have write across to the ROOL sources to fix appropriate issues, but have not yet pursued this. I may have a bunch of time this summer. Possibly I’d want to try and include some of the components in the GCCSDK autobuilder too; this can automate the build process, and let developers do actual development, and can spit out RiscPkg packages with little hassle.

Ok, reviewed yesterday the stuff I’d done last year to build things. Notes:

All the files in CVS (ugh, still wishing for SVN), are in RISC OS directory format, e.g. c/filename h/file instead of filename.c. I have a script which creates a symlink farm of the files and gives a unix-style build setup. This is all good from an autobuilder viewpoint.

Originally, many of the #includes used this format also, but I think ROOL purged all of those last year at my request. Also, there’s a bunch of case mismatches – I will check in some fixes for these shortly.

Last year I was building with the older GCC 3 (AOF output), and was linking Paint et al against a Norcroft generated RISCOS_Lib. For various reasons, I’d much rather use the GCC 4.1 ELF compiler, using ‘asasm’ to output ELF objects from the Acorn format assembler.

So the problem is building RISCOS_Lib with GCC, which may be somewhat challenging – fails horribly right now, and there are various historical assumptions about the C library which might need changing, or possible lots of #ifdef code. We’ll see.

Most of the problems I was having were actually with the SCL, which isn’t required, and I hadn’t paid attention as to what belongs to what. Anyway, I’ve made some more hacks to the GCC assembler, and RISCOS_Lib now builds, and I linked !Paint against it (not tried yet).

The immediate goal is to be able to build everything in Apps without requiring any commercial components, both natively in RISC OS and cross compile. This should be easy.

As for the SCL, it contains many things which are obscure compiler dependent things, and it will take some work for it to build with GCC. One option here is to simple replace it with Graham Shaw’s version. Anyway, that is not an immediate concern, and I hope that everything else in RISC OS is straightforward C and assembler that’ll pose no real challenges. We’ll see.

I hope that everything else in RISC OS is straightforward C and assembler that’ll pose no real challenges

C—probably. Assembler—less so, for example:

1. Include filenames are still somewhat variable, mixing hdr.foo and foo.hdr
2. There are various bits of magic that asasm still doesn’t grok (e.g. ?A to get the storage size of A, foo equ &f00:^ foo, and other things that probably need asasm to turn into a 2 pass assembler like objasm)

1. Include filenames are still somewhat variable, mixing hdr.foo and foo.hdr

Not so concerned. Those changes can be mostly one-time scripted. I’m prepared to much such wholesale changes as I go along (and the case fixes).

There are various bits of magic that asasm still doesn’t grok

Right. The terrible hack I did last night was to support objasm’s -PD flag. My objasm manual is packed away, so I couldn’t see the exact syntax, but what I put in sufficient in asasm to do -PD “UROM SETL {FALSE}” – see the Makefile for RISCOS_Lib. I did this by injecting a “GBLL <symbol>” and then the actual line before the main assembler is parsed. No doubt there’s a better way, but I’ll check in what I have.

I also found that objasm accepts “Entry”, where as asasm insists upon “ENTRY”. I put in some logic in objasm compat mode to suppor that too.

Er, “Entry” is a macro defined in Hdr:Macros, not a compiler directive.

The history of that is the sources used to be built with aasm, which didn’t have an ENTRY directive. At this time, the symbol “ENTRY” was defined as a macro and used on entry to the majority of assembler subroutines in the source tree.

“Entry” was added when we transitioned to using objasm, because objasm interprets ENTRY as the image entry point directive before it tried looking it up in the list of macros.

As a quick poll, is there anyone immediately interested in working on the “core apps” using GCC natively in RISC OS?

The reason I asked is that there’s a whole load of things I need to address to approach any kind of completeness towards building a ROM, but there isn’t any particular order. If there was genuine interest in this, I could work towards it pretty quickly. For reference, the list including the languages in use:

• Alarm – assembler (i.e. objasm/asasm) + Basic
• Browse – C + toolbox
• CDPlayer – C + RISC_OSLib
• ChangeFSI – C + Basic
• Chars – Basic
• CloseUp – Basic
• Draw/Edit/Paint/SrcEdit/MakeModes – C + RISC_OSLib
• Help2 – C + toolbox
• Maestro – Basic
• Sampler – C + toolbox
• SciCalc – Basic
• Squash – C + RISC_OSLib (app only, module source not in ROOL repo)
• TTXTViewer – C + Toolbox
• WebServe – C + Toolbox

Stuff in Diversions not catalogued. There’s other stuff elsewhere like HForm (that’s Basic) which I haven’t looked at.

Obviously the pure Basic apps don’t require anything for anyone to immediately work on – e.g. recent SciCalc thread. Everything using RISCOS_Lib I have at least cross compiled, but don’t yet have native Makefiles for (this includes RISCOS_lib itself too).

Let me know.

Peter, you may want to bear in mind that we’re hoping to be able to release the sources to srcbuild, the command line tool behind !Builder, in the near future. You should be able to cross-compile it then. This would (a) save you the effort of having to write another metamake system, and (b) mean that on an ongoing basis only one build configuration file (the Components file) has to be maintained for each build, irrespective of the toolchain in use.

The reality is that building on RISC OS is so very different to what you’d do anywhere else, that building most of a new system is unavoidable. e.g, the Makefiles are full of RISC OS commands (and often unsuitable for GNU make), and the filenames are all in RISC OS format.

The Components file is a different matter, and there’s certainly no reason not to reuse it – after all, it’s just an easy to parse text file, and of course, it makes sense to continue to use the Hdr2H tool (Perl).

Having said that, there’s any number of tricks you can do in GNU make and shell scripts to make the build system significantly less complex that it is presently for RISC OS. Don’t worry, I think you’ll like what I end up with.

Here’s the Makefile for Paint. Of course, this only builds the static binary, where as the native version can build a ROM version too.

objs=PaintLib.o AltRename.o ftrace.o guard.o Main.o m.o ToolWindow.o \
      Tools.o SprWindow.o PSprite.o Menus.o Colours.o jpeg.o writepixel.o

CFLAGS += $(rinclude)
LIBS = $(rlib)

App = Paint

include ../../../riscos.mk

Great minds think alike, but I sense some needless duplication going on. We already have a simplified makefile system that was designed with cross-compilation and toolchain neutrality in mind. You can achieve the above using included makefiles, amu and Norcroft like this:

# Makefile for Paint

OBJS = colours main menus psprite sprwindow tools toolwindow jpeg writepixel AltRename PaintLib
LIBS = $(RLIB)

include StdTools
include StdRules
include AppLibs
include CApp

# Dynamic dependencies:

I think you’ll agree that this is so similar to what you’ve come up with that it would be madness for every component to need makefiles of both formats. It would be relatively simple to come up with a hybrid format that will work for either toolchain, I think.

By starting with components that nobody’s done serious work on in years, what you’re seeing is makefiles that are several generations behind state-of-the art. Pace already did cross-compiled ROMs containing just the assembler components, so you’ll generally find that assembler components have already had their makefiles drastically simplified in this manner. I finished the job by adding the CApp, CModule and CLibrary makefiles for building C components – it’s just that I’ve only been converting components to use them as and when they needed working on anyway.

Good news everyone! I’ve just uploaded source for the following to the CVS repository:

• chmod
• diff
• find
• modsqz
• squeeze
• srcbuild
• romlinker
• unmodsqz
• xpand (alternate target for squeeze component)
• CLX library (required by all the above)

With a little effort it should therefore be possible to compile all these tools for other platforms. (A few trivial functions in CLX need rewriting for non-RISC OSes.)

I’ve just uploaded source for the following to the CVS repository:

Hooray!

With a little effort it should therefore be possible to compile all these tools for other platforms

Sure, but for ‘chmod’, ‘diff’, ‘find’ tools of those names are readily available on any system you’d sensibly be cross building on. I haven’t looked at the sources yet, but I guess that ‘diff’ might be the Acorn version from, which is exceptionally poor, and the other two are plain ports, with possible RISC OS improvements. I would want to eventually replace all these with versions from the GCCSDK autobuilder, which will be much more up to date, but that’s not a priority.

‘chmod’, ‘diff’, ‘find’ tools of those names are readily available on any system you’d sensibly be cross building on.

True, but they came “for free” when we’d reimplemented as much of CLX as was required to build the others, so we thought we might as well release them.

Maybe I’d better run down what they all are for those who aren’t familiar with them:

• chmod is a plain port of Unix chmod – done so that makefiles can use a single syntax for changing file permissions irrespective of the platform you’re building on
• diff is the diff from the Acorn C/C++ suite. True, it’s not as feature-rich as GNU diff, but from a brief glance it still has a couple of features that GNU diff lacks
• find is the find from the Acorn C/C++ suite. This is more akin to Unix grep than to Unix find
• srcbuild and romlinker are important parts of the RISC OS build system – srcbuild is a metamake tool, and romlinker produces the final binary ROM images
• squeeze and modsqz produce self-extracting compressed applications and modules respectively, using a compression scheme that’s specifically designed for ARM code
• xpand and unmodsqz do the opposite to squeeze and unmodsqz

diff is the diff from the Acorn C/C++ suite. True, it’s not as feature-rich as GNU diff, but from a brief glance it still has a couple of features that GNU diff lacks

Really? Like? ;-)

squeeze and modsqz

I think we’re missing the includes used for non-native compiles here.

#ifndef __riscos
#include "unsqueeze1.h" 
#include "unsqrm1.h" 
#else

Really? Like? ;-)

Acorn tools in not-entirely-superseded shocker. ;-) I can’t see that GNU diff supports tab stops other than 8, or Acorn diff’s “squidge” option. Not exactly killer features, but I don’t see the harm in making it available. Maybe it’ll inspire someone to improve GNU diff?

I think we’re missing the includes used for non-native compiles here.

Good point. I can’t see those anywhere. Presumably whoever did the port forgot to add the extra files to CVS.

Fortunately they look fairly easy to reimplement. Basically it looks like they correspond to two assembler source files that contain the raw ARM code to insert into the compressed binary. So if you want to build it for a non-ARM target you’ll need to include the ARM code as a binary blob – it looks like these headers will have contained a static const int hex array definition.

Peter, with your GCC build system I don’t suppose you can try compiling the apps with soft-float? I’ve just been looking at Draw’s grid-drawing code (which gets called a lot). Aside from plotting with VDU operations/OS_Plot, it’s very float-heavy. I just wondered if getting rid of the FPU ops would help? Rewriting it as fixed-point would seem to be roughly equivalent to soft-float, but much more work.

(Perhaps a better way to do it would be to plot the grid unit to a sprite and then plot the sprite – but probably the required XOR plotting wouldn’t get graphics card speedups. Perhaps a sprite with alpha might work)

About the Squeeze source release, does the shared source licence prohibit building Squeeze for a cross compiling environment, because Squeeze would then not be running on a ARM processor?

If Squeeze can be run on other processors, we can use it to compress NetSurf’s !RunImage on the NetSurf autobuilder. At the moment the binary is uncompressed.

You’re unlikely to get a definitive answer on that unless you ask Castle directly. However, in my opinion, the licence is not intended to prohibit such use – especially since the output of Squeeze is an ARM executable. You would definitely be crossing the line if you inserted some x86 inline assembler into the source code of Squeeze. But even if you take a more cautious interpretation of the licence wording than me, you’d be very unlikely to be prosecuted by Castle for using the Squeeze sources for the purposes of building a RISC OS application.