Building Sunfish

Is it just me or are so many RO programs protected by obfuscated build procedures?

I thought it was just me, struggling to understand why a simple build of something that “obviously” works, didn’t work for me.

If I understand this all correctly then various of the source setups work on the authors’ system because at various times in history the author used different compilers, libraries etc. for different components and then never did a ‘clean’ rebuild of the whole package to discover the kind of mess Dave R has struggled through.

I think the general issue is there has never been a culture of rebuilding each other’s code. Generally authors ship binaries, and if they ship source then it’s just an archive of their build tree. There’s been no effort in making builds easy, and nobody submitting bugs when they don’t work. This is in contrast to Unix where traditionally source was the means of distribution and so you couldn’t run it if it didn’t build (and latterly package building automated that process).

I wonder if there’s a way to start afresh with new projects? For example a set of baseline tools we can expect everyone to have, and a means of getting extra parts from PackMan. That would make things less dependent on whatever random stuff you happened to have lying around.

Once you’ve caught your breath, I hope you have equal success on the front ends.

I’m not touching the front end, despite it being in dire need of a crashinessectomy.

What I’m trying to do is change the modules to make untyped file have a load and exec address postfix rather, so they don’t change in to a text file with crazy dates when I back them up via NFS. e.g. filename,12345678-ABCDEF0

I’d previously identified the 3 routines responsible for munging and unmunging the names, but of course it’s snowballed in to fixing up lots of other places files are assumed to always have a valid type, and passing around load and exec addresses as well as the filetype through a cascade of functions.

That’ almost finished, and when reviewed I can start the real nightmare of debugging under RISC OS. I’ll probably be wearing out the mains switch on the wall as the Pi 3B repeatedly stiffs itself.

I wonder if there’s a way to start afresh with new projects? For example a set of baseline tools we can expect everyone to have, and a means of getting extra parts from PackMan.

Hmmm, fully fledged build tools and debug utilities. For free, otherwise someone(s) will always jump to the other ship with "interesting results¹.

That would make things less dependent on whatever random stuff you happened to have lying around.

Back to the complaint from Rick (and other people) about various libraries – clashes, incompatibilities not to mention possible bugs, if you can find the library at all.

We don’t make it easy round here, do we?

¹ Reading Druck’s comments I get the impression the Sunfish code had a mixed “childhood” since he’s mentions both GCC and AMU used in the process of getting a complete package, and he’s still not battled the building of the front end elements.

Hmmm, fully fledged build tools and debug utilities. For free, otherwise someone(s) will always jump to the other ship with "interesting results.

Not at all. It would be entirely possible to use the DDE, and standardise a way of working around that. In fact, that’s one of the things being suggested here.

The main problem is the environment – the other stuff that’s there to do extra magic, where all the files go and how things like system variables are set up.

Back to the complaint from Rick (and other people) about various libraries – clashes, incompatibilities not to mention possible bugs, if you can find the library at all.

You’re suggesting that all new projects roll their own library code from scratch? That sounds like an efficient use of developer time.

Also, I’ve got two versions of DeskLib here… The one from riscos.info where there’s source control and a history of what has changed, and the one that I got as a big Zip of files from Rick’s Pi-based web server. Even if I got around to doing what I do with SFLib¹, working out what was different in API terms between the two would be a nightmare, I suspect.

We don’t make it easy round here, do we?

You’re right, we don’t.

¹ When SFLib is built on Linux, it spits out a Norcroft library project with the source in “c” and “h” folders, which can be built in the DDE with a double-click on the usual Obey file. It therefore works in both systems, but has one single set of sources. That’s quite useful for maintainability reasons.

Is it just me or are so many RO programs protected by obfuscated build procedures?

I think there are rather more mundane forces at play.

Except for the simplest of makefiles, I find it more difficult to understand them than C source code. There are numerous good and comprehensive books on C, but all we have for makefiles is whatever tutorials we can find on the Internet. Never as comprehensive or well organised as K&R, for example.

So people create their build systems as best they know how, and they work reliably for them. They distribute their code. We all use the code, and it just works. Therefore we have no need to look at the source code or the build system for years. Then something goes wrong. Eventually some brave soul tries to build the code, and finds they can’t, but by this time the author has left the scene. There is now a problem of how to reverse engineer the build system and to find the library files that the author so took for granted that he never thought to mention it (and probably couldn’t include it for copyright reasons).

Alex Waugh looks to have done a fine job of creating a build system that would work with either gcc or Norcroft. That’s a rare achievement, and should be celebrated.

The use of Perl to process some files is unusual, but it’s in no way whatsoever an obscure language.

I’m sure it must be possible to update the build system to work with today’s tools and libraries, although it will doubtless take a few hours. I’ve often found that collaboration, in real time or near real time, is very beneficial; explaining something to someone else forces us to organise things in our own minds, and two minds are always better than one – no-one has the monopoly on good ideas. Talking on ChatCube, anyone?

Following on from the above, it would be good to be able to include Sunfish and Moonfish in the RISC OS HardDisc4 distribution. Alex Waugh can probably still be contacted (it’s not all that long ago that I exchanged a couple of emails with him), and I think he might well be amenable to changing the copyright terms.

Back to the complaint from Rick (and other people) about various libraries – clashes, incompatibilities not to mention possible bugs, if you can find the library at all.

You’re suggesting that all new projects roll their own library code from scratch? That sounds like an efficient use of developer time.

I was thinking it would be one set of libraries, all on one site.
For me the logic says this should be the ROOL site. That makes it way easier for new developers to find ALL the libraries.

So people create their build systems as best they know how, and they work reliably for them. They distribute their code. We all use the code, and it just works. Therefore, we have no need to look at the source code or the build system for years. Then something goes wrong…

Exactly that.

I was thinking it would be one set of libraries, all on one site.

The problem is when there are multiple different versions of libraries. Either different because of deviations, or different because somewhere along the way multiple copies of the same thing came to be.

At the very least, if one uses a library then either give a pointer to the source and credit the author (so one can attempt to find it later on) or (if it is small) drop it into the source archive so people can, like, hit the ground running.

Much better than to discover that a library is needed is the compiler complaining unable to open “obscure.h” and looking in the MakeFile it seems to refer to xyzzy.o so you’re left wondering what the hell?

Then something goes wrong. Eventually some brave soul tries to build the code, and finds they can’t, but by this time the author has left the scene.

Luckily the only “weird” build system was Ovation (it seems it is all written and built from the command line!), but it wasn’t too hard to DDEify it.
The main problem is that some of the code kicking around predates the era of “everything is on the ’net” so trying to track down bits required by applications that use numerous libraries can be…difficult.

You’re suggesting that all new projects roll their own library code from scratch?

No, just that if one uses third-party libraries, there’s some mention of where to get the library code that is used to build the code!

and the one that I got as a big Zip of files from Rick’s Pi-based web server.

The one that will remain DDE specific. :-)

working out what was different in API terms between the two would be a nightmare, I suspect.

Well, there’s the “probably okay” way and the “definitive way”.

Definitive? Diff the headers.

Probably okay? Over there you have source control and a list of changes.
For my “hack”? I take it you’ve not looked at the big zip yet, or you might have noticed a file called “ChangeLog” (actually, there are two, there’s one in the “Libraries” directory, with a ‘!’ suffix – this is so it can be served by the direct source access to the library).
This documents the changes that I have made, in one easy to read file and not fragments scattered across a version control system. ;-)
[ it is here → http://heyrick.ddns.net/desklib/source/%21ChangeLog ]

Once you get beyond the conversions (yes, the build system is different and there’s no longer any “h_doc”), most of the changes are either bugfixes or adding stuff that I found myself writing multiple times so decided to just put it into DeskLib and be done with it.

I was thinking it would be one set of libraries, all on one site.

That sounds pretty restrictive, for a whole host of reasons.

For me the logic says this should be the ROOL site. That makes it way easier for new developers to find ALL the libraries.

There’s already stuff on riscos.info, to pick just one example. There’s the advantage there that libraries there tend to bump through the Autobuilder and drop out as packages like this one for use when building sources – which dovetails with another recent thread. This doesn’t help with code built against the other version of that library, of course.

This is one of the reasons why I keep advocating that people use things like GitHub. The code is fairly safe, and can be tagged for different releases easily¹. We could, perhaps, have a page on the ROOL Wiki linking to different projects which are out there. I think the Autobuilder can make use of them, too – which would mean that they could be installed by PackMan.

¹ Need a version of that library from five years ago? No problem: you can check the snapshot out of the VCS and use it just fine.

Perhaps there would be a way to write a universal Makefile, that works with AMU and GNU make, Norcroft and GCC, native and cross build. Then all you do is fill in the gaps: list of source files, libraries, target output. Maybe those are defined in a separate fragment of Makefile so the main one can be easily upgraded. After all, I don’t think anyone really likes writing make rules, they’re just a means to an end.

A variation would be to embrace a build system that isn’t Make, in which you can specify all the things in some config file and pass it to some ‘build engine’ that does the job. There are plenty of tools around like that these days – it would mean porting and localising the tool. Some of those build systems know where to find things on the internet if you don’t have them installed – that has pluses and minuses, but it avoids the problems of where to find libraries. Some of those just generate Makefiles but others take care of everything themselves.

Another thing that would help from a crossbuild perspective would be to ship Linux binaries for Norcroft: we know they exist, but is there a licensing reason not to make them available to purchasers of the RO version?

I’m not touching the front end, despite it being in dire need of a crashinessectomy.

I believe Phil Pemberton is looking at the front end. He has my working native build environment using GCC.

I actually got this building using a version of GCC (and the Sunfish source) which Chris sent me.

I’ve in turn sent Chris a Makefile which fixes some awful mess with the build process — where it has to use one assembler for softfloat and another for hardfloat. It’s all rather nutty.

Given that the latest build dates back to 2013, I’d like to see what it does with the latest compiler and bugfixes before I start poking at it. Especially after the comments earlier in the thread about potential bad code (nullptr check followed by accessing the null pointer?)

In all honest my first step needs to be finding the latest version of the source and getting it into a modern source control system (Subversion or preferably Git)… then start fixing the cruft. A second repository with submodules to pull in RTK and other dependencies might be a nice way to set up a build environment.

If I got my brain in gear I could name a decent number of RO programs with peculiar build systems. I myself are guilty of doing weird things like using Lua or Python as a build tool.

If it’s of interest, the Haiku branch of Jam builds for RO. I could never figure out what it wanted from me in terms of environment stuff though. Jam is pretty easy for C/C++ program building.

This whole building layers over time using different tools with no clean builds theory gives me the willies.Do people really do that?

makefiles from Norcroft and Gnu don’t really like parsing each other’s files. It’s due to some slight syntactic differences, and silly things like tabs and spaces.

This whole building layers over time using different tools with no clean builds theory gives me the willies.Do people really do that?

Just a theory, but given that I’ve come across source where it compiles to a working binary except you notice some old build cruft and an old component or two and clean them out and the source fails to build the bits you cleaned out.
At that point my mind says “there wasn’t a clean build done using this source set”.
When you’re doing it all as a learning exercise you end up learning something other than what you set out to do.

Well, where are we all on this topic? I see there have been several steps on the way, but I haven’t seen anyone claiming complete success, and I haven’t seen any links to a newly built version.

I think we all agree that Sunfish and Moonfish are very useful applications, and I think we’d like to see them in the RISC OS distributions, technical and copyright issues permitting.

Who’s up for a holiday-time collaborative push to get it all to build and to work? ChatCube has proved to be a good real-time messaging system for RISC OS. We could set up a group for all collaborators.

The only useful contributions I’ve made are to run the Perl build steps successfully on Linux, and to identify a place in the source code that guarantees a null pointer access in certain circumstances – the fix is trival.

I’m concerned that the source references the RTK library, which seems to be abandonware. However, the first stage is clearly to get Sunfish/Moonfish to build, more or less as is. Deciding what to do about RTK is another matter.

It’s always built OK with GCCSDK.

Have you built modules and apps that crash less than the original ones? Have you made them available for download somewhere? (Original licence is GPL.)

There’s still a bug in the RTK library that causes the SunFish frontend to crash after clicking menu more than once. The original Sunfish module has always worked. Not looked at the Moonfish front end. Phil Pemberton sent me some changes to the make file. I haven’t got around to creating the recipe in the GCCSDK autobuilder with the makefile changes yet.

That’s reminded me, I never finishing getting SunFish to support untyped files. I made some initial changes, then found a whole lot more to do, but inconveniently got a new job.

I’ve had a reply from Alex Waugh. He is happy for Sunfish and Moonfish to be distributed along with RISC OS, and would consider re-licensing them if necessary.

Presumably anyone who has submitted patches will also need to relicence.

Which is the GCCSDK autobuilder package? Sunfish just packages the committed !RunImage and friends.

Actually, I don’t think re-licensing would be necessary. The Sunfish and Moonfish code is GPL, which permits “mere aggregation” and does not force its terms on anything merely aggregated. RTK is under the LGPL.

So unless Sunfish or Moonfish link with something that is a closed source part of RISC OS, I don’t see a problem.

I’ve just received another email from Alex, in which he said:

“I have put the sources on GitHub with an MIT license, which should be permissible enough for anything.”

https://github.com/ajw498/NFS

I have of course thanked him.