Build environment and CVS work

I’m trying to get myself set up with a build environment, but I’m slightly confused about how you’d go about a CVS update, and how things would work if eventually I was wanting to submit changes back into the CVS repository.

At the moment, I’m fetching the sources with Cygwin CVS, then moving them to somewhere they can be got at from within RPCEmu Spoon (0.8.6; RO 4.02; SA) in order to build the code. But I see that you have to combine several directories together (‘castle’, ‘gpl’, ‘mixed’) and also delete all the CVS dirs. In which case, if you were working on a build and made a bunch of changes to a number of files, wouldn’t you have to unravel all the changed files back into the original dir layout they came from before moving them back to within cygwin’s reach to do a cvs update or commit?

This sounds like a serious faff, so I’m sure I’ve probably missed the point somewhere by a wide margin. Could someone let me know how it really works in practice? I have only very superficial knowledge of CVS, if that isn’t already abundantly clear!

Unfortuantely, that is pretty much how it works in practice – you’ll have to fish around and copy your modified files back to their original locations.

GNU diff (not the crappy Acorn one that comes in the build tree that doesn’t do unified diffs!) is invaluable for finding exactly which files you’ve changed and where they came from. Unfortunately you still need to go through the diff output with a fine toothcomb to find what you’ve actually changed – all the CVS folders, object files and makefile dependencies mean that 99% of the diff output will be superflous.

Sometime tomorrow I’ll try and update the wiki with the tips & tricks I’ve learnt for dealing with CVS.

Seems like a tough job.. Using SVN it would have been quite trivial, ‘svn diff’ would cleanly show all the differences without much effort. Is there any objection to having a parallel SVN repository which can both be used for experimentation by people such as you and to enable quicker development methods? Lack of knowledge on my part in respect to CVS may also be debit to my opinion..

Guys, since about RISC OS 4 you can set the magic variable:

*Set Filer_Action$Skip Yes

in your boot sequence somewhere – this will cause all of the CVS-related files and directories (e.g. ‘CVS’ and ’.cvsignore’ files) to be skipped when you use the filer to copy stuff. Thus, when copying your checked-out sources from your Linux machine onto the RISC OS build machine (assuming you do it that way), you don’t get that stuff in your build directory.

As for putting stuff back, I don’t think it’d be too hard to throw something together which knows where each component came from in the repository – e.g. when you merge all of the ‘castle’, ‘bsd’, etc. bits into one combined tree, it tells you where a given component came from.

With a little more work, I could probably modify 7backup to take your build directory and your sandbox as parameters and move only the significant stuff that’s changed in your build over to the sandbox. It’d probably have to be a bit clever about how it works, but is doable.

And I would strongly warn against any parallel repository purely for the reason that you’d then end up with change logs being arbitrarily distributed over the two locations unless people are very disciplined in how they work – which they wouldn’t always be.

Even worse, we’d have people saying “oh, I don’t have your problem because I’m using version x from the svn repository”. Argh!

GNU diff (not the crappy Acorn one that comes in the build tree that doesn’t do unified diffs!)

Are you sure? gnu.diff—help on the command line seems to indicate there is a unified output option. Maybe you just mean the crappy FrontEnd GUI for it doesn’t include an option for unified output. That’s probably true.

Ah, I see what you mean. I reckon doing a front end for the gnu.diff tool would be helpful (I’m sure we used to have one back at Acorn) – especially if we add a RISC OS-specific command line argument to gnu.diff to ignore CVS-related cruft in its output.

Steve I can understand your concern. What bothers me a.t.m. is that though I can compile roms i need to jump through hoops to get the latest source code in the right place and format. Even the tar.bz2 files are cumbersome to be honest. Manually moving dirs and getting ‘file open’ errors really does not help. I noticed that after running !UnTarBZ2 it is best to type ‘close’ in a taskwindow to make sure everything gets copied correctly. Also i have a very small work-around to get everything properly compiled with an older version of the Norcroft c compiler. So each time i need to manually fix that too.

So I hope you understand that I am not advocating more problematic solutions but merely investigation how we can make things more streamlined.

When I’ve been poking around the code with the files stored on my Linux server (either via NFS or on RPCEmu), I’ve had a number of trees:

1. The plain tree direct from CVS
2. The tree with ‘castle’, ‘gpl’ etc directories merged together
3. The tree with CVS directories removed

3 is the one I build from, but it’s possible to diff between 2 and 3 to see changes. Converting 1 to 2 to 3 is a simple shell script.

It might be useful to have another tree 4 – the plain tree direct from CVS with changes between 2 and 3 passed in, which would then work with cvs diff, cvs update etc. That would require unpicking back into ‘castle’, ‘gpl’ etc directories. That might not be too hard if it could use tree 1 for reference. It would need to be told what happened to new source files.

Are you sure? gnu.diff

Ah, I hadn’t realised that there was a proper GNU version in the GNU folder. I was just running ‘diff’ from the command line and finding that instead of running the GNU version from my library folder it was running some non-standard presumably Acorn-ified one that didn’t provide an option for unified output.

I reckon doing a front end for the gnu.diff tool would be helpful (I’m sure we used to have one back at Acorn) – especially if we add a RISC OS-specific command line argument to gnu.diff to ignore CVS-related cruft in its output.

A smarter ‘diff’ tool that automatically ignore CVS files, object files, makefile dependencies, etc. would certainly be useful. However unless it’s 100% perfect I don’t think it would provide any real advantage over a diff + sed combination (which would probably take much less effort to get working to an acceptable degree).

Manually moving dirs and getting ‘file open’ errors really does not help. I noticed that after running !UnTarBZ2 it is best to type ‘close’ in a taskwindow to make sure everything gets copied correctly.

I can’t say I’ve experienced any problems like that myself – but then again, I rarely use !UnTarBZ2. What system are you trying this on?

Getting slightly off-topic, but the only time I ran into difficulty with !UnTarBZ2 was when I was playing around with it last night trying to set up a disc image from inside RPCemu. The fact that it requires additional modules to be installed doesn’t make it particularly nice to use from a bare system – plus it’s supplied as a zipfile, so you need to install one decompression program to allow you to decompress the other! I think I’ve mentioned this before, but it would be nice if we had a tool for generating self-extracting archives, just to remove all the hassle associated with setting up a disc image on a bare system. (Plus a standalone extraction tool for if you need to extract something that’s too big to fit in RAM)

As it happens, Steve & I were having a mutual moan about the pain of setting up RPCEmu from ‘blank’ just a day or two ago, since we’d both been doing it in order to test out the IOMD ROM.

We’ve had a few ideas on the best way forward but nothing concrete yet. The problem is more complex than it might appear “from the outside” due to issues about which OS/platform various things need to be built on and how repositories or binaries are synchronised with the web site.

Yes. And having a self-extracting UnTarBZ2 that doesn’t have external module dependencies and can also handle zip files would be a big step forward, I reckon. One for a weekend some time…

”!Raspberry”, because it gets us out of a jam.

When I’ve been poking around the code with the files stored on my Linux server

Yes. For the cross compiling work, I have a script which builds a symlink farm, and generally undoes a lot of the stuff that would be considered non-canonical if CVS/SVN was set up with RISC OS stuff now.

I too, think the current setup is a bit crazy, but I understand moving away from legacy setups can be hard. I’d very much like to see SVN in place – faster, easier to make light-weight branches, etc, etc, but ROOL have a lot of stuff tied into it.

Yes, it would be possible to maintain a separate SVN tree, and then have someone merge – but as Steve noted, you do need to be very careful, and there is lots of opportunity to miss stuff. On balance, however, given the slightly awkward alternative, this might be better.

Personally, I’m a bit reluctant to do any ROOL source work that I can’t cross compile (and therefore making way for compily natively with GCC), so I’d be looking at options in this direction.

Thanks for all the useful input, people.

Sometime tomorrow I’ll try and update the wiki with the tips & tricks I’ve learnt for dealing with CVS.

That would be really helpful, Jeffrey, if you can spare the time away from your video driver work. I wonder if you could elaborate on this statement in the https://www.riscosopen.org/wiki/documentation/pages/Cortex-A8+port+source+code wiki page:

Grab the source from CVS. You’ll probably want to do this from Linux or Cygwin otherwise the admin scripts won’t work.

Does that mean it’s effectively impossible to use the RISC OS CVS client to get the sources, or just more difficult, or just that a few (possibly non-essential) scripts don’t work?

Guys, since about RISC OS 4 you can set the magic variable: *Set Filer_Action$Skip Yes

Steve, thanks for the tip. I didn’t know anything about this. Could you explain a bit more about the syntax, and what exactly this does? Is it documented anywhere? This is definitely the kind of thing that would be very helpful to have on a wiki page somewhere…

I know very little about software licences, so forgive me if this is a really silly question, but… Can I ask what the rationale is behind having the sources split into the different directories for different software licences? It seems to me that from a development point of view, a much more streamlined approach would be to have the sources in CVS in a structure that mirrors the structure required for building.

And would it be very hard to have the build system scripts ignore the CVS directories and log files so that removing them wouldn’t be necessary?

If any work is going to be done on the build system as a whole, I’d say changing those two things would be the most beneficial changes that could be made. It seems a shame that developers will have to spend a significant percentage of their precious time on housekeeping files, not to mention the increased opportunities for bugs to creep in if you forget to copy some files back, or miss a change that you made in an obscure file somewhere.

And one more post, just for luck…

”!Raspberry”, because it gets us out of a jam.

”!Onion”, because it gets us out of a pickle?

If the system variable Filer_Action$Skip is set (to anything – even the null string), then during a Copy operation, Filer_Action will skip directories called “CVS” and files called ”/cvstag”. Variable is read whenever Filer_Action desktop task starts (ie. at the start of each copy). This dramatically reduces the time it takes to copy a checkout tree from (for example) NFS to RISC OS.

See changelog here.

Can I ask what the rationale is behind having the sources split into the different directories for different software licences? It seems to me that from a development point of view, a much more streamlined approach would be to have the sources in CVS in a structure that mirrors the structure required for building.

So that we and everyone else can be absolutely clear which bits of code fall under which license. If, for example, someone were to build a ‘GPL’ component into something else which is under the ‘Castle’ license, then they would be in a mess, because the two licenses are mutually incompatible.

If this is really such a big problem, we could easily change the build system so that we preserve this new directory structure by just modifying the paths in the BuildSys.ModuleDB to each component (and probably fix a couple of paths hardwired into components themselves referencing other stuff).

And would it be very hard to have the build system scripts ignore the CVS directories and log files so that removing them wouldn’t be necessary?

Yes. There are a million(ish) places where the contents of a directory are important. For example, think about how a makefile might copy a resources directory from the sources into the Install directory when building, say, !Paint. You’d end up with CVS subdirectories inside the installed application!

Your ‘problem’ here really isn’t a big deal. You can either use the system variable I mentioned, or run your build tree past the !EraseCVS application to remove the cvs stuff. See castle/RiscOS/Apps/!EraseCVS.

Your ‘problem’ here really isn’t a big deal. You can either use the system variable I mentioned, or run your build tree past the !EraseCVS application to remove the cvs stuff. See castle/RiscOS/Apps/!EraseCVS.

Thanks for the explanations here, Steve. I was aware of !EraseCVS already – and use it. The Filer_Action$Skip trick is new to me, and I can already see it will save a load of time.

I can see it’s not that big a deal to get the sources in a state that’s fit to build, but isn’t it a bit tricky when it comes to ‘cvs update’? You don’t have any choice but to copy any changes you’ve made back into a directory structure that still has all the CVS stuff in it, do the cvs update, and then go through the ‘copy back with CVS stuff removed’ step. Probably it’s not too much of an issue if not a lot is happening in the sources (i.e. few developers and not many files changing), but if more and more people start wanting to chip in, maybe this will become more of a stumbling block? What do you think?

In an ideal world, we’d move to a different source control system. This has been discussed many times in the past, but the main difficulty there is that there is currently no solution which can move our cvs repository over to a new repository format while still preserving our various branches, revision history and some odd edge cases. We’ve tried various tools in the past and they all failed leaving a mammoth manual job to finish it.

On top of that, we’d have to make major changes to the backends of our website if we moved to another system (plus various related tools, such as the autobuild system).

That’s not to say we wouldn’t want it to happen, it’s just such a massive job that it’s not even on our medium term plans.

And I’d rather see something more modern than svn (which has a list of problems almost as long as cvs) if we did bother migrating. Maybe Mercurial or something. But with modern source control systems, like Mercurial, I can’t immediately see how we’d manage having hundreds of separate components all with individual versions, tags, etc. being developed independently of each other in the same repository. Mind you, I’ve not stopped for very long to think it through…

I don’t really know anything about the pros and cons of the various source control systems. I was just wondering how difficult it would be if, staying with CVS, the sources that you check out could remain in that format for building, so it wouldn’t be necessary to run diffs all the time to see what’s changed where before doing an update or commit.

The way I was seeing it, was that if you could dispense with the different directories for ‘castle’, ‘gpl’ and ‘mixed’ licences, and add some extra code into the various build tools and scripts to ignore cvs folders and files, you’d basically have it.

But I can see that even that is quite a tall order. Sorry, just musing!

I’m looking at doing the first bit: modify the build so that you can keep the sources in the same structure as cvs. The second bit is simply not going to happen, but there may be ways to simplify the problems therein that I’m thinking about.

Here is a new wiki page detailing how to set yourself up with CVS write access, and how to use CVS and the admin scripts for day-to-day tasks. It’s basically just a wikified version of the guide that Ben sent me when I was first given access, plus the little bits that I’ve learnt after that. There’s also a fancy sed script for cleaning up the diff output when you diff your build tree against the three folders grabbed from CVS. Although I’ve been using a sed script for quite a while to help clean up the diff output, this one is much better – for my current build tree it reduces the diff size (of a relatively unmodified tree) from 900k to 46k!

Grab the source from CVS. You’ll probably want to do this from Linux or Cygwin otherwise the admin scripts won’t work.

Does that mean it’s effectively impossible to use the RISC OS CVS client to get the sources, or just more difficult, or just that a few (possibly non-essential) scripts don’t work?

The admin scripts are just perl scripts, so there’s a fair chance you might be able to get them working under RISC OS. Working without them is possible, but I don’t think ROOL would like it – apart from the bits the scripts make easier (e.g. getting the right components for a particular product) they also do important housekeeping operations when you submit changes (prompting you to write a sensible commit message, modifying the versionnum files, tagging components, etc.)

Also, as Steve says, the hassle of having your build tree in one location and your CVS tree in another isn’t a major problem – or at least not for me, since most of the time I’m the one submitting code, and often only once or twice a month since I’m making big changes instead of lots of little individual tweaks. If you can remember to maintain a list of all the files you’ve modified (or at least of the components you’ve modified) then that’s a help also, as it will save you from sitting around for several minutes waiting for a full diff of the trees.

I’m looking at doing the first bit: modify the build so that you can keep the sources in the same structure as cvs. The second bit is simply not going to happen, but there may be ways to simplify the problems therein that I’m thinking about.

Steve, thanks a lot. This would be a good step forward, I think!