Git cheatsheet - working with superprojects

Introduction

When you want to build a complete RISC OS ROM or disc image, you need a specific collection of source components, from specific branches or revisions. The way we manage these build trees uses one git superproject for each ROM or disc build. Each superproject contains references to hundreds of submodules. Each superproject, as well as each submodule, has its history recorded in a git repository and associated GitLab project. Many of the workflow concepts are the same for superprojects as for submodules: there is a shared central project (for superprojects, these live in Products) which you can clone, but to contribute back to it, you create a personal fork project on the server and raise a merge request from there.

Where a normal git repository tracks revisions of files, and you edit those files using a text editor, a superproject tracks revisions of its submodules, and you edit the submodules using git commands. Because submodules can be checked out in two different ways (by revision, or by remote tracking branch), a single superproject performs the function which under CVS required a pair of Products files (which used to be distinguished using a Dev suffix for the version which checked out by branch).

In continuation of the policy used in CVS Products files, there is a strict convention for the format of commit logs for superprojects. We have rewritten the prodcommit script to perform the equivalent function for git, and it is available from RepoTools. In order to use this, you will need to use the command line.

Adding a submodule to a superproject

This is perhaps best illustrated with an example. Suppose IOMDHAL didn’t already contain Alarm and you wanted to add it. You would create a personal fork of the super-project on the server, request access if necessary, then locally type:

git clone https://gitlab.riscosopen.org/Products/IOMDHAL.git
cd IOMDHAL
git submodule update --init --no-fetch
git checkout -b AddAlarmComponent
git submodule add ../../RiscOS/Sources/Apps/Alarm.git RiscOS/Sources/Alarm
git add .
prodcommit
git push https://gitlab.riscosopen.org/myusername/IOMDHAL.git UpdateAlarm

And then you raise a merge request. This should look familiar, as it’s almost exactly the same workflow as made for code commits to a submodule. The differences are that instead of editing a file, you use git submodule add and instead of git commit or srcccomit, you use prodcommit.

Some further notes on git submodule add:

If you look at the man page for git submodule add, you’ll see it takes two parameters <repository> and <path>. If <path> is missing, it defaults to the leafname of the repository URL, which is not how our source tree is structured, so for our purposes we must always cite it. For <repository>, use a relative URL as shown in the example, so that submodules inherit the HTTPS vs SSH URL scheme the user specified for the superproject.

For our purposes, <path> will almost always be the same as <repository> but with the leading ../../ and trailing .git stripped off.

A submodule actually has a separate name attribute which uniquely identifies it within the superproject. This defaults to be the same string as <path> which is fine in nearly all cases. However, since it is used as a directory name within the superproject’s .git directory, we make an exception if the path contains a pling. For example, to add !Builder, we would use:

git submodule add --name RiscOS/Apps/PlingBuilder ../../RiscOS/Apps/PlingBuilder.git RiscOS/Apps/!Builder

A reminder that if you’re using an interactive Bash shell, you’d need to escape the ! using a \.

There’s one more switch you need to know about if you want the submodule to be checked out on a branch other than master when users do git submodule update --remote. For example:

git submodule add -b HAL ../../RiscOS/Sources/HWSupport/IIC.git RiscOS/Sources/HWSupport/IIC

Updating the versions of submodules used in stable builds

Here’s another example: suppose you wanted to change Alarm to be the latest version:

git clone --recurse-submodules https://gitlab.riscosopen.org/Products/IOMDHAL.git
cd IOMDHAL
git checkout -b UpdateAlarm
cd RiscOS/Sources/Alarm
git checkout master
cd -
git add .
prodcommit
git push https://gitlab.riscosopen.org/myusername/IOMDHAL.git UpdateAlarm

In other words, the revision that is currently checked out in the submodule is the state that is tracked by the git add command in the superproject. Note that if you already have a clone of the superproject lying around, there’s no need to fetch a clean copy for every little change – the above is for the sake of being a self-contained example!

You can specify any other tag or branch name you like in place of master.

It is also valid to do the following to update every submodule to the latest versions. Typically, this would be the first step in producing a new stable release, before then regressing any components that are not considered stable.

cd path/to/superproject
git checkout -b UpdateEverything
git submodule update --remote
git add .
prodcommit
git push -u myfork UpdateEverything

It is common to combine git submodule update commands with any number of submodule git checkout commands between prodcommit calls.

Changing a submodule’s branch

Less frequently, you may need to change the remote tracking branch which a submodule is checked out on by a git submodule update --remote command. This is how to achieve it:

cd path/to/superproject
git checkout -b ChangeAlarmBranch
git config -f .gitmodules submodule.RiscOS/Sources/Apps/Alarm.branch HAL
git add .gitmodules
prodcommit
git push -u myfork ChangeAlarmBranch

Or, if changing to master,

cd path/to/superproject
git checkout -b ChangeAlarmBranch
git config -f .gitmodules --unset submodule.RiscOS/Sources/Apps/Alarm.branch
git add .gitmodules
prodcommit
git push -u myfork ChangeAlarmBranch

Removing a submodule

This is how you’d remove !Builder (chosen to illustrate when to use the submodule name and when to use its path):

cd path/to/superproject
git checkout -b RemoveBuilder
git submodule deinit RiscOS/Apps/!Builder
git rm RiscOS/Apps/!Builder
rm -rf .git/modules/RiscOS/Apps/PlingBuilder
prodcommit
git push -u myfork RemoveBuilder

Again, remember to escape the ! if using an interactive shell.

Creating a new superproject

One way of creating a new superproject is to start with an empty git repository and repeatedly add submodules to it using the instructions above. However, this will be a slow and tedious process due to the large number of submodules in a typical superproject.

A simpler way is to start from an existing superproject:

git clone https://gitlab.riscosopen.org/Products/IOMDHAL.git MyNewProduct
cd MyNewProduct
git submodule update --init --remote --no-fetch

Then you can add or remove submodules, or change their branches as necessary, as previously described. Then start a new git history as follows:

# Remove association with original superproject
git remote remove origin
# Detach HEAD
git checkout --detach
# Delete all tags
git tag --delete $(git tag -l)
# Delete all branches
git branch -D $(git branch -l | grep -v HEAD)
# Create new master branch with clean history
git checkout --orphan master
prodcommit
# Create association with new GitLab project
git remote add origin git@gitlab.riscosopen.org:myusername/MyNewProduct.git
git push -u origin

Then drop code@riscosopen.org an email to let us know about the superproject. We’ll create a new central GitLab project, configured to match all the others, and transfer your repository across to it.