BASIC 1.77 ARMv8 AArch32 opcodes

For information, BASIC 1.77 with an update to Assemble the later ARMv8 AArch32 opcodes has arrived unannounced in “Recent changes in GitLab” on the Welcome page in today’s, 30 June 2019, beta ROMs.

has arrived unannounced in Recent changes

It’s there in the longer list if you happened to know I wrote it at the start of June. For the very observant the Recent Changes list previously had FSLock next to the Kernel, but now it goes Kernel-SDIODriver-BCMSupport-FSLock, so 2 extras have been inserted mid-list.

From this we can infer that the date is the date the work was submitted rather than the date it got merged, where what might be more useful is to sort it by the merge date but display the author date?

In fact, the author date isn’t that useful if the date that pops out in the module (ie. controlled by VersionNum’s contents) is the merge date anyway.

In fact, the author date isn’t that useful if the date that pops out in the module (ie. controlled by VersionNum’s contents) is the merge date anyway.

That would certainly explain the changes on that page in the past few days when this:
“Correct exit of CallbackCode to pull PC not LR” appeared on the page before this:
“Export clock IDs in Hdr:BCMSupport” and this “Implement generic SDHCI version of GetTMCLK” but now the pages reads as though the JB submitted item was last.

Merge date and Version number being what the user would report in the event of a problem, so the list should be sorted in an order that reflects that and thereby gives the maintainers a better pointer to any specific merge which may need to be reverted (or fixed).
So, as you say, sorting by merge date is more useful.

For information,

I really miss the revisions page: What’s the point without it?

You seem to be “on top of things”! Could you host a page of changes that we mortals could understand until things get back to normal?

I really miss the revisions page.

The thing is, Git doesn’t really do revisions in the sense that older VCS did. That’s one of the reasons that it works better for more complex usages than things like CVS and Subversion.

What’s the point without it?

That the system works better for what it’s actually there to do: to manage the RISC OS source code?

You seem to be “on top of things”! Could you host a page of changes that we mortals could understand until things get back to normal?

That sounds like a lot of work!

to manage the RISC OS source code?

Given that an operating system is a huge project with many components that gets lumped together to create one final output (ROM image), would you not consider that a critical part of that is having easy access to everything that has changed from one revision to the next?

That sounds like a lot of work!

Okay, so something has changed around <date> which is causing <this> to happen. Possible culprits are… The filer, FileSwitch, or random bits of the entire Desktop system. It ought to be fairly simple to track down given a date of what did and didn’t work.
But, wait, can we scroll back to that date, or are we obliged to look at every single component to see if a change was made at the time in question?

Sane access to version logs, diffing, all those sorts of things are as important as the ability to merge this with that. And, of course, roll back. ;-)

Remember, also, that there are people who are not particularly nerdy who are running recent versions. Maybe they’d like to know what’s changed and to judge if it’s worth updating?

Finding out the merge date “from afar” is doable, but a bit clunky if you’re working from the web interface. The sequence is:

1. Scan the “Events” page, with a yesterday’s copy of the page to hand so you can spot new entries appearing further down the list.
2. Click on the URL title of the newly added entry.
3. Click on “Commits” (in the breadcrumb trail above the author’s name on the page)
4. Click on the commit entry that has the same name as the one that brought you here
5. Scroll down to the “VersionNum” listing on that page
6. Read off the date highlighted in green.

On the whole it would be a lot easier if ROOL adjusted the “Events” page to use the merge date instead of the commit date.

(Edited to correct faulty terminology in item 2)

All that being said…

I really miss the revisions page: What’s the point without it?

…is there anything particularly wrong with https://www.riscosopen.org/viewer/events ?
It appears to show a running list of what’s changed along the way. It’s different to the old way, certainly, but it seems to do the job. I can see FSLock got fixed “on Friday”. Tapping on the title takes me to a diff showing exactly what changed.

The only thing that stands out for me, and this applies to GitLab as a whole, is the widespread use of monospace type on the actual text of what’s changed in a given revision. This is a nightmare on a mobile device with the need to flick left and right endlessly to read anything.
It seems that GitLab does support a mobile view distinct from the desktop view. Would it be possible to consider dropping white-space: pre-wrap; into the CSS for the element, if it’s a pre block, to see if that improves things?

I am totally unfamiliar with Git … as are a lot of others here, I suspect.
What is the difference between ‘Merge’ data and ‘Commit’ date?
I would have thought that until a change was committed it would not be included in the ROM or disc images?

It would be really useful if there was a documentation page somewhere which started to explain how Git works (particularly in the RISC OS context), what the terminology means, and how to navigate the GitLab pages. It no doubt would be incomplete until things settled down, but it would be a start. It would be much easier to read and understand (and update) than trying to trawl through old forum posts. Any offers to start one?

Given that an operating system is a huge project with many components that gets lumped together to create one final output (ROM image), would you not consider that a critical part of that is having easy access to everything that has changed from one revision to the next?

Yes. However it’s never particularly clear in any complex project, and Git’s clever workflow can make it hard to see for those more familiar with older systems like Svn. I’m starting to find some clarity dawning now that I’m using Git in my day job (in the previous job, and for RISC OS stuff, it was/is Svn), but it can still take a lot of thinking about.

I suspect that this is going to be a work in progress for ROOL. I’m sure that constructive input will be appreciated, but complaints that it’s not like CVS probably don’t help much. :-)

Okay, so something has changed around which is causing to happen. Possible culprits are… The filer, FileSwitch, or random bits of the entire Desktop system. It ought to be fairly simple to track down given a date of what did and didn’t work.

But, wait, can we scroll back to that date, or are we obliged to look at every single component to see if a change was made at the time in question?

Sane access to version logs, diffing, all those sorts of things are as important as the ability to merge this with that. And, of course, roll back. ;-)

Remember, also, that there are people who are not particularly nerdy who are running recent versions. Maybe they’d like to know what’s changed and to judge if it’s worth updating?

My issue was more that John seemed to be expecting someone else to do this manually for him, pending an automated solution.

Remember, also, that there are people who are not particularly nerdy who are running recent versions. Maybe they’d like to know what’s changed and to judge if it’s worth updating?

Related to that, is beta testing by those users considered worthwhile? If they don’t know what they’re looking at…

What is the difference between ‘Merge’ data and ‘Commit’ date?

In git terms, I think it’s more correct to refer to these as the commit date and the author date.

The author date is the date the code was first committed to git (or if the code was submitted as a patch file, e.g. via email, git can use the timestamp information contained within the patch). Note that typically this first commit will be to the developer’s local copy of the repository – so it can still be several days or weeks until the code is exposed to the wider world.

The commit date is “when the code was last applied to a branch” – e.g. for a patch file this will be the date/time at which the code was committed, and for merge or rebase operations it’ll be the time of that operation.

Although git doesn’t have builtin support for tracking it, some systems also track the date of “push events” – i.e. whenever a developer pushes their local changes to a remote, shared server (like ROOL’s gitlab).

So a typical sequence of events could be the following:

• Developer starts a local branch to work on a feature
• Developer commits their changes to that branch on Monday (commit + author date set to Monday)
• Developer pulls changes from the remote server on Tuesday, and merges / rebases their local work into the main branch (commit now dated Tuesday, author still Monday)
• After some local testing, on Wednesday they push the changes to the remote server (push Wednesday, commit Tuesday, author Monday).

If the build machine spits out a new ROM just before midnight on Tuesday, and again just before midnight on Wednesday, then someone who tries to work out what’s changed between the two ROMs just by looking at the commit dates will be confused: the changes for Wednesday won’t show anything, because although the code was pushed on Wednesday, the commit date was Tuesday. But Tuesday’s ROM didn’t contain the changes, because they hadn’t been pushed yet. E.g. if you look at the history of Ben’s fork of the Pi HAL you’ll see that the most recent push event was just before midnight on the 24th, but if you click on the compare link you’ll see that it contains commits with several different author dates (and if he’d done any local merges during that time, you could see commit dates covering multiple days).

It looks like accepting a gitlab merge request results in a fake push event being generated (“fake” because the merge operation is applied server-side, so there’s no need to push anything), so we many end up with some confusing situations like the above.

Also there’s an extra layer of complexity with ROOL’s setup because everything is based around submodules (and the submodule references aren’t updated when merges are applied / when the build machine runs). Since committing every submodule update could result in merge conflicts for developers, perhaps the cleanest solution for handling this would be to make it so that every zip file contains a list of the commit hashes that were used to build the code. That way we’ll at least be able to precisely identify the code that was used, even if it’s still a bit of a headache to track down exactly what changed (or which commit caused a breakage – git has a handy ‘bisect’ command which allows you to binary search through the history to find which commit caused things to break, but when your repository is a flock of loosely-tracked submodules it can’t really help you much)

My issue was more that John seemed to be expecting someone else to do this manually for him

Ah. Okay. Fair point.

is beta testing by those users considered worthwhile?

Always.
I suspect non nerd users may have different ways of using the devices and/or different behaviour expectations. So they may encounter different issues.

I really miss the revisions page: What’s the point without it?

Without knowing what has changed, there seems little point in downloading and trying each new ROM image.

If I knew, for example, that something in the USB handling had changed, I could keep an eye on that aspect to try to notice any problems arising.
As I understand it, that’s the point of the beta ROM images.

You seem to be “on top of things”! Could you host a page of changes that we mortals could understand until things get back to normal?

That was addressed to Dave, who has twice recently very kindly alerted us to changes/new bits in the ROM.

If he is going to continue to point out changes of interest, and I hope he does, it might be more generally useful if he did so on a single thread, thus providing a partial replacement for the Revisions page for us less-clever people.

I was not expecting anyone else to take this on, certainly not Steve!

After some local testing, on Wednesday they push the changes to the remote server (push Wednesday, commit Tuesday, author Monday).

I think my point was, from a user perspective, only that last operation on Wednesday (which bumps the VersionNum file and stuffs Wednesday’s date in, triggers a new ROM to be built) has significance. Hence sort by that date might make more sense.

Under CVS I would similarly make changes, forget about them, weeks or months later I’d find them and submit. It was only the latter date that mattered not the date I originally did it.

If he is going to continue to point out changes of interest, and I hope he does, it might be more generally useful if he did so on a single thread, thus providing a partial replacement for the Revisions page for us less-clever people.

Did you miss the link to the replacement? It’s just here and also linked from the front page, and also from the downloads page.

from a user perspective, only that last operation on Wednesday (which bumps the VersionNum file and stuffs Wednesday’s date in) has significance. Hence sort by that date might make more sense.

Agreed.

Although if the activity on Wednesday was just a git push operation, there’d be no VersionNum bump – if we want VersionNum to reflect the merge/push date then that’s ROOL’s responsibility to build the relevant steps into their workflow. E.g. Ben’s kernel change – the VersionNum change in the merge request is dated the 20th, but the merge request was created (& accepted) on the 24th. The kernel activity page suggests that commit 9a1da777 is the merge request being accepted (with a VersionNum date of the 20th), but if you look at the history of the VersionNum file itself you’ll see that it’s commit fc49beaf which contains the merge (with VersionNum date of 24th). So it looks like their workflow does already have a step for updating VersionNum when accepting merges, but the act of doing so makes it harder to keep track of changes in git because the merge request thinks 9a1da777 is the merge, but that merge has been orphaned and replaced with fc49beaf.

I have found another little bit of confusion!!

On the ROOL Welcome page under recent changes is Pushed to branch master in SDIODriver (Ben Avison) – Implement generic SDHCI version of GetTMCLK which refers to SDIODriver 0.13 (24 Jun 2019) Pi4.1. What I actually have in the RPi ROM of 30Jun19 is SDIODriver 0.14 (29 Jun 2019). Both version look to be identical in that from the source 0.14 does have the change in c.probe.

The missing bit of information is anything to say that SDIODriver 0.14 has been merged, unless I have missed it!!

Perhaps another cup of tea might help.

Yep, that’s another instance of the problem I just mentioned. I think ROOL are being naughty and rewriting history when they accept merge requests, possibly because gitlab doesn’t directly support the kind of workflow that ROOL want to use.

I’m sure that constructive input will be appreciated, but complaints that it’s not like CVS probably don’t help much. :-)

For those new to git, but well-versed in the old centralised version control stuff, there are three things you absolutely need to know to understand git (and another thousand things that you should also know to clear up much of the confusion):

• what formerly was a simple “commit” is now a two-step operation: a “commit” which makes the change in the local repository, and a “push” that transfers this change to the remote (and probably in the usual simple git workflow the central) repository
• there are no automatically incremented version numbers – git only operates on hashes, and every commit (and every branch and every everything) is represented by a unique SHA-1 hash. So it is not possible to determine anything useful from the “version”, i.e. the hash, that git allocates to everything, apart from being an ID for that everything
• while CVS kept a history of every file, git (as SVN also did) versions the whole repository. So every hash represents exactly one state of the whole repository

Hope that clears up the confusion.

Git submodules complicate things further, and they seem to be not widely understood nor used. I only remember that there were various good reasons why we (at work) didn’t use them, but not what those reasons were/are…

perhaps the cleanest solution for handling this would be to make it so that every zip file contains a list of the commit hashes that were used to build the code.

Excuse me for asking, but wouldn’t that file be rather large? People routinely reference a hash by quoting its first 32 bits, but it’s actually 160 bits long (and being essentially random data it wouldn’t compress all that well). Since you can’t use “all commits since date X”¹ you would have to include every commit, every time. What’s the total number of commits on the GitLab server? I’d guess it’s well into the thousands (just from the CVS import) so it’s possible that a collection of hashes could be measured in megabytes…

¹ Or maybe you can. Perhaps Git does have a “flag day” function of some sort.

Did you miss the link to the replacement?

I did indeed. This obviously negates my concerns. Thank you! Pity it’s called events and not revisions, otherwise I might have discovered it myself!

Thank you!

Excuse me for asking, but wouldn’t that file be rather large?

You’ve misunderstood. The way commit hashes work in git is that they uniquely identify a node in the commit graph – they’re a hash of what the commit changed, and a hash of the parent node(s) that the change was applied to. So if you know the hash of a commit, and you have access to the repository, you can query the repository to get the full commit history relative to that known commit.

By “list of the commit hashes” I meant that we’d list the commit hash of each submodule – i.e. the commit hash of the kernel, the commit hash of the HAL, the commit hash of the build scripts, etc. So around 100 or so commit hashes.

The alternative would be for ROOL to have the build machine commit (and tag?) a change to the relevant “product” repository each time it does a build – that way we’d only need a single commit hash (or the tag) to find the relevant code.

Hope that clears up the confusion.

Sort of.
It does rather suggest to me that git is simply not the setup to use for the OS setup we have in place.
Unless of course we are all “holding it wrong” (thanks Apple for that immortal phrase)

I’m led to understand that git is appropriately named…

And, of course:

There’s also https://ohshitgit.com/ (NSFW, unless your W means having git inflicted upon you)