Subversion on Fat32FS

Tried moving over to a new 250GB Fat32FS drive (on RISC OS 5.28, Pi 4) and encounter an odd problem with subversion 1.6.23 from PackMan (or my older 1.6.17).

I can svn checkout a repository fine, svn update seems happy, but svn status reports that many of the local files are missing! Something is obviously being lost in translation twixt UnixLib and FAT32.

Does anyone have a file access logger that works on RISC OS 5 to see what filing system calls are actually being made?

[FileCore based filing systems working OK with svn on this system, of course.]

Worked ok here on Iyonix with a usb Fat32fs stick.
To get an output from svn status I had to use
svn -v status

I remember that ScummVM had the same issue a long time ago where occasionally you’d navigate into a directory on a Fat32FS drive using the built-in file browser and it would just show up as empty. I never figured out exactly what caused some directories to work and others to fail, but the problem went away when I rewrote the file handling in ScummVM’s RISC OS port to directly call the native APIs instead of using dirent from UnixLib, so I’d imagine that the problem is there somehow.

Have now tried that new large drive with latest Fat32FS 1.59 with no luck – still svn status is gebuggered.

Have now also done: svn checkout to dir onto Fat32FS, copy that dir to RAMFS, svn status of that dir on RAMFS is OK, so adding more weight to it being some name translation in svn status when used with Fat32FS that appears to be the issue. Maybe what’s returned by Fat32FS up to the OS interface of UnixLib dirent is not what it likes.

And also: svn checkout to dir onto RAMFS (all OK), copy that dir to A.N.Other Fat32 USB device, doesn’t help. About half the files are marked as missing still.

Is it likely to be down to the difference between c.foo and foo.c when stored on a FAT disc?

Is it likely to be down to the difference between c.foo and foo.c when stored on a FAT disc?

First thought, but svn status behaviour is not consistent between various subdirs in the repo! (or indeed other repos) Some are happy with their foo.h content but not their foo.c. Or both. Or neither. Some places it doesn’t like foo.png, others are fine. Some dirs it marks the entire tree below that place as missing.

I have, of course, failed to build a subversion on autobuilder to try to debug…

Another angle would be to try the troublesome fat32fs device/directories on a linux svn.
That would rule out a fat format/naming issue.

Ha! RISC OS svn is truly ancient and tree-incompatible with anything you’d find on a modern system.

I would assert that Fat32FS is getting the RISC OS < – > FAT32 filename translation correct as I can copy checked-out svn trees from FileCore to and then back from the Fat32FS devices fine using RISC OS, and that other svn operations are working OK regardless of which FS they are on. I think it has to be something in the Fat32FS enumeration that’s confusing svn status / UnixLib in just that one area.

The svn up must have to enumerate the directories to decide what to do next, so you are right it could be a weakness in the status command. Maybe it happens over a particular size, it didn’t happen here on a small project.

Without grokking the source, I’d think that svn update will be enumerating the remote repo (in this case, over svn: protocol) and checking each remote file against the local file, where svn status has to enumerate the local directory to warn you if you’ve deleted files (marks them as missing) or have additional files that you may want to add.

Could be the classical directory enumeration error that affect so many RISC OS programs: i.e. they stop when 0 objects are returned by the current call instead of checking that the next index is -1 (no more objects).

IIRC when Fat32FS encouters a deleted entry in the FAT, it simply returns (thus with zero objects if you asked for a single one), pointing to the next entry in the FAT.

This is beginning to sound like the age old problem of software authors misunderstanding the use of R4 in OS_GBPB 8,9,10,11 etc. It has been discussed at length many times.

E.g Here

Or the docs here

“Also, do not assume anything about the value in R4 – Filer_Action assumes that it increases by one for each entry read and comes horribly unstuck.”

Another possible issue, is that the program maybe assumes that the directory enumeration provides the objects in alphabetic order, which works only with FileCore as it is how FileCore store its directories on disk.

For example, ROM building on Fat32FS fails because one of the tools used to manage resources makes such an assumption.

I have built a current svn on my copy of the autobuilder. This version has no suffix swapping or filetype extensions but I can add back comma extensions easily.
Will have to work through the original svn patches to see what else will need put back in.
It is a large build and there are still few niggles to check up on.
For example there was no definition for uint and ulong so I just set them to unsigned int and unsigned long without looking into it deeper.
The wimpslot demand was nearly 6000k but after I changed the elfs to aif they didn’t seem to need memory at all?
Different hardware could react differently, if someone could advise what minimum to specify that would help.

This is beginning to sound like the age old problem of software authors misunderstanding the use of R4 in OS_GBPB 8,9,10,11 etc. … “Also, do not assume anything about the value in R4 – Filer_Action assumes that it increases by one for each entry read and comes horribly unstuck.”

Having had a look at readdir_r in SharedUnixLibrary, I suspect it might be down to it assuming that the R4 output from OS_GBPB 10 has been incremented by one for each object read – if the underlying filesystem skipped past certain internal/deleted entries in its dir structure, that may not be the case – and terminates if that is not satisfied.

[Amusingly there’s a comment near the top of the source saying this is indeed assumed for hacky purposes.]

Could be the classical directory enumeration error that affect so many RISC OS programs: i.e. they stop when 0 objects are returned by the current call instead of checking that the next index is -1 (no more objects).

Thankfully the authors were aware of that one!

The wimpslot demand was nearly 6000k but after I changed the elfs to aif they didn’t seem to need memory at all?

For ‘some time’, the OS has tried to determine a large enough WimpSlot when loading AIFs using the header data.

It makes it sound like the filesystem is not doing its job as a wrapper of the hardware level.
Off topic, but I noticed that
*fileinfo Fat32fs::Fat32_0.$.*
displays well formatted with Zap, but Edit and StrongEd seem to be missing the LF’s or looking for CR’s.
The standard dual table display of standard cat looks a bit messy in comparison.

It makes it sound like the filesystem is not doing its job as a wrapper of the hardware level.

Strong disagree there! It’s entirely up to the filesystem what it returns in R4 for that enumeration to locate the next object. It is most definitely NOT guaranteed to be unit monotonically increasing or interpretable by the caller.

And it’s not just svn that’s using readdir…

if it is a clear cut case, it will be easier than trying to provide a test case that is only on fat sometimes.
The changes would have to be done at the gccsdk recipe level I imagine. -Bedtime for me. -bye.

It makes it sound like the filesystem is not doing its job as a wrapper of the hardware level.

Three things that are very important to understand.

1. Do not assume anything about the order entries are returned. It’s normally alphabetic for FileCore, and order-added for DOS, but there is no guarantee to ordering.
2. Know that R3 can return zero (no entries) in the middle of a read. I’ve not seen this myself, but it could be provoked by, say, a network FS that has to go read data first.
3. Do NOT assume anything about the order of what is returned in R4. That FileCore returns sequential numbers is just an implementation detail. It could be a struct offset, or the next three digits of Pi. It’s for the use of the OS, not the caller.

Anything that makes such assumptions is wrong. Yes, FilerAction, we’re all looking at you…

I used the new svn to download gccsdk this morning -no problems, 24350 files and 174699406 bytes.
I am looking at dirent.c in the recipe, it looks like readdir_r() started life conformant with
} while (regs[3] == 0); causing re-reads but the monotonic clause in the middle comparing r3 to r4 with it’s return 0; could be the culprit.
It was introduced for the purpose of suffix directories, as was the nested inside itself readdir_()
I can rip that stuff out for my private build but the
gccsdk still builds suffix-swapping (for norcroft) capable binaries. So an improvement would have to be made to the existing.

Edit: I have rebuilt a gcc4 with the stripped down dirent.c and early tests show I haven’t broken anything (apart from suffix swapping.)
I guess the nested readdir_r was so it could recurse into the suffix folders simultaneously.
Many statements around the fact reg 4 but what are the possible
range of integers it can return?
Currently the dirent.c is comparing reg 4 to a MAX number as
well as the usual check for -1 (&FFFFFFFF)

Edit: I’m wondering, although the new svn is compiled static, you might need an accompaning unixlib module with the changed dirent.c to see any change?