Reading directories: Implementing FSEntry_Func 14

What is the valid lifetime of the “offset of next entry” value returned by OS_GBPB 9, FSEntry_Func 14 and friends? I can’t find this documented anywhere.

In particular what should happen if an object is created or deleted?

In particular what should happen if an object is created or deleted?

It all goes horribly wrong.

General rule: do not alter a structure while you are traversing it, particularly file structures.

General rule: do not alter a structure while you are traversing it, particularly file structures.

Of course, that isn’t an excuse for people to write bad iterators :-)

If you can fully deal with stuff being added/deleted while in the middle of iterating a directory, that’s great (e.g. making sure deleted items don’t show up in the results, and making sure any new items will show up). However this might be tricky to achieve within the limits RISC OS places (ideally you’d need a filesystem-owned block of memory to store the iterator state rather than a 32bit int)

If you can ‘gracefully’ deal with it then that’s good too (e.g. if the “offset of next entry” is a simple index into an array of files, added items might not be guaranteed to appear, and deleted items might cause other items to be skipped due to the array being shuffled).

Crashing horribly due to the offset not being validated correctly is the wrong way of doing things.

It all goes horribly wrong.

General rule: do not alter a structure while you are traversing it, particularly file structures.

That doesn’t consider the problem of other programs altering the filesystem, which in the general can happen at any time. This would imply the only safe way to use OS_GBPB is to read an entire directory in a single call, which requires allocating an unknown amount of memory.

Filecore provides a slightly stronger guarantee due to it being a local filing system that it won’t change if the client is not running in user mode and does nothing that may trigger callbacks. The simple option for my filing system is to read and sort the entire directory whenever FSEntry_Func is called with an offset of 0 (start of directory) or the path name is changed.

However this might be tricky to achieve within the limits RISC OS places (ideally you’d need a filesystem-owned block of memory to store the iterator state rather than a 32bit int)

The problem is there is no close operation in the API so there is no defined time to free said memory.

If you can ‘gracefully’ deal with it then that’s good too (e.g. if the “offset of next entry” is a simple index into an array of files, added items might not be guaranteed to appear, and deleted items might cause other items to be skipped due to the array being shuffled).

It seems that is what I have to do.

That doesn’t consider the problem of other programs altering the filesystem, which in the general can happen at any time.That doesn’t consider the problem of other programs altering the filesystem, which in the general can happen at any time.

This is true, although if it is really important you could try hooking into the various file vectors to detect files being created and deleted¹; though it might be simpler to make your code more bulletproof that it won’t fault if – say – you try to delete a file that has already been deleted (etc).

The alternative is to suspend multitasking (don’t poll) for the short time that you are evaluating the filesystem? But don’t get too paranoid over it – there’s nothing to say that a file wouldn’t be created in the directories you scanned a few moments later… ;-)

¹ Mmmm, what happens if a file is *Moved? It isn’t exactly created nor deleted…

Mmmm, what happens if a file is *Moved? It isn’t exactly created nor deleted…

 IfExist <path.file> ? 

After that you’re certain you’re trying to delete something that actually exists at the specified location.

I think I shall have to add a piece of advice to my application !Cat (which generates and displays a disc catalogue in graphical form) not to let other applications move, copy or delete files in the directory(ies) being listed until the list has been built up in meory and is being displayed. I though that this would be so self evident that I didn’t bother to mention it.

is to read an entire directory in a single call, which requires allocating an unknown amount of memory.

The directory iteration loop

*for leaf, ftype in riscos.dir (dirname) do ... end*

you try to delete a file that has already been deleted (etc).

That’s not the problem, consider not copying “Very_Important” if “Old_File” has been deleted. To illustrate the problem consider this BASIC program which will not find the file “Very_Important”:

*CDir RAM:$.Test
*Create RAM:$.Test.Old_File
*Create RAM:$.Test.Very_Important

DIM B% 1024
O%=0
REPEAT
  SYS "OS_GBPB",9,"RAM:$.Test",B%,1,O%,1024 TO ,,,N%,O%
  SYS "XOS_GenerateError",B% TO B$
  IF N% THEN
    IF B$="Old_File" THEN *Remove RAM:$.Test.Old_File
    PRINT "Found file ";B$
  ENDIF
UNTIL O% = -1

The alternative is to suspend multitasking (don’t poll) for the short time that you are evaluating the filesystem?

Not sufficient on Filecore as files may be created and deleted during Transient Callbacks, for example by ShareFS. And on a network filing system or similar, files can be deleted or created at any time by programs on other computers.

One of my ideas was for my filing system to split R4 into a stream handle and an offset and dispose of the data sometime later, what sometime later is was my original question. Permanently storing the offset to name mapping is not feasible except on a made for RISC OS filesystem; furthermore Unixlib fails unless the offset starts at zero and increases by one for every directory entry, the comments acknowledge this is broken:

It is a common mistake to assume that the offsets returned from OS_GBPB is monotonic increasing with unity steps from the start offset 0 onwards. Accoring to the PRMs, the only thing you may be sure of is that offset 0 starts the GBPB enumeration and offset -1 returned indicates the end of the enumeration.

After preaching the passion, now the hack … Well, we are going to rely on unity monotonic increase of the offsets and this up to GBPB_MAX_ENUM.

Yes, there was a thread on the GCCSDK e-mail list starting 4 March 2013 discussing this. I contributed to that thread and it appears I understood the issues at that point. I think the UnixLib problem is fixable, but I can’t remember exactly how I thought it should be fixed now.

It is a common mistake to assume that the offsets returned from OS_GBPB is monotonic

As far as I can tell the Filer when deleting files assumes that it is monotonic and decrements R4 for each file it deletes. This caused major problems for Fat32Fs which obeys the FAT specification and simply marks the catalogue entry as being a deleted file rather than removing it.
Incrementing R4 by the actual catalogue position in FAT32Fs is way faster than using a monotonic counter, so I sacrificed compatability with badly written programs for speed.

If you try the above program using FAT32Fs (or probably any other non-filecore FS) then the file “Very_Important” will be found!

As far as I can tell the Filer when deleting files assumes that it is monotonic and decrements R4 for each file it deletes.

On FileCore FSs, offsets are not monotonic, they’re not even offsets just a file count. Worse, the count is started from 0 every time. If you pass in ‘offset’ n then FileCore counts objects starting from 0 until it reaches n + 1 and will start returning objects.

This is why changes during enumeration can cause things to go wrong. To avoid this Filer_Action assumes that enumeration works as for FileCore FSs. For deletions it adjusts the ‘offset’ to continue at by subtracting the number of objects actually deleted.

Using FilerAction on FSs that don’t enumerate objects in a compatible way would be, er, interesting.

incrementing R4 by the actual catalogue position in FAT32Fs is way faster than using a monotonic counter, so I sacrificed compatability with badly written programs for speed.

Are the catalogue positions sequential with an increment of 1? If so, that should be ok but objects may be deleted more than once if using FilerAction. I assume FAT32Fs deals with this gracefully?

I get around the fact that the filer decrements R4 by using R4 = (Pos << 8) + 128

So when I receive it back again I do Pos = R4 >> 8

Then incement pos by 1 for each catalogue location.

If you try the above program using FAT32Fs (or probably any other non-filecore FS) then the file “Very_Important” will be found!

I’ve just tried it with RPCEmu HostFS on Btrfs, Sunfish on Btrfs, ArcFS, TBAFS, DOSFS, LinuxFS (my work in progress) on Btrfs and LinuxFS on tmpfs.

Only DOSFS and LinuxFS on tmpfs found “Very_Important”. LinuxFS currently use R4 a count into the results returned by getdents. and the directory is reopened on every call.