FileCore long filename fail

File this under “things I regret offering to look at”.

It’s been brought to my attention that using *cdir it’s possible to create directories with names longer than 255 chars, and that once created those directories may not be deletable via *wipe. Indeed, DiscKnight will complain that the directory name is bad and will attempt to truncate it for you.

As I understand it 1K is meant to be the limit for pathnames and 256 chars for leafnames (including terminators?). However it’s rather hard to track down any specifications which actually state these as being the design limits. The best I can find is the RISC OS 4 features list, which only says “the maximum filename limit is 1024 characters”, making no indication of whether that’s the pathname limit, leafname limit, or both. Presumably there’s a specification somewhere for how the E+/F+ big directory structure is meant to work, but wherever it is it doesn’t look like it ever found its way into our CVS.

As far as FileSwitch is concerned these limits are correct – see StaticName_length and util_objectname in FileSwBody (so is that 256 chars excluding terminator?). But for FileCore? It’s hard to say. BigDirMaxNameLen proudly proclaims that the maximum name length is 255 chars. But that value is only ever used for runtime stuff – it doesn’t seem to factor into the on-disc structure at all. The on-disc structure is entirely dynamically allocated, with no fixed length for the names. It even uses a word to store the length of directory or object names, so it’s fully capable of supporting names >= 256 chars. If you look at WriteName, which is used for old format discs, then you’ll see that it truncates the length of the name to 10 characters. But if you look at WriteBigName then you’ll see that no limits are applied whatsoever. Well, except for the remnants of some old limit-checking code, which was deleted a long time ago when long filename support was still in development (see manipulation of R2). Commence bogglement.

So the naive approach to fixing the problem would be to add back a limit check to WriteBigName (and its friend, WriteBigDirName). But that won’t fully solve the problem, because there’s still lots of code which uses the untruncated length to allocate space for the directory (e.g. MakeBigDirSpace). So it looks like I/we will have to track down and fix all of those as well. And that’s not to mention all the other bits of the OS which are currently falling over when given long filenames.

File this under “things I regret offering to look at”.

(-: Hands up who knows that feeling. :-)

So it looks like I/we will have to track down and fix all of those as well. And that’s not to mention all the other bits of the OS which are currently falling over when given long filenames.

Might I suggest you start by writing this into some documentation somewhere, so that it is clear that “this is the spec, dammit!”. As you commented above – is that with a terminator or without? ;-)

Most filesystems appear to support a filename of 255 bytes (so I suppose 256 with terminator) – http://en.wikipedia.org/wiki/Comparison_of_file_systems#Limits
The maximum pathname is OS-defined. In Windows, generally 260 bytes. Apparently Windows with NTFS supports 32,000 bytes, but a lot of bits of Windows will fall over in a heap if you hit it with that. Take a look at http://longpathtool.com/ Linux supports paths up to 4096 bytes.
So, file=256, path=1024 isn’t that bad.

Note to everybody – bytes does not necessarily equal characters. How many bytes would be needed to represent “天体のメソッドPV1.mp4”? ;-)

If you or somebody is having to go through and address this anyway, it could be an excellent opportunity to decouple the characters=bytes assumption that I’m guessing will have been made. At least that would be one less thing to worry about when the Filer and all its cousins get UTF-8 support. Although you may find that’s too much work. It often is nowhere near as trivial as it sounds, in my experience. Maybe I’m just slow, though! ;)

Then of course there is the complication of the Wimp Message_DataLoad which limits the full pathname to 203 bytes.

If you or somebody is having to go through and address this anyway, it could be an excellent opportunity to decouple the characters=bytes assumption that I’m guessing will have been made. At least that would be one less thing to worry about when the Filer and all its cousins get UTF-8 support.

I think that’s the kind of thing that’s best left to be done properly, because there’ll be a lot of work involved to make the code behave sensibly when faced with a mix of discs using UTF-8 and discs not using UTF-8.

Then of course there is the complication of the Wimp Message_DataLoad which limits the full pathname to 203 bytes.

At the moment I’m hoping to just concentrate on fixing bugs in 15 year old features, not design flaws in 20+ year old ones :-)

As I understand it 1K is meant to be the limit for pathnames and 256 chars for leafnames (including terminators?). However it’s rather hard to track down any specifications which actually state these as being the design limits. The best I can find is the RISC OS 4 features list, which only says “the maximum filename limit is 1024 characters”

You want the Ursula FileCore phase 1 and phase 2 specification which are document number 1309,207/FS and 1309,208/FS. There are mangled copies on the web, but it’s possible the catacombs under ROOL HQ contain the original master copies.

I believe the limit for FileCore is 255 characters in the name plus 1 for the terminator. My evidence is mostly the cunning <= assertion for FcbName in the File Control Block definition, plus the handful of uses of BigDirMaxNameLen which have loop exits that are BLO that value, so array indices 0-254 for the name. Directories have most of the same properties as files, so there’s no distinction between leafnames and directory names.

It looks like the RISC OS 4 press clipping was getting confused about 1k for the entire path, mostly imposed by the 1k command line limit. Of course in practice it’s a bit less than that since the shortest abbreviated command would be 2 letters then the name then a terminator. ie. 1024-1-1021-2=0.
That would also make copying and renaming tricky since you’d not be able to have two 1021 long paths on the same command line.

The on-disc structure is entirely dynamically allocated, with no fixed length for the names.

And that seems reasonable in my head. The guts are generic and you only need to put sentrys on the gates – so the limit only needs checking when names are passed into FileCore everything else doesn’t care.

And that’s not to mention all the other bits of the OS which are currently falling over when given long filenames.

The other option for the short term is just to run fsbash from the root directory (which is what I’m guessing caused the problem to surface). Clearly after 15 years it’s not a use pattern that trips normal humans up, I lose track of directories after they’re nested more than 6 or 7 deep!

You want the Ursula FileCore phase 1 and phase 2 specification which are document number 1309,207/FS and 1309,208/FS. There are mangled copies on the web, but it’s possible the catacombs under ROOL HQ contain the original master copies.

Thanks!

The other option for the short term is just to run fsbash from the root directory (which is what I’m guessing caused the problem to surface).

Yeah, pretty much. Find one loose end, start pulling on it, and end up with a whole ball of yarn.

I’ll probably end up expanding fsbash to allow it to generate leafnames longer than 256 chars (and a suitable warning to discourage people from running it on their main drive)

I ran this past Ben this morning:

The Ursula FileCore spec says several times that the filename (i.e. leaf name) length is limited to 255 bytes. The practical pathname limit was mainly enforced by the length of the command line buffers, which were increased from 256 bytes to 1K (that would be in the Ursula Kernel spec, not sure where that is though).

and later:

Ah yes, found it online – “The check for command line length in Ursula is now for a maximum length of 1024 bytes including the terminator, rather than 256 bytes.”

It looks like marutan.net has copies of 1309,207/FS and 1309,208/FS as well. Thanks, Peter!

The HardDisc4 is intended for use on all platforms and as such does not contain platform specific components such as the CPU tool, which is by Andrew Conroy. All we need is a link.

components such as the CPU tool, which is by Andrew Conroy. All we need is a link.

ROOL have made significant changes to the CPU tool since I originally wrote it, so I don’t feel it’s really mine to distribute any longer, I guess it’s become theirs. The original version, for the Pi 1 only, which I wrote (with considerable help from Jeffrey Lee and Theo Markettos) is at http://www.owlart.co.uk/Software but this is not suitable for use with the Pi 2.

If this is the ARM5/ARM7 (Alignment exceptions off)/ARM 7 (alignmnent exceptions on) then it is provided on RC14 in a form suitable for Pi1 and Pi2 (the first option being greyed out on Pi2). Separate similar tools are provided for Panda and Beagle and ARMX6 by R-Comp.