Path length limits

I’ve encountered an interesting problem while testing my disc error 20 fix; FSBash has managed to create several files/folders whose full paths are over 1024 characters long. This is problematic because FileSwitch has a max path length of 1024. Possibly I can delete them from the command line (Filer Action fails), but the reason it caught my attention is that instead of getting a “path too long” error I got a completely inaccurate “Not enough stack to call filing system” error due to naughty TopPath re-using the out-of-stack error if OS_GSTrans overflows the on-stack buffer.

Although I was thinking about fixing the error to be unique/correct, looking closer at how FileSwitch handles paths makes me think that it wouldn’t be too hard to remove the max path limit completely.

• TopPath only uses the stack temporarily, before it moves the cleaned-up string to a heap block. For long paths it can easily be changed to guestimate a heap block size, and/or use OS_GSInit + OS_GSRead to read into a resizable buffer instead of OS_GSTrans.
• DisplayDirectoryPath again only uses the stack for temporary storage so that it can output a path to the screen. But mainly it just wants a buffer it can read a system variable into, so for long paths it can easily read the variable length and then use a heap block of the correct size.
• Wildcard handling has a similar 1K limit, and again only seems to use the stack for temporary storage prior to moving the string into a heap block. Calculating the correct buffer length is going to be difficult (FSEntry_Func 14 can indicate buffer overflow, but not the required buffer size), so FileSwitch would just have to try increasingly larger buffers until it finds one that works. Although in this case it’s the leaf name that’s limited to 1K, not the full path – so we probably don’t have to worry too much about that one yet.

However there is this comment in TopPath which makes me worry that things aren’t as easy as they might seem. That comment is from changes merged in from ROL, and it’s 18 years old, so maybe it’s not relevant to RISC OS 5, but I was wondering if anyone is aware of any other issues that might crop up if I was to try removing the limit?

Possibly I can delete them from the command line

Yes I can (via a recursive FileSwitch op like *Wipe) – which makes the 1K limit on input paths feel even more arbitrary.

Removing the limit certainly sounds possible from your description.

Is there a knock on effect to programs? I know I’ll need to rewrite my filename handling, but it sounds like I need to do that anyway, as I believed the limit to be 255 chars.

Is there a knock on effect to programs?

Ordinary programs probably won’t care, at least until the user tries using a ridiculously long path. If a program is relying on a deprecated SWI like OS_GBPB 8 then now would obviously be a good time to make sure it’s using a more sensible alternative.

The more serious problem would be programs or modules which hang off of vectors (or similar) and monitor filesystems for activity. I’ve had ShareFS crash several times while deleting the 1K+ paths, so it seems that has some unchecked buffer limits somewhere.

Is there a knock on effect to programs?

I bet there are plenty of programs assuming an allocation of 255 bytes for paths. Certainly things could be… interesting regarding BASIC programs. ;-)

I will probably need to look to see if any of my programs have line length issues (possibly not as I use system variables to access stuff, let the filesystem worry about that) and just put a clamp on it to simply reject any path over 255 bytes. I did this before with Ovation where there are places that assume the path is “up to 128 bytes” (remember, Ovation was written in the days when harddiscs were rare and people used floppies) and since changing that meant breaking lots (possibly file compatibility), I just put in a twiddle to reject overlong names/paths.

Is there a knock on effect to programs?

I bet there are plenty of programs assuming an allocation of 255 bytes for paths. Certainly things could be… interesting regarding BASIC programs. ;-)

My Pic_Index (written in BASIC using DrWimp) program has suffered from path-length problems when used from too far down the directory tree – usually solved by relocating it to operate from nearer root.

Are these the sorts of problems which may be alleviated by this/these changes? In my case the path length limit seemed to be practically from somewhere round 235 bytes.

Remember when Impression Style could not handle files with a filename longer than 10 chars? Users learned to cope with it. IMHO we should just try to lift the limit and deal with the fallout later. Incompatible apps will either be fixed or the users will learn to circumvent the problem. For most users, nothing will change anyway because of the WIMP message limit.

Remember when Impression Style could not handle files with a filename longer than 10 chars?

Yeah, but that was just extreme stupidity by the programmers. There’s never been any reason for a desktop program to have a restricted filename limit.

However the 220ish character path length limit due to the WIMP message block size is a much bigger problem :-(

Yeah, but that was just extreme stupidity by the programmers.

Stupidity? Should have been 12 characters, to cater for DOSFS.

Seriously though, with the exception of alien filesystems (DOSFS, Omni, CDFS), did Acorn ever release a RISC OS filing system that did not have a ten character filename limit? I recall with RISC OS 3.70 using X-Files to get around limits on file name lengths and the 77 files in a directory restriction…

There’s never been any reason for a desktop program to have a restricted filename limit.

Template indirected text? Allocations of workspace?

However the 220ish character path length limit due to the WIMP message block size is a much bigger problem

And BASIC’s 255 character string size restriction may prove difficult to work around. I find myself wondering – is it necessary to have really big names/paths, or is this being done because it can? If the latter, it’s worth noting how much stuff stands to be potentially broken.

Example? I created a directory on my Pi ages ago that was something like l.l.l.l etc for 240 characters. The Filer flaked out at around 230 or so, it wasn’t capable of opening all of the folders within the standard-since-the-eighties allocation (possibly it bit the Wimp message limit?). With this in mind…….

Template indirected text? Allocations of workspace?

Yes, but a save box icon should take a path not just a filename, so should never have been only 10 characters long.

I remember accessing NFS shares from my A4 at work and that happily accessed long filenames back on RO3.1.

The implementation of strings as byte-arrays, adopted by RISC OS and BBC BASIC, was a compromise justified by the times that now is increasingly seen as an embarrassment. The trouble is that more modern OSes and languages build upon a base of garbage-collected memory management, a feature that would be hard to retrofit, I imagine.

Yeah, but that was just extreme stupidity by the programmers. There’s never been any reason for a desktop program to have a restricted filename limit.

To be fair until 1995 Impression saved its files as an application directory, making it easy to see the components – DDE text files and draw-enbedded sprites in chapters. The longest application sprite names were at the time 10 characters, 12 characters for ‘sm!filenames’ and if you called your file ‘paint’ it trampled over the app sprite for paint (not so good).

The limit on filename plus pathname was much more generous than 10 and I never encountered this as an issue, but then I never tried to bury an Impression file so deep as to encounter it either (the Wimp message protocol when saving to a directory window would probably be the limit here).

To be fair until 1995 Impression saved its files as an application directory

So they invented a new single file format in 1995 and then decided that 10 characters are enough for anybody. And crashing if the user renamed the file to something longer and trying to load it. Fileswitch supported longer filenames than 10 characters since day 1, and many FSes exploited this – DOSFS, NFS, CDFS, many image filing systems trying to work around Filecore’s limitations.

Maybe there were unsolvable internal structural problems supporting longer filenames, but I think this is very unlikely – at the same time, they created Style and Publisher (Plus) with significant new features compared to its predecessor.

If a program is relying on a deprecated SWI like OS_GBPB 8 then now would obviously be a good time to make sure it’s using a more sensible alternative.

OS_GBPB 8 was deprecated with RISCOS 3, so hopefully isn’t a big issue. What other non-deprecated calls are there that return a filename and/or pathname without first returning the size?

did Acorn ever release a RISC OS filing system that did not have a ten character filename limit?

Filenames were ten characters on all Acorn roms, Impression was obeying the filename rules at the time.

Quoting PRM2-12:

Filename elements may be up to ten characters in length on FileCore-based filing systems

I can’t find any reference to the path character limit though, but guess its probably 244 characters.

guess its probably 244 characters

Which with a 10-character element limit gives a 22-element structure depth. Anyone constructing paths with 22 elements deserves all the problems they’re going to get… imo

What other non-deprecated calls are there that return a filename and/or pathname without first returning the size?

OS_GBPB 12, perhaps? It won’t scribble over random memory as the command is given the length of the buffer. However, how long should this buffer be? And how can you tell if something has been missed because of stupid-long name? It is legal to return R3 as zero (no objects read) before the end of the directory, so you can’t say “R3=0 because buffer too small” or anything like that.

Frankly, anything (name plus path) over ~230 characters is asking for pain. You won’t be able to DataLoad (etc) it, because of the Wimp message protocol. Anything over 255 characters pretty much precludes BASIC from dealing with it, unless the user fancies writing custom string handling routines.

You know, I totally get the case for long command lines. A brief look at the rubbish that passes back and forth building a project of any sort of size will justify having long command lines.
But I wonder if there’s a solid case for having really long filenames/paths given the potential chaos this could trigger in other places.

Anyone constructing paths with 22 elements deserves all the problems they’re going to get…

I wonder how deeply nested the RISC OS sources go?

$.Coding.RISCOS.BCM2835Dev.castle.RiscOS.Sources.Kernel.s.PMF.mouse

Well, that was boring. Possibly the most deeply nested thing I have on my filesystem, and nowhere near.

You’re right, we should go back to BBC DFS. Using filenames longer than 7 characters, or directory names longer than 1 character, is just going to cause problems.

I don’t advocate a 10-character limit for filenames at all. But in a system that has them, I don’t regard a 244-byte limit on pathnames as noticeably restrictive!

I was wondering if anyone is aware of any other issues that might crop up if I was to try removing the limit?

My reading of what you’re proposing is to make a few low impact changes to FileSwitch to allow arbitrary length paths, right?

Changing the rest of the OS to handle correspondingly long command lines is something different (you’d want it to be 2*N+7 at least so there was space for ‘COPY very_long_thing another_very_long_thing’ for example.

For extra long command lines, off the top of my head, the kernel’s OS_CLI buffer would be impacted, DDEUtils probably, and although the limit was raised to 1k for Ursula (circa 1998) in more recent times I can recall some things finally catching up in system() in the CLib, stuff that intercepts or parses commands such as BootCommands, stuff built using OSLib, and there’s a good few 256’s still kicking around, I’m sure.

But FileSwitch should be able to delete something it created.

You’re right, we should go back to BBC DFS. Using filenames longer than 7 characters, or directory names longer than 1 character, is just going to cause problems.

Directories? Eee. Luxury. When I were a lad we had 300 baud cassettes w’no directories, and we were glad.

Cassettes? There’s fancy. I grew up with 7-hole ticker tape…

When I were a lad we had 300 baud cassettes w’no wi’ no directories, and we were glad.

ICL 1904 at Uni and I’m pretty sure the tapes were a bit less portable than cassettes.

Anyway, how big a can do you think Jeffrey has opened?

My reading of what you’re proposing is to make a few low impact changes to FileSwitch to allow arbitrary length paths, right?

Correct. Nothing more at this point in time.

“unlimited” command line length would obviously be nice, but as many people have pointed out there aren’t many situations where you’re going to encounter a very long path, so updating the rest of the OS isn’t a concern of mine at this point in time.

One key thing to take away from this thread is that programmers are lazy/short-sighted and like to take advantage of any situation where a system states its limits. If you know the target system only uses file names up to 10 characters, you’ll hard-code that length in your app. If you know the system only supports paths up to ~256 characters, you’ll hard-code that length as well. By removing the 1K limit from FileSwitch, it means that when a programmer reads through the FileSwitch docs they won’t see any mention of a limit, reducing the temptation for them to hardcode any limits into their apps (or if they do hardcode a limit, they’ll only have themselves to blame when a user reports that the software fails when given a long path)

programmers are lazy

Guilty as charged, apparently:

char fileToLoad[256];

One key thing to take away from this thread is that programmers are lazy/short-sighted and like to take advantage of any situation where a system states its limits.

If that is the way you wish to approach this, kindly implement an API that does indeed state what the current file limit is so those of us that read things dynamically have some sort of idea of how much memory to actually allocate…
[there’s actually a reason us lazy programmers like to know the sizes of things]

PS: Normally in this space you might see a comment regarding twigs (1s) and stones (0s) but, you know, off topic…

If that is the way you wish to approach this, kindly implement an API that does indeed state what the current file limit is so those of us that read things dynamically have some sort of idea of how much memory to actually allocate…

An OS_GBPB that tells you the length of the buffer you need? Yes, that would be a good addition.

A call that tells you the longest filename possible? No, that would defeat the point of removing the limit (the answer would be something like 2^32-1)