Filename length (again)

A user of SyncDiscs has been reporting a number of ‘unexplained’ crashes. During my investigations I have realised the crash regularly occurs in a !WebGet downloaded directory which contains a lot of files with long filenames. The longest I have found has a leafname of 217 characters! I know there is a limit (232 bytes) for any file path in a wimp message. Is there a limit to the length of a filename allowed in, say, a command line *copy or similar? When multitasking, SyncDiscs uses Filer_Actions for some of its processes, but when used in single tasking mode, it reverts to various command line instructions. It would be useful knowing what the limits on full file paths actually were for such commands. Any help appreciated.

‘unexplained’ crashes

What sort of crash – could you not call the X form of OS_CLI (or whatever) and report to the user that such-and-such a file failed to copy?

The longest I have found has a leafname of 217 characters!

Yikes.

With the copy command, path, leaf name, destination path, destination leaf name, copy options… What does it all add up to? Is their version of RISC OS capable of command lines that long?

Is there an OS_FSControl to copy objects?

Edit: OS_FSControl 26, PRM 2-110. Might that be better than using *commands?

The original SyncDiscs (ex David Pilling) was a single tasking app, and when I converted to Multitasking capability, I saw no reason to change the way the single tasking option worked – so yes, I could use other methods.

However, ignoring SyncDiscs for the moment, the user in question put the offending directory on an FTP server for me to download. Using FTPc, initially I crashed totally my Iyonix, several times – no stack for stack handler , filecore in use, etc. I found after several attempts that I could get the files if they were downloaded into a directory, named a single character, in the root of my hard drive. The full path was then just inside the 232 byte wimp message content limit. Attempting to copy one of these verylong filenames into a lower level directory gave – you’ve guessed – filecore in use, etc crashes, so the length is not being checked by the OS. Thus long path names can cause mayhem.

The original user is using a version of ROL’s Select, so is that more robust than RISC OS 5? The version of RISC OS on the Iyonix is about a week old.

Well, the FileCore Defns file says:

   438: BigDirMaxNameLen        * 255   ; maximum name length is 255 chars

So I’m guessing 255 characters was expected. I wonder where the difference lies?

https://www.riscosopen.org/viewer/view/castle/RiscOS/Sources/FileSys/FileCore/s/Defns?rev=4.12#l438

The original user is using a version of ROL’s Select, so is that more robust than RISC OS 5? The version of RISC OS on the Iyonix is about a week old.

Select/Adjust did increase the length of some buffers, command strings etc. I think wimp message was one of them. I think there was a note at the same time that there were still other limitations which affected files?

BigDirMaxNameLen * 255 ; maximum name length is 255 chars

Looks like my #defines of 256 are probably ok then, assuming that is a complete path and not just a leaf. The main question is how much checking do I do in SyncDiscs – checking every output file would slow things down considerably. At the moment, where possible, complete dirs are dealt with, so the individual files are not necessarily seen.

Select/Adjust did increase the length of some buffers, command strings etc. I think wimp message was one of them.

The Wimp message buffer is defined to be 256 bytes in the PRM, and applications all claim the requisite amount with a fixed constant because Acorn saw no need to provide an API to read it. If Select managed to increase the size of Wimp messages despite that, I’ll be very impressed.

The Wimp message buffer is defined to be 256 bytes in the PRM, and applications all claim the requisite amount with a fixed constant because Acorn saw no need to provide an API to read it. If Select managed to increase the size of Wimp messages despite that, I’ll be very impressed.

According to ROLs new PRM, the WimpMessage block is still 256bytes.

If Select managed to increase the size of Wimp messages despite that, I’ll be very impressed.

Unfortunately there is no way to set task “flags” in Wimp_Initialise other than by messing with version numbers.
So a Wimp passing back more than 256 bytes to a task expecting 256 bytes max could go about as wrong as a task trying to send huge message blocks to the Wimp; although arguably the later is less destructive as the Wimp can just ignore the excess, as opposed to potentially trampling on a block of memory.

At any rate, using something other than the expected 256 bytes would be tricky to implement and make your application compatible with only the systems that support it…

Unfortunately there is no way to set task “flags” in Wimp_Initialise other than by messing with version numbers.

Yes: I was just wondering how far Chris would go with the claim… :-)

At any rate, using something other than the expected 256 bytes would be tricky to implement and make your application compatible with only the systems that support it…

Just thinking, though… Could something be added that allows a task to subsequently negotiate the size of its poll block? Either an extra parameter (plus magic word, or bit set in poll mask?) to Wimp_Poll saying “I’m giving you a buffer of x bytes”, or a new SWI that tasks could call to negotiate the size during initialisation (or later?).

It’s still very messy, because broadcast messages would have problems and it can’t be reactive (the Wimp can’t easily say “I’m going to need a block of x bytes to return from Wimp_Poll, please”). The Wimp would also need to track each task’s block size. I suppose it would ultimately depend on whether fixing things for those apps still in development would be worth the complexity that this would add.

It’s still very messy, because broadcast messages would have problems

The (rough) idea that I have for a UTF8 native Wimp would use “UTF8” instead of “TASK” in Wimp_Initialise (precisely because there are no task flag bits). This would signal to use UTF-8 for the task, plus various extensions (larger poll block, etc).
But, as you say, the transition point between old and new is not exactly simple. The Wimp would need to handle bodging data between UTF-8 tasks and Latin1 tasks (as best is possible). Not the best approach, but we don’t really have the luxury of jacking in the old API and making a new one.

One possibility for handling long message blocks would be a “long message” user message. When sending a long message you’d construct your message block somewhere, and then construct a smaller regular-size block containing the length of the long message and a pointer to it. You’d probably also want to include the message code of the long message, so that the receiver can do some filtering without having to request the full block (e.g. filtering broadcast messages).

The receiver can then look at it and say “this is a long message”, allocate a buffer of the appropriate size, and Wimp_TransferBlock the long block into its buffer. If the receiver doesn’t support long blocks then it’ll just ignore it, which will at least ensure the system won’t crash, even if it means the application might not respond as expected.

The next logical step would be to make the Wimp handle it automatically, e.g. SWI Wimp_SendLongMessage which sends the long message directly to tasks which support large poll blocks, or constructs a shorter block if an attempt is made to send a message > 256 bytes to an older task. Perhaps the call should also have the ability to force a long message to be sent, e.g. if you need to send a sequence of messages and you know there’ll be a long message in there somewhere, you’d want to make sure all messages in the sequence are long so that a receiver which doesn’t support long messages won’t receive the first few and then get stuck waiting for the rest.

Not sure if it’s relevant but I found at:http://www.riscos.com/the_archive/acorn/products/phoebe/intro.htm

The size of the command line buffer has been increased so that operations such as *copy can be executed without running into the 256-character limit imposed on earlier OS versions.
The limit on command line length is now 1024 characters.

As this is a Phoebe document it’s may or may not be what I recall ROL increasing!

As this is a Phoebe document it’s may or may not be what I recall ROL increasing!

Yes, ROL did indeed do that (or at least took credit for Acorn doing it in RISC OS 3.8).

However, that’s the command line buffer. We’re talking about the Wimp message block, which often contains pathnames and can’t easily be increased without breaking a lot of stuff because, as it stands, the size is hard-coded into every application out there and the Wimp makes no checks…

My memory may be wrong but I thought ROL increased two or three important buffers/blocks/?
I recall at least one of them was 232? bytes i.e. less than 256 for technical reasons that were beyond me. They weren’t peripheral buffers like keyboard or sound buffers.
They may well not be relevant to this thread but as Chris says the reporting user was using Select and it is worth checking just in case ROL did find a way round the problem.

I recall at least one of them was 232? bytes i.e. less than 256 for technical reasons that were beyond me.

Sounds like a standard Wimp block – 256 bytes less the Wimp headers (20 bytes) gives 236 bytes for the filename.¹

More to the point, who seriously needs file names that long (as provokes the original problem)?

¹ Less for Message_DataLoad, Message_FilerAction, etc as these use some of the data words for flags, status, and such.

More to the point, who seriously needs file names that long (as provokes the original problem)?

Yeah, and who would ever need more than 77 files in a directory, right? ;-)

It’s pretty easy to end up with long file/path names without even realising it. E.g. look through one of the OS source archives and see how deep some of the directory paths get. But you might not know that when you extract the archive, so you might place it in some deep subfolder on your hard disc (ADFS::HardDisc4.$.JoeBloggs.SourceCode.ROOL.OMAP3.FixingSomeHorribleBug.14_04_2014 … etc.) and then bang! you hit a buffer length limit hidden deep within the OS and everything dies horribly. Or maybe you can extract the archive OK, but then when you go to do a backup to SCSI::Backup2014.$.HardDisc4.April.14 those few extra characters push it over the limit.

More to the point, who seriously needs file names that long (as provokes the original problem)?

The problem is, they are being generated automatically by, presumably, WebGet when it fetches the files. Here is one of the shorter leaf names – there will be the discname and top level dirs in front of this of course.

~tmpl~component~amp~link~aHR0cDovL3d3dy5NYWdoZXJhR2VuZ
WFsb2d5Lm9yZy9pbmRleC5waHAvb3RoZXJyZXNvdXJjZXMvNTMtbmV
3c3BhcGVycy84Ni1tYWdoZXJhZmVsdC1xdWFydGVyLXNlc3Npb25z
LTI4LWp1bmUtMTg3My5odG1s

(Maybe that won’t format properly so I have put in some LFs).

but then when you go to do a backup to SCSI::Backup2014.$.HardDisc4.April.14 those few extra characters push it over the limit.

Yes, I think that was what was happening in the original problem. The primary and secondary paths were at the top – ADFS::HD4a.$ and ADFS::HD4b.$ but a scrapdir was being used on a NAS device at a much lower level in the dir tree and attempts to move deleted files there provoked the nastiness.