DOSFS not cheking for disc sizes it can't handle?

Since I started the topic, I thought I might as well download a new version of the ROMs (my previous one dated from 2017-06-18) for my Pi3, so I installed the one from 2018-02-22. I have two oddities to report:
- The FAT32 HD shows “4” instead on the disc’s label. Clicking on the icon the first time usually gives an error like “Disc not understood, as it been formatted ?”. Clicking on the icon again changes the “4” to the disc labels and the disc’s root window shows it, Fat32Fs has taken control of it and it works like usual.
- With the first window I drag around, the release of the mouse button is not detected and the window follows the mouse till I fiddle with the middle and right buttons. After that it never occurs again.

I also tested somewhat LanManFS since the ROM had a newer version, but not change on that front, it still get stuck in “LanManFS is use” after a while without given a single clue to what trigger the issue.

- With the first window I drag around, the release of the mouse button is not detected and the window follows the mouse till I fiddle with the middle and right buttons. After that it never occurs again.

Interesting – there haven’t been any big changes to the USB stack during the period you mention, so this must be some kind of latent timing-related issue.

Occasionally I get stuck mouse buttons on my Iyonix (very annoying when clicking in a scroll well to scroll one page at a time), across all OS versions, but since that’s using a different USB driver/controller it’s hard to draw any conclusion other than “sometimes USB goes wrong”.

I forgot to mention that I also encountered the system shutting down be itself that was mentioned here

It might be worth posting in that thread, especially if you have hdmi_blanking=1 like we all seem to…

sometimes USB goes wrong

Could it be something turning FIQ’s off (USB is FIQ now isn’t it?), possibly when accessing media, the knock on effect being USB packets are lost?

When shutting down the JIT in ADFFS from the command line, quite often the ENTER key sticks. I’m certain that’s not USB related, but something is causing USB packets to be lost. Anyhow, the point being that USB issues are quite often triggered elsewhere.

- The FAT32 HD shows “4” instead on the disc’s label. Clicking on the icon the first time usually gives an error like “Disc not understood, as it been formatted ?”. Clicking on the icon again changes the “4” to the disc labels and the disc’s root window shows it, Fat32Fs has taken control of it and it works like usual.

Showing “4” until it’s clicked on is to be expected – the drive name comes from Service_IdentifyDisc, which DOSFS will no longer be claiming, because the drive is larger than can be represented using the basic disc record that DOSFS generates. Upgrading DOSFS to allow it to generate ‘big disc’ disc records would probably solve this – although until the 4GB image filing system limit is removed, it would probably make more sense for Fat32FS to handle Service_IdentifyDisc itself. (And possibly the OS should label unsupported drives as “:4”, to match the presentation of removable media)

Clicking on the icon the first time seems to work fine for me – regardless of whether Fat32FS was loaded before or after the drive was attached. This is with Fat32FS 1.53, with both removable and non-removable media.

Excuse my naïvety, but is there a reason why we persist with DOSFS instead of asking Jeff if FAT32FS (as I would capitalise it myself) could be absorbed into the OS in its place?

What would be any disadvantages in this course of action?

Because Fat32fs is an appalling bodge which was only intended to be a stopgap until Dosfs did it properly. Unfortunately the presence of Fat32fs removed the need to fix Dosfs.
Fat32fs does not look at the Service_IdentifyDisc because I could not make head nor tail of it.
Fat32fs uses the filter manager to intercept the mouse clicks on the SCSIFiler icons and attempts to mount the drive itself. It then has to write directly into the SCSIFiler workspace to create the discname for the icon. This is all pretty horrible!

This is all pretty horrible!

…and yet, works so much better than DOSFS ever did…

When you have time, could you do it properly, then?

And, of course, observe capitalisation conventions!

We are all most grateful for your efforts and achievements – but want more!

When you have time, could you do it properly, then?
And, of course, observe capitalisation conventions!

FFS! 😮

Fat32fs uses efsl which is GPL which precludes Fat32fs from being included in the OS. There are other libs out there which are BSD-licenced like the previously-mentioned FatFs http://elm-chan.org/fsw/ff/00index_e.html which has the additional bonus of handling exFAT (which leads directly into patent problem territory, but probably not in Europe).

Plan: extend Fileswitch to lift the 4 GiB-per-file limit, write a proper DOSFS-and-Fat32fs-replacement with the help of a known-good lib, all problems solved :-)

As a first prototype, maybe try FatFs in the context of Fat32fs?

The underlying problem isn’t DOSFS; it’s the inability for image filing systems to access anything beyond the first 4GB of a drive. Fat32fs is a workaround for that problem, not a fix.

On paper, a 64bit FileSwitch API (thereby allowing 64bit image filing systems) sounds simple. But looking at the filing system bounty roadmap, I’m reminded that there are other API-related things on the wishlist as well – like unicode filenames, an enhanced permissions system, and arbitrary metadata. So although we could implement a 64bit FileSwitch API right now (we even have a proposed spec for it), it would be foolish to introduce a new 64bit API and then shortly after introduce another new API which adds support for the other features. And it would be especially foolish when you consider that DOSFS/Fat32fs exist to aid interoperability with foreign filing systems – which is exactly the kind of thing that unicode filenames, enhanced permissions, and arbitrary metadata would also help with.

The filesystem bounty roadmap isn’t set in stone (bounties 1 and 2 have already deviated from the path), so if we can arrive at a sensible revision to the application-facing side of FileSwitch then there’s no reason why that can’t be tackled sooner than the initial roadmap indicated. And then once the new API is implemented and ready for use we can go back to worrying about the awful mess that is DOSFS.

we even have a proposed spec for it

Hehe, discuss with ROLtd for possible SIX compatibility. ;)

Having read the document, a few random things come to mind:

• What is the extra attributes? Wouldn’t it be better to have the file designed in such a way that the additional data is a part of the file and not something [ephemeral|ethereal] lurking in the filesystem?
  To give you an idea of what I mean, consider the RISC OS source code. Up to a certain point, all the changes were stored in text files (usually Blacklog). After that point, when proper source management came along, all of the changes are stored per file with fancy comparisons and all sorts. However, not only is all of this lost upon a download of the sources, it seems to be harder than it should to determine what has actually changed for any given component (the kernel, for example, is maybe 25 different files…). Do we want to have a similar situation with the filesystem in having some sort of extra odd lurking data buried in the filesystem itself?
• What happens to older applications that use the 32 bit API? Will there be some sort of error raised if the older API opens a file >4GiB?
• I notice load/exec addresses look like they’re finally going to be tossed in the rubbish heap. Somebody might want to tell *Save…
• I dread to think how many things will be assuming a filetype is a three nibble value. ;) What will the old API return if listing a directory full of extended type files?

Probably some other stuff but I’m cold and fed up and my memory is rubbish. It’s not nice when it’s minus something C and blowing a gale. I hope to defrost soon because right now I feel like one of those unidentifiable bits of meat in the supermarket freezer…

Hehe, discuss with ROLtd

First get a TARDIS and travel back to when ROL were still in existence

I’m cold and fed up and my memory is rubbish

Hmmm, walked from the bus stop (1.5 mile) directly East. I’ve been home an hour and a half and my face is still warming up. Passing the supermarket I noted gents in shorts – is that how you get your assets frozen?

What is the extra attributes?

“New filing systems may support extended attributes, to allow arbitrary metadata to be stored with any file or directory.”

At the basic level, a file is a blob of data (with a known size) that is identified by its name. All the other properties of the file – timestamp, filetype, permissions, load/exec address, etc. are metadata. Different operating systems & filing systems have different amounts of metadata, and it looks like all the major modern filing systems support arbitrary metadata. (In some cases you could also say the file name is metadata, if the FS allows identification of the file by some other method, e.g. inode)

Wouldn’t it be better to have the file designed in such a way that the additional data is a part of the file and not something [ephemeral|ethereal] lurking in the filesystem?

Perhaps. But for that to work you’d have to invent a time machine, travel back in time, and then convince all the OS/FS designers of the world to work together on a common file format which can support all the metadata any OS will ever need.

Assuming you’re not going to do that, we’re stuck with the current situation: some OS’s require some metadata (e.g. RISC OS uses filetype metadata, most other platforms use file extensions), and without end-to-end support for arbitrary metadata, that metadata may be lost or altered when files are transferred to foreign systems.

What happens to older applications that use the 32 bit API? Will there be some sort of error raised if the older API opens a file >4GiB?

Probably, yes.

I notice load/exec addresses look like they’re finally going to be tossed in the rubbish heap.

Deprecating them is good, but to me they’re just another example of metadata. So the new API could simply relegate them to being part of the metadata (e.g. ‘riscos:load’, ‘riscos:exec’). Arguably the RISC OS filetype should be part of the metadata as well. Essentially I’m thinking that for the new-style OS_GBPB directory enumeration calls, you’d specify a list of metadata tags you want to retrieve (potentially wildcarded), and the filesystem would return just those tags in its result set. Potentially we could support filtering the results by metadata as well (“only return the list of BASIC files”) – allowing for much faster lookups if the filesystem generates indexes for the metadata.

I dread to think how many things will be assuming a filetype is a three nibble value. ;) What will the old API return if listing a directory full of extended type files?

Presumably we’ll have one (or more) old-style filetype which will be used as a fallback. Maybe this could just be the bottom (or top) 12 bits of the 32 bit value – e.g. all text-based formats could be derived from &FFF, all binary formats from &FFD.

If someone’s thinking about new directory enumeration calls can we have an open and close calls so that protocols like SMB can be told when you have finished enumerating. You can call it OS_Directory to if you like to make it’s use clearer.

Good point – that’s a major omission in the current API.

Good idea about an OS_Directory API open, close. I would suggest that the open returns an handle and work with that handle for the other calls so that you don’t have to loose time manipulating the filename (putting it in canonical form, …).

Writing this for the second time

Rick asked:

What is the extra attributes? Wouldn’t it be better to have the file designed in such a way that the additional data is a part of the file and not something [ephemeral|ethereal] lurking in the filesystem?

Funny you should ask th…

Jeffrey replied:

you’d have to invent a time machine, travel back in time, and then convince all the OS/FS designers of the world to work together on a common file format which can support all the metadata any OS will ever need.

Well it’s not that bad. Firstly, this is exactly how Bruce Horn’s “forked” files worked on Mac OS. And thanks to John Tytgat’s MacFS, they even work on RO. However, that’s an implementation detail, not how the API should be.

You could use the same Image subfile wheeze that MacFS and ImageFS use (eg path.file is the data, path.file.thumb is the thumbnail) but then everything’s an Image and some programs will break (because they, or their author’s memories, were written before OS_File,5 changed subtly).

So metadata should be accessed by Some Other Mechanism, and its up to the FS how to store it. HostFS could take advantage of NTFS for example. FileCore could stuff it all into the file invisibly but that, like MacForks, results in a file that little else will be able to use directly.

Fortunately there is now an open and commonly used format that does allow metadata to be attached to files, which is used in Windows, Linux and MacOS – zip files.

DocX, XPS (which I had a hand in) and xlsx documents are all zip files really, and it’s really common on linux. This way the contents are still accessible on systems that don’t recognise the containerised format… as long as you’re prepared to change the file extension.