Filing system reorganisation

I just posted this on a Drobe thread and thought I might as well share it here. I’m just throwing a few ideas around, sadly I don’t have the energy to do any of this for the forseeable future…

I’ve been having a brief poke around the source of ADFS, FileCore and FileSwitch.

Firstly there’s a serious documentation need. All the documentation is in little text files scattered around – ideally it needs someone writing a basic overview of what happens where. Secondly FileCore and FileSwitch are inherently harder to understand (for me, anyway) because they do lots of structure twiddling which is harder to understand in assembler.

• Retaining ADFS as-is for its floppy support (on any machines that still have drives). All the floppy foibles take up a fair amount of code and I think it’s best left alone.
• Re-implementing the hard drive support as a new IDEFS (davehigton and I had a go in 1997 – it wasn’t too hard to get something working, the messy bit is coping with all the variations in drives out there)
• Bin RAMFS. Fix bugs in Memphis and use that instead.
• A new interface between block drivers and filing systems, with a FileCore ‘personality’ for compatibility. This might do disc partitions too.
• FileCore retained as-is, for supporting existing drives, but accessed through the new interface
• A module for a new disc format (FAT? NTFS? ext3?). If it can work on an extended image filing system API, so much the better (but would have to be reentrant)
• Lower priority: Rewrite HForm into a cross-platform FileCore filesystem generator – maybe plug it into acorn-fdisk that already exists.

Then it gets messy. FileSwitch is more closely tied in with the kernel, and there’s no nice list of tie-in assumptions in the source as there is for ADFS. I’m not convinced rewriting FileSwitch would help – you’d have to rewrite bits of the kernel too. Extending to 64 bit addressing would be a must, perhaps at the same time simplifying the API and putting the old one in a ‘personality’ module for compatibility. Are there any other problems with FileSwitch other than the 32 bit file size issue? Could they be solved by helper modules intercepting calls between FileSwitch and the lower layers in the stack?

That’s along the same lines as what I was thinking of as well. Other things:

• It would be nice to add the hooks to allow easy support in the future for things like RAID and LVM.
• Add support for arbitrary metadata on files. Possibly make filetypes just a special case of this.
• Remove the need for filesystems to do wildcarding. May need to move this in to FileSwitch for backwards compatibility.
• Add cacheing. Either in Fileswitch, or a layer between FileSwitch and lower layers.

I wonder whether it might be possible to extend the image filing system paradigm to do some of these things. I’m thinking of a hierarchy of image files something like this:

DiscFS:$    The root directory for all block devices 
DiscFS:$.Maxtor20GB    An image FS representing a raw disc image 
DiscFS:$.Maxtor20GB.partition1    The first partition on that image 
DiscFS:$.Maxtor20GB.whole    The whole disc image 
DiscFS:$.RAID    A directory for RAID devices 
DiscFS:$.RAID.RAID1    A directory for RAID1 volumes
DiscFS:$.RAID.RAID1.Seagate120GB+80GB    A image FS representing a particular RAID1 volume 
DiscFS:$.RAID.RAID1.Seagate120GB+80GB.LVM    A directory for LVM logical volumes 
DiscFS:$.RAID.RAID1.Seagate120GB+80GB.LVM.Music    An image FS representing an LVM logical volume 
DiscFS:$.RAID.RAID1.Seagate120GB+80GB.LVM.Music.partition1    The first partition on that LVM logical volume

This needn’t be too complex, especially if the FileSwitch API is simplified so the image FSs don’t have to do as much work. So a RAID module is just a case of exposing a bunch of partition files that are then picked up by the image FS that handles that type of partition (be it LVM, FAT or FileCore). The danger is that this might look outwardly neat but puts too much overhead on some operations. Any thoughts? Presumably you’d have to cache some things (eg that DiscFS:$.RAID.RAID1.Seagate120GB+80GB.LVM.Music.partition1 actually resides between block 1234567 and 2234567 on the IDE primary master and block 987654321 and block 988654321 on the primary slave).

Another issue is security – at the moment it’s possible to walk up to a machine and corrupt its HD by typing a few selected ADFS_DiscOp commands. By exposing everything like this will it make it much easier to do damage? Is there any way to make it harder for damage to happen (while at the same time admitting that the system is wide open at the moment)?

Another few limits I’d like to see removed are the 12 bit filetype (which would go with your arbitrary metadata, maybe we can have a textual filetype stored on disc instead) and the 8 character textual filetype limit (which is mostly a presentation thing I think?)

A module for a new disc format (FAT? NTFS? ext3?)

What are the constraints on the format chosen? ZFS was mentioned in another thread here. I think I saw Rob Kendrick point out somewhere that “modern” formats would be tricky due to their inherently multi-threaded nature or something?

I’d like to see [...] the 12 bit filetype [restriction removed] (which would go with your arbitrary metadata, maybe we can have a textual filetype stored on disc instead)

Possibly the filetype could be replaced with a mimetype? There is an official application/riscos mimetype which can be used to contain all the information of RISC OS filetypes. (And, of course, some file types map directly to other mime types.)

Adam

I wonder whether it might be possible to extend the image filing system paradigm to do some of these things. I’m thinking of a hierarchy of image files

That may be a good idea if the API doesn’t get messy. However I don’t think it should be exposed to the user as part of the pathname.

What are the constraints on the format chosen? ZFS was mentioned in another thread here. I think I saw Rob Kendrick point out somewhere that “modern” formats would be tricky due to their inherently multi-threaded nature or something?

I’d hope that if done properly, then any format could be theoretically supported. The choice of format (or multiple formats) could then be chosen at a later date based on license conditions, ease of porting, etc.

It may be worth introducing the concept of inodes to FileSwitch, to ease porting code and help interoperability with NFS.

Mimetypes sound like a good idea – make use of as much existing prior art as possible to make life easier. Inodes would seem like sensible reuse too.

I agree the disc partition structure shouldn’t be exposed to the user. The simplest way might be to have the same image file references in the root of another filing system – eg Files:$.HardDisc4.!Boot but that probably wouldn’t map onto existing users’ expectations very well so you might need to apply the current discname model – ie Files::HardDisc4.$. FileCore already does this in the case of DOS floppies (I think) – it presents the DOSDisc image file to DOSFS which provides the root directory (I think, I’ve never quite understood exactly how this differs from a DOSDisc image as a file in a FileCore FS).

Should the physical location of the volume be lost? It’s already visible in the volume-based hierarchy I outlined before, but do we want to turn ADFS::foo.$, SCSIFS::bar.$ and NFS::mountpoint.$ into Files::foo.$, Files::bar.$ and Files::mountpoint.$? There’s a problem of conflicting names (what if you take the drive called HardDisc4 from your other machine and connect it with a USB adaptor – will there be two drives called Files::HardDisc4.$?).

Please let’s shoot HardDisc4 as the default drive name :)

I’ve started putting some proposed details of these filing system changes in the wiki
Comments welcome.

Not being a developer, I’m afraid I can’t comment meaningfully on the detail of your proposals. But from a user’s point of view, this project seems to me to be the most important piece of work that needs doing to RISC OS, and your outline is extremely encouraging: I hope it prompts others to get involved. Being able to use large files, and large mass storage devices formatted on other machines, is becoming increasingly important.

If this gets off the ground, and non-programming work is needed (i.e. documentation, or template design, or something), please count me in.

Chris: Thanks for the offer.

I’ve now updated the wiki with some initial details of the userspace API changes. I haven’t yet thought of how to add some of the filetype/mimetype ideas whilst retaining a sensible degree of backwards compatibility. Suggestions welcome.