Filing system bounties

At ROOL, we’ve been trying to think up suitable projects to assign bounties to. One large area which seems suitable is filing systems, especially local filing systems (FileCore, ADFS, SCSIFS, RamFS etc). Common areas that people cite include

• Bigger files (facilitating bigger DOS discs, for example)
• Partition support
• Bigger discs (and more efficient use of big discs)
• More discs per filing system

A naive approach might be to assign each of these a bounty, but with a bit of thought, it becomes obvious that there’s a fair amount of overlap between them – for example, at many levels, lots of partitions look the same as lots of discs. It can be easy to see these simply in terms of API changes, but the more I’ve started digging into this, the more I see lots of fundamental issues that need addressing.

Bear in mind that a bounty needs to be:

• well-defined enough when the pot of money is opened, that anyone donating money can’t complain once the bounty is awarded that something they were expecting hasn’t been delivered
• limited enough in scope that someone doing the implementation doesn’t suffer too much from feature-creep once they have started work
• particularly important for filing systems, which are data-critical: there need to be well-defined acceptance criteria before the pot of money is released to the developer – obvious candidates would be automated “FS Bash” soak testing tools

Where there is an unavoidable dependency of one bounty upon another, I suspect we may have to only entertain donations for those with no dependencies, and add others as and when their dependencies have been completed – although in some cases this may pose a problem in justifying to donors why the available bounties are necessary, if they have no immediately obvious benefit. On the other hand, it would help keep the total number of bounties down, which is important to avoid dilution of available funds.

Some would advocate the only way to achieve this is to throw away the whole filing system stack and start again from scratch. My personal view is that this is probably unnecessary – that this would waste lots of reusable code which had been thoroughly debugged over many years of use, and most of which doesn’t actually need changing. Most of the changes involve “simply” changing the size of variables, which is amenable to “keyhole surgery” development, as has been proved successful before with things like the 32-bitting process, or the move from DiscOp to DiscSectorOp. I also think this approach is more readily divisible into orthogonal subtasks which would be suitable for assigning bounties to.

What follows is a collection of observations. There are a number of unknowns in there – it’s possible that there are other people reading the forums with more knowledge on the subject than me can rapidly fill these in. Consequently, I haven’t attempted to draw up candidates for bounties yet.

FileCore 31-bit file size limit

This is probably the most easily isolated task, and has no impact upon any APIs. Although FileSwitch already supports 32-bit file pointers (files up to 4GB) on many filing systems such as NFS, there is something in the way FileCore is implemented which further restricts it to 31-bit pointers (up to 2GB). This is why DOSFS partitions are currently limited to 2GB.

ADFS improvements

ADFS already knows the ATA LBA48 instruction set, and uses it instead of LBA28 for DiscSectorOp calls where bit 28 is set (bits 29-31 are the physical disc number). It should be relatively simple to fully implement the DiscOp64 API as defined here, and to test it, at least via a test harness which does block-level accesses to a large disc on an Iyonix. Discs up to 3TB in size are currently easily available. Because the new API supports more drives, you would probably implement support for more physical discs at the same time (perhaps testing it by installing multiple SATA PCI cards in an Iyonix, although drivers for these would need to be written).

Some investigation needs to be done into the impact of ATA-8’s larger sectors (4K rather than 512 bytes). Do any drives require this, or are they likely to do so in future? Is there a significant performance penalty to using 512-byte sectors on drives that natively use 4K sectors?

FileCore is documented to only support sector sizes of 256, 512 or 1024 bytes. How hard would it be to extend it to support 4K sectors natively? Nowadays, it wouldn’t be much of a restriction for the LFAU to be given a 4K minimum, so if it’s just a matter of support for LFAUs smaller than 4K sectors, that’s not a problem.

Note that one implication of 4K sectors would be that zones get bigger. One potential problem with that would be the size of the free space links in the map – but at 15 bits, they allow for links of 2^15 = 32 kbits from one free space block to the next within the map, which as luck would have it is just barely big enough!

SCSIFS improvements

SCSIFS would need extending similarly to ADFS to properly support DiscOp64. I’m not familiar enough the SCSI command set to say whether we’re facing a need to switch to larger commands yet. Testing large numbers of devices is easier than with ADFS – simply add lots of USB mass storage devices to an Iyonix or Beagleboard. Large discs can be tested using the same 3TB hard drives used for ADFS testing, but via an ATA-to-USB adaptor.

64-bit file pointers

This impacts FileCore upwards.

New variants of various interfaces need to be added at both the high and low APIs of FileSwitch:

• OS_File 5, 7, 11, 13, 15, 17, 20, 21, 22, 23
• OS_Args 1, 2, 3, 4, 6
• OS_GBPB 1, 3, 10, 11, 12
• OS_FSControl 28, 38
• FSEntry/ImageEntry_GetBytes
• FSEntry/ImageEntry_PutBytes
• FSEntry/ImageEntry_Args 0,1,2,3,4,7,8
• FSEntry/ImageEntry_File 5,6,7
• FSEntry/ImageEntry_Func 15,19
• FSEntry_GBPB 1,3

FileSwitch really just acts as a glorified telephone exchange between applications and filing systems, so I expect it will have limited amounts of processing to actually do on the pointers, beyond converting back and forth between applications using the 64-bit APIs and legacy filing systems that only support the 32-bit APIs.

Although I don’t agree with everything in this wiki page, it does at least begin to tackle defining 64-bit versions of the APIs.

One common user of the high FileSwitch interface is of course the C library. This is an issue which has been faced by many other OSes before us and 64-bit extensions to stdio.h have been developed – I suggest we copy these for compatibility’s sake. A new chunk to the stubs will be needed to support these new entries.

Any C applications wishing to use large files would need to be recompiled using the new headers, new functions and new stubs.

The Filer would need updating to use 64-bit directory enumeration calls, and display suitably large file sizes correctly.

FileCore’s E+ directory format is unfortunately limited to 32-bit file lengths, so another new variant would be needed (the obvious name is E++ format). Although this need not differ in other aspects from E+ format, reducing necessary code changes, it’s too good an opportunity to miss to evaluate whether other fields need extending. For example, while it’s not obvious to me from the Ursula FileCore spec what the limit it imposes on idlen is, my best guess is that this is 23 bits; we need to evaluate if that is big enough for our needs for (say) the next decade, or whether it should be increased.

Ideally, we need to ensure that existing versions of DiscKnight won’t attempt to “correct” E++ format discs.

DOSFS improvements

This is dependent on the above 64-bit file pointer work, but in other respects is an independent task. Once DOSFS is extended to use the above new ImageEntry reason codes, it will be possible to use FAT-formatted devices bigger than 4 GB via DOSFS (or 2 GB if you assume the FileCore 31-bit pointer issue hasn’t already been addressed). This is arguably not a high priority because of the existence of FAT32FS, but we should aim to fix it anyway.

Partitioning

There’s an obvious task to survey all commonly-used partitioning schemes used on third-party RISC OS IDE interfaces, as well as on other OSes, and add support to parse them in FileCore, and create/modify them in HForm.

I would suggest that each partition appears as a separate RISC OS drive, since for the majority of software won’t care about the partitions being on the same physical disc. We will need auditing the internals of FileCore to ensure no confusion between the “logical” drive number parsed from pathnames, and the “physical” drive number passed to ADFS/SCSIFS via the DiscOp API, but once this is done, there isn’t necessarily a direct dependency on work to support more than 8 physical drives per filing system. Letting the icon bar filer know what’s a floppy and what’s a hard disc may need some thought; using bit 2 of the drive number doesn’t scale well. What’s actually important from a user interface perspective is which drives feature removable media, although it doesn’t appear to be easy to identify this directly via the SCSI command set at least (currently the only things identified as hard discs are literally sealed spinning platters with an ATA-to-USB interface attached). It might be possible to do better by passing information through from the USB layer, but this is unclear – certainly, other OSes don’t seem to do a better job of identifying when a drive has removable media, but then other OSes tend not to have such good separation of the concepts of drive and media as RISC OS has.

We will probably need some additional APIs developing (probably at both high and low ends of the FileSwitch API) to enable software that cares to identify when multiple partitions are on the same device. This is needed for formatting software and disc information software, but also to handle ejection properly (you dismount a partition, but eject a physical disc).

Some thought needs to be given to the icon bar filer user interface. For one thing, having a large number of drives on the icon bar can be confusing. For another, partitioning support, while fine for fixed discs (which are displayed using the disc name, which would now be the partition name), doesn’t work as well for removable discs. Removable disc icons deliberately use the drive number rather than the disc name because they’re not mounted until you click on them – what happens if once it’s mounted it turns out to have two partitions? Do we have two “:0” icons? Does clicking on any “:0” icon remount all the partitions on that disc, or only if a media change had been detected? Perhaps if you dismount any “:0” partition, the icon disappears, except for the last one which then goes back to being a “click me to mount all partitions in this drive” icon? This needs some more thought.

Perhaps the only sensible thing to do with “:” disc specifiers is to keep them referring to (the first partition on) a given physical drive? Otherwise, as you mount partitioned media in removable drives in an arbitrary order, the drive numbers assigned to the newly-discovered partitions would become jumbled up.

Or perhaps we could use a new notation to access partitions – say, using the special field so that “ADFS#0::0” means the first partition on physical drive 0, “ADFS#1::0” means the second partition. But that breaks the hierarchical nature of sequential elements of the pathname, so perhaps “ADFS::0:0” is better, though that may break any software that tries to parse pathnames independently of FileSwitch, so maybe we have to settle for something like “ADFS::0/0” even though that would previously have been a valid disc name. But then that’s also going to be true for “ADFS::8” through “ADFS::255”, so it’s not an isolated problem.

Granularity of file allocation

Some have suggested that one way round this is to add partition support and split big discs into lots of partitions. Apart from this being an inconvenience to the end user, I don’t think this solves the problem given present limitations, especially since there are other pressures to increase the number of logical drives (both due to demands for more physical drives, and for partition support on each of those physical drives). I’ll try to explain.

Each instantiation of FileCore (one for each filing system) currently has a hard limit of 8 drives. For each drive, FileCore maintains a dynamic area containing a copy of the drive’s free space map. ADFS doesn’t support hot-plugging, so as many dynamic areas as there are drives area are allocated; SCSIFS doesn’t have this luxury since it’s the filing system used for USB mass storage, so dynamic areas for all 8 drives are allocated. These don’t use up RAM for drives which are absent, but they do use up quite a lot of logical address space. You may notice that at present, these dynamic areas are limited to 4MB each – so SCSIFS alone takes up 8*4MB = 32MB of precious global logical address space. The size of these dynamic areas is already a compromise – a 4MB map can’t make full use of formats with idlen=21, let alone idlen=23 which is enforced by current FileCore formats (as far as I can tell).

Why is this important? Well each bit in the map corresponds to a unit of file allocation granularity (LFAU) on the disc. So if the map is limited to 4 MB, one LFAU can be no smaller than 1/(8*4*2^20) of the disc. For a 2 TB disc this is already 32 KB. This may not sound too bad, but the smallest object on a disc is (idlen+1) LFAUs, so in this case it’s 672 KB – nearly as big as a double-density floppy disc! The impact is lessened to an extent by the fact that “small” files can share the same disc object as their parent directory, but this still sets a minimum size for each directory

And it’s only going to get worse. If we apply Moore’s Law to disc sizes, then in a decade we’ll be looking at 100 TB discs, and a 4 MB map would mean an LFAU of 4 MB and a smallest object size of 88 MB – 1/7th of a CD!

We could allow the disc map dynamic areas to be as large as is allowed by idlen=23, which is approximately 25 MB. Even this would only permit today’s 3 TB discs to have an LFAU of 16 KB and a minimum object size of 384 KB. And tomorrow’s 100 TB discs would be limited to 1 MB LFAU and 24 MB minimum object size. This doesn’t sound terribly efficient to me, but then are you going to care that much about the odd several megabytes of wastage if your disc is 100 TB large?

Leaving this aside for a moment, let’s look at the pressures on memory allocation for maps:

• larger maps per logical FileCore discs – as discussed above, 25 MB is probably the minimum we should aim for to support the next few years’ developments in hard discs
• allowing multiple logical discs (i.e. partitions) per physical disc – lets say 8 minimum?
• allowing more physical discs per filing system – the DiscOp API limits this to 256 (both by the new error reporting API, and by the definition of the drive word in the disc address triplet). Maybe for practicality we could limit this to 32 physical drives, so the limiting number of logical drives becomes 8 * 32 = 256?

So now we’re looking at a logical address space requirement for maps of 25 MB * 8 * 32 = 6400 MB. Per filing system. But the ARM architecture only has a 4 GB address space, and that has to include everything – applications, RMA, screen memory, I/O space etc, and in a typical current 32-bit RISC OS system, even with dynamic area clamping in use, about half the logical address space has already been claimed before any third-party software gets installed.

It should be noted that partitioning doesn’t help the tradeoff between LFAU and map size at all, it just changes the way you look at the problem. If you partition a disc in two in order to halve the LFAU, you just have two maps each of the same size as before you halved the LFAU – this takes just as much RAM and address space as simply halving the LFAU for the whole disc would have done, it just uses up an extra drive numbers and poses maintenance problems for the end user, who becomes responsible for ensuring that neither partition fills up.

So something radical has to change. This requires serious design work. What are our options? We could use an application slot per filing system (512 MB – 32 K available). This would permit 195 logical drives with idlen=20, 21 logical drives with idlen=23, or 1 logical drive with idlen=27. There’s little point in trying to address this by increasing the application slot size at the moment, as only the Iyonix can exceed 512 MB RAM (and only up to 1GB RAM at that). But there are potential performance bottlenecks here, since most file operations involve loading/saving the application slot – and so this would need lots of remapping.

idlen=27, by the way, would allow a 4TB drive to be mapped using an LFAU of only 2KB. This is back in the sort of granularity that appears to have been typical when the Ursula FileCore spec was written, and drives were nearer 4 GB than 4 TB. However, the efficiency of handling the corresponding map of 470 MB in size has got to be questionable – the search time to find a free space block could be considerable.

Perhaps we need to be considering on-demand paging of the disc map? This would save on RAM utilisation, but only while the disc map remains in a dynamic area since we don’t currently have support for holey application slots (not that that would be a bad thing to add to the bounty list in its own right). And this still doesn’t solve the logical address space exhaustion issues.

Something better than just allocating one 4 MB dynamic area for each drive on hot-pluggable filing systems might be another approach, maybe one big dynamic area for all maps for a given filing system, and then have the maps allocated using a crude heap within that dynamic area. Would have all the usual fragmentation issues that heaps have, of course.

Going up a stage in complexity but also in potential benefits would be a disc map cache, probably on the granularity of a disc map sector to avoid too much thrashing when zone auto-compaction occurs.

So in summary, it’s a complete can of worms. It’s far from obvious what the best approach is for most of these “more drives” / “bigger drives” bounties, and there is a lot of overlap between them, behind the scenes. It may be foolish to try to specify a technical solution up front at this stage, I can only guess at what the best approach is. What’s needed is a proper up-front study – more in-depth than this post – of all the important issues and how best to proceed, much like Acorn did before starting any of the Ursula FileCore work. But who’s going to do that, and should that have a bounty in its own right?

Of course, having had to investigate all these issues just to the extent that I had to do to write the above, I’ve gone and got my own interest piqued. Must… finish… objasm…

One common user of the high FileSwitch interface is of course the C library.

And BASIC. What’s going to happen to PTR#channel, EXT#channel et al.?

Some thought needs to be given to the icon bar filer user interface.

If we need an E++ format anyway, how about allowing customised icon bar sprites to be allocated to a disc, in addition to a name?

Also, something along the lines of the ‘hide USB’ functionality would be useful – maybe a ShareFS-like implementation which reduced icon bar bloat by only having commonly used drives in permanent residence?

Actually, now I think about it, ShareFS is an ideal user interface candidate for the philosophical issues you mentioned regarding the insertion of partitioned removable media. How about just displaying ‘Discs’, with none initially mounted but the number and details of partitions scanned, ready for the user to mount them and (optionally, see above) add them to the icon bar? Of course, some kind of configuration to add certain commonly-used partitions by default at the point of insertion would be useful, too.

In short: for large numbers of discs, think hierarchical rather than linear, but with optional shortcuts. Much like the RISC OS Filer and Pinboard, really. I think it dovetails quite nicely.

maybe we have to settle for something like “ADFS::0/0” even though that would previously have been a valid disc name.

Incidentally, what actually happens if you try to name a disc “0”? The infamous “Ambiguous disc name”?

Other than that, I don’t think that’s going to be an issue. It’s certainly preferable to inserting an extra colon, which will break every single program at once, right there.

What’s actually important from a user interface perspective is which drives feature removable media, although it doesn’t appear to be easy to identify this directly via the SCSI command set at least (currently the only things identified as hard discs are literally sealed spinning platters with an ATA-to-USB interface attached). It might be possible to do better by passing information through from the USB layer, but this is unclear – certainly, other OSes don’t seem to do a better job of identifying when a drive has removable media, but then other OSes tend not to have such good separation of the concepts of drive and media as RISC OS has.

Humble suggestion: devices connected via USB are removable, devices connected via IDE/SCSI are not.

Not sure of the implications for floppy and CD drives, but surely RISC OS has some way of distinguishing them already?

Random thoughts:

• one clearly separatable bounty would be to extend Fileswitch to 64bit and as a first client extend DOSFS to use the 64bit interface
• in theory, distinguishing between removable and fixed drives is easy in SCSI: fixed drives are device type 0, removable drives are type 1 (or 4 (MOs) or 5 (CD/DVD/BD))
• if someone extends filecore to be able to use 4k sectors, don’t forget to add 2k sectors – this would be nice for MOs and DVD-RAM and BD-RE
• currently, SCSIFS uses the Read(10) command which allows 32bit block addresses which is good for addressing 2 TB when using the usual 512 Bytes/block – using Read(12) or even Read(16) should be easy and buys us some more years ;-)
• another good idea for a bounty: provide a generic filecore client that handles the whole buffering and just needs a simple block driver – seperate ADFS as a first proof-of-concept
• another bounty: seperate ATAPI access from ADFS and implement it as a SCSISoft driver
• why do we want to extend filecore? I don’t think it scales very well, even today the map takes up a lot of valuable RAM – we should be able to find a better filing system somewhere in BSD land!

SCSIFS improvements

Also remember that another much-needed SCSIFS improvement is to add support for background transfers.

Granularity of file allocation

I think the best solution to deal with large free space maps eating RAM/logical address space is to change the way they’re stored in memory. Assuming that filecore only needs to access small amounts of the map at a time (or at least just a handful of pages), it may make sense to implement a system where a small dynamic area with can be used to access a much larger set of physical pages. This would keep the impact on the logical address space low, without requiring the wimpslot to be mapped in/out every time a free space map needs to be accessed. The system could also easily be reused for things like bigger RAM discs, bigger disc caches, etc. And somewhat tangentaly connected is the subject of supporting display rotation on OMAP – in order to support that we need a system where we can allocate physical RAM without mapping it into the logical address space (because it needs to be accessed via the rotation engine instead).

Just a quick comment to answer a question in the first post. A number of drives are already available that utilise 4k sectors, I believe. If memory serves, recent iterations of Western Digital Green drives 500Gb, 1Tb, 2Tb are 4k and have caused problems for some PC users. I think Seagate and Samsung also have some drives kicking around, although I don’t have time to look it up. But, they are here.

DOSFS improvements …

Partitioning …

FileCore’s E+ directory format is unfortunately limited to 32-bit file lengths, so another new variant would be needed (the obvious name is E++ format).

why do we want to extend filecore? I don’t think it scales very well, even today the map takes up a lot of valuable RAM – we should be able to find a better filing system somewhere in BSD land!

I think the first thing to do is have a proper architecture in place first to make it easy for everyone. What a disc drive is connected this it what should happen:

1. the IDE/SCSI/USB manager should register the drive to a central disc manager, along with an handle and the SWIS required to access the drive (DiscOp64, DescribeDisc, etc).
2. the disc manager registers the drive and checks the drive to discover how it is partitionned and builds a list of “logical” disks with their own handle, disc description and access SWIS (to perform address translation before calling the real drive’s SWIS).
3. for each “logical” disk the disc manager asks “logical” Filing Systems if they understand that disc (a sort of VetFormat).
4. if one of the “logical” FS understands the disc (Filecore E? FAT12? FAT32? NTFS? ExtFS?) it registers it and interfaces it with FileSwitch.

This scheme is I think easy to implement and test step by step:

1. you define the drive manager interfaces: (de)register drives, (de)register logical filing systems
2. you define the logical file system interfaces: VetFormat (Mount?), Dismount
3. you implement the drive manager, without partitionning support, just passing the physical disc handle, description and SWIS to the logical filing systems, and creating the drive’s icon on the iconbar.
4. you write a little logical filing system which registers each new drive as filing system “FSxxxx” with drive 0 (4?) where xxxx is an increasing counter so that you can handle many drives in your first tests without having to bother with the FileCore drive limitations.
5. you write a little tool to create FileCore disc images of xx MB.
6. you write a module which can register your disk images to the drive manager.

Once you get that running properly you can work your way step by step:

• Integrating the logical FS interfaces into FileCore and removing it’s number of drive limitations.
• Doing the same to FAT32FS and replacing the SCSI_DiscOp calls by the one provided when registering the drive. I’m mentionning FAT32FS here as I presume that it would be far easier than adapting an Image FS like DOSFS.
• Adapting ADFS, SCSIFS, RAMFS to register their drives to the drive manager instead of registerting themselves to FileCore.
• Extending the disc manager to support partitions.
• Integrate disk cache at the disc manager level.
• Defining how the disks are known to the world and how to pass that information accross the interfaces:
  • The naming for FileSwitch:
    • Do we ensure that they are still seen as “ADFS::0.$.” and the like?
    • How do we name partitions?
    • Do we do something so that disc images are accessible wih their filename as prefix just like DOSFS images?
  • Iconbar:
    • Disc images shouldn’t appear on the iconbar.
    • I’d like to see a more elaborate icon naming scheme like “dsk” where is “f” for fixed disks, “r” for removable disks and where is provide by the logical FS “riscos” for a FileCore disk, “fat” for a DOS disc, “none” for an unrecognised partition.
    • How do we organise the list of disks on the iconbar: all icons on the icon bar, a single icon opening a window listing all the drives, … ?
• Providing interface to list and provide access to physical/logical disks for tools like low level formatters, partionners, sector level verifiers, FS level verifiers (DiscKnight).

why do we want to extend filecore? I don’t think it scales very well, even today the map takes up a lot of valuable RAM – we should be able to find a better filing system somewhere in BSD land!

Even though Jeffrey appears to have some ideas to avoid this, I don’t think it’s a good idea for compatibility reasons. Do we want a new format that only works on RISC OS? The main reasons for a new format would be not leaving a big disk mostly unused, and allowing big files.

The first would be solved by a partitioning system, with 256GB filecore and Fat32 (8TB according to some documentation) partitions.

However what would be the main use of files of over 2GB on RISC OS?

The two uses I can see are DVD ISO images and video files. Both these would often involve other platforms, so to be really useful, ext3, NTFS or/and UDF support would be needed anyway. (an example would be downloading a large file to use with a media player.)

So I would agree with the comment for a separate reason.

There’s an obvious task to survey all commonly-used partitioning schemes used on third-party RISC OS IDE interfaces, as well as on other OSes, and add support to parse them in FileCore, and create/modify them in HForm.

Is there any need to support anything other than MSDOS and GUID partition tables?

It is unlikely that anyone would require (or even be able) to use an existing interface both under its 26 bit RO and RO 5.

It appears to be possible for an MSDOS type partition table to co-exist with a current style ADFS format, so this should mean that an ADFS partition at the start of a partitioned drive should be usable on non partition aware RISC OS. It would also imply that a partitioner could be written before the system can use partitions properly. (e.g partition 256 GB ADFS plus 750GB NTFS on a terabyte USB drive, for use by a PC and an RO system).

Or perhaps we could use a new notation to access partitions – say, using the special field so that “ADFS#0::0” means the first partition on physical drive 0, “ADFS#1::0” means the second partition. But that breaks the hierarchical nature of sequential elements of the pathname, so

Would it be possible to mount disks with the partitions appearing as subfolders in a read only root directory?

Using a linux style example: hda containing partitions 1,2,3

You could mount hda and see folders 1,2,3 which would correspond to the partitions, or you could mount the individual partitions directly.

You would see hda, hda1, hda2, hda3 in a sharefs style chooser.

Bigger files (facilitating bigger DOS discs, for example)

Since this will involve a new API, could it be non-blocking?

How do we organise the list of disks on the iconbar: all icons on the icon bar, a single icon opening a window listing all the drives, … ?

Would it be possible to mount disks with the partitions appearing as subfolders in a read only root directory?

If the “windowed approach” is to be implemented, we mustn’t forget to deal with users dragging a file directly to the iconbar drive icon (rather than a filer window)? Also, different hardware would be suited to different implementations, e.g. all icons shown would be less of a problem on a high resolution desktop than a UMPC/netbook. (Sorry to be stating the obvious to the technical wizards on here.)

It would be the same as with OmniClient. You can not drag a file to Omniclient but you can to the mounted remote discs.

I hate to slightly derail this thread, but I agree with Jess. From what I hear of the FileCore codebase, it’s somewhat lacking in maintainability. If we’re going to transition to a new format, I don’t see how we can justify keeping FileCore going much further, particularly given the major changes Ben has outlined above. I can accept that E++ might be worthwhile if it supports 64 bit file sizes with relatively minor changes, but if we have to address the map issues it sounds like it’s time to have a new filesystem. This might not be so much work if it re-uses code from elsewhere.

We would need a filesystem that’s:

• Stable
• Well-tested
• Re-entrant?
• Available as BSD/MIT licensed
• Single threaded
• Acceptable CPU/memory usage
• Not patent encumbered
• Has available fsck tools with appropriate licence
• Supports extensible metadata enough for filetypes (unless the ,xxx mapping can be made really bomb-proof, which it currently isn’t), ideally with some way to expose this metadata on other platforms
• Possibly supports datestamp granularity of more than one second
• Doesn’t have any upcoming limits (eg ext2 has max 32TB with 8KB blocks)

There are also some related issues that the FS and/or FileSwitch might need to deal with (eg what happens if I plug in a volume containing hard links?)

I hate to slightly derail this thread, but I agree with Jess. From what I hear of the FileCore codebase, it’s somewhat lacking in maintainability. If we’re going to transition to a new format, I don’t see how we can justify keeping FileCore going much further, particularly given the major changes Ben has outlined above.

I don’t think things are quite as bad as you think they are. Although there’s a lot of source code to FileCore, it does seem to be structured in a fairly logical manner, making it easy to identify the different subsystems. Apart from the 32bit conversion, the last big update that was made was the addition of the E+ format – about 15 years ago. 64bit disc addresses/file pointers will last us for much longer than 15 years. In fact, the only update I can think of us wanting to make after this one would be to update FileCore to be suitable for use in a threaded/multi-core version of RISC OS, which is still very much a pipe dream. Also remember that if we were to opt to use a different disc format for 64bit disc/file support then we’d still have to keep and maintain the existing FileCore code base, for compatability with old format discs.

One way to make FileCore easier to maintain, in my mind at least, would be to convert it to C. Since the disc format is well understood and the source is fairly well documented this shouldn’t be too hard, and could even be done in a piece wise fashion, one file or function at a time. In some cases it may even be possible to run both routines side-by-side at runtime in order to verify their behaviour. Since we’ll have to touch quite a lot of the source code in order to perform the 64bit updates, it would probably make sense to do the rewrite first. However since the rewrite won’t provide any immediate benefits, and would be a fairly large task, it may be difficult to justify it to people as a bounty item.

FileCore 31-bit file size limit

While idly browsing through the FileCore source, I found a comment relating to thils in FileCore.s.FileCore80:

KJB fix – with a nearly 2GB file, we may be asked to look up &80000000 – this used to lock up. This search has been converted to handle full 32-bit file addressing by checking for the Fcb terminator explicitly. We lose the oh-so-cunning signed comparison when going backwards, and unsigned when going forwards (phew).
If we went to 4GB files, we would have another problem, in that such a look-up would be impossible. We might have to restrict files to 4GB-1KB.

So it may be that the only thing that’s stopping >2GB files from working is the file cache (although judging by that comment, at least some of the cache code supports big files already)

Available as BSD/MIT licensed

Wouldn’t it be possible to use add-on filesystems with other licenses? (as with FAT32FS)

I would think it unlikely that 256GB volume and 2GB file size limits would be a problem for the boot volume.

If it were desired for tidyness to have a single big partition (on a beagleboard for example), couldn’t provision be made for loading the required filesystem from a separate file either from the bootloader, or using a similar method to the CMOS replacement. (Like a podule ROM, in effect.)

So it may be that the only thing that’s stopping >2GB files from working is the file cache (although judging by that comment, at least some of the cache code supports big files already)

This would give us 4GB files, like on FAT32?

Jess – it’ll be easier to read your posts if you use the "bq. " prefix for quoting, rather than "> ". There’s a link to the Textile reference given in the “Formatting Help” section immediately underneath the text area where you type in messages and replies.

Instead of this:

> So it may be that the only thing that’s stopping >2GB files from working is the file cache (although judging by that comment, at least some of the cache code supports big files already)

…which doesn’t stand out from surrounding text, you’ll get this:

So it may be that the only thing that’s stopping >2GB files from working is the file cache (although judging by that comment, at least some of the cache code supports big files already)

The only difference is the use of “bq.” instead of “>”.

Martin wrote:

devices connected via USB are removable, devices connected via IDE/SCSI are not.

I have 88MB Syquest and 1GB Jaz SCSI drives and an IDE 105MB Syquest that say you’re wrong. ;-)

The only difference is the use of “bq.” instead of “>”.

“>” used to display like that, did it get lost with the last update?

It’s not part of the Textile specification so if it ever worked, it worked by accident :-)

“Ideally, we need to ensure that existing versions of DiscKnight won’t attempt to “correct” E++ format discs.”

Good point, DiscKnight would very aggressively try to turn any E+ format extensions back in to vanilla E+.

On RISC OS 6 it already strips out the saved filer display options from directories, because if ROL refuse to supply the documentation, it doesn’t get implemented. Luckily RO6 users haven’t complained too much as long as their files are restored.

DiscKnight does have some built in leg room for increasing disc sizes, in that it wont fault standard E+ with idlen up to at least 27, as I expected this to have been increased years ago. Existing RPC and Iyonix hardware being physically limited or practically limited to 128GB, hasn’t made that worth while up to now.

If a E++ format supports 64 bit file lengths, I’d expect there to be a new set of magic values in directories which should prevent them being regressed by DK – I’ll have to check the source though! Some incompatible changes to the disc record in the boot block and map would prevent DiscKnight from recognising it all, and attempting any repair.

But the only guaranteed way to ensure DiscKnight doesn’t trash any new format would be for me to get a new version out before E++ hits the user base!

Not sure if this is the best thread to bring this up on, but just to add to the filing system suggestions, I know it’s been mentioned before but can we have some hooks added to the filer so that deletes & overwrites can be trapped by a suitable WasteBin-type App? (I’m naively envisioning something like a wimp_message being sent out when a file is about to be deleted/overwritten so that the wastebin can take a copy first.)

Not really necessary – the underlying OS calls can be trapped easily and handled appropriately.

Regardless, messages being broadcast (and then ignored, and then bounced, and only then acted upon by FilerAction) would make deleting multiple files insanely slow, so that’s not the solution.

Not really necessary – the underlying OS calls can be trapped easily and handled appropriately.

Ok, hopefully someone will be able to implement it fairly easily then :-)

Regardless, messages being broadcast (and then ignored, and then bounced, and only then acted upon by FilerAction) would
make deleting multiple files insanely slow, so that’s not the solution.

I did think after posting that wimp_messages weren’t really the way to go, but being a humble BASIC programmer of little talent that’s all I know about! Hopefully there’s a much more efficient system that can be used (well, BlackHole, Recycler, etc. etc. do it, so it must work somehow!).

One would expect a wastebin application (of which many have been written – doesn’t ROSelect come with Recyclone?) to work outside the desktop too. Consequently Wimp messages don’t really come into it.

The Filer asks Filer_Action to do recursive operations, so that can be trapped with a system variable redefinition if one wanted a UI to be able to select whether one wanted to really delete stuff – eg Shift-Del in Windows.

Just checked, and Recyclone doesn’t work outside the desktop! I saved a file to RAM, dropped to a command line and *Deleted it, and it wasn’t trapped! I think this one needs to work outside the desktop too, yes.

Yes, Recycler & Recyclone both offer Shift-Del to force a delete. I think we’d want to incorporate that too.

I’m not sure I agree. Deleting a file at either the Windows command prompt or in the Mac OS Bash shell results in it being deleted immediately.