Filing system bounties
Sprow (202) 1158 posts |
I reckon it’s about time to crystallise all this into some measurable progress. As I have a long train journey coming up in the next 2 weeks it should be possible to organise all these proposals into a plan which can get some milestones attached and hence bounties. The shopping list is large, complicated, and daunting, but none of the individual elements are impossible! So, I searched the interweb, and this forum, and collected the following list. Last call for any others? Note, I’m not interested in implementation detail at the moment, just the feature point that needs changing. From comp.sys.acorn.hardware thread “RISC OS VFS replacement” General From http://www.acornusers.org/wishlist/ ROOL forum and wiki |
Jess Hampshire (158) 865 posts |
Increase the limited number of drives on a file system |
Jeffrey Lee (213) 6048 posts |
Unicode filenames |
Sprow (202) 1158 posts |
I’ve edited my earlier post to reflect those 2 additions, and commenced the analysis. One train journey and several Eurovision rounds of voting later here come the results: First, I just pasted the list above into Excel. Then I coursely grouped them, for example ‘full permissions’, ‘UID on each file’, ‘Arbitrary meta data’ are all in the general theme of meta data not currently recorded. Then I combined related items within groups, for example ‘full permissions’ and ‘UID on each file’ is basically ‘Unix like file permissions’. Second, I classified each feature by affected component, for example some affect FileCore, others FileSwitch, others FileCore and FileSwitch, etc… The task “Rewrite portions in C” was eliminated as it is not a feature point, but an implementation decision. Some extra tasks were inferred, for example adding partition support requires going back an updating clients to use this new API. I colour coded each component. Third, I grouped the features by dependencies. For example, the 2GB file limit is not tied to any other activity, whereas 64 bit file extents are. The colour coding from step 2 showed that some groups overlap several components. Fourth, with grouped features by dependency I scored each task by ease of implementation and benefit to typical users. I’m naturally lazy so anything which is a big win for not much effort should go near the top of the pile. I sorted this list by the mean product of ease x benefit for each of the groups in step three. Fifth, I then estimated how long I thought each task would take. While it may take people less (or more!) time than me to do, this is just a scaling factor. Absolute time isn’t important. This revealed that the grouping had made some things massively longer than others which didn’t seem fair so I took out some (independent) stuff and spread it evenly into other groups. Three groups were eliminated as bounties as they were islands of only a couple of days of effort each. Voila. |
Sprow (202) 1158 posts |
In each of the proposed 5 bounties below, I’ve normalised my effort estimate to give prospective developers some idea of the size of the job. The cunning planning step means that the shortest job with the most immediate benefit is of course first, in bounty 1. Next, some scaffolding is put up to jiggle around the existing system to make is more sensible, basically sliding a drive manager in inbetween. Below the drive manager are the block data devices, these know how to turn requests for groups of sectors into commands on the wire (eg. ATA in ADFS, SCSI, SD cards, RAM, even ATAPI if you want FileCore CD-ROMs). The drive manager worries about partitions and presents physical drives as logical drives to the layer above. Bounty 2 probably provides relatively little end user benefit but is an important stepping stone. Then comes the biggie. My time estimate suggests the implementor will need to consider whether to apply these changes to FileCore or to start with a fresh media layout. Because of bounty 2 the choice doesn’t really matter since FileCore will still be around and able to talk to existing media, whether it is joined by a whole new media layout or whether it gains a new sub format (F++ or H?) gives the same user outcome after bounty 3. Some of the new features in bounty 3 are then made available through API extensions and C library functions. Things like having a ‘hidden’ bit in the filer attributes and updating it to support 64 bit lengths also get enabled here in bounty 4. Last comes UTF8 in filenames and corresponding updates to built in FileSwitch clients that can support this but don’t currently. DOSFS also no longer is an image filing system by bounty 5. Roughly speaking it goes
while there are so many other ways to sort these pretty coloured marbles, since the feature points have been collecting since 2007 there needs to be some vaguely objective Excel driven stake in the ground rather than 4 more years of talking. I have further sub-details on the tasks within the bounties which can be shared, assuming these get accepted as official! Bounty 1 [factor 1]
Bounty 2 [factor 2.3]
Bounty 3 [factor 3.5]
Bounty 4 [factor 2.5]
Bounty 5 [factor 2.3]
Non bounties
|
Jess Hampshire (158) 865 posts |
Some questions and observations: Bounty 1-3: This means making it 4GB? Bounty 3-7 and 4-2 (the same, different parts of the job?) if the new api (extensions) are non-blocking, then blocking could be allowed on old programs for ease of compatibility. (And a module created to use the API on old systems.) Bounty 2-1: shouldn’t a partioning tool be a separate job? (It would be useful before RISC OS supports accessing extra partitions.) |
Sprow (202) 1158 posts |
Correct.
In 3-7 the on-disc format is changed to allow for 64 bit extents, in 4-2 the APIs are extended to make use of this feature.
I don’t think so, you need to add support for partitions, so you’ll need to create the tool at the same time. The two are linked, and the implementor of bounty 2 is probably best placed to make a tool having designed the system. |
Steve Revill (20) 1361 posts |
Hi Rob. A good bit of spreadsheet bashing there. Maybe next time we meet we could go through this in more detail but from the looks of things, that’s as good a selection of file system-related bounties as any I could think of. |
Jess Hampshire (158) 865 posts |
I should have asked whether this is the ADFS format restriction, or whether it is a restriction that would stop things like FAT32FS from accessing 2GB – 4GB files. (I haven’t been brave enough to try). [edit: I just tried, the filesizes on FAT32FS and DVDFS both seem correct, but attempting a copy gives a stupidly large number of bytes to go, plus no readable memory at this address] However, unless this is a low hanging fruit, is it needed? Also, if it is a fix for ADFS format drives, what would be the implications of using a drive with 2 – 4 GB files on an old system? Also it occurs to me that the term ADFS is often used when referring to the format, rather than the filesystem, would it be good for ROOL (or better still ROOL and ROL) to suggest correct terminology? |
Jess Hampshire (158) 865 posts |
Wouldn’t this be better the other way round? Then network filesystems (i.e. with non RISC OS servers) could be updated to use it imediately. (Is it actually a good idea to update the ADFS format, anyway?) |
Jess Hampshire (158) 865 posts |
The important thing I see about partitioning is interoperability (and dual-boot) with other systems. Ideally the system would allow an ADFS format partition at the start of the drive, which could be accessed by current RISC OS systems, and a standard MSDOS style partition table, which would mark the ADFS format partition as used (there are several unused partition types), and work as normal on other systems. It looks like this should be possible. (After experimenting with a memory stick and gparted and scsiform) Even before partition support is added to RISC OS, it would have advantages. 1. It should allow dual boot RISC OS / Linux on a single drive. 2. It would allow USB drives to be used part with ADFS for 3. If a USB drive were partitioned ADFS / FAT32 and FAT32FS were modified to be aware of this and not dismount the AFDS partition we would have support for two partitions on the same drive without any changes to RISC OS. 4. Anyone replacing a drive would be able to use the tool, and hopefully not need to reformat the drive when partition support happens. 5. It would allow testing of the partion arrangement for backwards compatibility, before any work is done on implementation in the OS. (Note – the initial tool wouldn’t need to be RISC OS based, because most of the uses imply access to a second system) |
Dave Higton (281) 668 posts |
You can’t put in a disc of a format that the system doesn’t understand. It isn’t going to work. And why would you, anyway? |
Jess Hampshire (158) 865 posts |
I have read comments that the 2GB limit on ADFS format is due to a signed integer being used instead of an unsigned one. The implication being that fixing this would double the filesize. However what was not clear is whether this would work on existing formats or require a new system.
If you had a drive that was used on a select system, as well. |
Dave Higton (281) 668 posts |
Where do you think the bug is? It’s in the filing system, ergo a new one is required.
Sorry, Jess, that’s a daft suggestion. If you use a disc on multiple systems, it has to be formatted such that all systems understand it. |
Jess Hampshire (158) 865 posts |
If the the format is intrinsically 32 bit, and is only limited by wasting one bit, it would work. Some comments I have read imply this. (However it may be intrinsically 31 bit)
But that is precisely the situation we have with FAT32FS, it works fine on RISC OS, unless you put files over 2GB on it. |
Sprow (202) 1158 posts |
I think the origin of the misnoma is from the BBC Master era, when BBC Model B owners were used to Acorn DFS formatted floppies and then had the new ADFS format to choose from with a heirarchy of directories and 10 letter filenames. In that case ADFS was both the format and the thing that made the drive spin. In RISC OS the on-disc format is maintained by FileCore and it asks clients to do transfers to the physical drive. So RAMFS, ADFS, SCSIFS and so on are all saved in FileCore format. |
Matthew Phillips (473) 721 posts |
I don’t see how this fits in with the idea of partition support and a drive manager. DOSFS would have to be a client of the drive manager, would it not? As would whatever handles FileCore formatted drives. Bear in mind also that DOSFS is useful as an image filing system because you can currently take images of DOS discs and store them as appropriately-typed files on your hard drive. CPMFS and MacFS do this too. If you’ve lifted the file size limits, it seems to me the main reasons for moving away from the image filing system API for handling foreign disc formats have gone. But they ought to be talking to a drive manager, rather than via FileCore: the whole business of FileCore handling its own format and also driving the hardware drivers like ADFS is messy. Or do you mean that DOSFS would remain an image filing system and so mainly talk to FileSwitch? In which case, I agree. Sorry: this is all a bit confusing, as image filing systems which handle foreign disc formats talk to FileSwitch and FileCore at present. |
W P Blatchley (147) 247 posts |
Might be worth pointing out that even though the Filer could probably be adapted to cope with Unicode filenames, at the moment you’d be totally stuck at the CLI. We might need an interim step that shows unicode-named files in some ASCII-understandable way until such time as the CLI is updated to cope… |
Malcolm Hussain-Gambles (1596) 811 posts |
Is it 2012 already? ;-) |
Steve Revill (20) 1361 posts |
This bounty has been claimed, meaning work is underway towards achieving all the goals listed on that page. We’re considering opening the bounty for “File system improvements (Step 2)” so people can register their interest/support development by donating to that one. The only thing standing between us now and opening the Step 2 bounty is us agreeing internally about what should be in the second one. The biggest problem we face, to be honest, is that the really hard RISC OS tasks, the ones we envisioned the bounty scheme helping to drive forward, realistically represent many man months of work – which, when done on the odd evening, weekend, means they will take years to complete. Unless people are able to take time off work to dedicate a lot more time to the task. However, with the bounties not really fetching very much (in cash per hour terms) they aren’t really succeeding as a motivator. We’d need to see the bounties getting into the several thousand pounds mark for those harder tasks to stand a chance of actually getting done (or someone very skilful with lots of free time on their hands appearing out of nowhere). |
Malcolm Hussain-Gambles (1596) 811 posts |
Thanks for the explanation. |
David J. Ruck (33) 1635 posts |
Just to update the topic. DiscKnight 1.50 now correctly handles the new FileCore variant, allowing files of up to 4GB, the multi-zone disc map and the special partitioning for the Raspberry Pi. |
Jeffrey Lee (213) 6048 posts |
One thing I’ve realised is that the there doesn’t seem to be any mention of TRIM support in the filesystem bounty plans. Probably a good thing to slot in somewhere, now that SSDs are the norm for modern machines. |
Rick Murray (539) 13840 posts |
…and proper dismounting? ;-) |
Alan Williams (2601) 88 posts |
I would definitely like to see ‘resise to fit the device’ to work for RISC OS as it does for other OS on pies with ext4 etc when you download an OS image file and write it to an sd card. |