Filing system improvements (step 2)

We’ve added the next filesystem bounty – it’s a biggy so please give generously!

One of the most important one yet I think.

It would be nice if the formatting / partitioning tool (or suite of tools) included the ability to add partition information to existing FC drives. (The main reason being to allow a disk to be formatted partially with the current tools now, ready for the future, but also to protect drives from other operating systems thinking they are blank).

It would also be nice if there could be some presets in the new formatting tool. Purposes of this:

1. To create partitions the maximum size for efficiency (i.e. biggest partition for a particular sector size)

2. To create a first partition that is usable on systems without the improvement.

3. To create partition that do not span the 128GB UDMA/PIO boundary on the Iyonix

Some other desirables:

The improvements to be softloadable on older systems.

Filecore support to be added to something like gparted so partition can be done on other systems, and perhaps a port to RISC OS.

The partitioning tools to work on earlier versions of RISC OS. (Even though you may not be able to use the resulting disk on them)

To chuck in my 2p, I would suggest looking at Simtec’s IDEFS for inspiration.
The formatter natively handled partitioning and you could modify partitions without trashing the others as looking as you didn’t muck up the others. Partitions were attached to the filesystem in the order found, 4-7 then 0-3 (order is historical) and if there were more than eight partitions (I’ve maxed it out) then the others could be attached and removed fairly flexibly. The formatter also calculated the LFAU for the partition size, and you could change it.

Some things a new FileCore will need – to understand NON native formats and to be able to favour a native format over a foreign one. An example is the Pi boot where the first partition is FAT and the second would be FileCore. FAT is necessary for booting the hardware, FileCore for booting RISC OS.

I’m thinking if this comes to be, what effect would it have on DOSFS? And could this pave the way for extfs support?

I’m less concerned about support for previous systems. It would be nice, but I’d prefer something working faster than if worrying about legacy machines . . . in much the same way that older versions (<4) don’t have support for long file names or the extended disc formats with such.

It would be very nice for the formatter to detect a FileCore drive and write a proper partition table for it without otherwise altering the partition.

Some things a new FileCore will need – to understand NON native formats

Doesn’t it recognise that what it has found isn’t Filecore and offer it to other filesystems to attempt to mount?

I don’t think partitioning would really change this, other than offering partitions rather than disks.

And could this pave the way for extfs support?

In that it wouldn’t be much use without partition support. It would be nice to see a new filesystem for RISC OS. I wonder if UDF would be a better choice (because we’d get DVD and BD support too). I hope FC isn’t extended beyond any low hanging fruit.

I’m less concerned about support for previous systems.

The utilities would either rely on the enhanced facilities in the new OS or not. I’d think it easier if they didn’t, because then it would be easy to create the disks that the new system will be able to read, before making it.

But if the enhancements were softloadable, it wouldn’t matter anyway to end users.

It would be very nice for the formatter to detect a FileCore drive and write a proper partition table for it without otherwise altering the partition.

Or a separate utility.

Would it not be better for future developments to first split FileCore into 2 components:
- the first one handles a logical partition as FileCore disc, providing the interface for fileswitch (FS read, write, mount, dismount, …) as well as code for initialising such a partition, checking if the partition an FileCore one (format vetting), …
- the second would handle the physical drive, register pertitions to the first one (providing it with a disc read/write routines which perform the physical to logical adresses translation). The module also offer interfaces to register/deregister several modules of the first kind, allowing for easier integration of other files formats.

From that point the File System improvements could be done independantly in parallel:
- transforming the second module in a central disc management system not limited to 8 drives per FS, adding physical/logical disk enumeration so that specialised tools like !HForm or DiskKnight can make use of it.
- centralising there the management of disc icons in a single Filer module, using the disc enumeration facility and drive addition/removal notifications (IIRC something that is also only present in SCSIFS at the moment).
- turning Fat32FS into a a module of the first kind so its usage is not tied to SCSIFS.
- adding other modules of the same kind for Ext3, NTFS, …
- extending FileCore for larger partitions

I was going to write a detailed argument for why partitions should be treated as drives, but I realised there’s a greater problem:

It is not expected that the native RISC OS format, FileCore, will be extended in this bounty. Therefore its limitation of 2^29 sectors, 2^12 bytes per sector, 2^0 partitions per drive, and 2^3 logical drives will be retained.

That’s not a format limitation, it’s an API limitation, and one that needs to be solved before it impacts upon how partitions are implemented (or anything else), IMHO.

FileCore_SectorOp was coined to escape the 2^29 byte limitation of FileCore_DiscOp. There’s no reason why new interfaces could not be created to avoid drive number and related limitations. Removing the drive number from the disc address would be a welcome improvement.

FileCore_MiscOp’s Mount call already separates the drive number from the disc address (sort of).

FileCore_Create could support 510 drives with the addition of a single flag in the FS descriptor’s flags to turn R6 into a pointer to a byte array of map sizes (or we accept that the ‘map size/256’ is only a guess anyway and not rely on it at all).

If those FileCore API restrictions were solved, then we would be free to represent FileCore partitions as logical drives and all other difficulties would disappear (or be totally reframed).

traditional desktop filers present one icon per physical drive at present – whereas with partitioning the user will need some way to select which partition is desired.

Whether or not one has too many drives or too many partitions appears to be moot – the effect is the same. My preference is for lots of icons and a scrolling icon bar, others may prefer a stacked representation (or perhaps more naturally for RO, a Filer-like ‘My Computer’ window containing the drives). I don’t think that aesthetic consideration should effect how partitions are represented at the API level.

Similarly the boot process, which searches a drive number for a file called !Boot may need adapting to select the most appropriate partition, or a default one if none are natively formatted.

Again, this complication disappears if partitions are “drives”.

Finally, I should also point out that partitions have been handled for a very long time by various IDE and SCSI filing systems, and in all those cases partitions were represented as drives. I have a large number of dual-partitioned 640MB PD discs named after comedy duos to attest to that.

On the HForm question, I’d like to mention some prior art here which is acorn-fdisk
That’s probably not directly relevant here (it understands all the existing RISC OS partition formats) but it may (or not) be useful as a framework for an HForm replacement. However it is GPL, and I’m not sure if a ROM-able formatter is an advantage.

For such a replacement, I’d suggest separating the creation of partition tables (on Linux the job of fdisk and similar) and the filling in of those partitions with filesystems (mkfs.* eg mkfs.vfat, mkfs.ext4, etc). In particular, if the code for making a FileCore partition is in a library, it can easily be turned into a mkfs.filecore for use with tools like gparted – that means Linux live CDs can ship with it. Likewise Windows/Mac/etc tools can be written.

This would solve a lot of chicken-and-egg problems. For example, the BeagleBoard setup guide told you to use a FAT formatted device, put a self-extracting HForm and some U-boot files on it, *Settype HForm.bas, use that to format a USB stick, then self-extract Sparkplug, then download !Boot and unzip it with Sparkplug. I tried this when I got a BBxM but only had one USB stick, so re-formatted the stick with the HForm binary on it. Only HForm failed, so I’d now corrupted my stick and I couldn’t even re-run HForm as it wasn’t there any more. Would be much easier to say ‘download oursetup.exe, tell it where your USB stick is, and it’ll format and install a RISC OS distro on it’. Or ‘boot RISC OS with Ctrl-Alt-Shift-Win-Menu-Q held down and you’ll drop into the partition tool’.

Compatibility with existing tools like gparted would involve:

• A GPL-compatible licence (BSD is good)
• Not making assumptions about endianness and structure alignment. There are macros about to help with this.

I believe Ben has some code in this direction already.

I agree that partition support is overdue.

Even if a newly-created filecore partition were just to write the necessary partition table entries to protect itself (the area currently unused) to recognise other areas of the drive which contain other partitions (and ignore them) and to allow extra FAT partitions to be created in areas unused by any existing partition and to allow one special FAT partition that maps onto !Boot.Loader to be optionally created that would be a major step forward. (That can all work within the constraint that there is only one filecore partition on a drive.)

I was wondering if the money for the USB bounty could be diverted?
Or partially diverted/whatever….

I’m not sure if a ROM-able formatter is an advantage.

I would like to see a formatter (and partitioner) in ROM and a version of packman and working DHCP networking. The idea being if you just boot the ROM, and you have a harddrive and working internet connect you could install a full system.

It’s good to read a wide range of people’s thoughts and opinions on this really important bounty.

Given the importance of this piece of work and the relatively low number of donations to date, I’m looking at ways to speed this up, otherwise we could be waiting until 18 months before the bounty is at a sufficient level before any coding even begins.

what sort of $$$ figure do we think will be required before some wizard coder will take up the challenge of this?

Ubuntu Edge crowdsourcing managed to secure $10Million dollars in 25 days; We’ve managed ~$150. so there’s room for improvement.

Ubuntu Edge crowdsourcing managed to secure $10Million dollars in 25 days; We’ve managed ~$150.

And in what sense are those figures incommensurate?! ;-)

As no one is doing anything round here for the dollars, but for the love of it, you may get a better result from “£150 and a big hug”.

Yeah, I know what you mean. The huge amount of work done so far has been for free. Pretty amazing really.

I was just thinking perhaps we could up the ante; Donate:
$20 – signed photo of Steve & Andrew
$40 – signed photo of Ben ;-)
$60 – 1 hour free RISC OS Coding help by one of our experts
$100 – Big hug from Nemo

The more money we can get from outside our traditional community perhaps will speed up some development in some key areas.

There are some really talented people round here who might be persuaded to return to RISC OS if a reasonable amount of cash and hugs was available on a bounty. <looking at you Nemo ;)>

Just a suggestion, what about using !Store as a mechanism for fund raising?
There are a lot of apps that are badly needed on RISC OS.
I know ROOL aren’t generally interested in writing applications, but as a mechanism for fund raising for ROOL it may work.
I am thinking about doing something along those lines for my “fatter” applications.
Something like 15 quid for each bundle of applications and donating some to ROOL some to fund other RISC OS developments.
I’m not really interested in making money, there aren’t enough users for me to make enough to make it worth while at the moment but there are a quite a few RISC OS users out there, and there are likely to be more if there are more applications avaliable and thus more cash avaliable to fund ROOL.

Or how about some kind of donation mechanism? Something like 10 quid a month or something?

This may have been looked at before etc. just an idea…maybe a good one, maybe a terrible one…

Alan: Not wanting to belittle the work of those you mentioned, you omitted to mention Jeffrey Lee. He I think has been RISC OS’s biggest programming contributor for some years now. Thank you Jeffrey.

Why priced in $s?

@Malcolm Hussain-Gambles

Or how about some kind of donation mechanism? Something like 10 quid a month or something?

I suggested a long time back that ROOL should have set up a scheme for donations by standing order, but this was never followed up. See Forums —> Bounties —> ROOL Administration

$1000 – Pandora console signed by Uwe, Raik, Jeffrey and EvilDragon

I suggested a long time back that ROOL should have set up a scheme for donations by standing order, but this was never followed up.

It was around 8 weeks before the launch of the Raspberry Pi so it’s quite possible it got forgotten. I can also imagine it’s not as simple as the suggestion is brilliant: as ROOL are now VAT registered the standing orders appearing in a bank account would need to be accounted for separately, and some link established back with the bounties webpage. I’m not sure the how the bounty webpage works, but it updates automatically when you make a donation via Paypal by some hocus pocus (ie. no human involved).

As the only bounty that makes sense to have a recurring payment to is probably the admin bounty, it’s probably a disproportionate amount of effort to set up. Dunno. Paypal can do ‘subscription’ buttons which look reasonably simple HTML – maybe suggest that to ROOL?

What sort of $$$ figure do we think will be required before some wizard coder will take up the challenge of this?

Looking at the tasks listed in the bounty, it’s a pretty chunky amount of roadworks. Easily 3x more than the step 1 bounty, so the number is in £1000’s, not £10,000’s. Start hunting for coins down the back of the sofa!

Alan wrote:

$100 – Big hug from Nemo

I might pay even more than that.

Chris pointed out:

you omitted to mention Jeffrey Lee. He I think has been RISC OS’s biggest programming contributor for some years now.

Indeed, and what motivates Jeffrey is beyond my comprehension. Some people are just more saintly than others. OTOH Eric Hoffer said, “There is no striving for glory without a vivid awareness of the audience”. ;-D

I don’t think Hoffa has it quite right. I recall feeling very smug when I worked in a Hospital and drove to work on Christmas Morning at 6a.m. with no one on the roads. Even if no one else knows, you know!
I have though met people who will help others at the drop of a hat and I’m sure it never crosses their mind to feel proud or worthy.

More on topic for programmers I came across this video which I found very interesting.

Well I just tossed another 100 quid into the pot so let’s see if anyone nibbles…

I recall feeling very smug when I worked in a Hospital and drove to work on Christmas Morning at 6a.m. with no one on the roads.

6a.m. ?
Late start? Hospital network stuff: they like stuff as out-of hours as you can manage so finishing by 6am is good.
You’re right I do feel smug, but usually about the lack of traffic on the roads when I do those runs. I’d never thought of it as an automatic merit badge for working at that time. Watching mice run around the car park at 04:30 and avoiding the local ginger tomcat is quite entertaining when you’re doing a slow walk between one arranged job/timeslot and the next.

$20 – signed photo of Steve & Andrew
$40 – signed photo of Ben ;-)

Oi! I’m not cheap.

Mutter, mutter, mutter…

It was around 8 weeks before the launch of the Raspberry Pi so it’s quite possible it got forgotten.

It did.

I can also imagine it’s not as simple as the suggestion is brilliant: as ROOL are now VAT registered the standing orders appearing in a bank account would need to be accounted for separately, and some link established back with the bounties webpage. I’m not sure the how the bounty webpage works, but it updates automatically when you make a donation via Paypal by some hocus pocus (ie. no human involved).

Yes. The main blocker is some way to make the direct debit/standing order link into the bounties page so the totals get updated appropriately. I think the current code uses PayPal’s APIs to do what it does, rather than talking to our bank account, so adding this new feature is actually surprisingly tricky – or, we’d need to implement something that runs in parallel with the real direct debit to keep the bounties page in sync, but with the inherent risk that it could be overlooked if a direct debit stops, leaving things in a bit of a mess (especially if someone claimed the bounty with more in it that it should have!).

I need to chew this over with Andrew, who implemented the current stuff.