exFAT

It seems like every man and his dog are moving towards exFAT and leaving FAT32 and its cousins behind. I think therefore it would be beneficial if we could have exFAT built in as standard and leave FAT32 as a seperate app as this seems to be falling out of fashion now. My reasoning is it seems like there are a number of things that could be considered a standard that RISC OS still struggles with, or is unable to do, exFAT being one such thing. I do understand that finaces are thin on the ground and yes I massively appreciate everybodies hard work on keeping RISC OS going but I do believe we shouldnt have to consider what format something is before we plug in.

Today I was moving some files around between 2 hard drives and RISC OS failed me I had to use my mobile phone….

Anybodies thoughts on this

Greg

It seems like every man and his dog are moving towards exFAT and leaving FAT32 and its cousins behind.

Really? I don’t think I have one single exFAT device. The big harddiscs are NTFS and the smaller things (SD cards etc) are FAT32.

I think therefore it would be beneficial if we could have exFAT built in as standard

…only you will find that you probably need to install packages to get Linux to understand exFAT. Why? Because although a GPL implementation exists (which makes it difficult for RISC OS), exFAT is actually a patent encumbered format, and Microsoft does enforce their patents when they think it useful. The date of expiry of the first set of patents on exFAT is sometime in 2024.

and leave FAT32 as a seperate app as this seems to be falling out of fashion now.

Says who? There’s still plenty of life in “smaller” SD cards of the 16/32GB sizes. One might also wonder if there wasn’t some “irregularities” taking place with the SD alliance mandating that 64GB SD cards use a non-open format.

exFAT being one such thing.

I wonder if it wouldn’t serve RISC OS better to look at porting support for either NTFS or ext# to aid with Windows/Linux interoperability.

but I do believe we shouldnt have to consider what format something is before we plug in.

Use a standard format? ;-)

Today I was moving some files around between 2 hard drives and RISC OS failed me I had to use my mobile phone….

I’d use my PC. Quicker than any RISC OS device.

I’d like to see exFAT support, but when I mentioned it a few months ago there was no interest (someone even asked whether I was joking). It seems to be the only modern format supported by both big commercial OSes (Windows and MacOS) without third-party software.

• NTFS – Read and write on Windows, read-only on MacOS
• HFS – Read and write on MacOS, no support by default on Windows, read-only with Apple’s driver
• ExFAT – Read and write on both platforms
• Extx – Not supported at all on either platform without third-party software, as far as I’m aware

Once RISC OS is added to the mix, none of these formats can be used. I’m stuck with FAT32, which has a fairly low file size limit and therefore I have to use a separate ExFAT drive when moving video files around (ProRes files can easily be over 100 GB). It would be great to be able to use the same format for everything.

exFAT is actually a patent encumbered format, and Microsoft does enforce their patents when they think it useful. The date of expiry of the first set of patents on exFAT is sometime in 2024.

I was under the impression that software typically isn’t patentable under UK law.. I certainly don’t know the specifics around this though, and I’m too lazy to check :)

I was under the impression that software typically isn’t patentable under UK law

I believe that to be correct; but the caveat here is the moment something containing an “unlicensed” implementation is supplied to a person in a more screwed up jurisdiction, UK law ceases to be relevant.
Yeah. I know. It’s stupid. But lawyers get paid better than any of us. :-/

BTW, doesn’t Filecore currently have a 4GB object size limit?

BTW, doesn’t Filecore currently have a 4GB object size limit?

I think you’re right, but I also think it’s possible to avoid this limit when using alternative filing systems like FAT32/exFAT/HFS since they’re not FileCore-based.

Seems that FAT32 has a 4GB max file size and 8TB max partition size.

exFAT seems to be available for Linux – so some lib files may be available for other OS’s.

It seems like I stand corrected haha

I have always been led to believe that exFAT was the more popular over NTFS for things such as big hard discs. However it still stands that RISC OS fails to OR struggles to recognise anything other than FAT32 aside ADFS obviously and I thought that maybe RISC OS could benefit from these other formats.

Seems that FAT32 has a 4GB max file size and 8TB max partition size.

I agree on this, Ive found it impossible to copy of my phone a video 4.5GB onto a FAT32 USB Drive

I agree on this, Ive found it impossible to copy of my phone a video 4.5GB onto a FAT32 USB Drive

Ditto. For an ISO image (4.4GB I think), I formatted an SD card as NTFS.

It is my understanding that NTFS consumes vast amounts of memory. I recall there was a plan to add NTFS support to my PVR, but with 32MB RAM (64 really but half is given to the DSP), there just wasn’t enough memory to get it working.

Of course, this was from years ago. I can only imagine things are worse now that consumer drives have pushed the terabyte boundary.

I use 2 2TB USB HDD for backing up my phone and Iyonix / PI2.

The 1st one RISC OS wouldnt read it, obviously formatted to something other than FAT32, so proceeded to format it to FAT32. Been fine every since.

The 2nd one, bought a couple of years later, however RISC OS refused to format it to FAT32 and left it unreadable so decided to format using my Galaxy S6 phone. It is now unreadable by RISC OS.

I can still back up my Iyonix / PI2 stuff onto this drive, I just have to do it via a FAT32 USB stick which my phone will read. Hence my wish for RISC OS to have the ability for exFAT or maybe in this case it is NTFS.

It would be nice not to have to muck around

It is my understanding that NTFS consumes vast amounts of memory.

Extra facilities/features¹ in NTFS cause extra memory requirements. There are quite a few articles around like this which was among the first few in a quick search. Most of the features are of more use on a system drive than a removable data storage device.

¹ Journalling being one such feature.

I think you’re right, but I also think it’s possible to avoid this limit when using alternative filing systems like FAT32/exFAT/HFS since they’re not FileCore-based.

Unfortunately, the 4 GiB (minus a few KiBs) object size limit is deeply rooted in FileSwitch, which means every filing system on RISC OS will suffer from it. CDFS, NFS, LanManFS, FileCore, CloudFS…you name it.

Ugh. It seems that that would need to be resolved first; going to exFAT while still having the same file size limit as FAT32 would be of little benefit. I suppose that this is one of those “really hard” things too!

I now almost nothing of Filing Systems but wonder why EXT (EXT2, 3 or 4) hasn’t been mentioned!
Wikipedia is not very informative about their licences.

wonder why EXT (EXT2, 3 or 4) hasn’t been mentioned!

Uh… Fourth reply part of the second message. ;-)
I called it “ext#” where you insert whatever digit is appropriate in place of the #.

Ah sorry! I’d missed your reference and thought Chris M’s ‘Extx’ was was yet another Filing System, there does seem to be a few!

I made the “x” italic but admittedly it isn’t very noticeable :)

I’ve edited it to be a little more obvious, although it looks a bit funny (at least on my Mac).

The 4 GiB limit is also reflected throughout the FS API, so extending this would be quite a bit of work. And it is THE critical subsystem of the OS. You have to be extra careful not to break the old API for the legacy FS clients.

But perhaps the biggest problem is that the few people capable of doing it would immediately think of all the things that could be done during revamping FileSwitch. Rewriting it in C. Including a cache system. Revamping the whole FS stack to avoid multiple wheel reinvention for client FSes.

And then you end up with a mammoth task.

Not to mention, the FS itself probably needs a rewrite to avoid clobbering certain parts of the storage media. Not a big deal with spinning rust, could be an issue over flash media lifetime.
Not to mention (2 – The Sequel), in this day and age it is ridiculous that the FS is unable to repair itself and the little tool we have (*CheckMap) is broken…

So, yeah. Mammoth. With fur and tusks and everything.

So, yeah. Mammoth. With fur and tusks and everything.

First skin and joint the mammoth then slice out a steak or two
While people are at the barbeque perhaps you can interest them in further meals…

Wouldn’t the first step be defining a large file API? This would initially be implemented by a module that maps it to the regular calls, but divides large files into folders of parts (which I believe is how CDVDBurn does things). New file systems could then implement this directly. (I also wonder if they could present oversize files to the old API as a read only folder of parts, so that copying etc would still work.)

Could GPL filesystems be implemented as an analogue of podule ROMs? i.e. a second (optional) ROM in the same folder as the RISC OS ROM.

Would UDF be a sensible format for interoperability? (Benefit – DVD/BD support, downside maximum partition size and maybe supported names/paths)

NTFS is quite easy to write to on Macs, (I think even out of the box on modern ones). Linux Mint quite happily reads and write HFS and NTFS.

I’m tempted to stick my head above the parapet and have a think about the 64-bit file size API changes that’d be needed in FileSwitch (which is mostly my fault). Not interested in any huge rewrites or new file systems – let’s see how hard it is to support large files that are otherwise currently unavailable to use by RISC OS applications on NAS boxes using Sunfish/LanManFS, eh? FSs would need to declare 64-bit file size support, of course, otherwise the existing unchanged API would be used.

Don’t forget that the desktop drag-n-drop Load/Save file etc. protocols would need replacing too.

A minimal viable change set would probably be to extend FileSwitch with a 64bit API, then change Sunfish to support large files, then change the filer to use the new API and allow copying between Sunfish mounts.

Go, Stuart, Go!

I’m tempted to stick my head above the parapet and have a think about the 64-bit file size API changes that’d be needed in FileSwitch (which is mostly my fault).

Perhaps look at the analysis already done and donate some money to help the problem rool describe in the thread, we can all help as plain users in this way

There potentially is no safe method. You see, I was thinking that all we’d need is a 64 bit API and an application can query it and use it if available (else revert to 32 bit method).

But I thought a bit more, and realised that a 64 bit FS would imply a 64 bit drive format (else, why bother?). Well, how does one talk to a 64 bit FS using a 32 bit API?

I’m thinking that the overall best method may be to provide a 64 bit API that is entirely different to the 32 bit one (not 32 bit with bits bodged on) and then have a 64 bit FS to go with it. This will in turn be supported by a 64 bit aware filer, and… you see, how does one do EXT# or PTR# from BASIC? The only way is to treat it as an entirely separate thing and migrate stuff to be compatible piece by piece.

In other words – it’s “difficult”. ;-)

Could GPL filesystems be implemented as an analogue of podule ROMs? i.e. a second (optional) ROM in the same folder as the RISC OS ROM.

IANAL, but I cannot see any great problem as long as it is not part of the RISC OS code base and it isn’t directly supplied with the RISC OS ROM image.
Of course… https://www.gnu.org/licenses/gpl-faq.html#NFUseGPLPlugins is heavily slanted to specifics of how Unix works. The “plugin” method could in theory describe the entire RISC OS API. Hmmm… This is covered more further down the page (https://www.gnu.org/licenses/gpl-faq.html#MereAggregation) that says “An “aggregate” consists of a number of separate programs, distributed together on the same CD-ROM or other media.” and “Where’s the line between two separate programs, and one program with two parts? This is a legal question, which ultimately judges will decide. We believe that a proper criterion depends both on the mechanism of communication (exec, pipes, rpc, function calls within a shared address space, etc.) and the semantics of the communication (what kinds of information are interchanged).” and “If modules are designed to run linked together in a shared address space, that almost surely means combining them into one program.”.
As you can see, RISC OS itself could technically fall foul of this. Modules interacting and passing structures to and fro in a shared address space?

But the best bit, IMHO, is the “ultimately judges will decide”. That alone is the reason I simply do not touch the GPL. I don’t want arbitrary judges in a litigation-happy country with a screwed up protectionist legal system deciding what does and does not fly. If it isn’t stated clearly and concisely in the licence, then the licence is useless.
That the GPL’s own FAQ includes the words “judges will decide” in response to a decision on what is and isn’t “a program” says to me the entire GPL is flawed, and indeed dangerous. These sorts of things should be clearly stated in advance, not “to be decided later by somebody else”.

Read the paragraph after. If programs pass data using sockets or pipes or CLI arguments then it’s two separate programs. If communications are “intimate enough”, one could consider the two parts to be combined as a larger program. This, again, shows (yet another) failing in the GPL for not recognising that there is a world outside of how Unix works. The entire RISC OS SWI API is based upon intimate sharing of data structures too and from modules (including the core kernel). When you get down to it, the entire Wimp API that every program uses is probably lower level than the majority of GUI calls on any flavour of Linux ever written.
(aside note: and has anybody actually ever used PipeFS in anger? ugh!)

In short, I do not foresee any problem with making a GPL “module” that is loaded separately akin to podule ROMs, in much the same way that supplying NetSurf (etc) with a RISC OS SD card image isn’t causing any crisis. Just don’t expect any such thing to be a part of the official RISC OS. GPL is too vague and too damaged to take the risk.

I do not foresee any problem with making a GPL “module” that is loaded separately akin to podule ROMs

That was pretty much what I was thinking, the question would be how viable would it be to do on things like the Pi? (You’d need it loaded that way so you could boot from the filesystem.

how does one talk to a 64 bit FS using a 32 bit API?

Wouldn’t you have a module that mapped the 32 bit calls to the new 64 bit ones? As far as I can see you’d need to trap any files that are too large and present them differently. If it presented them as a folder containing 1GB read only files, programs shouldn’t be able to make a mess, and you could even use 32 bit FS programs to copy them, providing the 64 bit system was clever enough to recognise them and save them as a single file.