DOSFS not cheking for disc sizes it can't handle?

A few months ago I had an issue in my boot sequence which I did not notice directly and which prevented the loading of FAT32FS during boot. Clicking on the 320GB USB disc showed the root directory correctly but accessing files or opening directories caused errors. It took me a while to figure that FAT32FS was not loaded and that it was DOSFS which was handling the disc.

Wouldn’t it be wise to had checks to DOSFS so that it rejects drive sizes it cannot handle?

I asked about this half a decade ago, when DOSFS couldn’t handle >2GB and trashed the filesystem of a larger device in attempting to deal with it – https://www.riscosopen.org/forum/forums/2/topics/1056

320GB? That’s brave. I’m too cynical to give DOSFS a live filesystem of any size with important data on it. I usually use FAT32FS, but it’s a USB stick of ~4GB size with no “live” (as in not copied elsewhere) data…

I asked about this half a decade ago, when DOSFS couldn’t handle >2GB and trashed the filesystem of a larger device in attempting to deal with it – https://www.riscosopen.org/forum/forums/2/topics/1056

Relicense RISC OS under an MIT style license – problem solved?

Relicense RISC OS under an MIT style license

Yes. Because relicensing an entire OS made of many parts from many sources when you’re not the rights holder… That’s easily done. ;-)

Had a quick look at FAT32FS. The library part is LGPL, so it might be possible to come to some sort of arrangement with Jeff if we’re going to lay DOSFS to rest…

That said, FAT32FS is loadable and usable. It would just be really good if DOSFS could be modified to reject media it clearly can’t cope with. No if but maybe, just a simple “can’t”.

Relicense RISC OS under an MIT style license

Yes. Because relicensing an entire OS made of many parts from many sources when you’re not the rights holder… That’s easily done. ;-)

I would have thought the bigger problem with that statement was the implication that relicensing the OS will make the heavens open and deliver us a flood of motivated developers.

Isn’t that what crowdsourcing is? Wishful thinking and the expectation of something from nothing?

In reality I think any “interested” developer is going to bail the moment they got their first STMFD and have the horrible realisation that it isn’t created by bolting together bits of frameworks of the current trendy language…

That said, FAT32FS is loadable and usable. It would just be really good if DOSFS could be modified to reject media it clearly can’t cope with. No if but maybe, just a simple “can’t”.

That’s the only sane solution, isn’t it? (“Tried RISC OS on the Pi and it TRASHED MY USB DRIVE !!! !!! This would never had happened, if it were open source!! SAD!”)

https://www.riscosopen.org/viewer/view/castle/RiscOS/Sources/FileSys/ImageFS/DOSFS/

Patches welcome.

“Tried RISC OS on the Pi and it TRASHED MY USB DRIVE !!! !!! This would never had happened, if it were open source!! SAD!”

Why? Open Source is not a proof of quality. I can prove it: the DOSFS module IS open source software. But not Free software, as OEM must pay for its use.

Quick hack time… :-)

https://www.riscosopen.org/viewer/view/castle/RiscOS/Sources/FileSys/ImageFS/DOSFS/c/DOSFS?rev=4.30#l373

    /* A few more sector sanity checks */
    dprintf(("","DOSFS: testforpartition: bbsize=%u, winisize=%u, fatsize=%u sectors\n", bbsize, winisize, fatsize));
    if (bbsize == 0) return 1; /* Bad start */
    if (bbsize > winisize) return 1; /* Image is bigger than the partition table has set aside. */
    if (fatsize > bbsize) return 1; /* FAT is somehow bigger than the entire image. */
    if ((rootsize / DOSsecsize) > bbsize) return 1; /* Root directory takes more sectors than the entire image. */

Add this just after to reject anything over 4GB:

    /* Test: Reject any device larger than ~4GB so we don't claim what we can't handle.
       The value below was determined by looking at the number of sectors on a FAT32 formatted
       8GB USB stick (bbsize=15515648), halving it (=7757824), and then "adding some". */
    if (bbsize > 7790000) return 1;

It will report “Error when reading SCSI::0.$ – Disc not understood – has it been formatted?” because DOSFS will no longer ‘claim’ the device as something it recognises.
The convoluted method is because I had an 8GB device (that it failed at correctly reading, including crashy-crashy) but no available 4GB device. There was one, but SCSIFS rejected it for having a bad sector size. Hmmm!?

I rather suspect the trouble comes a few lines further down when:

    discsize(dr) = bbsize * DOSsecsize; /* Use bbsize incase FileCore was guessing discsize */

“discsize()” is a defined macro that writes “((drec)→dr_discsize)” where “dr_sizesize” is defined as a word (which is itself defined as an unsigned int).

Thus, anything over 4,294,967,295 (i.e. 4GB) simply can’t be correctly dealt with.

A better “solution” might be to try assigning the disc size calculation to a long long and seeing if the result overflows UINT_MAX?

At any rate – this “quick hack” will sanitise DOSFS from touching what it can’t handle.

There might be another bug here, too. It’s an 8GB device being read, so not sure if it’s the directory is beyond the 4GB range, or what. It seems to me mighty suspicious.

Here’s directory entries being read, like this:

DOSFS 20:39:27: read_dir: lfnp = 201eb22c act:00000000 holds:NULLptr
DOSFS 20:39:27: read_dir: cdir = 201eb154
DOSFS 20:39:27: read_dir: cfile = ‘Ore No Imouto 06.avi’ 201eb228 2b
DOSFS 20:39:27: DOSFS: convertDOStoRISCOS: “Ore No Imouto 06.avi” converted to “Ore No Imouto 06/avi”
DOSFS 20:39:27: filetype lookup,found in mimemap 0xFB23
DOSFS 20:39:27: read_dir: file name = Ore No Imouto 06/avi
DOSFS 20:39:27: read_dir: loop = 44
DOSFS 20:39:27: read_dir: returned *dentry = 201ebcfc
DOSFS 20:39:27: read_dir: loop = 45
DOSFS 20:39:27: read_dir: returned *dentry = 201ebd1c
DOSFS 20:39:27: read_dir: loop = 46
DOSFS 20:39:27: read_dir: returned *dentry = 201ebd3c
DOSFS 20:39:27: read_dir: loop = 47
DOSFS 20:39:27: read_dir: returned *dentry = 201ebd5c
DOSFS 20:39:27: read_dir: lfnp = 201eb23c act:00000000 holds:NULLptr
DOSFS 20:39:27: read_dir: cdir = 201eb154
DOSFS 20:39:27: read_dir: cfile = ‘Higurashi no Naku Koro ni 16.avi’ 201eb238 2f
DOSFS 20:39:27: DOSFS: convertDOStoRISCOS: “Higurashi no Naku Koro ni 16.avi” converted to “Higurashi no Naku Koro ni 16/avi”
DOSFS 20:39:27: filetype lookup,found in mimemap 0xFB23
DOSFS 20:39:27: read_dir: file name = Higurashi no Naku Koro ni 16/avi
DOSFS 20:39:27: read_dir: loop = 48
DOSFS 20:39:27: read_dir: returned *dentry = 201ebd7c
DOSFS 20:39:27: read_dir: loop = 49
DOSFS 20:39:27: read_dir: returned *dentry = 201ebd9c
DOSFS 20:39:27: read_dir: loop = 50

However look what happens here:

DOSFS 20:39:27: read_dir: loop = 251
DOSFS 20:39:27: read_dir: returned *dentry = 201ed6dc
DOSFS 20:39:27: read_dir: loop = 252
DOSFS 20:39:27: read_dir: returned *dentry = 201ed6fc
DOSFS 20:39:27: read_dir: lfnp = 201eb570 act:00000000 holds:NULLptr
DOSFS 20:39:27: read_dir: cdir = 201eb154
DOSFS 20:39:27: read_dir: cfile = ‘Kimi ni Todoke 06.avi’ 201eb56c fc
DOSFS 20:39:27: DOSFS: convertDOStoRISCOS: “Kimi ni Todoke 06.avi” converted to “Kimi ni Todoke 06/avi”
DOSFS 20:39:27: filetype lookup,found in mimemap 0xFB23
DOSFS 20:39:27: read_dir: file name = Kimi ni Todoke 06/avi
DOSFS 20:39:27: read_dir: loop = 253
DOSFS 20:39:27: read_dir: returned *dentry = 201ed77c
DOSFS 20:39:27: buildFILEname: dentry:201ed77c
DOSFS 20:39:27: buildFILEname: got:€IÓ§«ƒ”·.ß`q
DOSFS 20:39:27: read_dir: lfnp = 201eb580 act:00000000 holds:NULLptr
DOSFS 20:39:27: read_dir: cdir = 201eb154
DOSFS 20:39:27: read_dir: cfile = ‘€IÓ§«ƒ”·.ß`q’ 201eb570 fd
DOSFS 20:39:27: DOSFS: convertDOStoRISCOS: “€IÓ§«ƒ”·.ß`q” converted to “€IÓ§«ƒ”·/ß`q”
DOSFS 20:39:27: read_dir: file name = €IÓ§«ƒ”·/ß`q
DOSFS 20:39:27: read_dir: loop = 257
DOSFS 20:39:27: read_dir: returned *dentry = 201ed79c
DOSFS 20:39:27: read_dir: loop = 258
DOSFS 20:39:27: read_dir: returned *dentry = 201ed7bc
DOSFS 20:39:27: read_dir: loop = 259
DOSFS 20:39:27: read_dir: returned *dentry = 201ed7fc
DOSFS 20:39:27: buildFILEname: dentry:201ed7fc
DOSFS 20:39:27: buildFILEname: got:QÌØõ[‰✘Ï.§ÈÍ
DOSFS 20:39:27: read_dir: lfnp = 201eb590 act:00000000 holds:NULLptr
DOSFS 20:39:27: read_dir: cdir = 201eb154
DOSFS 20:39:27: read_dir: cfile = ‘QÌØõ[‰✘Ï.§ÈÍ’ 201eb588 103
DOSFS 20:39:27: DOSFS: convertDOStoRISCOS: “QÌØõ[‰✘Ï.§ÈÍ” converted to “QÌØõ[‰✘Ï/§ÈÍ”
DOSFS 20:39:27: filetype lookup -§ÈÍ- not found.. default to DOS
DOSFS 20:39:27: read_dir: file name = QÌØõ[‰✘Ï/§ÈÍ
DOSFS 20:39:27: read_dir: loop = 261
DOSFS 20:39:27: read_dir: returned *dentry = 201ed81c
DOSFS 20:39:27: buildFILEname: dentry:201ed81c
DOSFS 20:39:27: buildFILEname: got:.–3õ
DOSFS 20:39:27: read_dir: lfnp = 201eb594 act:00000000 holds:NULLptr
DOSFS 20:39:27: read_dir: cdir = 201eb154
DOSFS 20:39:27: read_dir: cfile = ‘.–3õ’ 201eb590 105
DOSFS 20:39:27: DOSFS: convertDOStoRISCOS: “.–3õ” converted to “/–3õ”
DOSFS 20:39:27: filetype lookup -- not found.. default to DOS
DOSFS 20:39:27: read_dir: file name = /–3õ
DOSFS 20:39:27: read_dir: loop = 262
DOSFS 20:39:27: read_dir: returned *dentry = 201ed83c
DOSFS 20:39:27: buildFILEname: dentry:201ed83c
DOSFS 20:39:27: buildFILEname: got:É©Íqò·âW
DOSFS 20:39:27: read_dir: lfnp = 201eb598 act:00000000 holds:NULLptr
DOSFS 20:39:27: read_dir: cdir = 201eb154
DOSFS 20:39:27: read_dir: cfile = ‘É©Íqò·âW’ 201eb594 106
DOSFS 20:39:27: DOSFS: convertDOStoRISCOS: “É©Íqò·âW” converted to “É©Íqò·âW”
DOSFS 20:39:27: filetype lookup -- not found.. default to DOS
DOSFS 20:39:27: read_dir: file name = É©Íqò·âW
DOSFS 20:39:27: read_dir: loop = 263
DOSFS 20:39:27: read_dir: returned *dentry = 201ed85c
DOSFS 20:39:27: buildFILEname: dentry:201ed85c
DOSFS 20:39:27: buildFILEname: got:>®.VŴê
DOSFS 20:39:27: read_dir: lfnp = 201eb59c act:00000000 holds:NULLptr
DOSFS 20:39:27: read_dir: cdir = 201eb154
DOSFS 20:39:27: read_dir: cfile = ‘>®.VŴê’ 201eb598 107
DOSFS 20:39:27: DOSFS: convertDOStoRISCOS: “>®.VŴê” converted to “>®/VŴê”
DOSFS 20:39:27: filetype lookup -VŴê- not found.. default to DOS
DOSFS 20:39:27: read_dir: file name = >®/VŴê
DOSFS 20:39:27: read_dir: loop = 264

read_dir() scans through 384 entries, and not one single one with a loop value over 255/256 reads correctly. They’re just utter gibberish.
Is the offset being held as a byte value or something?

Open Source is not a proof of quality. I can prove it:

I can prove it better. OpenSSL

Okay, fair enough, that’s just one OSS project, and crypto is hard.

Oh, wait, what about bash? Or dealing with fancy MMS messages in Android (the OSS part)?

And we need to talk about “Dirty Cow” (Linux and derivatives) just because it’s logo is pretty awesome: https://en.wikipedia.org/wiki/Dirty_COW

This isn’t to say that non-OSS software is bug free. Far from it. As Meltdown and Spectre have shown, even the hardware has issues that can be used to gain access to data that isn’t supposed to be accessible. The difference is that it’s far easier to pretend that there’s nothing wrong with closed source software (FAKE NEWS!!!) and fail to provide fixes – Adobe is quite famous for this, which is why their name will be forever associated with one of the least secure internet technologies ever devised. And their idea of adding random scripting to PDFs is pretty awful given their woeful history with Flash and its gaping chasms (not security “holes”, Flash is a gaping chasm complete with waterfall).

Furthermore, OSS can quite easily trash things just as much as anything else. Wanna talk to me about how utterly miserably awful Brasero is and how it ignores a specific user request to write DVDs slowly, spins them at max speed, tries to write that, fails, and down near trashes the disc or the drive or both in the process? Test repeated twice until I gave up, dropped the files on an SD card, and wrote them to DVD (peacefully) using Nero with Windows.
While we’re here. That nice video editor suite that is in the Ubuntu package repo? You know, the one that looks kind of unfinished (something of a trend with OSS), and is lucky if it stays running more than five minutes without crashing.

ALL software can suffer from quirks, bugs, and problems. Anything else is just a delusion.

read_dir() scans through 384 entries

I should point out that while there appear to be this many entries in the directory structure, there are only 75 files actually present…

Why? Open Source is not a proof of quality. I can prove it: the DOSFS module IS open source software. But not Free software, as OEM must pay for its use.

Just because it’s not closed source doesn’t mean it’s open source. “OEM must pay for its use” disqualifies it as open source software:

6. No Discrimination Against Fields of Endeavor
The license must not restrict anyone from making use of the program in a specific field of endeavor. For example, it may not restrict the program from being used in a business, or from being used for genetic research.

Yesterday was the 20th anniversary of the introduction of the term “open source software”
https://opensource.com/article/18/2/coining-term-open-source-software
https://www.engadget.com/2018/02/03/20-years-of-open-source-software/

ust because it’s not closed source doesn’t mean it’s open source. “OEM must pay for its use” disqualifies it as open source software:

I think you’re confusing “open source” with “free to all”, which is not surprising I suppose given the the person credited with coining the term actually seems confused:

A quick Google search indicates that “open source” appears more often than “free software,” but there still is substantial use of the free software term, which remains useful and should be included when communicating with audiences who prefer it

So you see, even she seems to thing that “open source” and free software are equivalent which, given the existence of free to use software that originates from closed source, is rather strange.

Do note that anyone can use the source free of charge provided they don’t try to sell anything derived from it because that specific point (sale of the derived item) is where the cost kicks in.

On the issue of quality and open source software I agree, there is no general correlation. Other factors could be much more relevant, e.g. the people writing the code, funding, etc.

(Maybe I should have added a irony smiley to my stupid comment with the exclamation marks)

So you see, even she seems to thing that “open source” and free software are equivalent which, given the existence of free to use software that originates from closed source, is rather strange.

Did you even read that stuff?

Let’s go back to the original announcement
http://www.catb.org/esr/open-source.html

I already used software in the 80s and 90s and free to use software was called freeware. Free software aka libre software is this Stallmann / FSF / GPL thing.

Did you even read that stuff?

Erm, how do you think I came by the bit I quoted?¹

It’s toward the end of the article in which she covered the subterfuge of using a techie to deliver the phrase because perhaps the suggestion of a non-techie might fall flat but if a techie delivered it then it might fare better.

I read her references to Stallman as carrying elements of what many others think of him – rarely positive.

I already used software in the 80s and 90s and free to use software was called freeware. Free software aka libre software is this Stallmann / FSF / GPL thing.

Part of the fog and confusion Stallman generates, free software is simply free. Freeware is a term concatenating free and software because it’s “hip”/trendy/whatever. He would like people to believe that free software must have available source code and that simply isn’t so, it might be nice but then if a vendor releases a slightly limited version of their software as a reader for the files produced by the full fat version and accompanies it with the source what’s to stop the user modifying the free version into a full fat version and putting the amendment on the net thus putting say Adobe out of business?
Or, more locally, someone fiddles the source for the read-only version of SparkFS and makes a read-write version for free – not that DP makes a living, or even beer money, from SparkFS

BTW. You might notice the hyphens in her article: free-to-use
Which I take to imply that the whole phrase is the term just like open-source

¹ It’s a habit, I read stuff

6. No Discrimination Against Fields of Endeavor

But blatant discrimination of “open source” (as a true concept) is perfectly acceptable when it is presented as a positive ideology and one that must not be questioned?

Yes, I’m referring to the GPL and the fact that it exists and is compatible with only itself, which is not my definition of “open source”. The moment you start adding restrictions (OEMs must pay, GPL only compatible with GPL, etc) it ceases to be “open source” and becomes “source available”. For all the talk about “freedoms”, GPL is just that. Talk. Hot air. Religious pontification.

Just because it’s not closed source doesn’t mean it’s open source. “OEM must pay for its use” disqualifies it as open source software:

Just because it’s not OSI compliant doesn’t mean it’s not open source. Anyway, here source is visible and there are bugs.

DOSFS not checking for disc sizes it can’t handle?

FYI I’m partway towards a fix for this – there are a few different places that need tweaking for out-of-range offsets to be caught correctly.

Fat32Fs uses the efsl for its FAT work (and, if I understand Jeff Doggett correctly, it had to be extended to work with long filenames), which is GPL. Has anyone investigated alternatives to efsl, like e.g. FatFs http://elm-chan.org/fsw/ff/00index_e.html which seems to also support exFAT and has a very libre BSD licence? So we could throw away DOSFS and live happily ever after?

I think this should be handled properly now (or better than it was before, at least). Let me know if anything’s broken which shouldn’t be!

https://www.riscosopen.org/viewer/revisions/logs?ident=1518998757-800500.html

In c/DOSdirs now that the malloc has been removed , can the checking for NULL pointer also be removed ?

Yes – well spotted!