Magic at ROUGOL and SASAUG

Before a packed audience in a room at The Blue Eyed Maid, near London Bridge, at a meeting last night of RISC OS User Group Of London, I demonstrated “Now You See Them Now You Don’t” with a USB stick. Inserting the stick in an ordinary Raspberry-Pi, provided by a member of the audience, and mounting it with the SCSI filing system 15 objects were seen (5 directories and 10 files); but, next, mounting it with the FAT32 filing system only 10 of these 15 objects were seen (scroll bars fully extended). The audience were aghast and even ROUGOL’s most knowledgeable experts present (Keith Dunlop was absent) could not explain the phenomenon.

A week earlier I had demonstrated the same ‘trick’ at a meeting of the Surrey And Sussex Acorn User Group with the same results, completely baffling the members present including several respected experts in the field.

For the sceptics, I point out that the beer was off at the Blue Eyed Maid and SASAUG meet in a church hall and so all were sober at both meetings.

How “big” is the USB stick? Does it (and did it) contain any private info?

I ask because those who develop such things might appreciate an image to see if they can replicate the issue. For instance, have you ever run doschk against the drive? Are both copies of the FAT identical and complete? Were the missing files created/opened/last modified in a way different to the others?
There is surely a reason for five objects apparently disappearing, but…
…without some live data to work on, tracking this down could prove to be difficult.

To give an example of my own, I have some files on a FAT stick using FAT32FS. They have long file names, appear on RISC OS as such. Insert the thing into a Windows box, it shows me stuff like LONGFI~1.ext and MYFILE~1.ext – yet it appears as it should under RISC OS… Hmmm…

those who develop such things might appreciate an image to see if they can replicate the issue.

Does anyone not have problems with fat formatted memory sticks/memory cards?

Does anyone not have problems with fat formatted memory sticks/memory cards?

Given that XP’s GUI disc scanner can’t find its backside with both hands and a big pointy arrow, and given that half the time one of the DLLs underlying chkdsk (command line) will crash with damaged formats, I certainly would not trust Microsoft to be competent to detect faults in and correctly repair their own disc format. For large-sectored¹ FAT flash sticks, I use a Debian etch port of dosfsck running on my PVR (!).

In other words – are you sure that’s a question you want to ask? ;-)

On the other hand, by Xperia U phone only supports MTP file transfer, it cannot be a USB mass storage device. Because of this the internal format is possibly not FAT-anything. ext2? ext3? At any rate, I tend to find my storage depletes rather rapidly, and the LOST.DIR folder fills up with large files that appear to be random fragments. Nothing appears broken or corrupted, but it is a worry that a closed-loop filesystem completely free from outside interference is incapable of keeping itself in good order!

¹ Small-sectored devices or overly large devices fail, the PVR only has 64Mb inside (30Mb for Linux and 34Mb for the DSP codecs (invisible to Linux)) and no swap, so it runs out of memory…but then it was designed to record video, not deal with Microsoft’s losingness.

It’s a 1GB USB stick, but I can repeat the phenomenon with a 256MB USB stick. It does have some private information on it, but the filenames shown in an image wouldn’t give away anything to matter. I don’t know how to include images in a post to this forum, but, as I said above the experiment was witnessed by a large crowd at ROUGOL and a substantial gathering at SASAUG. If you are interested in the characters in the filenames as one spectator was, they are alphanumeric plus some underscores, spaces, hyphens, forward slashes, round brackets and + signs. Nobody at the demonstration thought these had any bearing on which files disappeared. For example, of the two objects that had a space in their filenames one disappeared and the other did not.
I don’t know what you mean by “Are both copies of the FAT identical and complete?” as there is only one FAT version.
One file with a filename of over 40 characters doesn’t disappear while another with a 13-character filename including the ‘extension’ /zip , does . The only differences in the filenames of non-disappearing objects, was that all non-initial letters of filenames of directories under SCSI FS were in lower case except one that had both its first and second characters in upper case and another that had an upper case letter after a space, this affected two of the directories filenames.

I don’t know what you mean by “Are both copies of the FAT identical and complete?” as there is only one FAT version.

File Allocation Table
The system stores two copies of the space allocation table marking where the files are stored. One of the fun things about the FAT system is that it can (and often does) end up with the two copies out of sync. Unsurprisingly MS OS implementations don’t always notice or care. Other systems that read the table(s) differently give different results and this may be the effect you are seeing.

Unsurprisingly MS OS implementations don’t always notice or care.

Neither does Fat32fs!
Basically because comparing the FATs would impose a massive performance hit, it’s always worth using chkdsk or fsck to validate the media every now and again.

Other systems that read the table(s) differently give different results and this may be the effect you are seeing.

I’m assuming that these files are in the root directory, so the FAT shouldn’t be an issue. Also if it’s reproduceable on a 256MB stick then that’ll be FAT16 which (iirc) has a fixed size root directory. Only FAT32 can have a root directory that is based on the FAT and hence can be extended.

If you’re using FAT32fs then you can use the mount -v option to dump the sectors. If you do it after viewing the ‘missing’ files (ie use mount -v after it’s already mounted) then it should show the whole of the root directory. The first mount will only show as much of the root directory that FAT32fs needed to read to find the volume name.

The standard way round the problem is to only ever write to a FAT format device from either Windows or RISC OS. You should then be able to read from either OS. So one pen for writing with RISC OS and another Pen for Writing with Windows.

Below is a screen dumpout (reduced) showing the two filer windows.

The original larger, version is here.

I think that when you Select click on the USB stick it is being opened by DOSFS and when you Adjust click it is opened by Fat32FS, and they are using different copies of the FAT.

What happens if you RMKill DOSFS before inserting the stick?

In fact, is there any reason to ever use DOSFS rather than Fat32FS?

I think that when you Select click on the USB stick it is being opened by DOSFS and when you Adjust click it is opened by Fat32FS, and they are using different copies of the FAT.

While I don’t know how that screenshot was produced, its point is clearly that the two Filer windows are showing different contents.

In fact, is there any reason to ever use DOSFS rather than Fat32FS?

Demonstrably.

While I don’t know how that screenshot was produced, its point is clearly that the two Filer windows are showing different contents.

The files which go AWOL under Fat32Fs are all typed as DOS. Is there a clue there?

Forgot to say, I examined Dave’s USB stick under Linux and there is only a single FAT partition on there. I couldn’t persuade dosfsck to give me any more useful info :-(

I have found this many times It’s not just RISC OS.

I have Windows Xp and 7, both view the same USB stick differently (32 GiB).

More often than not, RISC OS is the one which shows the stick ‘correctly’ (all files I expect to see there).

Watch out, files can be over-written! (Obviously)

Eject your media people :@) always resolves it here!

@Chris Evans
Nice Try, Chris, but this USB stick has never had any files copied to it from an MS Windows computer.

@Chris Johnson

The files which go AWOL under Fat32Fs are all typed as DOS. Is there a clue there?

I have another USB stick, a 256MB one, for which some of the files typed DOS don’t disappear.

Try again! :-)

I have ported disktype and added a function to return Filecore information in the case of a RISC OS drive.
To use it,
*disktype SCSI::0 or whatever the drive name
It can work with disk image files too I think.

http://homepages.woosh.co.nz/ron.may/disktype.zip

It is only an info display program, but might be helpful.

Is it that one obeys hidden files on FAT and one doesn’t?

If that’s true – if something corrupts the flag of the file so that they become ‘hidden’ one filesystem would ‘see’ them and one would ‘not’.

@Ronald May
I haven’t been able to get your application to work. I get "File ‘disktype’ not found.
On doing a *Run command, the response is “Usage: disktype <device/file>…”. Does it need to be run at Boot time?

@Jeff Doggett
I have outputs, using the mount -v option, for mounting the USB stick under SCSI FS and mounting under FAT32 FS and they are different. After mounting with SCSI (adjust button), it gives information on 7 sectors, the final one starting with the name “NEW VOLUME”; while, after mounting with FAT32 FS (select button), it gives information on 9 sectors with the first 7 being identical to the 7 given WITH SCSI FS.

@David
yes the disktype binary would have to be somewhere the system can find it.
The CSD or if you plan on using it regularly, !Boot.Library
No it doesn’t need to be run at boot time, just has to be where the CLI can find it, or (harder) use the full path, eg *ADFS::4.$.discutils.disktype ADFS::0
disktype may help to show what the format was done with, for example Fat32 could be type &0b or &0c. I updated it so it displays an exFat drive also.
It should show all known disktypes, atari, linux or whatever.

Is it that one obeys hidden files on FAT and one doesn’t

Fat32fs ignores the “hidden” flag.

I have outputs, using the mount -v option,

You can send them to me if you wish. I’m expecting to see a NULL byte corrupting the root directory. Fat32fs obeys the FAT spec and stops searching the directory when it encounters a null byte as the first character of the filename.

@Bryan Hogan

What happens if you RMKill DOSFS before inserting the stick?

Answer: Get errrors when attempting to mount stick with SCSI FS, “Disc not understood – has it been formatted?” To be expected, surely?

@Jeff Doggett
I have just emailled you the two text files of the sector dumps.

I have just emailled you the two text files of the sector dumps.

I have it.
Below is an example of a corrupted entry:

01E0 : 41 70 00 61 00 6E 00 64 00 61 00 0F 00 D8 62 00 Ap.a.n.d.a…Øb.
01F0 : 6F 00 6F 00 74 00 2E 00 7A 00 00 00 69 00 70 00 o.o.t…z…i.p.
Sector 100A (Logical FEA) (Cluster 2 offset 2)
0000 : 20 00 1B 1F 06 00 1B 0F 2F 00 00 20 91 0A 41 88 ……./.. ‘.Aˆ
0010 : 63 43 63 43 02 00 41 88 63 43 8E 22 E6 33 03 00 cCcC..AˆcCŽ"æ3..

This shows that the longfilename is “pandaboot.zip”, but there is no short (8.3) format present, just junk bytes (20 00 1B 1F 06 00 1B 0F 2F 00 00), so FAT32fs has ignored that entry.

Fat32fs has been updated to 1.44

http://sites.google.com/site/jeffreyadoggett/

Fat32fs has been updated to 1.44

Well done, Jeff. And a nice number for a filing system software to reach, if you remember floppies. ;)

@Ronald May
Below is the report from your “disktype” binary which I put in !Boot.Library .

*disktype SCSI::0

Regular file, size 0.988 GiB (1060356096 bytes)
DOS/MBR partition map
Partition 1: 0.988 GiB (1060356096 bytes, 2071008 sectors from 32, bootable)
Type 0×0B (Win95 FAT32)
Windows NTLDR boot loader
FAT32 file system (hints score 4 of 5)
Volume size 0.986 GiB (1058271232 bytes, 258367 clusters of 4 KiB)
*
As I said earlier, it is a 1GB USB stick. It has never been reformatted since I have had it and, as I said earlier, I have not saved files to it from a MS Windows computer.