Magic at ROUGOL and SASAUG

as I said earlier, I have not saved files to it from a MS Windows computer.

How about Linux?

http://www.osnews.com/story/21766/Linux_Kernel_Patch_Works_Around_Microsoft_s_FAT_Patents

To quote here:

In a Q&A about the patch, more details were revealed about how it avoids the patents:
The claims of both of the VFAT patents involve the creation (or storing) of both a long filename and a short filename for a file. The 2nd patch only creates/stores either a short filename or a long filename for a file, but never both. The 11 bytes created by vfat_build_dummy_83_buffer() to pad the field for short filenames cannot be used to access the file, and contain bytes which are invalid in FAT and VFAT filenames, and therefore are not filenames as that term is and has been used in the technical community.
So, are there any downsides to this patch? Well, only if you are unlucky enough to still be using Windows 9x-based or MS-DOS operating systems. Tridgell believes this scenario is too rare to worry about, and there’s always the msdos file system, which forces 8.3 and doesn’t use long file names altogether.

So either you have used Linux or something derived from Linux, or a component used within the RISC OS world uses more recent Linux-derived code?

At any rate, it looks like thanks to some American patent bo*****s, we possibly shouldn’t rely on short file names to mean anything at all if a valid long file name exists; and as such, that corrupted entry may in fact be entirely intentional. :-/

Yup.

Just looked at the kernel patch – that is exactly what this is.

To quote Jeff:

but there is no short (8.3) format present, just junk bytes (20 00 1B 1F 06 00 1B 0F 2F 00 00),

And to quote the kernel patch:

/* we start with a space followed by nul as spaces at the
* start of an entry are trimmed in FAT, which means that
* starting the 11 bytes with 0×20 0×00 gives us a value which
* cannot be used to access the file. It also means that the
* value as seen from all Windows and Linux APIs is a constant
*/

Additionally, the ‘extension’ is a slash and two nulls (check!) and the rest is random bytes.

This is a file with a Linux broken&patent-friendly long filename.

@Jeff Doggett
Looking at the sector dump, what you have said about the disappearing pandaboot.zip also holds for the other 4 disappearing files: No short (8.3) filename format. So, you are telling us, that all 5 filename entries are corrupted, but how and why? I ask because none of the directories disappear and the 5 disappearing files, and only those, have all been saved to the USB stick on the same computer (all 5 downloaded from the internet on that computer) running a flavour of Linux. Also, you may have observed that the disappearing file OMAP4Dev/5/21/zip has a twin, actually a copy, with filename OMAP4Dev/5/21xx/zip which doesn’t disappear. I have repeated this with other disappearing files: copy the file with a slightly different filename and the copy doesn’t disappear.
I was hoping to charge a fee for performing my experiment at various RISC OS groups throughout the land, but I fear that I have now given too much away. :-(

@Rick Murray
Gosh, you’re quick! You replied with the 2 previous posts while I was replying to Jeff. So, putting together Jeff’s post and yours, the corruption of filenames is no accident. As I said, while you were writing your posts, the 5 disappearing files were all downloaded and saved on a Linux computer (Android) and then copied to the USB stick for insertion into a RISC OS computer.
There seems to be quite a lot of nulls inserted into filenames, if I am reading it correctly. For example, the following.

I don’t know how this will be interpreted by Textile, but the character looking a bit like an x preceding the second e has hex number &84. The two dots following the forward slash are two null characters (&00).

The hex numbers for the above would be better perhaps.

Bye! Bedtime!

There seems to be quite a lot of nulls inserted into filenames, if I am reading it correctly.

Try looking at a Kanji filename, you’ll see a lot less null bytes. Long filenames are 16 bit Unicode, and since the ones you are looking at are plain alphabet letters, the second byte serves no purpose…

@David
That’s odd compared to a range of Fat32 sticks that I have. They report
Regular file, size 640 KiB (655360 bytes)
I dont know what that means exactly though.
I think it is more space efficient to format to Fat16 on drives smaller than 2GB, I wonder if it is a poor format job that has been cloned onto the stick. It obviously originates from Windows, you wont see NTLDR boot loader if you reformat.
One Emtec 8GB stick I have came with no partition map, and the stick appeared in Linux as /dev/sdb instead of the more normal /dev/sdb1.
I used it like that for quite a while then it started with ‘not writeable’ errors and such, so I reformatted it, and it worked OK again.

@Ronald
Thanks for your application and interest. The USB stick was given to me by my employer a few years back as a blank formatted 1GB stick.

I have made a file reappear under Fat32FS by renaming it. Can anyone explain that?

I am still experimenting. It doesn’t seem to be simply that objects with long filenames disappear as suggested by the explanations of Rick and Jeff, because some of the objects with longfilenames do not disappear (see the screen shot in an earlier post).

In the sector dumps I notice that the initial letter of some filenames has been replaced by the character with number &E5. These files do not appear under Fat32fs or SCSI FS. Is that how a file gets ‘deleted’?

In the sector dumps I notice that the initial letter of some filenames has been replaced by the character with number &E5. These files do not appear under Fat32fs or SCSI FS. Is that how a file gets ‘deleted’?

Exactly. If there are no directory entries beyond that one, the first character is replaced with 0, else it’s replaced by &E5.

An initial 0 means “don’t bother looking beyond here”.

I have made a file reappear under Fat32FS by renaming it. Can anyone explain that?

Possibly because that rename generates a valid short file name.

because some of the objects with longfilenames do not disappear (see the screen shot in an earlier post).

Because they have valid short file names as well…

Let me explain – it is supposed to work like this: every file has a short 8.3 name. A special directory table holds the associated long file name entries. These were optional and you could always use the 8.3 name.

Thanks to the patent, the patent applies to using long file names in association with short names. Therefore the patent workaround is to generate long file names without any associated 8.3 name. The short name will be junk.

What we are seeing here is the result of that. It has implications, and reminds me I can never use Linux to save files I want played by mplayer under Windows (can’t handle kanji, needs the 8.3 name).

In the sector dumps I notice that the initial letter of some filenames has been replaced by the character with number &E5. These files do not appear under Fat32fs or SCSI FS. Is that how a file gets ‘deleted’?

I think so. I recall the old DOS undelete used to ask for the first character of the name of restored files.

A special directory table holds the associated long file name entries.

There’s no special table; long names are held in a contiguous series of entries in the normal directory table.

@Dave Higton, Rick, Ronald and Jeff
Thanks for this information and your help. I have done a *disktype on several USB sticks and the problematic ones, for which I can perform my amazing disappearing trick, have been the only two with the line “Windows NTLDR boot loader” in the output. For one USB stick the same line has “Windows / MS-DOS boot loader” instead, but I can’t do my disappearing trick with it. The latter is a Type 0×06 (FAT16) whereas the other two are Type 0×0B (Win95 FAT32).

For one 4GB stick the report is:

*disKtype SCSI::0

Regular file, size off the scale (18446744073436921344 bytes)
FAT32 file system (hints score 4 of 5)
Volume size 3.738 GiB (4013944832 bytes, 979967 clusters of 4 KiB)

*
I like the “size off the scale” bit. What can it mean?
The FAT32 sticks only score 4 of 5 hints as against the FAT16 sticks that score 5 of 5.
Bye. Bedtime.

I dont know for sure what the regular file size is about, it may be for showing the size of disk image files primarily.
Here is a link to disktype online documentation.
http://disktype.sourceforge.net/doc/

I have read about the progression of Fat filesystems before and it is enough to make your head spin. At least when working with Jeff’s formatter there is some hope for some support, or to establish some groundrules.
Chris Evans noted that writing to a stick from one OS only guarantees keeping it intact. However this may be only a problem with Windows, and I haven’t had any trouble using sticks to play/record media files on my satellite box, or samsung bluray player, RISC OS, and Linux.
Windows has always matched the criteria to be called a virus, but that’s just my opinion. (-: