SD Card Filer

Presumably the BeagleBoard boot SD card must be FAT formatted, though?

Well yes – X-Loader and U-Boot only talk FAT, of course.

file is now dated 2/1/2098, presumably because it was written at some time before NetTime had done its stuff

I think I’d have expected 2054 (2012 + (2012 – 1970)) though.

Just remembered from looking at my BBC micro DOSFS, FAT uses a 7 bit year offset from 1980, whereas Unix uses 1970, so 2098 = 1980 + 128 – (1980 – 1970). One mystery solved.

Here’s how I’m understanding the situation now.

Time and date on FAT media are stored as a set of bitfields containing (near enough) year, month, day, hour, minutes and seconds. Under Windows and MacOS, these are used to store the local time (UTC + timezone + daylight savings) when the file was modified. Under Linux, they are instead used to store the UTC time, so right away there are incompatibilities even outside the RISC OS world.

There seem to be a number of bugs in the RISC OS implementation. DOSFS uses the C library’s localtime() function as part of the conversion process from a RISC OS time (a centisecond count based on UTC) to FAT time. This function is defined by the C standard (and its name suggests) that it outputs a time as expressed in the current local timezone and daylight savings setting. Yet, for many years, our C library has been incorrect and its localtime() function outputted a time as UTC – this is something which, coincidentally, was again fixed by Sprow a couple of weeks ago.

Logically, this wouldn’t be a problem if DOSFS did a direct reverse operation in the other direction – in this case, the C library mktime() function is the reverse of the localtime() function. The only change would be that RISC OS would move to the Windows/Mac way of storing time on FAT media, rather than the Linux way. But there are two reasons why this isn’t working:

• For some reason, DOSFS tries to do the reverse operation longhand rather than by using mktime(), and its longhand conversion is the reverse of the old, non-conformant localtime()
• Even if (as I was about to do) DOSFS was changed to use mktime(), the C library mktime() hasn’t been changed in line with localtime() – another bug

There is another implicit question: do we actually want to change DOSFS to the Windows/Mac way of doing things? We’ve been using the Linux way for years, and there are distinct advantages to UTC – it doesn’t get confused about which files are newer than which others if they are written around the time when the clocks go back each year, and backup utilities on RISC OS (as with Linux) normally compare UTC values to determine if a file has changed because the native file APIs are UTC-based. It would be a relatively simple fix to DOSFS to change it back to storing UTC – just substituting localtime() with gmtime(). (There doesn’t appear to be a C library function to do the opposite of gmtime(), so the longhand code in DOSFS would have to stay!)

Ben: Sorry I opened a can of worms that kept you out of bed! I understand the problem: years ago I had similar problems with a purely Microsoft Windows network with some file timestamps being +/- 1 hour. The cause: after FAT came NTFS – and that (AFAIR) stores times as UTC. So a M$ Windows network with FAT and NTFS → time problems!

Re DOSFS, I think I would vote for making it consistently use UTC. But either way could cause problems, so it is a matter of deciding which is the lesser of the evils. I assume there is no way to distinguish a ‘FAT linux’ from a ‘FAT Windows/Mac’ ?!

Having slept on it, I think going back to the way DOSFS used to work, using UTC, is probably the lesser of the two evils. If we did move to storing local time, it should probably be a conscious decision after evaluation of any side-effects anyway, rather than an accidental result of a bugfix in the C library. (The fact that mktime() is wrong is still an outstanding bug though.)

There isn’t anything in the FAT directory entry to indicate whether the time is a local time rather than UTC. It’s probably the sort of thing we could work around if we added support for extended attributes – one of those attributes could be the timezone in effect when the file was modified, thereby enabling conversion between local time and UTC as required. But that’s a whole other chunk of work!

Wouldn’t the best option be a mount option? With the default being set in configure, the current status quo being the supplied default.

I think the phrase “mount option” belies a familiarity with the way Unix filesystems work. That’s quite a different model from how RISC OS works – the equivalent of a Unix mount operation is something that happens implicitly whenever you reference a pathname containing a drive specifier, if the media in that drive has changed (or may have changed, for certain types of drive where media change can’t be detected, and this is triggered based on a timeout). As a result, there’s nowhere for an option to be added – no commands get issued, and there’s no equivalent of /etc/fstab. We do have a *Mount command, but what that does is to set the currently selected directory and library to the canonicalised versions of the root and library of the disc in the specified drive, which is something else entirely.

Having said that, it would be relatively easy to have a global option that would apply to all DOS discs. It’s the sort of thing that would probably warrant a bit of CMOS, since the boot disc might be DOS formatted and you probably don’t want the timestamps of all the files on the boot disc to change part-way through the boot sequence. We certainly wouldn’t want to make such a change until mktime() is fixed though.

The nearest RISC OS gets to a ‘mount option’ is the filing system special field – ie adfs#special::DiscName.$ (I may have the syntax wrong there). Various filesystems like SerialDeviceDriver use it for that purposes, but it’s pretty ugly and not something I’d want to inflict on users.

I’d caution against making it configurable, because it would potentially cause more confusion: the same files would appear differently on two different machines, or on the same machine with the setting different. That’s far more likely to happen with a differing config setting than someone having two machines in different timezones (though the latter can easily happen, especially if DST setting isn’t automatic). So I’d recommend picking one and sticking to it.

the same files would appear differently on two different machines

But isn’t that what happens already? (Just not 2 different RISC OS machines).

I think it would be bad if the setting were able to be globally set to anything other than UTC. The main reasons being, that is how it is done at present and also there is a big possibility of machines being dual booted with Linux with a common DOS partition, (and obviously very little with windows).

When I was thinking of mounting drives, I was thinking in terms of clicking on the icon, (perhaps adjust clicking would allow selection of UTC, local time or user defined offset.)

However if such a facility were desirable, would the offset be needed to be set on mount? Could there just be a menu option on the icon for different timezone? (And a corresponding command line option, which could be put in a dummy app on disks that need a non utc timezone).

Also if it were to be “fixed” properly, as per windows, would it need to take into account the DST shifts for different regions?

I notice Apple seem quite happy to dumb down their filing systems to fit in with windows, while we are talking about DOSFS, could it be made to show apple ._ files as resource forks and ignore their extension?

If a file’s date/time is interpreted as local time, then there is an implicit assumption that the time zone that was used to stamp the file is the same as that of the machine on which it is being read. That asusmption won’t always be true.

It still doesn’t help in that local time doesn’t give enough information to know whether daylight savings were in force – unless you make another assumption that local time includes daylight savings.

Ben: with today’s ROM, files are saved to SD card and FAT USB stick with the correct timestamp. Before NetTime has worked, they get 00:00:00 01 Jan 1980. So it all seems much better! (at least within RO!).

Do you want me to do a more controlled change of the SD card label to see if I can (a) provoke the SD trashing and (b) get some more diagnostics? I now know how to get a dump of the card afterwards.

I will now read up on Sprows changes to BST etc, as I do not understand what is happening there, and I need to because Organizer can set/unset the flag (rather like Alarm) and I am not sure if it will have to be changed.

If you can investigate what’s going on with renaming DOS discs, that would be great. Do check if the same thing is reproducible with DOS discs on SCSIFS (for example with an USB SD card reader), because that would point at a(nother) bug in DOSFS. Touch wood, so far none of the reported bugs have actually been anything to do with the new modules SDFS or SDIODriver, which is promising (anybody had a card not work at all yet, for example?)

For what it is worth, I wrote a program to scan a folder and report on which files have changed. There is a very clear difference between W98/FAT (where it worked) and XP/NTFS where each timezone change would report all files as having changed. It appeared at the time that NTFS was holding UTC, which XP was then dumbly correcting to local timezone…

From the horse’s mouth: “The FAT file system stores time values based on the local time of the computer. For example, a file that is saved at 3:00pm PST in Washington is seen as 6:00pm EST in New York on an NTFS volume, but it is seen as 3:00pm EST in New York on a FAT volume.”

Source: http://tinyurl.com/74yw6yr [is actually http://msdn.microsoft.com/en-us/library/windows/desktop/ms724290%28v=vs.85%29.aspx but the forum breaks the URL]

It looks, therefore, that FAT is just expected to screw up its times. If there is any decision here, I think to revert to UTC and apply our own timezone corrections on top would perhaps be the most appropriate as we can at least (within the RISC OS world) make it behave in a consistent and predictable manner across all platforms. Ideal? No. Workable? About as much as FAT itself is…

note: copy-paste as forum breaks URL

The "IBM":http://www.ibm.com format seems to work, to give File Times.

I will now read up on Sprows changes to BST etc, as I do not understand what is happening there, and I need to because Organizer can set/unset the flag (rather like Alarm) and I am not sure if it will have to be changed.

In general, applications shouldn’t be writing to CMOS bits via OS_Byte that they don’t own (by ‘own’ here, I’m probably thinking ‘he who prints out the *STATUS value’). I have updated, but not yet checked in, Alarm to remove its manual DST stuff – since that’s now either handled automatically by Territory manager or can be set with !TimeSetup configure plugin (erm, also not checked in yet!).

You are of course free to use *CONFIGURE from your application. If your application needs to find out if automatic DST is in use, there’s a new Territory_DaylightSaving SWI to detect this (as a more robust check than *RMENSURE).

Touch wood, so far none of the reported bugs have actually been anything to do with the new modules SDFS or SDIODriver, which is promising (anybody had a card not work at all yet, for example?)

Seems to be working OK for me so far. Nice work!

anybody had a card not work at all yet bq.

Card seems to be working in general. However, having copied !Boot to the SD card alongside Mlo and U-boot etc, I now find that it is very difficult to delete it. Setting access to unprotected seems to make little difference. It does seem that individual files can be deleted, but not directories, or anything in directories, unless each individual file in those directories is deleted one at a time. Individual empty directories can be deleted, but access has to be set to unprotected immediately before deletion. The chances are theat I’m doing something wrong, but does anyone have any thoughts?

I wonder if this is another one of those bugs due to the odd way FilerAction uses R4 in calls to OS_GBPB? Can you delete whole directory structures from the command line (e.g. *wipe mydir ~cfr~v)?

(anybody had a card not work at all yet, for example?)

Yes on the Pi a Maplin ‘CnMemory’ 8Gbyte SD card just gives Disc error 08 at .. C00

Writes on the Pi to an SD card are s…..l….o….w (very)! (This is not a criticism just for the record, reads seem quite fast.)

Writes on the Pi to an SD card are s…..l….o….w (very)!

Hmm, the first time I wrote to the SD card (A 4GB SanDisk Ultra) it was nice and fast. Second (and third) attempts it started to write and then everything ground to a halt (Mouse still moved but nothing else).

Saying that I think I’ve also seen this when writing to the USB HDD, so not sure if this is an ‘SDFS’ related issue or just down to the pre-alpha nature of the RPi ROM.

Oh well, once I’ve gotten this ROUGOL demo thingie out of the way I might find time to do some debugging!

it started to write and then everything ground to a halt

The write eventually failed with a ‘command timeout’ error… If that means anything to anyone :)

On the Pandora does not work :-(

Yes, there is no support for the Pandora, Touchbook or OMAP3EVM at the moment. If the board detection is working correctly, you should not see any SDFS drives for those devices. I missed them out because it was hard to find the schematics, and I don’t have access to them to test the software (except for the Touchbook). I might try to add support when I’m not so busy, if nobody else has done it first.