Please make *CheckMap do something USEFUL!

So I wrote a little program to try to extract the CVS files and strip off the rubbish – essentially to take files unpacked by !UnTarBZ2 to there and write the files here – with the idea of “there” being a temporary place and “here” being the place where the sources go.

It was progressing well, but the file copy can be a bit slow for larger files, so I thought I’d drop in some Hourglass action so you know the thing hasn’t, you know, crashed. Press F3 in Zap to save the code when…

Now, while I have never experienced this on the Pi, I recall from days of old that once you finish swearing, you do not switch off.

I used *CheckMap which went through the filesystem and then told me that the maps were inconsistent. I wanted to punch whoever wrote that bit of code – why am I wasting time discovering what I already knew?

DiscKnight (the check-only version) told me the cross-check for map #1 was wrong, but map #2 was good.

If this can be easily determined, why can’t *CheckMap fix this? This wouldn’t be a problem re. DiscKnight, for there is a lot more stuff DiscKnight can fix. *CheckMap should, at the very least, try to recover the situation to a state where you can at least mount the filesystem for data recovery.

I was able to recover everything¹ (that I checked) by the tortuous process of:

• DO NOT SWITCH OFF OR REBOOT!!!!
• Found myself a 2GB SD card (actually, it is a uSD)
• Downloaded the RC11 image from RPi.org and imaged it to the second card
• On the Pi, *RMKill FAT32FS, as it doesn’t seem to want to allow access to the FileCore partition
• Put the second card into a USB reader and plugged it in to the Pi
• With access still available to the filesystem, I copied all of the files from the original SD card on top of the ones on the second card.
• …with special attention for the Pi’s firmware (you can’t just copy !Boot.Loader, you need to copy the stuff within)
• For the important stuff, run a Count on old and new and make sure the file count and byte size match.
• Hold your breath while you switch off the Pi and swap cards around, then boot from the second card.

It took a while, but it is working. Everything appears to be present.

To make things a little simpler in the future, I am currently making an image of the working card, so I’ll have a snapshot of it as it is today.

As for the old card? Useless. RISC OS will boot, but attempting to access the filesystem via SDFS will tell me that the disc drive is empty. Attempting to open the filesystem via SCSI hangs the machine with an hourglass (hang as in ^Break won’t even work). At this point, it would be safe to consider – without access to external products – the data on this SD card would be “lost”. DiscKnight might be able to fix it.

Really, at the very least, *CheckMap should attempt to patch simple errors so that the filesystem can, at least, be mounted. There will still be a call for third party utilities (there was more wrong than just the map corrupt); the thing is, I think it is a bit ridiculous for a filing system to trash its own filesysytem irrecoverably!
[what might also need looking at – why does SDFS report “Drive empty”? this implies DiscKnight may be unable to provide further repairs, if the OS itself doesn’t seem to realise that there is something there?]

By the way – while it may be that there was an existing problem with the disc structure (files not in directory or the like), the machine worked and booted and was happy to save files to the disc. The problem occurred between pressing F3 to save a BASIC file in !Zap, typing three Hourglass SWIs, and pressing F3 again. A taskwindow was open, doing nothing, and several filer windows were open. No file accesses other than that described, and no other applications were running.
Bizarre.

¹ The sources and stuff was backed up, the problem is the major amounts of hassle involved in recreating/reinstalling everything.

Which is why my answer to Steve P re ‘how many SD interfaces do you need’ a week or so back was ‘2’.

RISC OS still FUBARs it’s filesystem with alarming frequency.

Keeping a regular mirror of your SD card is really the only safe way forward. Also helps if swapping your ROM image fails (as it did for me this morning).

In answer to your question: is this something Druck or Sprow (the only people I know with good enough knowledge of the filesystem) could be persuaded to do; or an extension of the filesystem bounty? I think ‘is it useful’ is a no-brainer – ‘can someone do it?’ Is a much more challenging question!

Looking (briefly) at the source, it looks like CheckMap won’t offer to fix the map if it is inconsistent with the directory tree – though one might wonder which copy of the map is being looked at, at this time? Certainly, I was not asked if I wanted to fix the map…?

I used *CheckMap which went through the filesystem and then told me that the maps were inconsistent. I wanted to punch whoever wrote that bit of code – why am I wasting time discovering what I already knew?

Checkmap will already try to recover the map (see *HELP CheckMap) if it can, but it will only do so if it’s confident that wont make things worse. What it does is

1. Try to load the 1st copy of the map, checking checksums & cross checks
2. If that fails, try loading the 2nd copy of the map, checking checksums & cross checks
3. If neither copy is good it errors before even looking at the files
4. Then, try to “delete” each file on the drive so that you should end up with an empty map (bar the root directory and the map itself)
5. Check all you have left is the root directory and the map itself
6. If the map is consistent, but the 2nd copy was loaded in step 2, offer to copy the 2nd copy over the 1st

In your situation it has loaded the map off the disc, found it to have good checksums, but found it doesn’t match the directory layout. There’s nothing to be gained by copying the 2nd copy of the map over, since the 1st copy is has good checksums already.

DiscKnight is a tenner and well worth having in your armoury.

I wrote a little program to try to extract the CVS files and strip off the rubbish

Is that not what a CVS client is for? That wheel was already invented.

There’s nothing to be gained by copying the 2nd copy of the map over, since the 1st copy is has good checksums already.

That’s not what the check-only version of DiscKnight says. Copy #1 cross check was wrong.

I believe there is something to be gained – namely that the device can be mounted and ‘seen’ by RISC OS. Yes there is a problem, but wouldn’t it be better to allow the user to gain access?

well worth having in your armoury.

I would agree – if only the SDFS filer didn’t report “Drive empty”, making things a little more complicated. It’s moot now, anyway, I imaged the older SD card from the newer one; so don’t have the damaged filesystem any more.

Is that not what a CVS client is for?

Yup.

That wheel was already invented.

Only partly. This Pi is not, ordinarily, connected to the Internet. That’ll change if/when RISC OS can talk to WiFi devices. But until then it is a sort of messy arrangement using a way-too-short patch cable and Windows’ connection sharing. As you can imagine, I don’t do this as a matter of course.

Sprow: I may have miss-understood things but if Checkmap finds Map1 is good, map2 is not checked and could be bad. If so couldn’t CheckMap if it finds Map1 good but Map2 bad, copy Map1 over Map2?
In fact I’m wondering when in normal use is map2 read or written to. Surely every change to map1 is replicated to map2?

if Checkmap finds Map1 is good, map2 is not checked and could be bad.

Correct.

If so couldn’t CheckMap if it finds Map1 good but Map2 bad, copy Map1 over Map2?

If it finds Map1 to be good it wont have even looked at Map2. It only allocates enough RAM to hold one copy of the map, and stops as soon as possible whether a good or bad outcome.

In fact I’m wondering when in normal use is map2 read or written to.

In normal use the map in RAM is read/written to, the copies on disc aren’t touched. On dismount the RAM copy is written twice to disc.

I just had Andrew complaining to me on Skype this morning that CheckMap really should have an option to not output every file it finds (that’s not actually broken) because that just massively increases its runtime. :)

If it finds Map1 to be good it wont have even looked at Map2. It only allocates enough RAM to hold one copy of the map, and stops as soon as possible whether a good or bad outcome.

That seems absolute madness, it would probably add less than 1% to the time of a checkdisc and make the system much more robust. I’d have thought it worth comparing maps for incongruity every time a disc is mounted. I’d go as far as to call it a bug.

In normal use the map in RAM is read/written to, the copies on disc aren’t touched. On dismount the RAM copy is written twice to disc.

That doesn’t sound like it could be correct unless I’m still missundestanding things. I just created a directory and copied a small text file into it and the root directory on an RO3.71 A7000+ and then turned off at the wall, on rebooting everything was there!

I just had Andrew complaining to me on Skype this morning that CheckMap really should have an option to not output every file it finds (that’s not actually broken) because that just massively increases its runtime. :)

Sounds a great idea, but it needs to put something on the screen every few seconds, say as DiscKnight does.

Funny Chris writes about this. I was thinking about this at work today.

If it finds Map1 to be good it wont have even looked at Map2. It only allocates enough RAM to hold one copy of the map, and stops as soon as possible whether a good or bad outcome.

In which case, I think if it is true, it implies that *CheckMap is fundamentally broken. But, then, a filesystem that can’t provide basic recovery could be said to be broken – even Microsoft provided ScanDisk right back to the DOS days… and Unixen come with fsck…

In normal use the map in RAM is read/written to, the copies on disc aren’t touched. On dismount the RAM copy is written twice to disc.

Can I call you on that? The only system I have seen where that would make any sort of sense is the SJ MDFS that built its map from the disc structure at power-up (and take a fair while doing so).
Any other time, any other system, that does not sync the media structure and layout with what is actually there, is fundamentally broken. It would be insanity to keep maps in memory and only write on dismount, any power cut or idiot user could easily trash the media in that case – and remember, FileCore filesystems do not appear to be robust enough to recover from these sorts of situations.

I’d have thought it worth comparing maps for incongruity every time a disc is mounted.

Weren’t we talking about *CheckMap ?

I think RISC OS does, as a matter of course, look at both maps. Nothing else would explain the message at the top of this thread…

Sounds a great idea, but it needs to put something on the screen every few seconds, say as DiscKnight does.

You could perhaps mimic the look and feel of the old CheckMap without spitting out thousands of files… just print directories.