Help - lost data

A little disappointed to see aspersions cast over the ARMX6 service packs, there. I have checked a recent one (12 in this case), and HForm 2.75 was most definitely included. That has been available since April 2018 I believe.

With the disc wiped, I can’t check, but I do have RISC OS 5.25 Aug 2018 on SDFS::1 and SDFS::1.$.Utilities.Caution.!HForm.!RunImage is dated 2016.

Perhaps I failed to apply the service pack correctly and it’s my fault. Perhaps I’ll know once I have restored the data.

Edit:I applied service pack 8 but not 12 for reasons I cannot explain so it is, indeed, mea culpa.

Only one post (from Sprow) has offered help – I did say ‘keep shape’ during the format and I am particularly keen to recover $.Internet.!NewsDir.MessPro.* which (hopefully) should be far enough down the sub directory list that I can recover the sub directory content record in full as the records should be a long way away from the map that was initialised. Hence any directory entry of !NewsDir or MessPro should point to all its sub directories.

From past experience, I’d say you need to speak to Druck, he’s about he only one who can talk you through using DiscKnight to recover your data. The important thing is not to write anything to the disc until that point.

The important thing is not to write anything to the disc until that point.

Yes. The disc has been removed and stored in a plastic bag. A new SATA disc has been (is being) created using my latest backup (March 2015) plus e-mail (March 2018) and I’ll see how that looks tomorrow…

I’d say you need to speak to Druck,

Yes. His mobile number gives number unobtainable. The FlexTel ‘personal’ 07xx number that follows you wherever you go – that didn’t work then.

Chris – as I said on the phone, I have sent him two emails on your behalf.

In the meantime, have you tried ticking the box to recover lost files/folders and specified the locations you want to recover (eg. $ or $.NewsDir etc)?
Also tick “recover deleted files” (try both ways).

Essentially you need to make it do a bit more than just check the disc, which it should do if those options are set.

I have had confirmation today that Druck as helped other people recover from Format problems (as I said on phone) but sadly no-one can quite remember how.

The one thing not to do (as I said on the phone) is write anything to the formatted disc, as that’s asking for bad things to happen.

Chris – as I said on the phone, I have sent him two emails on your behalf.

And I am very grateful for that. I don’t think DiscKnight (without magic words from druck) will let me recover things on a disc with a valid map. I don’t intend to try anything until I have taken a copy of the disc image and created a second, identical disc. I do have another, identical SanDisk SATA 128G solid state drive and my Titanium has a fixed SSD plus a plug in (with power off) slot which takes a second SATA drive as ADFS::5 so I can examine a removable drive easily (and do backups easily). Then I can safely try things…

I have now told Messenger Pro to receive e-mails (on top of my restored March 2018 e-mail database) in such a way that they are not deleted from the server so that, if and when I recover the e-mail up to date I can simply resume normal working and it will receive everything from the server. The e-mail database is about 8Gbytes in 508 files, most of the files being quite large. Now all I have to do is to remember my freeuk password…

If all you want to do is upgrade HForm you should grab version 2.75 as mentioned by Sprow.

Whilst this runs a lot better, it is unfortunately crunched so I will have to download the proper source and interpret the ‘#’ lines from the crunched version. Fortunately include Stdio does nothing.

I have been away all day Saturday and Sunday so I haven’t expected any contact yet but am hoping to get my e-mail working later today and thus offer a more convenient medium for any contact.

Is there any way I can tell Messenger Pro to receive e-mails (on top of my restored March 2018 e-mail database) in such a way that they are not deleted from the server so that, if and when I recover the e-mail up to date I can simply resume normal working and it will receive everything on the server please?

You configure Hermes (if that’s what you’re using) to fetch and “leave on server”, “don’t delete from server”, or whatever the wording it uses. It does depend on your mail provider supporting this part of POP, but I’ve never found one that doesn’t.

Whilst this runs a lot better, it is unfortunately crunched so I will have to download the proper source and interpret the ‘#’ lines from the crunched version.

Can’t you just build through the Build application? If you’re planning to offer the changes back to ROOL, working on the unmodified text source would make that much easier (as you just send them the file, and Git does the rest).

Fortunately include Stdio does nothing.

I’m not sure what that’s referring to, but I’m fairly sure that

REM Gratuitous use of an include file
#include "Global/FSNumbers.h"

does do something. The code has a lot of references to FS Number constants which aren’t defined in the BASIC file, as far as I can see. My guess would be that “Global/FSNumbers.h” #defines a lot of values, for use by both C and BASIC sources.

My guess would be that “Global/FSNumbers.h” #defines a lot of values

HForm only uses 3 of the values:

fsnumber_adfs = 8
fsnumber_SCSIFS = 26
fsnumber_SDFS = 192

I congratulate whoever designed github in making it difficult.

Why not say:

# include filename
fs_adfs%=fsnumber_adfs

then at least the code would then say

fs_adfs%=8 (albeit that it would then be crunched)
and references would be a variable

Many thanks to all those that have helped. Sprow was correct in that it is impossible to recover my e-mail data in a useful form as would have been possible in earlier disc formats which included absolute disc addresses within each directory of file names. Now only a pointer into the disc map is included which is meaningless without the disc map. HForm carefully does not store a copy before wiping it out making sure that any forseeable error on the part of the user is completely fatal.

Messenger Pro stores e-mail in encrypted form so it is not easy to work out (from the raw data) where each file starts and ends and where it sits in the heirarchy (Spool.0.0.0 etc.). Hermes keeps a text version of incoming mail in plain text but expires and deletes it after a month. So I may find something in the 1893 files each of which is 64Mbytes if I go through them looking for plain text. Wibble.

Looking to the future I think that three separate HForm programmes, one each for SDFS, SCSI and ADFS, would be sensible. It would then be easy for the user to banish the one that would write onto the hard disc, putting it in a directory that contains something perhaps a little stronger than ‘Caution’. It would also be sensible to bar any format of the drive on which the HForm software is residing or on which the Boot: path is pointing to without something better than ‘Are you sure?’, which appears even when the disc being formatted is clearly blank and so is expected by the user. The print is also rather small because it is a command line programme so it discriminates against those whose sight is not perfect. Perhaps there should be legislation about this? Oops, some frustration seems to have showed through, sorry.

Having said that, the updated HForm is much better at warning you not to format a drive that contains RISC OS data as it makes it clear what the disc name is etc.

Sprow was correct in that it is impossible to recover my e-mail data in a useful form as would have been possible in earlier disc formats which included absolute disc addresses within each directory of file names. Now only a pointer into the disc map is included which is meaningless without the disc map. HForm carefully does not store a copy before wiping it out making sure that any forseeable error on the part of the user is completely fatal.

I suspect/hope keeping a copy of an old map (and the one previous to that, and the one…) wouldn’t be a massive amount of work unfortunately it doesn’t seem to form part of the “Filing system improvements (step 2)” bounty. I wonder if the bounty claimant can be persuaded to add it! An extra sub bounty?
It would have the knock effect of needing an option to NOT keep the previous copy to enable for more secure data wiping. Are any of the data wiping utilities RO5 compatible? and will they need updating for when partitions are supported?

Can someone help me, please, to recover my data?

The most important advice was already given: do not try to write anything to that drive.

The second most important advice is: do a sector-level copy from e.g. Windows or Linux. And make multiple backups.

Then you have various options, including mounting that sector-level copy as a .hdf in VirtualRPC or RPCEmu and let DiscKnight run in recovery mode.

Another option is to look into filecore-understanding tools, e.g. Gerald Holdsworth’s “DiscImage” project written in Pascal/Delphi. I have a work-in-progress tool called FilecoreImageReader which does basically the same, but in Java. If it helps, I can give you the source code. But if the central map is lost, you need a good understanding of Filecore format (I knew a bit when writing the code, but seemingly have forgotten most of the details already).

In ancient times – long before DiscKnight became available – I was able to recover a lot of data from a filecore disc by just inspecting all 512 byte blocks for known text patterns. This allowed to extract known data types like archives and text and BASIC. Unless the disc is very full, Filecore is very successful in keeping things defragmented, but it is still a lot of manual work to recover all the stuff intact.

Hope that helps.

It does help but the crucial files that I need to recover are my e-mail – these are compressed binary files totalling about 8Gbytes in 500 or so files. I would be able to find their directory entries with their filenames (usually a single or double digit) but will not be able to interpret their position on disc as the ‘where do I live’ entries are an offset into the free space map giving start and finish disc addresses and not themselves absolute disc addresses (as they were in earlier formats such as the 77-file version). What DiscKnight has been able to do for me is to extract each bit of free space on the disc into separate 64Mbyte files (1893 of them, starting at File00000003, CB, 196, 261, 32C up to File00005DC4D) on another drive.

I think the e-mail data are thus lost. Pining for the fjords. Late. This is a dead parrot.

offset into the free space map giving start and finish disc addresses

Eh… That’s complicated!

There was a time when FAT drives could be “unformatted” if you screwed up. And files could be undeleted. It’s a shame we have generally gone backwards since then. My phone, my photos, one moment of inattention and all gone forever. Very lucky I only lost a couple of weeks in real terms, thanks to a recent backup.
But, you know, people make mistakes. There ought to be undo options (so long as data isn’t written in the meantime).

for more secure data wiping

Yeah, I rather imagine somebody worried about securely wiping data… likely wouldn’t be using an insecure OS in the first place.

I think the e-mail data are thus lost.

Unless the compression prefixes an obvious header that you can search the data for. But if it’s just a wodge of data with just a size word prefixed, you’ve nothing definitive to scan for.

It is a long time ago when I last used Messenger Pro, and I always used it in conjunction with Newsbase, but I remember that the “compressed” option was just using Squash, which is simple LZW compression. But I am sure Andrew is able to supply definitive information.

If they are Squashed, they should have ‘SQSH’ in the first 4 bytes, followed by words holding length, load, exec & zero.

That’s only if they are Squash files. It’s perfectly possible to use the Squash module with different data layouts. One of my old (circa Y2K) programs loaded its resources from a file that consisted of a central directory indicating where things were, followed by raw squash data one wodge after another.

In the case of an email program, you only need one single word – the uncompressed length. If you clip messages to a max length of 16MiB, then this will mean the first byte in a compressed file will be 0, the MSB of the uncompressed length. If the first byte is ASCII, not zero, then it can be considered uncompressed.

So, yeah. One word plus raw data is all that’s actually necessary.

these are compressed binary files totalling about 8Gbytes in 500 or so files.

That all being said – 8GB in 500 files equals “about 16MB per file”, so maybe there’s some sort of index table or the like at the beginning?

Many thanks to all those that have helped. Sprow was correct in that it is impossible to recover my e-mail data in a useful form as would have been possible in earlier disc formats which included absolute disc addresses within each directory of file names. Now only a pointer into the disc map is included which is meaningless without the disc map.

D format was the last HDD format to use old map, where files are held contiguously and the directory entry is just a disc address. E/F format (and E+/F+) support fragmented files in order to not have to *Compact ever.

E/F format are described in the PRMs as “new map”. More recent FileCore versions build on that (bigger maps, more zones, longer fragments) but fundamentally it’s all similar. You could be lucky if the zone never needed compacting then the file will still be contiguous, but there’s still the issue of knowing where the start is if there’s no easily spottable header mark.

If they are Squashed, they should have ‘SQSH’ in the first 4 bytes, followed by words holding length, load, exec & zero.

Only if created by the desktop application, IIRC. The Squash application added the header, not the underlying Squash module, so things which called the SWIs direct could just use the compressed data stream.

Individual lumps of mpro data begin MSGE followed by a bunch of binary/compressed data. There is also a short “IndexV2” file which is also unintelligable. I believe this acts as an index for quick listing of messages, and pointers to data in the bigger files. Each folder contains these files plus a “Group” file which contains the name of the group eg. “Email.bob” or “comp.sys.acorn.announce”.

In all honesty, you’d be better looking for the Backup folder which contains at least 30 days of email. These files will begin #! rmail and are much easier to extract because they are human readable.

These are kept in this form for just this reason – it allows recovery of recent data in case of disaster – the logic being that recent emails are most likely to be in need of reply.

Address books can also be recovered because they are stored in plain text.

Sadly the compression is still needed (although could be optional, I guess), as compressed my email approaches 10GB in size. Emails have also ballooned in size thanks to stupid mail servers that append reams of analytics to each email. In the old days, a plain-text email was typically 1-4 KB roughly. The smallest email I receive these days is 50-60 KB per message. Thankyou MS/Google etc. I will give some thought to how to make it an option, but don’t hold your breath – it’s a fairly core part of the database system.

For what it is worth, the tutorial for SafeStore specifically describes how to automate backup of your email with specific reference to NewsDir, as we know how important email is to people. I do try to be as responsible as possible when encouraging good practise.

There is also a short “IndexV2” file which is also unintelligable. I believe this acts as an index for quick listing of messages, and pointers to data in the bigger files.

Does that mean that like Marcel’s ‘overview’ file if the “IndexV2” file is missing it is automatically recreated?
Or would Chris need to find the matching IndexV2 file.

Chris E, there is a “repair” feature in Mpro, and also a “Re-index” which may help in these situations. However, I don’t have time to go over all that code to figure out exactly what and how – it’s a lot of rather complex code, written by others over a period of years. The source to Mpro is available to anyone with a serious interest, albeit existing owners preferred!

as compressed my email approaches 10GB in size.

Jeez. I think it bad enough that my email app for all my mailboxes takes a gig and a half (and knowing Android, that’ll be uncompressed and with loads of wastage).

Emails have also ballooned in size thanks to stupid mail servers that append reams of analytics to each email.

Some of that is for trying to detect and kill spam.

In the old days, a plain-text email was typically 1-4 KB roughly. The smallest email I receive these days is 50-60 KB per message.

You’ll probably find that has less to do with analytics and more to do with emails these days being sent out MIME formatted, with a copy in HTML just in case one wants to use markup to stick an important word in italics.

I kind of wish mobile/tablet email clients had an option to “send this as something the ’80s would have understood”, but no. Instead next year your emails will not only be marked up, but will contained embedded CSS and webfonts…

<sigh>

Jeez. I think it bad enough that my email app for all my mailboxes takes a gig and a half

My WORK mailbox has a 512MB limit – I could raise that limit, or get one of the server boys to do it officially, but I prefer to delete the cra… erm, unwanted dross.

more to do with emails these days being sent out MIME formatted, with a copy in HTML just in case one wants to use markup to stick an important word in italics.

Nah, important stuff is in CAPITAL LETTERS
Maybe I should concoct a rule that replies to those¹ with a suitable rejection message >:-)

¹ She’s a repeat offender

ⒹⓄⒺⓈ ⓉⒽⒾⓈ ⒸⓄⓊⓃⓉ?

Well it’s halfway to the response in WingDings that I’d thought about.
Oh, there’s an aspect of her posts I missed – colour. I’ve corrected that.