Help - lost data

I have just done something stupid and would appreciate some help.

I have just used !SystemDisc to create a split partition on the SD drive 0: on my ARMX6. I had just formatted the card using SDformatter under Windows (16Gbytes). !SystemDisc said I needed to format it as filecore and dropped me into the RISC OS format utility to do this, showing the default ‘S’ for medium.

Due to the stupidity of programmers who chose ‘S’ for SCSI and ‘M’ for SDFS and my forgetfulness, I accepted ‘S’ as SDFS and formatted drive 0.

It then formatted drive 4 on SCSI, loosing all my data (128 Mbyte solid state drive).

How much of the data are recoverable please? I have DiscKnight.

I strongly recommend contacting David Ruck via the Disknight/ARMclub site ASAP. He’s still around, and might be able to give you guided recovery advice.

Probably best to remove the disc ASAP from the machine so that nothing else happens to it in the meantime.#

I believe you should be able to get most things back, since an “I” format (Initialise) should only replace the disc maps and not any disc content. However, I can’t say for sure as I haven’t had first hand experience.

Many thanks for this.

Just tried his mobile, number unobtainable, so my concern ‘charged at Flextel rates’ was misplaced (!)

With no e-mail at the moment am hoping some one van talk me through recovering the sub directories.

I’ll drop him an email on your behalf – that’s probably the easiest way.

An excellent idea, please tell him I can’t currently receive e-mail… (but can get text messages on my mobile which number you know).

I gave him your land line number… which I now realise might not have been the best thing. I’ll have a look around for your mobile.

Due to the choice of programmers who chose ‘S’ for SCSI and ‘M’ for SDFS and my forgetfulness, I accepted ‘S’ as SDFS and formatted drive 0.

It would be good if Elesar, who I think are the only supplier now, could arrange for a rather more robust user interface for what is obviously a destructive operation. Using “S” to choose between “SCSI” and “SDFS” is clearly regrettable

It would be good if Elesar, who I think are the only supplier now,

The problem is HForm, not SystemDisc. And yes, HForm should have been thrown away around 1988.

could arrange for a rather more robust user interface for what is obviously a destructive operation. Using “S” to choose between “SCSI” and “SDFS” is clearly regrettable

Indeed. Additionally, HForm should scan the drive you selected if it is already formatted and e.g. show the content of the root. I am sure a seasoned BASIC developer could do that in a few minutes (even in the compressed BASIC – I never understood why HForm was delivered in compressed form).

A replacement for HForm is one of the deliverables of a bounty that is now underway – I’ve no idea of the timescales involved, mind.

I agree with all your views on the horribleness of HForm, the inappropriateness of it being in crunched BASIC (can someone point me to the proper, uncrunched source on github?) the lack of sense in !SystemDisc dumping you into HForm with the wrong default options etc etc.

Can someone help me, please, to recover my data?

can someone point me to the proper, uncrunched source on github?

Here’s a direct link

More of work around than a fix, but I have changed my copy of the messages file in !HForm so that it asks a different question:-

Is your drive connected to SCSIFS or a MicroSDcard (S/M) ?

There are three lines to change

the inappropriateness of it being in crunched BASIC

“No user-serviceable parts inside”

IFgotadfs% availablefs$+="A"
IFgotscsifs% availablefs$+="S"
IFgotsdfs% availablefs$+="M"

https://www.youtube.com/watch?v=gbJfRe6iGlo

Just in case someone asks “when was your last backup” I can say March 2015 for the full disc image and March 2018 for !NewsDir (which I think contains all my Messenger Pro e-mail data).

So on Monday I hope to get myself to the position where I have a working machine again, albeit with all e-mails from March 2018 to date missing. Plus an untouched drive that might or not be recoverable. The next question therefore is, during the period when I use my e-mail based on the March 2018 version, is there any way to use POP3 to obtain my incoming e-mails but NOT delete them from the server so that (if and) when I recover the e-mail data and restart, I can then just resume and thus bring it up to date without losing the e-mails I receive in the interim?

And yes, HForm should have been thrown away around 1988.

With many thanks to Jeffrey, HForm was updated to version 2.75 in December 2017 addressing exactly the problem that tripped me up:

Previously devices would have been listed in the order they were found by the SCSI bus enumeration code, and the code would have asked for an index into that list, without giving any indication of the SCSIFS drive numbers that are assigned to each device. Until, that is, the SCSIFS drive number is shown to the user at the final "Are you sure?" prompt, potentially resulting in confusion because in many cases the SCSIFS drive number won't match the device number assigned by HForm.

it is unfortunate that this update has not yet made it to the ARMX6 service pack update…………. (I felt that a single ellipsis was not enough here)

Many thanks to Stuart. I recommend users download the updated BASIC programme from the link above and replace Utilities.Caution.!HForm.RunImage immediately. I have never seen ‘#’ commands in a BASIC file before – how do they work? Are they compiler directives for RiscBASIC? I thought that they had to have ‘REM #’ before them?

Can someone help me, please, to recover my data?

If the disc was formatted for the 2nd time with different parameters than the 1st, your hopes are reasonably high:

• The disc is divided into N zones, the map is placed in zone N/2, but the old map will still be wherever it used to be
• For long filenames discs the root directory is just the first fragment added so isn’t at a fixed address
• To recover, you’d need to replace the boot block (held at &C00 usually) with the old disc parameters then FileCore will go looking for the original map wherever it used to be
• You’d lose any file(s) that happened to where the new map was blatted, but as the map is typically 2-4MB at most that shouldn’t be too disasterous

If the disc was formatted for the 2nd time with the same parameters as the 1st then your recovery options are much worse:

• The map is where all the fragment ids are, which in turn tells FileCore where to go looking to reconstruct the bits of the files
• For smallish files in smallish directories they’ll be using sharing of the same minimum object size that the directory in which they live are, so don’t need reconstructing – once you’ve found the directory markers (‘SBPr’ ‘oven’ for long names) the sectors immediately after that will contain the data
• For larger files, in the absence of a copy of the map, they’re pretty much gone. The data’s there on the disc of course, but you’d have to reconstruct the files manually. Not too bad if you have a big text document, rather harder for binary data

With many thanks to Jeremy, HForm was updated to version 2.75 in December 2017…it is unfortunate that this update has not yet made it to the ARMX6 service pack

I think you meant Jeffrey, I should also plug that the latest app is also available as a packaged ZIP or in the RISC OS 5.24 disc image.

I have never seen ‘#’ commands in a BASIC file before – how do they work?

They don’t. At least not in BASIC. They are directives to the preprocessor, so need to be unpicked before the program will work. This means you have to manually insert the contents of the file referenced by the #include statement at line 14, and remove the “#ifdef DEBUG” constructs at three other places (e.g. the one at lines 24-28 can be slimmed down to a single line of BASIC: “Verbose%=FALSE”).

Finding a copy of “Global/FSNumbers.h” is left as an exercise for the reader.

I recommend users download the updated BASIC programme from the link above and replace Utilities.Caution.!HForm.RunImage immediately.

If all you want to do is upgrade HForm you should grab version 2.75 as mentioned by Sprow.

I recommend users download the updated BASIC programme from the link above and replace Utilities.Caution.!HForm.RunImage immediately.

I would very strongly suggest that they do not do this.

I have never seen ‘#’ commands in a BASIC file before – how do they work? Are they compiler directives for RiscBASIC? I thought that they had to have ‘REM #’ before them?

That’s because the source isn’t valid BASIC. Replacing your copy of HForm with it will render your copy unusable.

A lot of the BASIC source for RISC OS is stored as de-tokenised plain text, which allows CVS/Git to do useful things with it when it is changed. I do a very similar trick with all of my software, for use with Subversion.

However, whereas I pass my BASIC through Tokenize in order to generate valid tokenised BASIC, the RISC OS build system does two things. It tokenises the code using the BASIC module (AFAIK), but before it does that, it passes the plain text through the C preprocessor from the DDE. This latter stage leads to things like this from the HForm source that you refer to:

#include "Global/FSNumbers.h"

#ifdef DEBUG
Verbose%=TRUE
#else
Verbose%=FALSE
#endif

which pulls in some standard constant definitions from another file and results in either Verbose%=TRUE or Verbose%=FALSE – but not both – being included in the final tokenised BASIC depending on whether the DEBUG preprocessor constant (not BASIC variable) was set.

This is a very common thing to see in the source for BASIC programs in the RISC OS source.

PS. I’ve often considered adding such “conditional tokenisation” to Tokenize. A tokeniser that I used for similar purposes with BB4W in my last job had this ability, but so far it’s never proven to be something that I’ve needed for my RISC OS work.

That said, Tokenize can already do both of the above things in slightly different ways. It will “in-line” LIBRARY references in the same way that the #include in HForm does, and it can also replace all references to a variable with a constant value. So instead of the above, Tokenise would expect

Verbose%=TRUE

in the source. Should

-define Verbose%=FALSE

be passed as a command line parameter, Tokenize would remove the line and then update references such as

DEF FNdoalloc(Verbose%,Log%)

to read

DEF FNdoalloc(FALSE,Log%)

or whatever value was passed to -define. I use this for baking build dates and version numbers into releases.

Tokenize’s approach does have the advantage that the text source is still valid, untokenised BASIC that the TEXTLOAD command will accept. The downside is that Tokenize can’t skip arbitary blocks of code, such as this from the HForm source:

#ifdef DEBUG
    PRINTprompt$;" ";MID$(autoreply$,start,end-start)
#endif

An interesting discussion of pre-processing BASIC. Pace Steve’s Tokenise and the RISC OS source, I have a been doing this for a while myself. I will leave what Basalt does, as it is rather complex and is a process I call ‘Translation’, but mention Crunchie .

At its normal level it takes a running, tokenised BASIC program and crunches it. However, during that process it can apply some optimisations such as Steve mentions, and the program runs both before and after crunching.

• variables that are assigned literal values with the keyword LET (which is never required) early in the program text will be substituted as constants in the rest of the program, as above.
• the keyword VDU (which does nothing without parameters) at the start of a line marks code that will be removed, such as debugging statements.

There are command line switches to control these and other aspects of crunching.

I am under some pressure to implement in-line LIBRARY references. ;-)

But, Crunchie can also implement some features of ‘Translation’, that takes a non-running BASIC program and produces a running one from it. For example, in a different topic I mentioned the use of the IF...THEN...ELSEIF...ENDIF structure that Crunchie implements by adding enough ENDIF lines to match the ELSEIF lines. A more sophisticated feature is implementing procedure and function calls without the PROC or FN keywords, like COMAL.

A warning, Crunchie is still an active project, so it comes with a limited guarantee, but it is used to crunch a large commercial application successfully.

When looking at libraries, you might want to think about a “feature” I’ve made use of.
The set of programs I use communicate by means of TCPIP meassages. Some of those messages are sent unsolicitated, and need to be handled. For most of the programs they can be discarded,. The dispatcher calls a routine name to handle them, and the shared library has a routine for each which discards them. The programs which need them have an identically named routine to process the messages. This is used in preference to the “discard” routines, because it is found first.

When looking at libraries, you might want to think about a “feature” I’ve made use of.
The set of programs I use communicate by means of TCPIP meassages.

Interesting, that raises two immediate questions in my mind:

1. Does your setup allow a task sharing among a number of machines?
2. Is the messaging setup processor and OS agnostic? – i.e. Could it be used to “cluster” ARM and x86-family machines?

Other questions spin off those.

1 Does your setup allow a task sharing among a number of machines?

Depends on what you mean by task. It’s a client/server system, so the task of entering data and processing it is shared among machines.

2 Is the messaging setup processor and OS agnostic? – i.e. Could it be used to “cluster” ARM and x86-family machines?

It’s written in BASIC V, and firmly tied to RISC OS, so no, sadly.

and firmly tied to RISC OS, so no, sadly.

Never say never. As MS prove regularly almost anything is doable if you have a big enough budget, enough time, and competent developers.¹
There’s always the possibility of your message being in a format a similar application can understand.
Out in the normal world there’s stuff like ØMQ ²

¹ I feel sure someone will fail to resist the urge.

A little disappointed to see aspersions cast over the ARMX6 service packs, there. I have checked a recent one (12 in this case), and HForm 2.75 was most definitely included. That has been available since April 2018 I believe.

Indeed, since that accompanied the 5.24/25 update, I am more than a little surprised to hear that it wasn’t present and correct on the system in question.

For reference, I always start every SP by going back to the baseline RISC OS disk image, then begin building up from there, checking each application in turn.

Some nemobaiter mentioned zmq. I concur.

Some nemobaiter mentioned zmq

Moi? :)
I hadn’t realized it was in your area of interest, although on thinking about it the opportunity to open up so much non-RO based IT is actually very much in your area of interest.
As a person wanting to do control of various systems and general message passing I think Dave Higton would come to mind first though.