The case for the rehabilitation of RamTransfer

I have written at great length about the complexities of the Data Transfer Protocol; how it has never been implemented correctly; and how the later addition of DataSaved effectively broke Wimp_SendMessage (to say nothing of the Clipboard Protocol breaking the DTP itself).

That article dismisses RamFetch/RamTransmit as a badly designed work-around for floppy-only machines in the late 80s, which can (and should) now be safely ignored. But an alarming thought occurred that I ought to discuss.

To recap: I have argued that DataSaved was very badly documented, not only in failing to explain precisely what it was designed to achieve and why it ought to be implemented, but also in the description of the Wimp message itself – the PRMs only mention two fields.

Crucial to that argument is this most important sentence from the DTP preamble in the PRMs:

The message blocks are designed so that a reply can always use the previously received message’s block just by altering a couple of fields (my emphasis)

Thus I argued that as DataSaved is (usually) constructed from a DataLoadAck, it consequently inherits all of the usual DTP fields but for two fields that must be changed – the two fields that the PRMs describe. i.e. unlike the verbose message description of DataSave to DataLoadAck, the DataSaved documentation failed to list the unchanged fields, leading to the erroneous belief that these were the only fields in that message type.

DataSaved was a much later addition to the DTP, so it is plausible that the documentation may not have been as thorough at that point.

However it now occurs to me that a similar argument can be presented for RamFetch – for it is a reply to a DataSave, so by the “Most Important Sentence” above, ought to inherit all of its fields. This is quite shocking.

This would mitigate much of the problems with the RamFetch/RamTransmit protocol, not least of which is that with the present “short” messages, at the point the transfer is complete all metadata has long been lost – filetype, leafname etc – unlike the DataSaveAck/DataLoad/DataLoadAck route (when implemented correctly!).

It would also allow a non-final RamTransmit to supply a hint as to the amount of data still to come – by far the greatest failing of the current understanding of these messages. The mislabelling of the “estimated size” field is one of the worst failings of the PRMs – that field is not a size, it is the Unsafe flag, which can be used as a hint for buffer allocation in DataSave alone – see the above article for much more detail.

Now although this interpretation is very attractive, RamFetch/RamTransmit date back (very nearly) to the creation of the DTP. So it’s harder to argue that there was some laxity in the documentation of RamFetch that was not present when DataSave was documented.

But it’s not impossible, and the application of The Most Important Sentence to the RamFetch reply is irrefutable.

I’m still of the opinion that RamTransfer is an irrelevant complexity and unnecessary burden on the programmer in the 21st Century, but rehabilitating these messages from badly-thought-out hack to first-class members of the Data Transfer Protocol is attractive.

I’d be interested if anyone else cares.

[I’m also of the opinion that the Type17/Type18 distinction regarding ongoing versus terminal RamTransmit ought to have been emphasised, as that would further optimise the usual case and clarify the embarrassing “estimated” nonsense forever.]

I got as far as the Filer and the unsafe flag. I think I’ll stop and go put the kettle on. I have a feeling it will need much tea and a pillow on my desk to make it through the rest of the document.

Thank you so much for writing this. And I’m glad I wasn’t the only one confuddled with the your ref my ref bollocks.

PS: I added drag to save to Tea (my Freesat programme guide) and after looking at the RAM Transfer stuff in the PRM and noting that it seemed to work perfectly well dragging to Zap using an intermediate file, I thought “sod it, too much bother for no useful gain”. Maybe if Econet, maybe if floppy discs, but that era is now referred to as a “classic music station” ¹ so isn’t of much consequence these days…

¹ Not “classical”, that’s something else.

If you ever want even a hint of inter process security in RISC OS, the Wimp_TransferBlock SWI¹ has to go, or to be severely restricted to only the addresses specified in the RAM transfer protocol. Luckily RAM transfer was always an optional part of the data save protocol, so could dropped altogether and everything should fall back to file transfer, which isn’t much slower now we have flash storage².

¹ That would result in a problem for other programs such as my WindOpen module which needs to read the text of icons, as for an indirected type the text lives in it’s application’s workspace, and Wimp_TransferBlock is currently the only way to read it.

² It’s not like in the days of single floppy drive, when you had to dig out the disc with !Scrap on it for file based transfers to work.

Luckily RAM transfer was always an optional part of the data save protocol

Maybe this ought to be entitled “The case for the retirement of RamTransfer”?

which needs to read the text of icons

This would also impact the ability to create accessibility software such as a screen reader. Perhaps the Wimp ought to have a way of reading the definition of an icon including the indirected data (pasted to the end of the block? ¹) which would be less of a problem than furtling around in the memory of another application.

If you ever want even a hint of inter process security in RISC OS

Security? RISC OS?

<walks away giggling>

¹ There’re actually two bits of data, there’s the indirected icon text, and there’s the validation string, so it might be a two part process, like call once to get the amount of memory to allocate and call again to actually read. If the Wimp is being helpful, it’ll set two registers to directly point to the start of each part so one doesn’t have to mess around to work out where one finishes and the other begins.

Druck ventured

If you ever want even a hint of inter process security in RISC OS, the Wimp_TransferBlock SWI has to go

Hence BlockTransfer

However, PostFilter laughs at your attempts at security and offers a safe way of scouring app memory and there’s nothing anyone can do about that.

It’s not like in the days of single floppy drive

Which is a point made in the article.

which needs to read the text of icons

There’s an interesting point to be made here, and I have a nearly-complete module for the purpose – reading the text of an icon is less straightforward than you would think.

RO5 provides Wimp_Extend,14 to read the Validation string (which is a partial solution for that problem space) but no similar call for the text buffer – presumably because they think that’s easier, when in fact it’s harder.

And if you think an indirected icon tells you how long the text buffer is, you’ve seriously misunderstood icons. That last word is not the length of the buffer, it’s the maximum number of bytes that can be inserted into writeables. It has absolutely no effect on the display of icons, and hence the length of the buffer contents.

Reading the text of an icon is therefore the same problem as reading the validation – except you also have to deal with the unterminated unindirected case.

Nobody ever gets anything right.

Also, whereas an author might use the max field to store their text buffer length (and then again they might not), there’s certainly nowhere to store the length of the validation string buffer, so a technique employed for mutable validations is to pad the validation string, either by semicolons or semicolon-then-spaces. The aforementioned module allows read and write of icon text and validation by using these various methods.

Not that the model works in the presence of BlockTransfer. Perhaps that ought to be more discerning about the mode of the caller.

Some time ago I started Copy and Paste, I needed it for the !PariGp editor. This one uses two TextAreas. This was not easy.
I studied Benoit Gilon’s Genie program and used Clipper to do transfer checks.

Using this information I built a flowchart, which represents events and the program to test with 2 TextArea.

This is a good basis to use and test the protocol and my PariGp application works, but not on a RPi, I have to fix…:-(

If it can be useful, the program, its source, the flowcharts, be indulgent:-)
Note: To have information during the transfert, !Reporter must be running.

https://jeanmichelb.riscos.fr/Download/Temp/TA_CutPaste.zip

And if you think an indirected icon tells you how long the text buffer is, you’ve seriously misunderstood icons. That last word is not the length of the buffer, it’s the maximum number of bytes that can be inserted into writeables. It has absolutely no effect on the display of icons, and hence the length of the buffer contents.

Not surprising people think it’s the buffer length when PRM 3-101 says it’s the buffer length. So you are saying that RISC OS does not check the buffer length when displaying the contents of the buffer in the icon, but does prevent the user adding more characters to the icon if the length of the text exceeds the declared buffer length (less one for the terminator?)? So you could have a situation where the writable has 40 characters displayed, but a “buffer length” of 20, and the user will find they cannot type anything into the buffer unless they delete 22 characters first.

I wonder how many application authors have actually relied on this quirk?

I wonder how many application authors have actually relied on this quirk?

None, I hope, as…

Not surprising people think it’s the buffer length when PRM 3-101 says it’s the buffer length.

…it is documented as being the buffer length. ;)

But as was pointed out, there can be two buffers (icon text, validation string) and only one has a buffer length provided (even if it isn’t really a buffer length, sigh).

Lesson kiddies: The importance of accurate documentation.

I’d just like to record my gratitude to Nemo for a very interesting read. I got right to the end, understood it all (I think) and did not actually swear at any point.

I’m not sure I agree about the most important sentence. I think that the person writing that was just saying “look how nice the protocol is: you don’t have to do very much to construct your replies”. You have not considered the word “can” in the article. That implies it’s optional to some extent.

I completely agree that without that interpretation, the DataSaved message description is incomplete and less magical than it should be. I think your interpretation is stretching it a bit. The most important sentence is part of a paragraph on PRM 3-252 (printed 1992 edition) which starts “The messages used in the above descriptions are described below.” but the above descriptions do not include DataSaved. So I think that the most important sentence probably was not intended to apply to that message. I agree that implementing it the way you have done would give an extremely nice effect.

You won’t therefore be surprised when I say that in my view the most important sentence cannot possibly apply to Message_RAMFetch. The structure of the message is, in any case, just too far different from the core DTP.

I really enjoyed all the nooks and crannies of RISC OS your have explored and the historical perspective. The way poor documentation builds up to the car crash of the clipboard is most unexpected, and the prize at the end was excellent!

I am going to have to go and look at my applications again and make sure I improve them. I’ll try and add a few comments here later, as I have found some interesting features at times because of how other applications implement the DTP. Ovation Pro and Impression come to mind.

Well that was a very enlightening read!

I remember it being very normal to read the doc, conclude it made little sense and hack stuff about until it appeared to work.

It also reminds me of trying to write a filecore filing system that supported removable media.

The documentation I remember was confusing to say the least about who was who in the contract. Despite the section being labeled writing a driver, I eventually got an email saying yes actually we documented the filecore point of view.

Matthew complained

Not surprising people think it’s the buffer length when PRM 3-101 says it’s the buffer length.

Repeat after me: “The PRMs were written by mere mortals and are not the irrefutable word of a Deity”. They also used to be regularly corrected with an Errata. Those were the days.

Anyway, the so-called “buffer length” only applies to text editing of writables. And these days is only part of the decision as to whether something can be inserted into the buffer – the U validation will also play a role when the Alphabet is UTF-8.

So you could have a situation where the writable has 40 characters displayed, but a “buffer length” of 20, and the user will find they cannot type anything into the buffer unless they delete 22 characters first.

Yes. If the ClickType is not 14/15 Writable (and the cursor hasn’t been inserted by other means) that final field is simply not consulted. And, after 35 years now, cannot be.

I wonder how many application authors have actually relied on this quirk?

Every app whose indirected display icons don’t have a sensible value? Can’t be changed now.

cannot possibly apply to Message_RAMFetch. The structure of the message is, in any case, just too far different from the core DTP.

That’s a cyclic argument. If viewed as a delta of the DataSave that it is a reply to, then the only fields that are changed are the Handle and Icon/Ref (which are, thanks to the various other protocols that end up becoming these messages, not necessarily window and icon anyway). So the documentation is entirely consistent with the only-change-what-you-need-to design that is explicitly defined in the Most Important Sentence.

And I suggest now that the MIS should probably apply to the RAMFetch messages not because I want it to (I want those messages to die) but because I can’t find any logical reason to exclude them. The MIS is not restricted to specific messages but to the whole DTP. That does include the RAMFetches and, as of RO3, DataSaved too.

Put it this way (and this is always a useful tool) imagine the opposite – imagine that DataSaved, RamFetch and RamTransmit had been fully documented from day 1 as being fully-fledged DTP messages… would you now be arguing that they ought not to be? No, you absolutely would not.

And in particular, both RAMTransmit and DataSaved are rendered valuable and useful by virtue of the other DTP fields (beyond what their subsection documents, which are only the changed fields) – as they include the filetype, filename and unsafe flag from the original message. Without those fields, the messages are rendered much less useful (DataSaved becomes meaningless) and the apps at both ends are forced to cache the rest of the DTP fields because they will be needed when the RAMTransmit or DataSaved is received. It’s painfully obvious they were intended to be there in the first place because of this.

[Edit: To clarify the DataSaved point – if you receive a DataSaved without filetype, you have no way of knowing whether your data has been saved – eg if you are a Word Processor and the remote task saves a Plain Text version, then the DataSaved should not result in your document becoming ‘unmodified’, as the remote task has thrown important context away. And the ability to take on the filename under which the data has been saved seems like a most important point]

I am going to have to go and look at my applications again and make sure I improve them

Wonderful. I keep promising to complete a test harness that would probe every detail of a task’s implementation of the DTP and give a score out of 100… but I suspect nothing would get higher than 15!

Getting the messages the correct length without extra junk on the end or omitting fields; with the correct reference; and dealing with bounces correctly is 90% of the job. Actually handling message refs the way that DataSaved requires is the icing on the cake. It’s a lot of faff. But being able to save a file from this app and have the document in that app become ‘unmodified’ and adopt the new filename really is magical – no other OS does that and we really should have worked harder to implement it everywhere in 1991.