ShareFS protocol or opensource module

I’m playing around with David Boddie and Mark Woodcock’s AccessPlusPython server software, with the intent of fixing some of the bugs (notably the one that’s clobbering filetypes when I try to compile code across an Access share). I’m curious if it’s a protocol issue.

I noticed that the ShareFS module is only present in the Git repository as a binary blob. With that said, Freeway is there, so network discovery is at least documented in some fashion.

Is there any documentation available on the ShareFS protocol, or any plans to open-source the ShareFS module with the rest of RISC OS?

Cheers,
Phil

I can’t help with the documentation for the ShareFS protocol but I was also poking around with AccessPlusPython the other day after a hankering to get my Pi booting via a network !Boot. I had AccessPlusPython running using Python 3 under macOS.

I had to fix a few things to get it working correctly. IIRC, it really needs to use SO_REUSEADDR before binding any of the UDP sockets and the broadcast code needed some attention (e.g. rather than figuring out the broadcast address, just use '<broadcast>' as the address). I delete my local repo with the intention of rewriting it in Perl because it’s a language I’m much more familiar with. But if you want to fork AccessPlusPython on GitHub I’ll send some PRs your way (and learn a bit of Python).

We (RISC OS Dev) have gained permission from all the original guys for the release of ShareFS etc, so if you get in touch with ROOL via email, they should be able to send you a copy of the source. Alternatively, get in touch with us at ROD, but it is better to have a proper submission chain via ROOL, in case you wish to feed back changes.

@dgr: I’ve already converted it from a Mercurial repository to a Github one (https://github.com/philpem/AccessPlusPython) — I’d be happy to incorporate your changes into that :) My email is philpem at gmail :)
I’ve been seeing issues where APP screws up the file name mappings during a C compile and ends up crashing the listener thread.

@Andrew: Cheers – will drop an email to ROOL shortly.

We (RISC OS Dev) have gained permission from all the original guys for the release of ShareFS etc

Good news! Thanks, Andrew.

For what it’s worth, I’ll mention that I did do some work on ShareFS getting on for 15 years ago (with the intention of making it work over TCP) that was never finished because a change of job meant I no longer had the time to spend on it. However, somewhere on an old hard disc I do have a version where the source code has been considerably cleaned up to make it more comprehensible, and I also have a notebook with some notes on how the protocol works. When the ShareFS code makes it into GitLab publicly I’ll try and motivate myself to dig all this out from the attic and make it available, because while I suspect I’m unlikely to find the time to be able to finish what I was doing I think it could be beneficial to anyone else who wants to work on it.

We (RISC OS Dev) have gained permission from all the original guys for the release of ShareFS etc

Excellent!

I do have a version where the source code has been considerably cleaned up to make it more comprehensible

Even better!

I do have a version where the source code has been considerably cleaned up to make it more comprehensible

= less incomprehensible?

I did do some work on ShareFS getting on for 15 years ago (with the intention of making it work over TCP)

Funny you should say that. It occurred to me a long time ago that a new version of ShareFS that worked over TCP might be a good thing – specifically on condition that it lost none of its ability to run with existing old versions of ShareFS, so no compatibility would be lost.

The first question to ask is: would it be a good thing?

We all seem to have got used to setting ShareFSWindow to 1. Well, that’s an easy default to change, but only the tiniest deal.

I can imagine that it would be nice to lose the double hourglass, during which nothing else works, when communication is lost. But what else, if anything, is there to gain?

But what else, if anything, is there to gain?

TCP could be a stepping stone to SSL/TLS (although I’d imagine there’d still be many more stones on that path)

That might make it possible to use ShareFS between two sites without needing to set up a VPN first, although the broadcast traffic might present an interesting problem.

We all seem to have got used to setting ShareFSWindow to 1

While having the too high a setting grinds everything to a halt I find a too low setting slows transfers down. At one stage I was using ShareFSWindow 8 but I can’t remember the combination of machines I was using for that to work. Switching to tcp has to be a good thing for maximizing performance.

@dgr: I’ve already converted it from a Mercurial repository to a Github one (https://github.com/philpem/AccessPlusPython) — I’d be happy to incorporate your changes into that :) My email is philpem at gmail :)
I’ve been seeing issues where APP screws up the file name mappings during a C compile and ends up crashing the listener thread.

I’ve created https://gitlab.com/dboddie/accesspluspython from the original Mercurial repository on Bitbucket. I may well replace the original with a git repository once Atlassian delete it.

Thanks for that David, I’ll look at archiving my repository and linking it to yours. No sense in stepping on each others’ feet.

I have seen the patch you left but haven’t had a chance to test it yet. Too much time spent playing with Econet!

AccessPlusPython seems to work quite well on my Linux Pi4, but is very slow even under pypy. I was hoping it might help me track down the issue of ShareFS dieing after a few tens of thousands of files, when copying from an ARMx6 mini.m to an Iyonix.

Does it die if you fetch from the iyonix instead of sending from the mini.m? Are you running a large wimpslot? If so does changing it back to 640k fix the issue? Basically anything you can do to slow down the fast machine when sending from fast machine to slow machine is a good thing and sometimes sharefswindow 1 isn’t enough. If it is a speed difference issue I would expect fetching from the slow machine to work as then it controls the data rate.

I seem to recall from a discussion a few years ago, that when shareFS pushes a file, the remote end first creates the file, then fills it with zeros before accepting and writing the received data. This delay can cause timeouts for the sender. Does your sender always die when it reaches a significantly large file?
I may of course have totally misunderstood it.

Thanks for the tips. I normally only get issues when synchronising almost the entire disc between two machines, the changes are mostly small lots files, and I’ve had issues with both push and pull. I’ll try to set up a reproducible failure, and experiment with sharefs windows and wimp slots etc.

Having just tried my iyonix for the first time in ages it is painful when used over VNC compared to my armx6 no wonder you have problems transferring lots of data.

As the first program I saw after dusting off my Iyonix was Discknght I thought I’d do a test so I set *wimpslot -next 320k and sharefswindow 1 on my armx6 and iyonix and dropped a 1GB file onto my Iyonix hard disc. It took 7/8 minutes of hourglassing before the filer action window opened – oddly enough I think this was normal at least during this time the disc activity light on my iyonix was flashing. It’s steadily transfering the file at the moment at less then 1MB/s if it fails I’ll let you know. You can tell that you are having transfer problems with sharefs with a large file because it starts to stutter and optionally hourglass while the filer action window is open. I think you can avoid the houglassing at the start by fetching instead of sending. – Transfer is still looking smooth 500MB to go.

It’s steadily transfering the file at the moment at less then 1MB/s

I changed the flow control setting on my 8 port ethernet switch and that changed to 4MB/s

I seem to recall from a discussion a few years ago, that when shareFS pushes a file, the remote end first creates the file, then fills it with zeros before accepting and writing the received data. This delay can cause timeouts for the sender.

The problem is probably that ShareFS probably creates and sets the filesize before copying the data to ensure that it has enough space on the target. You have two possibilities, creating the file and altering its size with SWI OS_Args,3 or creating the file directly with the correct size using OS_File,7. With OS_Args,3 the extended size is automatically filled with zeroes, with OS_File,7 the behaviour is left to the underlying FS.

I had the same issue with Samba when I still had my RISC PC and that the connection timed out when attempting to transfer large files to the RISC PC. I had to recompile the provided sources with OS_Args,3 commented out to get it working.

Does this still work? I have set this up on my DietPi NAS and nothing is picked up by any of my machines. Is there some weirdness to the config .access that i am missing?

I have set this up on my DietPi NAS and nothing is picked up by any of my machines. Is there some weirdness to the config .access that i am missing?

You could try to run WireSalmon on your RISC OS machines to see if they are receiving the broadcasting from the SahreFS on Linux.

http://www.cp15.org/networking/

I do not know if there are newer releases somewhere else, sorry.

I saw Andrew has the code could I have it ?.