SMB2/3 protocol

Hi,

I’ve successfully ported the libsmb2 library to RISC OS and it can be compiled using the DDE.

The sources for it is currently available in my GitHub repos. Please see https://github.com/Rob-McKay/RiscOS-smb2/tree/main for details.

I’m in the process of making the library a module.

I know that there is a bounty for `More recent LanManFS protocols` and I think that this library achieves:

• Dialect negotiation at SMB 2 or 3 level
• Use of direct TCP on port 445
• Change to use DNS for name resolution in preferences to NetBIOS name service
• Discovery of printer resources for Printer Manager
• Following symbolic links
• NTLMv2 authentication

Other authentication options are available if/when the libraries libsmb2 uses for them are available on RISC OS.

I’ve currently only tested the library with a couple of the example apps which come with the library connecting to a Windows 2019 server.

I hope this is useful for the RISC OS community

Rob, good to hear the news, sounds promising. :)

This is excellent news that I have been waiting for for years. With every new update in Windows or in my NAS, I worry that I will no longer be able to connect to it. I know there is an experimental SMB2/3 client for RISC OS, but that’s a stand-alone application that doesn’t merge with the filer. So yes, this is great news.

Excellent news. I’ve never been able to get RISC OS to talk to my Mac and I believe that’s down to SMB versions, so it sounds like this is a big step in the right direction :)

With the latest version of SMB, even my old MacBookPro is unable to see the M1 MacMini. At least the the MacMini can see the MacBookPro, and they can both see the older (pre-ARM) MacMini (which is now serving as a file server).

Excellent news! I’m not trying to steal your thunder, but to avoid any confusion with similarly-named threads, I hope you won’t mind if I share my own recent plans and progress towards claiming the bounty for this highly-anticipated feature. I’ve started a private email thread with ROOL to discuss the specifics of the SMB2 work items, including some that go beyond what’s listed on the bounty page. What I wanted to share so far is a link to the SMB1 fixes that I’ve been committing to my public GitHub repo to familiarize myself with the code, and what’s needed to get SMB1 working properly.

https://github.com/jhamby/RiscOS-OmniLanManFS/

• SMB1 support has issues now but becomes more usable after fixing three specific bugs that I’ll describe shortly.
• I’m actively fixing all of the SMB1 bugs on my fork (jhamby_smb2 branch) and I hope some/all of my commits will be considered safe enough to make it into the upcoming 5.30 release.
• The libsmb2 port is still very useful to have, for ports of apps that can use it. Linux has at least 3 different SMB clients: smbclient from Samba, a kernel SMB client, and a GNOME GVfs module. OmniLanManFS is equivalent to a kernel module, so I plan to claim the bounty by adding all of the features to that kernel module. There’s still a need for Samba and libsmb2, but what people actually want is for the Omni client to work properly again, and libsmb2 is neither necessary nor sufficient for that.
• LanManFS can be easily updated with a newer module so you don’t need to replace the ROM to test new/debug builds.
• Changing the DEBUGIT (“!List-Debug – Wimp based debug system (MChallis)”) to !Reporter has made debug logging quite convenient, without which I wouldn’t have made this much progress in two weeks.

And here are the three SMB1 bugs I’ve found that make OmniLanManFS quite a bit more usable once fixed:

• there are 40-second timeouts when any SMB2/3-only servers are on the LAN because the client isn’t detecting either the remote end closing the TCP socket or the fact that the server responded saying it doesn’t support any of the requested dialects.
• adding direct TCP support was easier than one might think, because the 32-bit message header is almost the same as for NetBIOS-over-TCP, so the main difference is removal of an initial NetBIOS session handshake. I then discovered that the deprecated SMB1 RAW_READ / RAW_WRITE commands don’t work as expected with Samba, and with Windows 2008 / Vista and newer, the server actually disconnects you if you try to use either command, despite advertising the RAW feature in the capabilities flags. The only safe fix is to only use the standard 4KB read/write commands, which are infinitely preferable to non-working deprecated SMB1 commands that Windows closes the socket if you attempt to use!
• if the caller requests more directory entries than will fit in the buffer (Filer likes to request 18 entries, with 20-byte information field, in a 512-byte client buffer), the code complains “Oh dear – client ran out of buffer space!\n” and then drops the last entry on the floor. So the correct fix without adding any additional directory caching would be to request the full directory from the server and then hand it to the client as they ask for entries.

This leads me to observe that Filer directory queries should benefit by increasing the client buffer size from 512 bytes to perhaps 4KB, to have a better chance to get the full 18 entries it requested, although the speedup would be minimal assuming the entries are being copied out of a cached copy of the entire directory. Unfortunately, LanManFS is currently broken whenever the filename lengths add up to larger than the client buffer, because it’s losing all the entries that didn’t fit the buffer after being received from the SMB server.

If you have any comments, suggestions, questions, or feedback, I’d definitely appreciate it. At my current rate of progress, and considering my professional expertise with these types of protocols, I’m aiming to collect the bounty some time in the next 4-6 weeks, depending on various factors. In order for that to happen, I’ll have to put in quite a few hours of work writing new code, but my intention is to earn every penny by making the donor community 100% satisfied.

As for the SMB1-related bug fixes, I’d love to see them get into RISC OS 5.30, along with adding direct TCP and removing the obsolete NetBEUI transport layer. Those release engineering decisions aren’t up to me, but of course I’d love to see 5.30 ship with my changes so that SMB1 works as well as it did in the past, rather than being completely broken with lengthy timeouts.

@Jake: All sounds good to me … but I can see no commits in the last month on your github link – but I could be doing something wrong!

Look at the other branches. jhamby_smb2.

@Jake: That sounds good.

I did consider trying to fix the CIFS (SMB1) routines but opted to go down the libsmb2 route for the following reasons:

• provides support for both SMB2 and SMB3
• is (currently) well supported by an active community, reducing maintenance burden
• has been used in bare metal projects so doesn’t require OS support
• supports kerberos when combined with other libraries
• supports SMB3 signing and encryption which Microsoft are in the process of requiring to be able to connect to any new windows 11 machine (currently only the latest servers).

The downside is that it is LGPL licensed which is why I am wrapping it in a module and the modifying OmniLanManFS to call that module. I would have preferred to split OmniLanManFS into two parts, the filesystem and the Omni Client interface as is done for nfs so that the code is cleaner.

I’m not interested in the bounty but perhaps we should pool our resources and work together.

@Rob: That sounds even better!

@Jake, not sure if it is work trying to maintain the current version, apart for the fact that it is SMB1 only, it is not very reliable as soon as you do anything outside standard file transfers.

Check these topics:
LanManFS problems
and attempts by Colin to begu some of them:
LanManFS debugging

I’ve nothing useful to add, but just wanted to say: this thread is one of the best I’ve seen in a while. There’s a great effort in porting an active library to RISC OS, and someone seriously looking into a bounty. And in the case of the latter, source code – which is totally public/open. I take my hat off to you both, and think the openness of both approaches is wonderful.

I had been meaning to ask, with LanMan98 being ‘open sourced’, if there was anything in there which was useful. I remember that back in the day, it offered long filenames and LanManFS didn’t. But LanManFS gained those a while back, so I’m not sure what LanMan98 actually brings to the party.

I remember browsing the LanManFS source code once, as there was some strangeness with files named ‘/[something]’ (they became invisible). I mentioned my issue, and before I got as far as building it, Colin was in there with a fix. Not my area, obviously!

@André, thanks for the links!

@Rob, yes, I’m happy to work with you on this, especially since you’ve investigated areas that I haven’t yet because I’ve been approaching the task from the opposite direction (familiarizing myself thoroughly with how the current client code operates).

Here’s what I’ve discovered about OmniLanManFS: it’s overgrown with backwards-compatibility code that I’m currently in the process of stripping out altogether, now that I know that only the “NT LM 0.12” dialect of SMB1 (aka NTLM, aka CIFS) will ever be used. I’ve already finished taking out the NetBEUI transport as well as the block read/write raw transfer callbacks that also aren’t needed.

By keeping only the “-DLONGNAMES” code path and removing all dialects except DIALECT_NTLM (currently “DIALECT_NT”), then we have a minimal working subset of SMB1 with NTLMv2 challenge/response authentication, Unicode filename support, etc. to build on top of. I expect my pruned-down version of SMB1 to be 1/2 the size and much easier to follow, as well as more reliable. More importantly, it’ll be a solid foundation for building SMB2/3 protocol support on top of.

The reason I’m not super enthusiastic about using libsmb2 is first of all the incompatible license, because the most useful part would be the header files and utility functions for the SMB message and response structs, and the whole thing really should fit into a ROM. LGPL makes that library useless, sadly, but Samba’s libsmb has the same issue and is much worse in terms of complexity.

What’s really going to be useful for this project are the “other libraries”, specifically for Kerberos and cryptography. There’s no Kerberos domain login support in the current LanManFS, and that appears to be a requirement for the bounty work items. Also, I assume it makes the SMB3 signing and encryption more useful because you’d have client computer certs and domain user certs to check the signatures against. Or am I misunderstanding the benefit of signing and encryption? It seems like signing would be much less useful for a home workgroup, other than making replay attacks harder, and who’s attacking you with those on a home network?

Fortunately, I have both Samba 4.15.x and Windows Server 2022 180-day trial ISOs to create test domains. I’m just curious if there will be any RISC OS users who have domain single-sign-on at home or your office, perhaps with Samba 4.x, perhaps for a heterogeneous Windows / Linux / macOS / BSD environment? For this part of the project, I’m the most interested in learning about the subset of users who have any interest whatsoever in Kerberos domain logins, presumably because you’re already using Active Directory (Samba’s or Microsoft’s).

The good news is that despite all of the fearmongering about SMB1 being insecure and unsafe, the concepts are similar enough to SMB2, and the number of RISC OS users who also have retro versions of OS X (PowerPC perhaps?) or Windows XP they’d like to use is probably greater than the number of people interested in Kerberos, and definitely non-zero. After stripping out the 8.3 filename and MS-DOS / OS/2 / NT3.5 / NT4.0 compatibility code, it’ll be much more maintainable than it is now. I’m also cleaning up the socket handling in preparation for SMB2/3.

I should have a relatively large commit to my SMB2 branch of my fork of OmniLanManFS in the next day or two with all of the old code removal, socket refactoring, directory query caching, etc. that will be necessary for SMB2. BTW, one thing that isn’t listed on the bounty page but I consider of great importance is that the client should use mDNS DNS-SD (aka Bonjour) to find local servers and not regular DNS with the system resolver. Fortunately, I have prior experience with mDNS clients on a variety of OS’s, and the good news is that it works very similarly to the NetBIOS Name Server code that’s already in the client. I just have to copy-and-paste those blocks, change the port number, and put in different strings to query for “smb.tcp” parse the mDNS response.

Perhaps surprisingly, you can add mDNS client support to user apps without any special permissions (Windows apps may pop up a firewall notification), and it’s simple enough that it doesn’t really need a special library. I was planning to copy-and-paste and modify the relevant sections of a public-domain mDNS client I found on GitHub:

https://github.com/mjansson/mdns

So that’s mDNS, Kerberos, message signing and encryption, and one more thing, which is message sizes. As I mentioned, the “READ_RAW” and “WRITE_RAW” commands don’t actually work because you’re supposed to use “AndX” commands instead, which can be chained on the client side, but don’t have to be (I’d forgotten about that part). Now that I’ve made the executive decision to remove all vestiges of pre-Win2K versions of SMB1 support, I can assume the presence of the AndX variants of the read and write commands, which let you pass a higher maximum number of bytes than the normal message size that’s been negotiated.

Long story short, after I finish removing the backwards-compatibility code and fix the sockets and directory handling, but before I work on adding mDNS / DNS name lookup and then the SMB2/3 protocol (which, as I alluded to, should plug nicely into the RISC OS interfacing portion of LanManFS quite easily, however we decide to implement the SMB message parsing), I’ll take a quick diversion into adding support for CAP_LARGE_READX and CAP_LARGE_WRITEX capabilities, which need to be negotiated between the client and server, along with passing a larger max receive buffer size (Samba appears to go no larger than 63 KiB, presumably to avoid going over 65535 bytes for the full packet), and then see how much of a performance boost that provides compared to 4KB messages.

Theoretically, SMB supports direct TCP packets up to 128 KiB – 1 byte, and handling these sorts of limits is the sort of mess that using an existing SMB client library would definitely help with. However, now that I’ve already spent far too many hours studying the daunting CIFS, SMB1, SMB2, and related specs, I think it makes more sense for me to write the RISC OS version from scratch, if only because the current client uses mbufs so heavily and I have no confidence that a cross-platform library would function well in such a restricted environment as far as memory allocation, stack size, being called from supervisor mode, etc..

Once I’ve finished preparing the way for SMB2/3 client support, I’m definitely going to need some help from the community in terms of figuring out how to implement and load sub-modules for Kerberos, and any other optional feature that’s big and doesn’t need to be in ROM. That way the ROM version will work for booting and 99.9% of RISC OS users, including encryption and code signing (those algorithms aren’t too big by themselves), and only people using domain logins will load the Kerberos module. Maybe there can be code reuse with other RISC OS programs that support Kerberos (like SSH?).

I did eventually find exactly one suitably-licensed SMB2/3 client library, from Apple, via this post:

https://news.ycombinator.com/item?id=16087969

Apple also keeps a GitHub repo here.

https://github.com/apple-oss-distributions/smb/

I thought I’d read that Apple had stopped using their homegrown SMB client/server code in favor of Samba, but Samba is GPLv3, which is strictly forbidden by Apple (that’s why they were stuck with such old versions of bash and GCC/binutils). So Apple is still using their own implementation and people have found exploits in it:

https://blog.talosintelligence.com/vuln-spotlight-smb-mac-deep-dive/

For those who aren’t aware, you can’t just mix-and-match GPL or LGPL code with Apache / BSD / similar licenses. The GPL license “infects” the whole project if you don’t strictly follow the rules, and many companies refuse to deal with it at all, particularly for something ROMmable, like RISC OS. libsmb2 is LGPL 2.1 and not the hated LGPL 3, and if there are already LGPL libraries “tainting” the ROM, then I have no other objection to linking it into the OmniLanManFS module statically.

Now that I know Apple has an open-source BSD-licensed SMB1/2/3 library that they use for their SMB client and server, it seems like a no-brainer to start from Apple’s SMB library.

• it’s in production use, maintained by one of the largest software companies in the world
• the BSD-style license presents no issues whatsoever for RISC OS integration
• since it’s written for use in a BSD-style kernel environment, it already uses mbufs, so memory management will plug into RISC OS’s mbufs module quite nicely

Well, I’m sold on not writing a bunch of boilerplate code for the SMB2/3 and signing, Kerberos, encryption, etc.. I just didn’t want to use libsmb2 for the inverse of the reasons why I do now want to use Apple’s “SmbX”.

I’ve looked at the Apple SMB repo and it’s from approx 2013 so is probably quite out of date and seems to confirm that they’re no longer using it.

The whole reason that I’m using libsmb as a separate module is to allow dynamic linking against it. That way a user can load another version of it which they have built from source or obtained from elsewhere to fulfil the terms of LGPL 2.1

The main reason I chose libsmb is that it has an active community using and maintaining it which reduces the maintenance burden from the whole implementation down to the RISC OS specific bits (like the module interface).

For those who aren’t aware, you can’t just mix-and-match GPL or LGPL code with Apache / BSD / similar licenses.

Large percentages of the people round here are very aware of the general nastiness and “infective” nature of GPL of any version.
3rd party applications sometimes include it, no aspect of the OS has it.

Many years ago (Iyonix era 2002) someone made the mistake of using some GPL for part of the HAL. Much upset and aggravation ensued.
The HAL was rewritten, sans-GPL code, but 20 years on it still raises hackles.

So, yes, keep things GPL free. Who wants a licence that doesn’t agree with itself anyway?

The module I started is separate to the lanmanfs module it connects using smb2/3. It uses mbedtls from the risc os sources. I’ll bundle it up sometime today. If it’s useful feel free to use it.

Just a quick post to mention that I have a mixed home office network with RISC OS, Windows and Linux. I’m currently in a domain with AD, although I have Linux using an LDAP server and will probably migrate everything to LDAP when I my Action Pack Windows licences expire.

The sources for my module is available here – SmbFSSources.zip.

Who wants a licence that doesn’t agree with itself anyway?

Who wants a licence that runs on fuzzy ideology and is therefore poorly defined and utterly unfit for purpose.

With respect to the GPL v2, the whole thing about static and dynamic linking etc is some rubbish that was discussed at length in the FAQ. The licence itself does not make any such distinction, only talking about whether or not the code is a part of a whole. This was “fixed” in v3.

Relating to this, quote from the same FAQ: What constitutes combining two parts into one program? This is a legal question, which ultimately judges will decide.
So it’s a licence with a somewhat onerous side effect that they (GNU) are clearly passing the buck on, leaving it to the legal system to do what the licence should have been doing.

Of course, by failing to specify exactly what they mean, it might not quite turn out the way they want it to: https://thehftguy.com/2021/08/30/french-appeal-court-affirms-decision-that-copyright-claims-on-gpl-are-invalid-must-be-enforced-via-contractual-dispute/

So, yes, keep things GPL free.

If you really support “open source”, you wouldn’t touch it with a bargepole. Thanks, but no thanks.

I’m delighted to be reading this discussion. It looks very promising in terms of results.

I’d like to offer my encouragement to all of you who do anything towards creating a working SMB2/3 client for RISC OS.

Sadly I’m unlikely to be able to do anything concrete towards the programming. Any help I can offer is most likely to be in testing; I have an OpenMediaVault installation on a RasPi, and a WD MyCloud, both of which of course support SMB2 and 3.

What communication medium are you going to use for dev?

Just reading though this… Would it perhaps be beneficial to place the mDNS part in a separate module? I know IPP relies upon mDNS, as does this, and using Service Browser on my phone all sorts of things can turn up.

Therefore, perhaps there’s some value in having a general solution for tracking (and sending?) mDNS/Bonjour messages within RISC OS rather than everything doing it for itself?
Then, an app (any app) just needs to ask the module if it knows about “x” in order to get a useful reply – “yes, tomato.local is a printer at 192.168.1.14 port 9000”, however “telnet.tcp” returns nothing…

An mDNS module apparently exists, according to Gerph, but whether it’s generally available is something that I’ve never followed up.

Here’s what I’ve learned so far. It’s mostly good news. First, Apple did stop releasing sources for their SMB client after OS X Yosemite 10.10.5 (released July 2017), which is the “smb-759.40.1” code drop (the “main” branch on GitHub only has “smb-759.0” but the .tar.gz archive of 759.40.1 is available in “releases”). It supports SMB 1.0 through 3.0.

I believe Apple is still using this code, but they stopped releasing the source code after OS X 10.10.5, as is their right. The original code was released under the BSD 4-clause license, which is slightly annoying because you have to find a place in your distribution to credit the author, in this case “Copyright © 2000-2001, Boris Popov”. On macOS, there’s a massive “/Library/Documentation/Acknowledgements.rtf”, which includes the Boris Popov copyright for the “(smb)” package (one reason why I’m sure the code is still being used).

On the RISC OS “Nut Pi” microSD that I’m using (purchased in July 2021, so I need to acquire an updated DDE to be able to build the current RISC OS source tree), there’s a Sources.LICENCE/bsd text file containing “The Ancestral BSD Licence”, “The 4-clause BSD Licence”, 3-clause, and 2-clause, with copyrights lines concatenated together above each type of license. Someone will need to add the Boris Popov copyright line to that file (I don’t know if it’s generated automatically or not) and that will be the only legal obligation.

I can see two major challenges with this code, and then I’ll describe my current progress. The first issue is Kerberos authentication. Apple uses Mach IPC ports extensively, and this is no exception: the kernel code in smb_gss.c talks to “gssd” via Mach ports. FreeBSD’s Kerberos uses a UNIX domain socket at /var/run/gssd.sock. The userland API is called GSS-API, and RISC OS will need to supply some equivalent that the LanManFS module can interact with to ask for credentials.

For the regular NTLMv2 authentication, the code is self-contained. The second challenge has to do with enumerating shares and printers once connected to a server and its IPC$ share. LanManFS currently only supports the original “RAP” (Remote Administration Protocol), which connects to \PIPE\LANMAN. Apple’s code supports this, plus the newer MS-RPC (an extended version of the old DCE/RPC) using their own extended version of DCE/RPC to generate the client stubs from the .idl files, which aren’t supplied in their archive, I assume due to copyright reasons.

I’ve tried to generate client stubs with dceidl using the command in the “make-idl-client” shell script, but it complains about the syntax of the ms-dtyp.idl, lsarpc.idl, and srvsvc.idl files that I copied-and-pasted from Microsoft’s specs. I assume Apple modified the .idl files to be compatible with the DCE/RPC syntax, and they’ve added autoconf tests to the version of DCE/RPC on GitHub (and the one listed alongside the other open source packages for 10.10.5) to detect and use their own SMBClient.framework interfaces if the library is found. So I’m going to fork the DCE/RPC repo and try to modify it to generate the necessary (but not supplied by Apple) lsarpc_client.c and srvsvc_client.c stubs to use with the other files in lib/librpc. Some references to MS-RPC and SMB:

https://www.thehacker.recipes/ad/recon/ms-rpc
https://learn.microsoft.com/en-us/openspecs/windows_protocols/ms-wpo/4de75e21-36fd-440a-859b-75accc74487c
https://wiki.wireshark.org/DCE/RPC

The most full-featured open-source implementation of these MS-RPC interfaces would be Samba’s, but I can’t reuse their work for two reasons, the first being GPLv3 “contamination”, but also their generated code is tied to Samba-specific libraries (talloc, tdb, etc.), which would be more work to adapt than figuring out how to get dceidl to work by modifying Microsoft’s .idl files. I looked to see what libsmb2 is doing, and they appear to have hand-hacked together stubs based on commented blocks of .idl listed above each function, and it all looks a bit flaky, since the function names don’t match the spec, and not very maintainable.

I also looked briefly at Wine’s “widl”, which can generate Windows-style client and server stubs, like Microsoft’s MIDL compiler, and that might be a reasonable alternative if for some reason I can’t write .idl files compatible with dceidl. I’ll likely copy the client stubs directly into my OmniLanManFS repo, with a text file explaining where to find my repo with the patched version of DCE/RPC and corresponding .idl files. There shouldn’t be any copyright associated with the generated files, or else Apple wouldn’t have taken this approach (and DCE/RPC was originally designed as a closed-source product that customers could use to write their own commercial software).

Setting aside those two challenges for now, I’m currently stripping out the original SMB1 implementation from OmniLanManFS, leaving only the RISC OS interfaces and stubs to call into the new code, which I’ll add as a separate commit. I discovered that the error messages should be implemented more like SDFS and SCSIFS, with an s/LanManErr assembly file to import the FileCore errors (OR’d with 0×0001nnxx, where “nn” is the filesystem number, in this case 0×66), with custom error types added. Then the error block generation code can get the localized error message text for both the standard and custom errors by searching for the error name without the “ErrorNumber_LanMan” prefix. SDFS has a MESSAGE_ERRORLOOKUP() macro that calls a helper function, constructing the necessary params for the numeric error and the short text string.

I was a bit confused by the meaning of the #{DictTokens} line in the Messages file. It appears that it applies token-based string compression (I think this is what’s meant by “tokenised”?) and also doesn’t add a C null terminator to the strings, so you can’t use them directly as, say, string arguments to sprintf(). But you can use the looked-up string in error blocks. It looks like RISC OS can terminate strings with ‘\n’, ‘\r’, or ‘\0’ in cases like this. That would explain the trouble I had modifying the “CmdHelp” file to remove the ‘\0’ at the end of each line, which are necessary to print just the looked-up help line and not the entire rest of the file.

Apple’s code (in kernel/netsmb/smb_subr.c) converts all SMB error returns to NTSTATUS error values, which it then converts to POSIX errno values, with a couple of extra values for errors like “violated password policy” and “password expired” that their SMBClient library knows about. I’m looking forward to replacing the errno values with FileCore + LanMan custom error values so that the SMB client will be able to return useful error messages for all types of errors, instead of combining several different errors into a single EAUTH, ENOENT, etc..

For memory management, I plan to use both Mbuf Manager and ModMalloc, since Apple’s code uses BSD-style mbufs and also the kernel’s malloc(), wrapped with an SMBMALLOC() macro, which I can easily change to call ModMalloc. OmniLanManFS does have a couple of uses of malloc(), which could fail if the user program is out of memory (according to the docs), which appears to be a mistake.

Apple’s code routes all socket I/O through the code in smb_iod.c, which creates a thread to handle I/O when nothing else is going on. When a client sends an SMB request, it appears to lock the necessary mutexes and then directly writes/reads from the socket, rather than, say, adding a request to an internal queue and then waiting for the thread to do the request and then wake it up. I’m going to leave the mutex locking code in place and try to use the mutex_*() functions in SyncLib, even though RISC OS only has the beginnings of SMP support. At least if there are any reentrancy bugs, this should catch them.

Rather than a thread, I’m going to rely on the Internet_Event callback to wake up what’s currently a kernel thread to handle async data and remote socket closes whenever a client isn’t calling into smb_iod.c. There’s a userland ioctl-based interface that macOS uses to connect to servers and manipulate the kernel driver (by opening /dev/nsmb0), and it looks like I won’t need to include much of that code or the associated data structures because I can tie the Omni_ and fsentry_ SWIs directly to the functions that the filesystem calls when it receives the ioctls from userland. After mounting a share, the BSD-style vfsops and vnops function tables can similarly be removed and replaced with direct calls to the functions to do the work.

I can see why this code hasn’t been used by anyone besides Apple (yet!). The kernel portion was written for NetBSD and should be usable in any BSD-like system, but all the BSD’s have either removed or are talking about removing their old SMB1-only version of the client because everyone who cares about SMB is using Samba with a usermode filesystem interface. The ArcaOS (OS/2 descendant) SMB client and server is based on Samba 3.6. That means the only SMB implementations that matter are Microsoft’s various versions (which Apple’s 2017-era code has 12+ years of interop testing with), Samba, and Apple’s, plus any proprietary NAS implementations. Apple has added a few NeXT-isms like CFDictionary (formerly NSDictionary), but I don’t need to port any of that code.

In conclusion, I’m cautiously optimistic about my progress so far, and my chances of success with this porting project.

I hope I’m not messing up anyone’s forum browsing experience with my chapter-length posts! I’m trying to limit myself to one post per day in this thread when I can.

First, if there’s an existing RISC OS mDNS module that’s suitable, then SmbX can plug into it, and I will have to replace the current version’s calls to Apple’s Bonjour implementation of mDNS with something. As much as possible, I’d like to leverage whatever modules people are using these days that are well-respected.

After giving the matter a great deal of thought, I think I’ve managed to come up with a clever bidirectional Unicode to/from RISC OS filename conversion algorithm that includes some ideas from the current LanManFS encoding but will hopefully cause the fewest international incidents. :-)

It looks like I can pass all of the Latin-1 Supplement characters directly from SMB to FileCore, and vice versa, with the exception of some chars on each end that have to be escaped somehow, as well as ASCII " ", which apparently has to be converted to non-breaking space (0xa0) for RISC OS?

I haven’t seen anything in the RISC OS programmer manuals saying anything about having to use 0xa0 as the space character and not the normal ASCII 0×20, and I’d prefer to remove pass both the ASCII and “non-breaking” versions of " " through directly, if that works for the OS. Does anyone know anything about any convention to use 0xa0 non-breaking space in place of ASCII 0×20 as a way to avoid things breaking? If so, it seems like it should be documented with the other FileSwitch conventions.

So what about the chars that need to be escaped in translation, particularly anything above 0xff converting to RISC OS? For that, I propose using a slightly modified version of UTF-7 with the changes being to use something less common than “+” as the escape character (which I’d like to pass through directly, since “+” appears to be safe on both FileCore and SMB side), as well as the list of chars in 0×0000-0×00ff to escape on each end.

My criteria for an escape char to introduce the Base64 sequences, as described in RFC 2152 is that I’d like it to be printable on both ends for debugging purposes, in the ISO Latin-1 range, but not a character that anyone uses in practice in any locale. I came up with macron: “¯”, which is a common diacritical mark on top of vowels, but I don’t think anyone has ever used it by itself, in a filename, unless they were trying to do leetspeak for filenames on a demo disk. :-)

With this approach, Japanese (or any other non-Latin language) filenames on the SMB side would encode to a macron plus a string of Base64 characters, followed by nothing, the next non-Base64 character, or the “-” terminator, which is discarded, and then back to direct translation until another macron (escape) is seen. You wouldn’t be able to read the Base64 unless the RISC OS software handled Unicode and did its own conversion, but at least it wouldn’t be a long string of hex digits, so you’d have a far better chance of seeing the (encoded) filename in the Filer. Then when you copy that file or directory in the other direction, the original Unicode filename is preserved and viewable on your Mac, PC, Linux box, etc. and there’s no mojibake (fingers crossed).

You could even use the macron for your leet filenames, except wherever it’s followed by a Base64 char, in which case you’d need a “-” terminator in between to turn off the escaping (so macron immediately followed by hyphen would be valid, and discarded). I really like how UTF-7 avoids anything superfluous, e.g. you only need the “-” terminator to end a Base64 sequence when the next char in the string is a Base64 char.

To sum up, by using UTF-7, except with all of ISO Latin-1 minus the special characters on each end passed through directly, and using the printable-but-uncommon standalone U+00AF macron (not the combining macron code point, which is U+0304) as the escape character to introduce a Base64 sequence instead of the more-likely-in-a-filename “+”:

• passes through most of the chars directly, so “¡¦¤¥®±¿¼½¾×¢£ÆØ÷þÿ¬” and all of the accented Latin-1 chars will “just work” on the RISC OS side and be preserved bidirectionally
• escapes any length sequence of “special” chars, namely the ones that aren’t allowed by RISC OS or by SMB, depending on the direction of conversion, as a Base64 sequence, without any trailing “=”, with a trailing “-” to avoid ambiguity with following chars
• preserves all Unicode filenames from the SMB side using the modified UTF-7 based system described
• uses an uncommon ISO Latin-1 symbol (non-combining macron) to introduce Base64 sequences so that “+” can be passed directly
• does the usual RISC OS swapping of “.” and “/”, plus “#” and “?” (as a wildcard char).

If this idea works, then it could be useful for other apps, and as a way for UnixLib to give the appearance to ported software that the filesystem is really UTF-8 and allows the ASCII characters including special ones, which is how open(), fopen(), and friends should appear. The previous thread on this filename conversion topic got completely sidetracked and I wanted to propose a solution that should work for any of the languages, emoji, flags, and all the other Unicode filenames people are likely to have on their file servers that they want to work with using unmodified RISC OS apps (with a minimum filename size expansion in the escaping process).

BTW, the RISC OS UnicodeLib source has a UTF-7 encoder/decoder that I’m going to copy to the LanManFS folder and then modify as described, taking UTF-16 directly from SMB with no intermediate UTF-8 step in the middle. The reason UTF-7 use is discouraged is it’s a security nightmare if your parser is buggy, which happens to be the same reason SMB1 is discouraged. But I think the method I’ve described is as simple as it can possibly be, but no simpler.