SMB2/3 protocol
Rob McKay (8401) 4 posts |
Hi, I’ve successfully ported the libsmb2 library to RISC OS and it can be compiled using the DDE. The sources for it is currently available in my GitHub repos. Please see https://github.com/Rob-McKay/RiscOS-smb2/tree/main for details. I’m in the process of making the library a module. I know that there is a bounty for `More recent LanManFS protocols` and I think that this library achieves:
Other authentication options are available if/when the libraries libsmb2 uses for them are available on RISC OS. I’ve currently only tested the library with a couple of the example apps which come with the library connecting to a Windows 2019 server. I hope this is useful for the RISC OS community |
Andrew McCarthy (3688) 605 posts |
Rob, good to hear the news, sounds promising. :) |
Paul Sprangers (346) 524 posts |
This is excellent news that I have been waiting for for years. With every new update in Windows or in my NAS, I worry that I will no longer be able to connect to it. I know there is an experimental SMB2/3 client for RISC OS, but that’s a stand-alone application that doesn’t merge with the filer. So yes, this is great news. |
Chris Mahoney (1684) 2165 posts |
Excellent news. I’ve never been able to get RISC OS to talk to my Mac and I believe that’s down to SMB versions, so it sounds like this is a big step in the right direction :) |
Clive Semmens (2335) 3276 posts |
With the latest version of SMB, even my old MacBookPro is unable to see the M1 MacMini. At least the the MacMini can see the MacBookPro, and they can both see the older (pre-ARM) MacMini (which is now serving as a file server). |
Jake Hamby (8915) 21 posts |
Excellent news! I’m not trying to steal your thunder, but to avoid any confusion with similarly-named threads, I hope you won’t mind if I share my own recent plans and progress towards claiming the bounty for this highly-anticipated feature. I’ve started a private email thread with ROOL to discuss the specifics of the SMB2 work items, including some that go beyond what’s listed on the bounty page. What I wanted to share so far is a link to the SMB1 fixes that I’ve been committing to my public GitHub repo to familiarize myself with the code, and what’s needed to get SMB1 working properly. https://github.com/jhamby/RiscOS-OmniLanManFS/
And here are the three SMB1 bugs I’ve found that make OmniLanManFS quite a bit more usable once fixed:
This leads me to observe that Filer directory queries should benefit by increasing the client buffer size from 512 bytes to perhaps 4KB, to have a better chance to get the full 18 entries it requested, although the speedup would be minimal assuming the entries are being copied out of a cached copy of the entire directory. Unfortunately, LanManFS is currently broken whenever the filename lengths add up to larger than the client buffer, because it’s losing all the entries that didn’t fit the buffer after being received from the SMB server. If you have any comments, suggestions, questions, or feedback, I’d definitely appreciate it. At my current rate of progress, and considering my professional expertise with these types of protocols, I’m aiming to collect the bounty some time in the next 4-6 weeks, depending on various factors. In order for that to happen, I’ll have to put in quite a few hours of work writing new code, but my intention is to earn every penny by making the donor community 100% satisfied. As for the SMB1-related bug fixes, I’d love to see them get into RISC OS 5.30, along with adding direct TCP and removing the obsolete NetBEUI transport layer. Those release engineering decisions aren’t up to me, but of course I’d love to see 5.30 ship with my changes so that SMB1 works as well as it did in the past, rather than being completely broken with lengthy timeouts. |
Martin Avison (27) 1494 posts |
@Jake: All sounds good to me … but I can see no commits in the last month on your github link – but I could be doing something wrong! |
Simon Willcocks (1499) 513 posts |
Look at the other branches. jhamby_smb2. |
Rob McKay (8401) 4 posts |
@Jake: That sounds good. I did consider trying to fix the CIFS (SMB1) routines but opted to go down the libsmb2 route for the following reasons:
The downside is that it is LGPL licensed which is why I am wrapping it in a module and the modifying OmniLanManFS to call that module. I would have preferred to split OmniLanManFS into two parts, the filesystem and the Omni Client interface as is done for nfs so that the code is cleaner. I’m not interested in the bounty but perhaps we should pool our resources and work together. |
Paul Sprangers (346) 524 posts |
@Rob: That sounds even better! |
André Timmermans (100) 655 posts |
@Jake, not sure if it is work trying to maintain the current version, apart for the fact that it is SMB1 only, it is not very reliable as soon as you do anything outside standard file transfers. Check these topics: |
Richard Walker (2090) 431 posts |
I’ve nothing useful to add, but just wanted to say: this thread is one of the best I’ve seen in a while. There’s a great effort in porting an active library to RISC OS, and someone seriously looking into a bounty. And in the case of the latter, source code – which is totally public/open. I take my hat off to you both, and think the openness of both approaches is wonderful. I had been meaning to ask, with LanMan98 being ‘open sourced’, if there was anything in there which was useful. I remember that back in the day, it offered long filenames and LanManFS didn’t. But LanManFS gained those a while back, so I’m not sure what LanMan98 actually brings to the party. I remember browsing the LanManFS source code once, as there was some strangeness with files named ‘/[something]’ (they became invisible). I mentioned my issue, and before I got as far as building it, Colin was in there with a fix. Not my area, obviously! |
Jake Hamby (8915) 21 posts |
@André, thanks for the links! @Rob, yes, I’m happy to work with you on this, especially since you’ve investigated areas that I haven’t yet because I’ve been approaching the task from the opposite direction (familiarizing myself thoroughly with how the current client code operates). Here’s what I’ve discovered about OmniLanManFS: it’s overgrown with backwards-compatibility code that I’m currently in the process of stripping out altogether, now that I know that only the “NT LM 0.12” dialect of SMB1 (aka NTLM, aka CIFS) will ever be used. I’ve already finished taking out the NetBEUI transport as well as the block read/write raw transfer callbacks that also aren’t needed. By keeping only the “-DLONGNAMES” code path and removing all dialects except DIALECT_NTLM (currently “DIALECT_NT”), then we have a minimal working subset of SMB1 with NTLMv2 challenge/response authentication, Unicode filename support, etc. to build on top of. I expect my pruned-down version of SMB1 to be 1/2 the size and much easier to follow, as well as more reliable. More importantly, it’ll be a solid foundation for building SMB2/3 protocol support on top of. The reason I’m not super enthusiastic about using libsmb2 is first of all the incompatible license, because the most useful part would be the header files and utility functions for the SMB message and response structs, and the whole thing really should fit into a ROM. LGPL makes that library useless, sadly, but Samba’s libsmb has the same issue and is much worse in terms of complexity. What’s really going to be useful for this project are the “other libraries”, specifically for Kerberos and cryptography. There’s no Kerberos domain login support in the current LanManFS, and that appears to be a requirement for the bounty work items. Also, I assume it makes the SMB3 signing and encryption more useful because you’d have client computer certs and domain user certs to check the signatures against. Or am I misunderstanding the benefit of signing and encryption? It seems like signing would be much less useful for a home workgroup, other than making replay attacks harder, and who’s attacking you with those on a home network? Fortunately, I have both Samba 4.15.x and Windows Server 2022 180-day trial ISOs to create test domains. I’m just curious if there will be any RISC OS users who have domain single-sign-on at home or your office, perhaps with Samba 4.x, perhaps for a heterogeneous Windows / Linux / macOS / BSD environment? For this part of the project, I’m the most interested in learning about the subset of users who have any interest whatsoever in Kerberos domain logins, presumably because you’re already using Active Directory (Samba’s or Microsoft’s). The good news is that despite all of the fearmongering about SMB1 being insecure and unsafe, the concepts are similar enough to SMB2, and the number of RISC OS users who also have retro versions of OS X (PowerPC perhaps?) or Windows XP they’d like to use is probably greater than the number of people interested in Kerberos, and definitely non-zero. After stripping out the 8.3 filename and MS-DOS / OS/2 / NT3.5 / NT4.0 compatibility code, it’ll be much more maintainable than it is now. I’m also cleaning up the socket handling in preparation for SMB2/3. I should have a relatively large commit to my SMB2 branch of my fork of OmniLanManFS in the next day or two with all of the old code removal, socket refactoring, directory query caching, etc. that will be necessary for SMB2. BTW, one thing that isn’t listed on the bounty page but I consider of great importance is that the client should use mDNS DNS-SD (aka Bonjour) to find local servers and not regular DNS with the system resolver. Fortunately, I have prior experience with mDNS clients on a variety of OS’s, and the good news is that it works very similarly to the NetBIOS Name Server code that’s already in the client. I just have to copy-and-paste those blocks, change the port number, and put in different strings to query for “smb.tcp” parse the mDNS response. Perhaps surprisingly, you can add mDNS client support to user apps without any special permissions (Windows apps may pop up a firewall notification), and it’s simple enough that it doesn’t really need a special library. I was planning to copy-and-paste and modify the relevant sections of a public-domain mDNS client I found on GitHub: https://github.com/mjansson/mdns So that’s mDNS, Kerberos, message signing and encryption, and one more thing, which is message sizes. As I mentioned, the “READ_RAW” and “WRITE_RAW” commands don’t actually work because you’re supposed to use “AndX” commands instead, which can be chained on the client side, but don’t have to be (I’d forgotten about that part). Now that I’ve made the executive decision to remove all vestiges of pre-Win2K versions of SMB1 support, I can assume the presence of the AndX variants of the read and write commands, which let you pass a higher maximum number of bytes than the normal message size that’s been negotiated. Long story short, after I finish removing the backwards-compatibility code and fix the sockets and directory handling, but before I work on adding mDNS / DNS name lookup and then the SMB2/3 protocol (which, as I alluded to, should plug nicely into the RISC OS interfacing portion of LanManFS quite easily, however we decide to implement the SMB message parsing), I’ll take a quick diversion into adding support for CAP_LARGE_READX and CAP_LARGE_WRITEX capabilities, which need to be negotiated between the client and server, along with passing a larger max receive buffer size (Samba appears to go no larger than 63 KiB, presumably to avoid going over 65535 bytes for the full packet), and then see how much of a performance boost that provides compared to 4KB messages. Theoretically, SMB supports direct TCP packets up to 128 KiB – 1 byte, and handling these sorts of limits is the sort of mess that using an existing SMB client library would definitely help with. However, now that I’ve already spent far too many hours studying the daunting CIFS, SMB1, SMB2, and related specs, I think it makes more sense for me to write the RISC OS version from scratch, if only because the current client uses mbufs so heavily and I have no confidence that a cross-platform library would function well in such a restricted environment as far as memory allocation, stack size, being called from supervisor mode, etc.. Once I’ve finished preparing the way for SMB2/3 client support, I’m definitely going to need some help from the community in terms of figuring out how to implement and load sub-modules for Kerberos, and any other optional feature that’s big and doesn’t need to be in ROM. That way the ROM version will work for booting and 99.9% of RISC OS users, including encryption and code signing (those algorithms aren’t too big by themselves), and only people using domain logins will load the Kerberos module. Maybe there can be code reuse with other RISC OS programs that support Kerberos (like SSH?). |
Jake Hamby (8915) 21 posts |
I did eventually find exactly one suitably-licensed SMB2/3 client library, from Apple, via this post: https://news.ycombinator.com/item?id=16087969 Apple also keeps a GitHub repo here. https://github.com/apple-oss-distributions/smb/ I thought I’d read that Apple had stopped using their homegrown SMB client/server code in favor of Samba, but Samba is GPLv3, which is strictly forbidden by Apple (that’s why they were stuck with such old versions of bash and GCC/binutils). So Apple is still using their own implementation and people have found exploits in it: https://blog.talosintelligence.com/vuln-spotlight-smb-mac-deep-dive/ For those who aren’t aware, you can’t just mix-and-match GPL or LGPL code with Apache / BSD / similar licenses. The GPL license “infects” the whole project if you don’t strictly follow the rules, and many companies refuse to deal with it at all, particularly for something ROMmable, like RISC OS. libsmb2 is LGPL 2.1 and not the hated LGPL 3, and if there are already LGPL libraries “tainting” the ROM, then I have no other objection to linking it into the OmniLanManFS module statically. Now that I know Apple has an open-source BSD-licensed SMB1/2/3 library that they use for their SMB client and server, it seems like a no-brainer to start from Apple’s SMB library.
Well, I’m sold on not writing a bunch of boilerplate code for the SMB2/3 and signing, Kerberos, encryption, etc.. I just didn’t want to use libsmb2 for the inverse of the reasons why I do now want to use Apple’s “SmbX”. |
Rob McKay (8401) 4 posts |
I’ve looked at the Apple SMB repo and it’s from approx 2013 so is probably quite out of date and seems to confirm that they’re no longer using it. The whole reason that I’m using libsmb as a separate module is to allow dynamic linking against it. That way a user can load another version of it which they have built from source or obtained from elsewhere to fulfil the terms of LGPL 2.1 The main reason I chose libsmb is that it has an active community using and maintaining it which reduces the maintenance burden from the whole implementation down to the RISC OS specific bits (like the module interface). |
Steve Pampling (1551) 8170 posts |
Large percentages of the people round here are very aware of the general nastiness and “infective” nature of GPL of any version. Many years ago (Iyonix era 2002) someone made the mistake of using some GPL for part of the HAL. Much upset and aggravation ensued. So, yes, keep things GPL free. Who wants a licence that doesn’t agree with itself anyway? |
Colin (478) 2433 posts |
The module I started is separate to the lanmanfs module it connects using smb2/3. It uses mbedtls from the risc os sources. I’ll bundle it up sometime today. If it’s useful feel free to use it. |
Grahame Parish (436) 481 posts |
Just a quick post to mention that I have a mixed home office network with RISC OS, Windows and Linux. I’m currently in a domain with AD, although I have Linux using an LDAP server and will probably migrate everything to LDAP when I my Action Pack Windows licences expire. |
Colin (478) 2433 posts |
The sources for my module is available here – SmbFSSources.zip. |
Rick Murray (539) 13840 posts |
Who wants a licence that runs on fuzzy ideology and is therefore poorly defined and utterly unfit for purpose. With respect to the GPL v2, the whole thing about static and dynamic linking etc is some rubbish that was discussed at length in the FAQ. The licence itself does not make any such distinction, only talking about whether or not the code is a part of a whole. This was “fixed” in v3. Relating to this, quote from the same FAQ: What constitutes combining two parts into one program? This is a legal question, which ultimately judges will decide. Of course, by failing to specify exactly what they mean, it might not quite turn out the way they want it to: https://thehftguy.com/2021/08/30/french-appeal-court-affirms-decision-that-copyright-claims-on-gpl-are-invalid-must-be-enforced-via-contractual-dispute/
If you really support “open source”, you wouldn’t touch it with a bargepole. Thanks, but no thanks. |
Dave Higton (1515) 3525 posts |
I’m delighted to be reading this discussion. It looks very promising in terms of results. I’d like to offer my encouragement to all of you who do anything towards creating a working SMB2/3 client for RISC OS. Sadly I’m unlikely to be able to do anything concrete towards the programming. Any help I can offer is most likely to be in testing; I have an OpenMediaVault installation on a RasPi, and a WD MyCloud, both of which of course support SMB2 and 3. What communication medium are you going to use for dev? |
Rick Murray (539) 13840 posts |
Just reading though this… Would it perhaps be beneficial to place the mDNS part in a separate module? I know IPP relies upon mDNS, as does this, and using Service Browser on my phone all sorts of things can turn up. Therefore, perhaps there’s some value in having a general solution for tracking (and sending?) mDNS/Bonjour messages within RISC OS rather than everything doing it for itself? |
Dave Higton (1515) 3525 posts |
An mDNS module apparently exists, according to Gerph, but whether it’s generally available is something that I’ve never followed up. |
Jake Hamby (8915) 21 posts |
Here’s what I’ve learned so far. It’s mostly good news. First, Apple did stop releasing sources for their SMB client after OS X Yosemite 10.10.5 (released July 2017), which is the “smb-759.40.1” code drop (the “main” branch on GitHub only has “smb-759.0” but the .tar.gz archive of 759.40.1 is available in “releases”). It supports SMB 1.0 through 3.0. I believe Apple is still using this code, but they stopped releasing the source code after OS X 10.10.5, as is their right. The original code was released under the BSD 4-clause license, which is slightly annoying because you have to find a place in your distribution to credit the author, in this case “Copyright © 2000-2001, Boris Popov”. On macOS, there’s a massive “/Library/Documentation/Acknowledgements.rtf”, which includes the Boris Popov copyright for the “(smb)” package (one reason why I’m sure the code is still being used). On the RISC OS “Nut Pi” microSD that I’m using (purchased in July 2021, so I need to acquire an updated DDE to be able to build the current RISC OS source tree), there’s a I can see two major challenges with this code, and then I’ll describe my current progress. The first issue is Kerberos authentication. Apple uses Mach IPC ports extensively, and this is no exception: the kernel code in For the regular NTLMv2 authentication, the code is self-contained. The second challenge has to do with enumerating shares and printers once connected to a server and its I’ve tried to generate client stubs with https://www.thehacker.recipes/ad/recon/ms-rpc The most full-featured open-source implementation of these MS-RPC interfaces would be Samba’s, but I can’t reuse their work for two reasons, the first being GPLv3 “contamination”, but also their generated code is tied to Samba-specific libraries (talloc, tdb, etc.), which would be more work to adapt than figuring out how to get I also looked briefly at Wine’s “widl”, which can generate Windows-style client and server stubs, like Microsoft’s MIDL compiler, and that might be a reasonable alternative if for some reason I can’t write .idl files compatible with Setting aside those two challenges for now, I’m currently stripping out the original SMB1 implementation from OmniLanManFS, leaving only the RISC OS interfaces and stubs to call into the new code, which I’ll add as a separate commit. I discovered that the error messages should be implemented more like SDFS and SCSIFS, with an I was a bit confused by the meaning of the Apple’s code (in For memory management, I plan to use both Mbuf Manager and ModMalloc, since Apple’s code uses BSD-style mbufs and also the kernel’s malloc(), wrapped with an Apple’s code routes all socket I/O through the code in Rather than a thread, I’m going to rely on the Internet_Event callback to wake up what’s currently a kernel thread to handle async data and remote socket closes whenever a client isn’t calling into I can see why this code hasn’t been used by anyone besides Apple (yet!). The kernel portion was written for NetBSD and should be usable in any BSD-like system, but all the BSD’s have either removed or are talking about removing their old SMB1-only version of the client because everyone who cares about SMB is using Samba with a usermode filesystem interface. The ArcaOS (OS/2 descendant) SMB client and server is based on Samba 3.6. That means the only SMB implementations that matter are Microsoft’s various versions (which Apple’s 2017-era code has 12+ years of interop testing with), Samba, and Apple’s, plus any proprietary NAS implementations. Apple has added a few NeXT-isms like CFDictionary (formerly NSDictionary), but I don’t need to port any of that code. In conclusion, I’m cautiously optimistic about my progress so far, and my chances of success with this porting project. |
Jake Hamby (8915) 21 posts |
I hope I’m not messing up anyone’s forum browsing experience with my chapter-length posts! I’m trying to limit myself to one post per day in this thread when I can. First, if there’s an existing RISC OS mDNS module that’s suitable, then SmbX can plug into it, and I will have to replace the current version’s calls to Apple’s Bonjour implementation of mDNS with something. As much as possible, I’d like to leverage whatever modules people are using these days that are well-respected. After giving the matter a great deal of thought, I think I’ve managed to come up with a clever bidirectional Unicode to/from RISC OS filename conversion algorithm that includes some ideas from the current LanManFS encoding but will hopefully cause the fewest international incidents. :-) It looks like I can pass all of the Latin-1 Supplement characters directly from SMB to FileCore, and vice versa, with the exception of some chars on each end that have to be escaped somehow, as well as ASCII " ", which apparently has to be converted to non-breaking space (0xa0) for RISC OS? I haven’t seen anything in the RISC OS programmer manuals saying anything about having to use 0xa0 as the space character and not the normal ASCII 0×20, and I’d prefer to remove pass both the ASCII and “non-breaking” versions of " " through directly, if that works for the OS. Does anyone know anything about any convention to use 0xa0 non-breaking space in place of ASCII 0×20 as a way to avoid things breaking? If so, it seems like it should be documented with the other FileSwitch conventions. So what about the chars that need to be escaped in translation, particularly anything above 0xff converting to RISC OS? For that, I propose using a slightly modified version of UTF-7 with the changes being to use something less common than “+” as the escape character (which I’d like to pass through directly, since “+” appears to be safe on both FileCore and SMB side), as well as the list of chars in 0×0000-0×00ff to escape on each end. My criteria for an escape char to introduce the Base64 sequences, as described in RFC 2152 is that I’d like it to be printable on both ends for debugging purposes, in the ISO Latin-1 range, but not a character that anyone uses in practice in any locale. I came up with macron: “¯”, which is a common diacritical mark on top of vowels, but I don’t think anyone has ever used it by itself, in a filename, unless they were trying to do leetspeak for filenames on a demo disk. :-) With this approach, Japanese (or any other non-Latin language) filenames on the SMB side would encode to a macron plus a string of Base64 characters, followed by nothing, the next non-Base64 character, or the “-” terminator, which is discarded, and then back to direct translation until another macron (escape) is seen. You wouldn’t be able to read the Base64 unless the RISC OS software handled Unicode and did its own conversion, but at least it wouldn’t be a long string of hex digits, so you’d have a far better chance of seeing the (encoded) filename in the Filer. Then when you copy that file or directory in the other direction, the original Unicode filename is preserved and viewable on your Mac, PC, Linux box, etc. and there’s no mojibake (fingers crossed). You could even use the macron for your leet filenames, except wherever it’s followed by a Base64 char, in which case you’d need a “-” terminator in between to turn off the escaping (so macron immediately followed by hyphen would be valid, and discarded). I really like how UTF-7 avoids anything superfluous, e.g. you only need the “-” terminator to end a Base64 sequence when the next char in the string is a Base64 char. To sum up, by using UTF-7, except with all of ISO Latin-1 minus the special characters on each end passed through directly, and using the printable-but-uncommon standalone U+00AF macron (not the combining macron code point, which is U+0304) as the escape character to introduce a Base64 sequence instead of the more-likely-in-a-filename “+”:
If this idea works, then it could be useful for other apps, and as a way for UnixLib to give the appearance to ported software that the filesystem is really UTF-8 and allows the ASCII characters including special ones, which is how open(), fopen(), and friends should appear. The previous thread on this filename conversion topic got completely sidetracked and I wanted to propose a solution that should work for any of the languages, emoji, flags, and all the other Unicode filenames people are likely to have on their file servers that they want to work with using unmodified RISC OS apps (with a minimum filename size expansion in the escaping process). BTW, the RISC OS UnicodeLib source has a UTF-7 encoder/decoder that I’m going to copy to the LanManFS folder and then modify as described, taking UTF-16 directly from SMB with no intermediate UTF-8 step in the middle. The reason UTF-7 use is discouraged is it’s a security nightmare if your parser is buggy, which happens to be the same reason SMB1 is discouraged. But I think the method I’ve described is as simple as it can possibly be, but no simpler. |