SecureSockets Bounty

I think the first step would be to define what the bounty is. For example what I have in mind does not necessarily fit what other people have in mind and far exceeds the scope of just SSL/TLS

SSH as both client and server would be good; as would be the ability to work with server fingerprinting (akin to PuTTY) instead of the current SSL setup of (mostly) blindly trusting third party certificates.

Unfortunately SSH and SSL are quite different. But I would agree we need to use CACertificates, it’s a fundamental part of SSL.
I’d also like to see a revamp of ssh and a top notch terminal implementation – like Mobaxterm for Windows.
We already have an ssh client, which I use with putty – it’s not awesome, but it does the job for now…
But SSH would have to be out of scope for this particular bounty I would suggest.

Following on from SSH would be SFTP… All the plain unencrypted protocols just don’t cut it these days¹.

¹ By way of example, I can only access my website using WinSCP as it is SFTP with a freakishly long line-noise style password, and then secondary authentication using a key pair known only to the server and my machine. Paranoid? Maybe. But spend long enough reading The Register, you come to understand that security measures are important.

We already have an ssh client, which I use with putty – it’s not awesome, but it does the job for now…

You have Putty on RISC OS?
I use Nettle, but terminal emulation is really not perfect.
And of Course, there is the SFTP problem.

I’m agree, that a SecureSockets Bounty should just address the SSL problem, both as server or client.
So it could be used for SSH clients, web servers, or simply to fetch an HTTPS web page.

You should be able to use sftp and ssh using the openssh package in packman.
Two factor authentication isn’t paranoid, it’s now the norm.
FYI As part of the next PCI standard all admin access to boxes will have to have two factor authentication.
If you are using ssh keys, I would personally recommend the ecdsa keys – they are far shorter and stronger.
Personally I prefer using openvpn (with password and keys) and ssh.
If you are using just SFTP, it’s better to have a separate daemon e.g. proftpd.

Start Rant…so ignore if you wish!
There are loads of things missing from RISC OS, git would be number one.
It’s all well and good porting software, but at the end of the day, if RISC OS is going to be the same as linux people may as well use linux.
I think getting the basics of socket and protocol integration into proper RISC OS modules is the only way to go.
Especially as that enables the BASIC programmers to make use of all of these functions relatively easily.
Whilst I may not like BASIC at all now, its where I started to learn programming and it’s one of the things I think that makes RISC OS really stand out.
The weather application, just released is a classic example.
The fact it uses wget is awful, but if you look at the size of the code compared to my weatheruk, it’s staggering, really beautifully done I think.
The whole BASIC program is around the same length of a couple of the functions in mine.
All because there is no decent way of using http/https in RISC OS.
It really shouldn’t be that hard for people to use native RISC OS functions to do what is basic functionality nowadays.
End rant

If you are using just SFTP, it’s better to have a separate daemon e.g. proftpd.

IMHO, it’s not really a good idea to multiply entry points. I use sftp server from ssh for the main account, and restricts the others to ‘sftp only + root = userdir’.

All because there is no decent way of using http/https in RISC OS.

Yeah, that’s a pity, since it would be very fun to convert HTTP services to RISC OS applications. Oh, wait, that’s exactly what 80% of Android apps do :)

There is a nice set of modules which were part of Acorn’s Browse called URL and AcornHTTP which simplify HTTP fetching. But there is currently no HTTPS equivalent. We use the AcornHTTP module in RiscOSM — with HTTPS supported we could add one or two more nice features. So I would support a bounty that got SecureSockets updated and also introduced an HTTPS equivalent of AcornHTTP. I think Justin Fletcher did this for ROL using the curl library: I read it in one of his rambles.

My aims would be to have a pluggable module library.
There is a lot more than http(s) to support. (imap/pop/nntp/ftp)
Also http/1.1 and queued async downloading for me would be a bare minimum.
Then there is support of XML, JSON etc. so having a framework to allow pluggable modules seems to me to be the only way.

For me, implementing any of these individually is relatively easy. The big problem comes when you have to integrate it into something.
The code quickly becomes unmanagable – hence the “pluggable” module idea.
That way each component can be managed separately.
To give you an idea, I was nearly there with a Jabber client – it connected to a Jabber server and I could send messages to/from RISC OS to Google Talk (via a third party jabber server, as there was no TLS).
The multi-tasking became a nightmare to debug with sockets when I combined it with all the other stuff so I gave up.

There is a lot more than http(s) to support. (imap/pop/nntp/ftp)

Okay, can somebody confirm something for me. I know that SMTP, POP and HTTP are basically simplified telnet-like¹ data exchanges because I have performed both manually from a terminal by connecting to the intended port and issuing the relevant commands.

Is SSH, HTTPS, encrypted POP, IMAP, etc basically the same thing with the addition of an encrypted channel in between? In other words, if I had an SSL module I could open my socket to the https port and then start up the SSL stuff, then issue the exact same commands as I would on HTTP?
Does this hold true for other basic encrypted channel methods (such as POP or IMAP)?

I am asking this because what concerns me about the post is that the idea says There is a lot more than http(s) to support. (imap/pop/nntp/ftp) and I can’t help but wonder if a “secure socket” service should concentrate purely on doing that, and not getting involved in the specifics of any particular protocol unless such involvement is unavoidable.

Also http/1.1 and queued async downloading for me would be a bare minimum.

Would be nice to have a built in wget (or a simplified version, at least) but this should be a separate thing and not part of the base networking protocols.

The multi-tasking became a nightmare to debug with sockets when I combined it with all the other stuff so I gave up.

I’m not familiar with Jabber, but something that concerns me is that RISC OS does lack sensible debugging tools that work in a complex multitasking environment where one doesn’t have the luxury of freezing the entire machine in order to examine something.
I use a combination of DADebug and writing to the serial port (for viewing on a PC), but it does make an annoying thankless task (such as debugging is) all the more annoying and thankless.

¹ Telnet LIKE in that it is an exchange of (usually) readable information in a predefined form; there are plenty of guides of how to access a POP3 server using a terminal…

SSH is completely different to the others.
But there are two ways of “doing” SSL/TLS, one way is the “old” way i.e. an SSL/TLS tunnel, the other way is using STARTTLS.
i.e. you start in plain text, then say STARTTLS and then you enter encrypted mode.

My point was not that we should support ALL protocols etc. just that we should bear it in mind, so protocols can be added.
e.g. if we have SSL as a protocol and HTTP then we can do https.
If we can support IMAP then we have IMAPS automatically.
If we can support STARTTLS then we have IMAP with TLS etc.
If we add FTP then we have FTPS automatically etc. etc.

The first thing for me is to support SSL/TLS as this is the most crucial to ALL methods.
Then I’m going to add HTTP once this is done and carry on.
If people want to just use the SSL part, then that is fine – if they want to use just the HTTP part and do their own SSL – that would be fine too.
Whether this is what other people want is another question.

I’ve managed to get aes.o created – small steps, but it’s nice to see!
Was quite easy (norcroft) to be honest – here’s hoping the rest is easy.

I’ll provide a link to what I’ve done once I can at least get the library compiling.
My Makefile skills are “interesting” to say the least!

If anyone is interested my “interesting” makefile and my attempts so far is at:

https://www.paymentlabs.com/mbedtls_r.zip

I’ve managed to get most of the libraries compiled. Looks like there are some includes required that I’ve missed for types.h and others.
But it’s a good start, threading compiled – which looking at the name I assume wouldn’t!

Got to say it’s looking very promising at getting at least the ability to use the library on RISC OS fairly soon.

I should really bother to read up on Makefiles, it would save me a lot of effort.
But thanks to a great keyboard from Elesar and a nice soundbar it’s not that painful!

I’m on a train with patchy wifi so I can’t look at your archive, but mbedtls was pretty painless to crosscompile for me – it’s in the autobuilder here. I do agree it’s a much more sane way forward than OpenSSL.

To revise what I said previously, after a discussion on the GCCSDK list it turns out that using shared libraries from supervisor mode is somewhat tricky. So if we want to implement the SecureSockets API, reluctantly we’re looking at running mbedTLS in supervisor mode as a module. Ideally application level programs (C at least) would use the shared library rather than the module, for the added security and reliability that provides. It would also be good to gain a way to do that from BASIC (there may be trickiness here too).

In terms of the SecureSockets API, the bits that require work are:

• Reimplementing the API in a sensible way. For instance, I suspect SecureSockets just ignores certificate mismatches, which you really cannot afford to do in this day and age.
• Make a module that can be easily rebuilt when the underlying library changes. That suggests taking the existing library and writing module code that uses the library’s API, rather than changing the library. That way we can build the library and module code and link them together into a new SecureSockets module. When the library changes, the autobuilder can pull in new code and rebuild, rather than requiring a human to re-apply the changes. That means we can easily keep up with security patches.

Having a module that’s easily updatable fixes one of the vulnerabilities, that of the code being ancient. It still means the code runs with all the privileges in the world, but the RISC OS API is already fundamentally broken in that respect, so it won’t be easily fixed.

From mine and a Norcroft/DDE perspective:
I’ve managed to get most of the library compiled, the only specifics that don’t work are the net library (not a surprise) and another library that uses SIGPIPE – not sure why.
But that’s not a surprise and of no real consequence as those portions will need a rewrite to support any level of multitasking and are the parts that I need to abstract into a separate module anyway (the socket handling module) as it needs to read data during wimp_poll to get any level of reasonable performance (unless I’m missing something that I can use IRQ’s or something to get a lot finer timer than 100 per second) [17k*100 is a dismal download speed, even 1000 times isn’t great]

I’ve not yet looked at the code, so this is just a guess at this point – but I think a reasonable one.

As regards this thread, I’m going to duck out.
Largely because I think for what I want, using GCCSDK is not going to work for me.
I’m just getting my head around Modules and am just about making progress with the DDE.
Then there is the fact that I’ve ploughed a lot of time, and a bit of money into it too!
Also, I’m assuming a straight “port” would just hang on issues, or wouldn’t be smooth – so a rewrite of some parts will be required for me either way.
However I do think it’s (GCCSDK) probably the best bet to get something (aka securesockets) working quickly.

Not sure how using shared libraries provides any more security or reliability – on the whole for RISC OS I’d say less on both counts personally – but hey!
I would totally agree with abstracting the library from the module though, the fact that mbedtls is a clean build is great.

If anyone is interested in helping me out though, I’d be more than grateful.
I’ll keep posting updates on any progress.
I’ve updated my zip with my latest attempts…If anyone wants a chuckle.
Again, yes it needs a Makefile rather than a huge obey file.

But that’s not a surprise and of no real consequence as those portions will need a rewrite to support any level of multitasking and are the parts that I need to abstract into a separate module anyway (the socket handling module) as it needs to read data during wimp_poll to get any level of reasonable performance (unless I’m missing something that I can use IRQ’s or something to get a lot finer timer than 100 per second) [17k*100 is a dismal download speed, even 1000 times isn’t great]

The Internet event is your friend. For code running as a module you basically want to enable event generation for the event, claim EventV, and then whenever “something interesting” happens to the socket (RX data arrived, TX space available, error) you’ll receive a call for that socket. Within the event handler you then register a callback so that you can process the data as soon as the OS returns to the foreground (although it may be possible to directly interact with the socket from within the event handler in some cases – I’m not sure what the rules are)

I recently implemented support for this in vncserver in order to improve the TX throughput, so you could take a look at that for a simple example (sockevent.c/.h and the EventV handler in mcode.s)

I would have thought that the huge benefit of using GCSSDK and the autobulder, coupled with shared libraries, is that we would be more like to get timely fixes which ‘just work’ in all applications.

If each person does their own thing, through static linking and/or heavily modified bespoke library builds, then we risk delays and inconsistencies. For example, Messenger Pro might have a 3 month old SSL codebase, but Store might be 2 weeks old.

Modules are effectively shared libraries. That’s really the point of them

Also, I’m assuming a straight “port” would just hang on issues, or wouldn’t be smooth – so a rewrite of some parts will be required for me either way.

I suspect we may run into issues porting anything complicated from a POSIX style system due to differences in how the operating systems work internally.

as it needs to read data during wimp_poll to get any level of reasonable performance

If I was going to write something that needed to retrieve data from a socket quickly, I would do the following:

• Claim a block of RMA
• Assemble some code into it that reads data from the socket into a buffer in a Dynamic Area
• Sort out some way of starting this code, either from TickerV or an event

The Wimp part will retrieve data from the DA, independently of the lower level socket reading code. You might ask why I don’t make a module. It isn’t intended to be a module, just a bit of code that can be called regardless of what task is currently running (doesn’t get paged out).

Not sure how using shared libraries provides any more security or reliability – on the whole for RISC OS I’d say less on both counts personally – but hey!

I’ve not looked as I have zero interest in GCCSDK’s shared libraries (doesn’t work with the DDE), however a statement like that would suggest to me that shared library code most likely works in USR mode; unlike every SWI which is entered in SVC mode.
Okay, it means practically diddly/squat in RISC OS, but in some mythical future day when RISC OS attempts to enforce some sort of memory protection, having everything that isn’t a straight app running in a privileged mode isn’t the best of starts…

is that we would be more like to get timely fixes which ‘just work’ in all applications.

ALL applications?

If each person does their own thing, through static linking and/or heavily modified bespoke library builds, then we risk delays and inconsistencies.

I think this depends greatly on whether or not the code is open. It’s nice that the SSL module code is available, it’s just a shame it is a decade too late. The important thing for any new implementation is to touch the original source code as little as possible (break rules if need be to leave the original code alone) and document every change made to the original code.
Why? Because these changes will need to be performed for every single incarnation built.

Back when these things were young, people actually used 40 bit SSL, and Google didn’t exist, it probably made sense to make an SSL module and, well, that’s that.

Now? Now it’s a different game where an outdated bit of encryption software is worse than none at all, to the ability to merge in updated code (to patch flaws) needs to be one of the main design considerations.

Modules are effectively shared libraries. That’s really the point of them.

Remember that those from a Linux background will be looking at it from the DLL Hell perspective, namely the ability to have lots of independent versions of the ame library, like MehLib 1.5.6.4 for this program and MehLib 1.6.1.1 for that program.

Don’t look at me, it it was my ABI, I’d consider MehLib 1.6.x to be broken if it couldn’t work with software expecting MehLib 1.5.x unless there were insurmountable reasons preventing it. I mean, you don’t see the latest CLib baulking at APCS-R software, on platforms capable of running such.

I tried the internet event, looks like I misunderstood it’s implementation. That explains the disaster I had.
It seemed to go well…until I tried interacting directly that didn’t go well.
I’ll take a good look at vncserver, thanks Jeffrey

OK, now I’m seriously impressed. I theorised that using the event_handler or IRQ’s would be the best way, no idea you could combine the two! Which seems to be what dark magic you are up too from a quick glance.
I did read the s.mcode as “Implemented in assembler so no-one can understand it”, I don’t have a clue what the actual assembler functions do (or care).
Easy to follow assembler that you don’t need to understand? wha? Again impressed.

Malcolm,

If you want to carry on using DDE, don’t let me stop you. I’m happy to help out with the build system side to make sure we have something that can be easily rebuilt using whatever tools (DDE, GCCSDK, Jenkins, whatever). But at this stage it’s more important to have some code that does something useful than worrying about build systems (mbedtls is relatively germane in that respect, so the build system isn’t the barrier that it is in other projects).

Your progress so far looks fine. I haven’t looked in the code in great detail but I think that threading will probably become a problem when you try and link – it’s calling pthread functions which aren’t there. UnixLib supports that in user-mode programs using callbacks, but not in modules. SCL has never heard of the concept of threads.

Unfortunately a lot of porting existing codebases to RISC OS stumbles on scheduling. Not much pre-existing application code (with the exception perhaps of some very asynchronous Javascript) expects an event-driven/interrupt-driven environment – it expects to be pre-empted. In addition, some crypto such as RSA is slow (can be a second or more), which makes it harder to break up into event-sized chunks (to make this more fun, changing this code is also security critical). It would need some more understanding of the mbedtls source code to work out whether there is a sensible way to run it in a module in an interrupt-driven way. Possibly the pthread interface is enough, but I suspect you’d end up reimplementing your own pthreads to run the rest of the code in user mode out of a module.

Which kind of brings us back to the beginning… userland avoids some of these problems, unfortunately the API we already have is a module API. So it looks like we’re stuck with it :(

Theo,

I’m undecided about the DDE or GCCSDK, I keep on being certain on one then on the other!
Both are great tools that provide benefits either way. It’s frankly driving me nuts.
Seeing Jeffrey’s code didn’t help either, now I’m back thinking GCCSDK may be better.
I guess tomorrow I may think it sucks and the DDE is better!

But thankfully it largely doesn’t matter as you said, the main barriers are getting a scheduler done.
My progress so far isn’t much.
But my conclusion so far is that the compiler is irrelevant.
The differences between the code for modules for the DDE or GCCSDK look small.
From what I see the DDE module code looks “easier” as I’m more familiar with it (which isn’t much TBH)

I’m just really glad that mbedtls is so portable.
I’m also happy enough to drop the TLS/SSL portion for the moment and get a scheduler and socket module written.

One thing I would really like is to have the ability to use other cores for the data processing.
A module that would provide the ability to call a C function and execute it on “spare core” if it can’t (i.e. all cores in use).
Again plugging that into the scheduler module would be cool so the scheduler would handle the queuing of jobs and the async portion of it from a user perspective.
Not sure it’s possible to do that, I assume the C functions would have to be “self contained” – i.e. all local variables/structures or pointers to specific defined memory areas.

Quite a lot of arm hardware has hardware acceleration for AES, a shame that can’t be used or looked at when selecting the “next” arm hardware for RISC OS.

Also I’m not so interested in keeping to the current API more defining a new one that allows me to abstract far beyond sockets and ssl.
That said, there is no reason why the SecureSockets module couldn’t hook into that for legacy applications, that would be relatively simple to do (and securesockets would in theory never need updating)

Hence why I think at this point I need to forget about SSL and get the bare bones done.
I’m not sure everyone will agree with my idea, but I think it’s “cool” and fun.
The other point is I am not interested whether it works well and smoothly on “old” hardware.
Hence my suggestion that if there is a need to get it done quickly the GCCSDK may be the best option.

Either way it’s not wasted work.

From what I see the DDE module code looks “easier” as I’m more familiar with it (which isn’t much TBH)

Module code should be pretty much the same on both compilers. There’ll be some differences due to Norcroft using CMHG for generating the module header code & .h file, while GCC uses cmunge, but the tools should broadly be compatible with one another (cmunge effectively started off as a free clone of CMHG).

One thing I would really like is to have the ability to use other cores for the data processing.

TBH that’s probably the best approach in terms of (network) security. Have a microkernel running on another core which provides a nicely sandboxed environment for running the TLS code. If bad data is received over the network then it won’t be able to do a thing to break out of that sandboxed environment.

Quite a lot of arm hardware has hardware acceleration for AES, a shame that can’t be used or looked at when selecting the “next” arm hardware for RISC OS.

Citation needed ;-)

ARMv8 has AES instructions as an (optional) extension to the instruction set. If you can find an ARMv8 board with support for the crypto extensions then we should be able to use it. if it’s some other AES hardware which is present then using it would obviously be dependent on documentation from the SoC manufacturer.

Hence why I think at this point I need to forget about SSL and get the bare bones done.
I’m not sure everyone will agree with my idea, but I think it’s “cool” and fun.

If you’re interested in getting other cores working, you’ll have a bit of an uphill struggle, as there are some features that the OS is lacking (mainly, proper handling of shareable memory).

But then again I think it’s possible to get a multi-core taskwindow module working on Pi 2/3 without needing to change the OS sources, so it’s not an insurmountable task to get something multi-core running (ignoring task windows and rolling your own scheduler may even be easier, as then you wouldn’t have to bodge the code into the taskwindow module!).

Thanks Jeffrey, appreciate the education. The thought of me using GCC on RISC OS, just seems wrong to me – but being wrong is what I enjoy most in life.
Or having people who can prove you wrong anyway.

I also like toolbox too, so that may show how warped I am?

OK, possibly perspective error on my part of “quite a lot” of arm hardware, I’ve used a few that have it – and I’ve only used a few.
So there maybe quite a lot, but only 0.01% – there isn’t exactly a shortage of arm hardware and I’m hardly even an amateur.

It was more of a nudge to be honest, so anyone looking at porting to anything new, put’s it on a “nice to have” list.
As always documentation is the nightmare.

I was more thinking from a multi core perspective (this is at a dumb persons level – i.e. mine) if you could copy a segment of memory to a specific area then say to the other core(s) go do stuff, use this memory. So is it possible to not use shared memory, as long as it’s allocated in the main OS as “in use” so nothing else uses it whilst the other core is “in use” if the cores are locked to a certain memory segment?
From that you may get an idea of my deep understanding of how multi core works ;-)
I think that’s kind of what your saying on the thread, but again more reading and thinking required on my part.

If not I don’t mind digging around, it’s vaguely interesting – something I should really know about to be honest, at least I’d know more than multi core means more than one core!

Both are great tools that provide benefits either way. It’s frankly driving me nuts.

In a way it is good that both are working for you.

Seeing Jeffrey’s code didn’t help either, now I’m back thinking GCCSDK may be better.

? I didn’t think his code was bad…?

But thankfully it largely doesn’t matter as you said, the main barriers are getting a scheduler done.

I have a horrible suspicion you’re going to need a wodge of ARM code for that. While you can, no doubt, set up the memory areas and stuff in C, you will likely need to set up some sort of environment on the other core, which would mean trapping and dealing with the hardware vectors. Quite possibly initialising the MMU and sorting out memory pages. If the other cores are essentially stalled when RISC OS is running, that means there is no kernel, no environment, nothing.

From what I see the DDE module code looks “easier” as I’m more familiar with it (which isn’t much TBH)

In a way you’re lucky that you are trying out both. My main objection to GCC is I’m too lazy to bother with learning the intricacies of a different toolkit. I have stubbornly resisted all that shared MakeFile rubbish that has turned up in the DDE in preference to hand-built MakeFiles – while the build process is probably simpler, at least I know what is getting built and how. I absolutely detest setups that pull in data from dozens of directories and files. Okay, this may make sense if you’re building something like the Wimp, but for an application it really ought to be as self contained as possible. Then projects can be moved between machines, and potentially different versions of the DDE, with less risk of breakage due to X not being found.

socket module written.

IPv6? ;-)

Not sure it’s possible to do that, I assume the C functions would have to be “self contained” – i.e. all local variables/structures or pointers to specific defined memory areas.

I know SFA, but I don’t expect that it would be that difficult once the low level code is running. You should be able to assign an area of memory, copy data/code into it, then instruct the other core to run it. I think that was the basic principle behind how the x86 card worked, and there may be some ideas there regarding how to handle “other cores”. However as Jeffrey points out, RISC OS isn’t really set up for such a thing, despite Acorn machines supporting (possibly alien) coprocessors since long before the concept was even understood on home micros (and networking, and proper I/O, and…).

This said, I think before any of this begins, we ought to have a protocol devised for the claim/use/release of any additional cores. You want to use core(s) for the socket stuff. Jeffrey has mentioned the TaskWindow doing so. We may arrive at a day when the Wimp itself is able to spread tasks among the available cores. Well, how does any of this know if a core is in use or is available? We need some sort of protocol, even if it piggybacks on the Device Claim, to prevent this becoming a free for all.

Quite a lot of arm hardware has hardware acceleration for AES, a shame that can’t be used or looked at when selecting the “next” arm hardware for RISC OS.

There is a lot of hardware acceleration in many ARM based SoCs. My little PVR chip has hardware Huffman tables, hardware bright/contrast/focus, hardware realtime scaling, a powerful DSP, and some other stuff I forget. I have no doubt that there are many good things lurking within the Pi silicon, perhaps as co-processors, perhaps as things located in the memory map.
The difficulty, documentation. Some things are documented. A lot isn’t. Compare the thousands of pages of OMAP datasheet with…the Pi’s minimal documentation that doesn’t even (last I looked) describe the GPU mailbox interface or even how to use the dumb frame buffer.

I’m not sure everyone will agree with my idea, but I think it’s “cool” and fun.

Well, there would be some interesting experiments to do with a module that could schedule pieces of code to run on a co-processor (other core). There is also lots of fun involved in writing a decent schedular. Have you read the MINIX book? It talks about these sorts of things in detail (though not multicore, I think it predates that sort of technology), so it’s worth a look if you can find a copy (the one linked is one Google found on somebody’s Google Drive, it’s a ~9MiB PDF).

The other point is I am not interested whether it works well and smoothly on “old” hardware.

Smoothly is one thing, but your idea may hit obstacles if it isn’t able to sensibly degrade to something that will run (even if less effectively) on a single core machine. How do you define “old”?

Hence my suggestion that if there is a need to get it done quickly the GCCSDK may be the best option.

If you’re writing most of this yourself, either toolchain ought to work. ;-)