Failing to program an application that uses asynchronous socket IO

Howdy!

So, I have gotten stuck trying to figure out how to get asynchronous socket IO to work.

I have verified that, when using blocking IO, I see the expected results (a connection is successfully established, data is exchanged, etc).

When trying to set up “internet events”, however, I am running into problems.

The first thing I tried is the following: I start by claiming the event vector with SWI OS_Claim, passing 16 (Event vector), and then call SWI OS_Byte 14 (Enable event), passing 19 (Internet event). Then, a call to SWI Socket_Ioctl, passing FIONBIO, and another call to SWI Socket_Ioctl that passes FIOASYNC. Before calling Wimp_Poll, I make sure to call SWI OS_DelinkApplication, and call SWI OS_RelinkApplication after Wimp_Poll has returned. Unfortunately, when trying to connect to the socket that this application opens, the event vector routine that I passed to OS_Claim simply does not get executed, and I am failing to figure out why. Otherwise, the application runs and terminates without any issues.

The second thing I tried was calling OS_Module 6 (Claim) to allocate some memory from the RMA, and then call Socket_Ioctl, while passing FIOPOLLWORD, as well as the pointer to the allocated memory. I was hoping that this would cause Wimp_Poll to return with the PollWord_NonZero whenever an internet event had occurred. But alas, this ended up with fairly disastrous results, like crashing other running programs, like FTPs or NetSurf, which in turn would lock up the entire system, forcing me to reboot. Clearly I am missing some crucial things here.

Some additional context: I am writing this application in C, paired with inline assembly. Using GCC and LD as compiler and linker, I end up with an ELF binary. The application does not use the C runtime (-nostdlib). As implied, this application uses Wimp, but creates only one simple (empty) window. I’m running RISC OS 5.28 on the Raspberry Pi 3B+.

I would like to learn what I might be doing wrong, and what I’m missing. I can imagine that I may have misinterpreted how exactly to use FIOPOLLWORD, but I am a little surprised that my event vector, in the first attempt, was never entered, as the documentation seems fairly clear about how this should be set up.

To be a little more clear on my actual goal: I would like for my main loop to be exclusively event-based. That is; whenever RISC OS has work that is the job of my task / application, then I want to be notified and awoken somehow (rather than explicitly having to poll to see if the OS has any work for me to do). Ideally, everything would go through Wimp_Poll, thus the pollword (FIOPOLLWORD) approach sounds particularly good to me. For reference; on Linux I would use epoll_wait(), while on Windows it would be a function like MsgWaitForMultipleObjects().

Well, if I have forgotten relevant information, just let me know. Your insight would be appreciated. :-)

PS: Since this is my first post here, I suppose I’ll do a bit of an introduction. :) I was given a RPi in December. At the time, it had a Linux-based OS on it, but it all went to hell after a system update. I tried for about 2 hours to recover, before deciding that I might as well just start from scratch. So I started looking around to find out what my options were for the RPi, and that’s when RISC OS showed up for me. It looked and sounded fun, so I figured I might as well give it a shot. Installation was seamless. I quickly grew fond of the GUI experience, and I thought that StrongED was quite the cool text editor, so I ended up hanging around for long enough to try and write some programs for it. I have an older chat program that I initially wrote for Windows, and later ported to Linux, and it seemed like a fun project to port this chat program over to RISC OS. I’ve spent a number of weeks learning about RISC OS, slowly working my way to this point. Unfortunately, getting asynchronous socket IO to work seems to have gotten the best of me, so here we are. :P

For my non blocking code, I use select() to check the status of the socket when PollIdle returns.
I think the Internet Event is to do with buffer thresholds. I never got it to work correctly (fired once, and only once), so probably doing something wrong too.

People may be more able to spot issues if you could post some actual code.

Howdy Matthias, welcome to the forum :-)

make sure to call SWI OS_DelinkApplication, and call SWI OS_RelinkApplication after Wimp_Poll has returned. Unfortunately, when trying to connect to the socket that this application opens, the event vector routine that I passed to OS_Claim simply does not get executed, and I am failing to figure out why.

I expect the event is happening whilst you are delinked. If you wish to claim an event, you either need to be running as a ‘single-tasking’ program (i.e. no Wimp_Polls or not in a task window), or have your code in the RMA (i.e. you are a module or have copied a tiny bit of code into an RMA block).

and then call Socket_Ioctl, while passing FIOPOLLWORD, as well as the pointer to the allocated memory

I don’t think FIOPOLLWORD is available under RISC OS 5.28.

To get woken up by a poll word, you’ll need to have some code sitting in the RMA hooked up to the event. Since you are using assembly you might attempt to do that yourself, otherwise I believe there are third-party modules such as ‘SocketWatch’ (used by the !Nettle SSH client) available for this job.

Here’s some past discussion of this.

Welcome to the forum Matthias,

I don’t think FIOPOLLWORD is available under RISC OS 5.28.

Agreed. AFAIR, FIOPOLLWORD was something “added” to RISC OS Select (which is a different branch off RISC OS 3.80) that started to “evolve” separately from RISC OS 5.

Hey, and thanks for the welcome! :)

@Paolo Fabio Zaino
@Julie Stamp
Regarding FIOPOLLWORD; it’s present in the headers of the 5.28 source code that I downloaded (from here), so I figured I’d give it a shot. The documentation is not always clear / complete on everything, so I wasn’t sure whether or not it was actually supported. But it appears that you are correct. Alas!

@Rick Murray

People may be more able to spot issues if you could post some actual code.

You’re right. Unfortunately, most of my code is quite messy and experimental, as I’m still familiarising myself with OS functionality. I also use quite an unconventional coding style that would probably upset people. :P

For my non blocking code, I use select() to check the status of the socket when PollIdle returns.

I would prefer not to use wake-up timers like that. But we’ll see; I might just have to resort to this. Thanks for the suggestion.

@Julie Stamp

I expect the event is happening whilst you are delinked. If you wish to claim an event, you either need to be running as a ‘single-tasking’ program (i.e. no Wimp_Polls or not in a task window), or have your code in the RMA (i.e. you are a module or have copied a tiny bit of code into an RMA block).

Right, this made good sense. After moving the routine itself into the RMA, and not de-linking it, the routine indeed gets called successfully. It seems, however, that it still does not pick up my internet events, while it does pick up events from other applications. (As my first post implied, I’m following the steps here.) I also put a pollword into the RMA, which has proven to work with Wimp_Poll(), so I can potentially set that pollword to non-zero when an event is caught in the vector routine. Cheers!

So, I still don’t know why my events are not happening. For now I’ll just assume that something about my implementation is not right.

In case any of you know of a link to source code of a module or application that has implemented this successfully, I’d love to check it out to learn how to implement this properly. Edit: I just found the source of SocketWatch 0.7, so hopefully I’ll learn something. :)

Anyway, for now I’ll go over my code again with the assumption that I’ve made a mistake somewhere. Thank you guys for the tips and suggestions.

Regarding FIOPOLLWORD; it’s present in the headers of the 5.28 source code that I downloaded (from here)

I think you may have been led astray by the sources including 3rd party components. The only reference I can spot to FIOPOLLWORD is in OSLib and GCC, there’s no implementation backing up those #defines.

It seems, however, that it still does not pick up my internet events, while it does pick up events from other applications.

You’ll only get an event for your socket if you enable ASYNC notifications (FIOASYNC). It’s worth a read of the StrongHelp Internet manual which might be a bit more approachable than rummaging in header files. Go to Internet event → Event reason for example.

Unfortunately, most of my code is quite messy and experimental, as I’m still familiarising myself with OS functionality.

That’s alright. I’m sure we have all “thrown together a quick hack to test something”. My entire IPP finder program is exactly that.

When asking for help, the code is necessary in order to see what you are doing and how you do it.
For example, perhaps in BASIC one is putting/reading the values into/from the wrong registers due to a simple miscount of how many commas to use, such as SYS "OS_File", 23, file$ TO ,,,,,type% would be reading the attributes, not the type. It’s all too easy to miss something like that and wonder why something elsewhere is messing up. I speak from experience!

I also use quite an unconventional coding style that would probably upset people. :P

Oh, god, you’re one of those people who put opening braces on the same line as the conditional, aren’t you? :-p

Really, don’t be afraid to post code. Seeing what you’re actually feeding the compiler is better than a description of it.

So, I still don’t know why my events are not happening.

How are you setting async?

The code I use to set non-blocking (I don’t use events so no async) is:

   // Make it non-blocking
   value = 1;
   reply = socketioctl(sock, FIONBIO, &value);

value is an int. For your use, change FIONBIO to FIOASYNC.

I do this after bind() and before listen() (server) or just before calling connect() (client).

So, I still don’t know why my events are not happening.

Are you absolutely certain they aren’t working? The problem I ran into (and why I don’t bother with them) is that I’d get the “socket has input waiting to be read” event once but no more. Never figured out why, so I just use select() to check the socket instead.

Some of this socket stuff is a dark art, and since the RISC OS stack harks back to the Mesolithic, many of the examples you’ll find on-line won’t work. It’s a real shame that EasySockets never made it to RISC OS 5.

so I just use select() to check the socket instead.

To expand on this a little, it’s not so different to using an event loop.

I claim NULL events, and call Wimp_PollIdle with a 20 centisecond timeout. That means the Wimp will call my program five times per second, or if there’s other work to be done (like window/icon activity).

When I get a NULL event, I check the socket to see if there’s anything for me. If so, I process it. Otherwise, I just call PollIdle again (effecively, go to sleep).

In a module implementation (background service, not a Wimp task), I do pretty much the same basic mechanism, but it is controlled by a callback. I set up OS_CallAfter to call me after so many centiseconds (I use 25 so it’ll be called four times a second). When this fires, I then call OS_AddCallBack, which will call my event handler when RISC OS is not busy.

I’ll quickly explain the distinction. CallAfter is hooked to the timer, and when it fires you’ll be in SVC mode with interrupt disabled. More importantly than that, RISC OS may well be in the middle of doing something, like loading data from a file. It’s not “safe” to do much in this state. So AddCallBack allows us to have our demand deferred until RISC OS is not busy. There is no time guarantee here (it depends on what is happening) but it’s usually quick enough to not be noticed. At this point, you can safely do whatever needs done.
There are other things to keep in mind, like you must remove pending CallAfter and CallBacks when closing down, and you must keep track of whether a CallBack is pending to avoid setting it multiple times.

That being said, if you’re running entirely in the Wimp environment and using polling, the above paragraph won’t apply to you. Just mentioning it in case you experiment in that direction.

Simple example of a rubbish server at https://heyrick.eu/blog/index.php?diary=20160104 :-)

Simple example of a rubbish server

In the PC world there’s a handy little utility (several actually) with the name “Port listener”
Its job is to allow you to check connections through a firewall by sitting there listening on a specified port and giving you a quick message when you connect.

Simple example of a rubbish server

Far too complicated, look what you can do with a few lines of Python and the flask module

#!/usr/bin/env python3
from datetime import datetime
import flask

app   = flask.Flask("Date and time Server")

@app.route("/")
def get_date_and_time():
    return flask.make_response(datetime.now().isoformat(' '), 200)

if __name__ == '__main__':
    app.run(host="0.0.0.0", port=9500, debug=True)

We need this sort of thing on RISC OS

We need this sort of thing on RISC OS

And the question now is… Can it be done in fewer lines of Lua? ;-)

Far too complicated

Hmm, and can one write a full fat desktop application in Python that runs on RISC OS and doesn’t take an eternity to do anything?

We need this sort of thing on RISC OS

That all being said, I agree. Simple(ish) and capable scripting would make a big difference. It’s long gone time to look beyond the confines of BASIC.

@ Rick

Can it be done in fewer lines of Lua? ;-)

Technically it could be done yes. We now cam make a fast WEB server in C (or C++) and control/use it from Lua, this is now allowed by the changes added supporting Dynamic Binary Library load (at runtime).

Hmm, and can one write a full fat desktop application in Python that runs on RISC OS and doesn’t take an eternity to do anything?

Well we can look at what can be done on the exact same hardware under Linux, and identify areas where RISC OS lacking performance (or maybe easier to stop the few places where it isn’t lacking).

But getting back to the case in hand, socket stuff in RISC IS is much harder than it should be due to the last of pre-emption, and you don’t really have any other option but to make your socket code non blocking, put it in a module and communicate with the desktop application via poll words and SWIs.

you don’t really have any other option but to make your socket code non blocking, put it in a module and communicate with the desktop application via poll words and SWIs.

Or to make it a bit simpler, use the Socketwatch mocdule to set the pollword, and then the rest can run in application space. You don’t even need to allocate the RMA space for the pollword – Socketwatch can do it for you.

Isn’t that exactly what I described?

Well, a bit of a late reply, but, gladly, it did end up working in the end. :-)

What ultimately seemed to be the problem is that RISC OS is apparently quite picky about when you can perform certain operations on sockets. For example, if I set both FIOASYNC and FIONBIO on my listener-socket (after bind(), but before listen()), then FIONBIO simply does not take effect. Similarly, trying to set FIONBIO on a socket that had just been returned by accept() also does not take effect. So what I ended up doing is; after the first event for a given socket is received, then I set the socket to non-blocking. It took me quite a bit of trial-and-error to figure out that this was the problem all along.

So I’ve ended up with something pretty simple: I allocate some memory from the RMA for some workspace, and some space for the vector routine itself. Then I memcpy() the vector routine in there so that I don’t have to make an actual module (as I don’t know how yet, and perhaps GCC can’t generate these?). Then, if one of the sockets generates an event, all that the vector routine does is store the event in the allocated workspace, and set the pollword to non-zero to get wimp_poll() to return. It’s not very elegant, but at least it keeps things simple. :>

Initially, I tried to send a message from the vector routine to my wimp task, but the message simply doesn’t arrive. Perhaps that a message can’t be sent from SVC / IRQ mode; I’ll have to investigate. (It also seemed that a wimp task can’t send a message to itself?)

@Sprow

I think you may have been led astray by the sources including 3rd party components. The only reference I can spot to FIOPOLLWORD is in OSLib and GCC, there’s no implementation backing up those #defines.

Ah. :/ Yeah, I had taken the # define from the OSLib header, indeed. I did not realise it that wasn’t an OS component.

@Rick Murray

Thanks a lot for all the insight, as well as the example program. I’m a fan of your comments. :P

The callback method sounds intriguing. I’m not familiar enough yet with ARM and RISC OS, so I have a bunch of reading to do first. ^^

I do this after bind() and before listen()

Yeah, it seems that the order in which these functions are called is crucial. This is what ultimately cost me the most time to figure out, unfortunately. Hours of trial-and-error, admittedly. :E

Well, again, the advice and suggestions were much appreciated. Thanks for your time :-)

Edit I realised later, that I had been reading old documentation, which states that the r0 register is preserved by socket-related SWIs that do not return any value. This was changed at some point however, and r0 is simply set to 0 in stead of being preserved. This was the cause of some of my bugs, as the compiler assumed that the value in r0 was still the same after one of these SWIs. :(