HTTP requests from C

Is there any compatibility at all between the GCC and DDE “worlds” or is it all one or the other?

I think we time goes on, they’ve diverged more and more.

The spec states that “Bits 8-15 of R2 are the type of data wanted”

Oh well done Acorn… I’m guessing this stuff never got properly tested?

I’ve had a bit more time to look into this. I figured that it’ll be better in the long run to attempt to fix the module instead of hack around the bugs in it. It turns out that not only is the contents of data_wanted ignored, but it’s not even set correctly in the first place. The docs state that it’ll get the body if you pass in 0, the header if 1, or both if 2. Meanwhile data_wanted is set with the first two options flipped (0 for header, 1 for body). [Edit, six months later: This post is wrong; it does set data_wanted correctly but I’d previously misunderstood what it was putting in there.]

But the next step of the puzzle is to answer a design question. If you call AcornHTTP and specify that you want a GET request but only the header, should AcornHTTP switch that over to a HEAD request, or should it still do a GET and then throw away the body? The former is obviously more efficient, but if the developer has specifically asked for GET, then…

If you call AcornHTTP and specify that you want a GET request but only the header, should AcornHTTP switch that over to a HEAD request, or should it still do a GET and then throw away the body?

Doing a GET and throwing away the body is the safer of the two options. A non-standard server (e.g. as used with REST) might not implement HEAD.

Yes, just to conmfirm that we use the AcornHTTP fetcher in both RiscOSM (for interacting with the Geograph API) and Nominatim (for the Open Street Map name database API, Nominatim ). As I understand it, the network stack bounty first stage will add support for https. This will open up the possibility of using various other interesting APIs in RiscOSM.

As our applications are compiled using Acorn C/C++ I don’t think using the Unix port of libcurl would be an option for us, and I wasn’t keen on using wget via the command line. We have some socket code for another applition, implementing SMTP, but I thought we would get a better result using AcornHTTP rather than reimplementing it all ourselves (though I’ve driven HTTP from telnet before, so I am familiar with the protocol).

The APIs we are using so far only need http, but we have ideas for other APIs which require https. The SecureSockets module would have been one option, but that would have required us to implement the application layer on top, as it just provides socket-level stuff.

We’ve had to write C wrappers round the SWIs, but that is easy enough. Our code adds a high-level interface that allows the rest of the application to receive data from the HTTP response in the same way as teh application deals with data from the RAM transfer protocol.

So the AcornHTTP module works well for us. We’ve not had to compile it up ourselves as there was a binary version somewhere on the ROOL site at some stage. It’s a shame it’s not distributed with the OS, but users only have to merge the !System we provide once and then it’s done.

Doing a GET and throwing away the body is the safer of the two options. A non-standard server (e.g. as used with REST) might not implement HEAD.

Understood!

(Matthew’s stuff)

Thanks; it looks like it does work “in the real world” then :)

It turns out that not only is the contents of data_wanted ignored, but it’s not even set correctly in the first place. The docs state that it’ll get the body if you pass in 0, the header if 1, or both if 2. Meanwhile data_wanted is set with the first two options flipped (0 for header, 1 for body).

It occurred to me that “data” does not necessarily mean “body”, so I decided to track down the initial definition and see whether there were any comments. Technically, yes, there is a comment:

unsigned int	data_wanted;	/* No idea */

Doing a GET and throwing away the body is the safer of the two options. A non-standard server (e.g. as used with REST) might not implement HEAD.

Just so long as HEAD actually works (if requested) and not the horrid hack of getting the entire thing and throwing away the body. Because doing that works if you want to HEAD a 20K document, but it fails if you want to HEAD a 20M data file.

Furthermore, we should really treat broken servers as a special case, not “situation normal”. :-/

It appears that HEAD works if that’s what you actually request (by setting R2 to 2). I know what you mean about broken servers though!

I think what we are running into is that there are actually three items of information that could be returned:

• Headers
• Body content
• Both of the above

To my mind, as the API provides a specific means of performing a HEAD request, perhaps if the user specifies “GET but only return headers” then the module should do exactly that, even though it might seem nonsensical? This could be the “broken server” workaround – use HEAD for servers that support it, and fake it for those that don’t.

Yep, and that’s my intention. Do what the developer asked for, even if it doesn’t make sense.

Meanwhile I’ve found that if I perform a HEAD request, it does respect the value in R5 (ie. if it’s set to “body only” then it doesn’t return anything). I haven’t had a chance to track down why this one works.

Rick,
Thanks for the help, it’s working pretty good now, I sent curl to silent and now TaskWindows no longer appear, but the downloads still run multitasking in the background. I got rid of the text printing to screen just by redirecting stdout, so all is working quite well at the moment.

Thanks a lot, you were a great help.

Cheers

Garry

The spec states that “Bits 8-15 of R2 are the type of data wanted” (where you can pick head, body or both) but it always gets both.

I’ve finally looked into this again and I have it working now. I’m going to do a bit more testing to make sure that everything works then I’m going to submit a patch to ROOL; it’s only a few lines that need to change. Hopefully we can get the HTTP module added to Bonus Binaries at the same time, to make it easier to track down.

Hopefully we can get the HTTP module added to Bonus Binaries at the same time, to make it easier to track down.

I look forward to that. Thank You.

I see from the CVS logs that it will soon be in the main HD4 download :) (although it doesn’t yet contain my patch).

Edit 19/5: HTTP 0.98 is now available with these changes implemented!

I would like to know what the current situation is with AcornHTTP, AcornURL and AcornSSL since I hope it has changed since 2018: Is it possible to issue HTTPS calls by using AcornHTTP and is it correct that AcornSSL now supports TLS 1.3?

I would like to know what the current situation is with AcornHTTP, AcornURL and AcornSSL since I hope it has changed since 2018:

The current situation is that all work, the SSL is kept up to date with mbedTLS and most of the releases have the same version number. ;-)

Is it possible to issue HTTPS calls by using AcornHTTP

Why would you want to do that? Use URL and let it figure out what to do.

Example here: https://heyrick.eu/blog/index.php?diary=20191226

and is it correct that AcornSSL now supports TLS 1.3?

It may well be more up to date than your browser…

and is it correct that AcornSSL now supports TLS 1.3?

It may well be more up to date than your browser…

While that might be the case for a RO browser, there’s less certainty when the likes of Firefox¹ update pretty much monthly plus patch intermediates. Sometimes RO is ahead of the big boys though.

¹ Firefox 75.0 April 7, 2020

I think it’s still TLS 1.2 at present; if I understand correctly then TLS 1.3 support comes with mbedTLS 3.0.

Aside from that, I agree with Rick that you should be using URL, rather than AcornHTTP directly. I’m also going to take this opportunity to plug HTTPLib (discussion | download) as it handles the fiddly bits, so long as you only need to do one transfer at a time (I really need to sort that out one of these days…)

Thanks for the answers, so it seems that the current situation is much better than I expected! And yes, I meant to use AcornURL and not AcornHTTP directly but did not make it clear. Also TLS 1.2 is totally fine for me – as long as it is a quite recent version and not SSL from 1998 or so ;-)

Chris: Thanks for the link to your library – this really seems to be very usefull for me and makes it much easier to use than calling AcornURL directly. Before getting started I just have to learn C/C++ now – but I hope I can get into it quickly since I have experience with C# and Java and hope it is not too different – exept for the memory handling (and probably more things I don’t know about just yet ;-) )

It supports 1.3, because it works with my site, and that doesn’t support anything else (so apologies if you’re using an old iPad or ancient browser, it no longer works for you…).

It supports 1.3, because it works with my site, and that doesn’t support anything else

Are you sure about that, Rick? All the stuff about mbedTLS that I read says that TLS 1.3 support is not there yet. But there seems to have been a lot of activity within the last month (mostly 26 days ago), and one problem is that it will break the API.

Are you sure about that, Rick?

Probably not. :-) Unfortunately my site went through a spot of chaos (server failure) so what is running now isn’t what was running before. I think it’s TLS 1.2/SNI at the moment. And still not got MediaWiki sorted, thanks to bloody PHP and the total lack of useful backwards compatibility that seems to exist in the “Unix” domain (let’s not even talk about Python 2 vs 3).

Thankfully, since I wrote my own blog code, and I understand how it works, and it’s dead simple, fixing that was more of an annoyance than anything else.
I also discovered that I’d been using some old Regex expressions for something that could more effectively be done with a case insensitive string search, so the new code ought to do the same job quicker and with fewer overheads. Plus, it’s obvious what is going on, not something I can say about Regex.

I think it’s TLS 1.2/SNI at the moment

Looks to me like someone scanned the site heyrick.eu at 22:11:01 UTC yesterday to check.
As you say TLS1.2/SNI and no weak cyphers