HTTP requests from C

Warning: Me again :)

What’s the “best” way to perform HTTP requests from a C app? I’ve found some references to the AcornHTTP and URL modules and the associated documentation, but it seems that these modules are difficult to find so I question whether they should be used for any new development. It’s also unclear whether there are any C veneers for these modules or whether I need to “manually” call the SWIs.

My first question is therefore “Should I be using AcornHTTP and URL, or something else?”

I’ve also seen Wget mentioned but I’d prefer not to call into a CLI utility if I can help it.

Before finding AcornHTTP/URL I did a bit of playing around with Socklib and I have a basic HTTP GET working… but that feels like reinventing the wheel so I decided to stop and ask for advice :)

My first question is therefore “Should I be using AcornHTTP and URL, or something else?”

I think you’ve hit one of the pitfalls of the antiquity of the Acorn¹ stack in that you’re either going to try and use antique unsupported modules or start making direct access to the socket code and throwing your requests in there as you’ve noted.

Shame you’re not looking just after the network bounty is complete.

¹ Be honest folks, it’s that old Acorn were in full flow when it was written. Some minor mods since but it really does need the planned re-write.

Shame you’re not looking just after the network bounty is complete.

I actually learned of the existence of AcornHTTP by reading the bounty description! I’d previously searched for “risc os http” with Google but it didn’t find anything useful.

As noted, my existing GET does work so I wonder whether it’s worth finishing it off; it’s a bit basic but perhaps some other people might find it useful if I were to share the code.

Writing in BASIC, I have found Justin Fletcher’s EasySockets module to be the easiest way of performing HTTP requests. (I see no reason why it would be different in C and it is easy just like it says on the tin.)

However, there are two caveats:
– On modern hardware, like a Raspberry Pi, you will need Aemulor to run EasySockets.
– But on a Pi 3 or a second generation Pi 2, Aemulor will not work. (so, neither will EasySockets!)

So, you could be best with your existing code, as you have it working.

Sockets aren’t that hard, just a bit more fiddly than opening a file and reading.

Here’s a dumb server: https://www.heyrick.co.uk/blog/index.php?diary=20160104

I ought to write an article on the other side. Already written the code (Manga, for instance, has it’s own http fetcher).

Good example of Arm assembler in ‘C’- Rick :-)

Have you thought of doing a replacement of ’EasySockets"?

Manga, for instance, has it’s own http fetcher

I thought it might; I saw the change log entry about using \r\n instead of \n :)

saw the change log entry about using \r\n instead of \n :)

Hehe. Worked just fine with the mangareader site. Worked just fine with WebJames. Failed miserably with the HeyRick server. Working out why indicated that CRLF was the expected line ending. No idea why, seems a waste of data and bizarre that Apache (on a Unix-like box) would choke without it, but there you go…

seems a waste of data and bizarre that Apache (on a Unix-like box) would choke without it,

Imagine a laser printout, from a Unix box, where the data only included LF and no CR.
Had that and the application developers couldn’t understand why the printouts were only the top line.
When I explained they said “you need to change the printer so it automatically adds CR”, no said I “you need to write your reports correctly to include CR then when a printer fails and we replace it the change only involves plugging in the new unit with default settings”.

Printing is a little bit different – the CR and the LF actually mean specific things (clue in the names!). For a web server, an “end of a line” is an “end of a line” (and while Windows uses CRLF, Unix and RISC OS don’t).

BTW:

the application developers couldn’t understand

Given the organisation you work for, I’m not sure I’d go as far as to refer to them as “developers”. Has IE6 been laid to rest yet? ;-)

Worked just fine with the mangareader site. Worked just fine with WebJames. Failed miserably with the HeyRick server.

For what it’s worth it works with IIS too. The first build of my client used \n but I peeked at the code for another client when trying to track down a bug, and noticed that it was using \r\n. Like you I thought “that can’t be right; this protocol is from the Unix crowd” but sure enough, the spec says \r\n!

Given the organisation you work for, I’m not sure I’d go as far as to refer to them as “developers”.

Application from GE Medical systems, and I’d agree with you.

Has IE6 been laid to rest yet? ;-)

Oh, we nailed the lid down on that on our systems a while back. Can’t speak for other Trusts.

I might as well share what I’ve done so far. Here is what I’ll call “alpha 1”. It’s certainly not ready for prime-time yet as it has several limitations:

• Only supports GET requests
• Can’t follow redirects (it probably can’t even tell you there is one – this is completely untested)
• Only connects on port 80
• If the server uses chunked encoding then only the first chunk will be returned
• Does not multitask, although it does still function in a desktop app (but will freeze the UI until transfer complete)
• Will probably crash and burn if you feed it a largish file
• Will probably crash and burn if you feed it a binary file, especially if it has a \0 in the middle!

The zip file contains both the code and a pre-built library (in the Export directory). It’s fairly easy to use; you call httplib_get and give it the hostname, the file path, and the name of a callback function. Here’s a bare-bones example to display raw HTML:

#include <stdio.h>
#include "httplib.h"

void show_response(char *html, httplib_metadata meta, void *handle)
{
    printf("%s\n", html);
}

int main(void)
{
    httplib_get("www.example.com", "/", show_response, NULL);
    return 0;
}

Your makefile will need to include ${NET5LIBS}.

You can find the total content size and HTTP response code in the httplib_metadata struct. The fourth parameter to httplib_get (NULL in this case) is passed verbatim up to the handle parameter of show_response.

As for the underlying code, it’s a bit messier than I’d like. As I’ve said in the past, I’ve been a bit spoiled by C# so I’m probably not doing things the most efficient way (eg. doing a full strcpy before a strtok then only accessing the first few bytes) so any suggestions are appreciated. But it’s a start! :)

Now that the bounty has been funded I’m having second thoughts. While the bounty description doesn’t mention AcornHTTP specifically, this does make me wonder whether it might be worth sticking with the “official” stack. Any thoughts?

Now that the bounty has been funded I’m having second thoughts. While the bounty description doesn’t mention AcornHTTP specifically, this does make me wonder whether it might be worth sticking with the “official” stack. Any thoughts?

You might note I said earlier “Shame you’re not looking just after the network bounty is complete.”

I think the answer is you’re better off working with the official stack, but at the moment you have nothing to work with really.

Maybe someone with inside knowledge could give you an indication of a target date for completion, then you know how long you’re likely to wait.

Why not use libcurl?

Because I didn’t know of its existence. I specifically looked for it before starting, and Google was unable to find a RISC OS version.

What’s the “best” way to perform HTTP requests from a C app? I’ve found some references to the AcornHTTP and URL modules and the associated documentation, but it seems that these modules are difficult to find

They’re here and here amongst the other fetchers.

Other than minor makefile twiddling it doesn’t look like anything’s happened to them for 12+ years, presumably because nothing’s using them (they’re not involved in any nightly build), but I believe RiscOSM uses them because I have a thread from May 2016 in my outbox asking where the sources are. The demo version has some prebuilt ones by the looks of it.

Thanks; the “resources” zip on the RiscOSM site also contains prebuilt copies of the latest versions. Now to have a play…

Unfortunately, the ready-to-run RISC OS port of curl is outdated and without source:

http://www.strcprstskrzkrk.co.uk/curl

Maybe you could contact James Bursa, the porter, for advice?

It might be a good idea to have a look at NetSurf, ISTR that it uses libcurl.

NetSurf uses curl. I expect it uses sources/libs from riscos.info, but it’s a bit hard to check since the site seems to be down at the moment.

riscos.info, but it’s a bit hard to check since the site seems to be down at the moment.

Yeah, I think it was Theo looked at things last time that happened so I emailed him yesterday

riscos.info is up again, here is the newer-but-not-quite-the-latest libcurl readily ported:

http://www.riscos.info/packages/arm/Library/libcurl3-dev_7.47.0-1.zip

Thanks, but unfortunately there doesn’t seem to be any RISC OS-specific documentation included, only generic stuff. There’s a file typed as text called “libcurl/a” which appears, from its size and the fact that it’s actually binary, to be the library. However, Link tells me that it’s “not an object or library file”.

I think last time I ran into this problem it turned out that the library was only compatible with GCC. Is libcurl known to work with DDE 27?

And I suppose this is as good a time as any to ask: Is there any compatibility at all between the GCC and DDE “worlds” or is it all one or the other?

Well, I’ve given AcornHTTP a go and aside from one bug¹ that I already have a workaround for, it seems to do what I need it to do :)

Thanks for all the suggestions. It looks like this is the path of least resistance as it does the bulk of the work for me, and we have the source if there are any showstoppers.

¹ The spec states that “Bits 8-15 of R2 are the type of data wanted” (where you can pick head, body or both) but it always gets both. I looked at the source and it reads the value you pass into it, uses it to fill ses→data_wanted, and then never looks at it again. I also looked at the source to Browse, which only ever passes in 2 (ie. get both head and body) so it looks like nobody at Acorn noticed that it doesn’t work.

But in any case I already have some code to split the head and body apart so it’s not really much of a problem.