POST request with URL_Fetcher and AcornHTTP

Having found yesterday how much easier it is to do a simple HTTP request using the URL_Fetcher and AcornHTTP modules than knife-and-forking it myself (thanks to Rick Murray!), I thought I’d try something a little more advanced today. I need to send a POST request with a body. I can create the body, and I can pass it to URL_GetURL, but it ends up in the header, not the body. The host I’m connecting with wants it in the body. I get a 400 response.

The docs (which are very out of date) suggest that data passed to URL_GetURL can appear in the body, but in HTTP the data are in the header.

So is there a way to do what I need? Maybe something in bits 8..15 of R2?

I found the answer. I scanned enough of the source code of AcornHTTP to give me the inkling that it wants to see a Content-type header in the data passed in R4, so I added “Content-type: application/json” and a blank line before the stuff that I want in the body. Lo and behold, it works.

Even better: I changed the URL in my test app from http to https, and Wiresalmon/Wireshark tell me that it’s sent encrypted – and it still works.

It’s hugely easier than doing all the sockets stuff myself. I’m a convert to this method.

Lo and behold, it works.

Oh, wow. Is that it? I found POST didn’t work either.
I mean… there’s a POST method, why doesn’t it just… do it?

I changed the URL […] from http to https […] it’s sent encrypted

Great, isn’t it? There’s a hidden flag in Manga’s setup that can switch to https if MangaReader ever decide to follow the rest of the world again and go back to encrypted. And, yeah, it’s about as simple as changing “http” to “https” ¹ and… it just works.

¹ It’s a little harder in that all of the links in the content need to be analysed and patched to the correct schema. I don’t know if they’ve fixed it yet, but for a long time (and possibly still?), when they went to plain http, all of the site content referred to https, which would reply with a 301 redirect to the non-https version. Such a waste of resources when it could be fixed up in the master script (I’m assuming the site is build dynamically from some sort of database – anything else would be a big bucket of crazy).

I mean… there’s a POST method, why doesn’t it just… do it?

I can think of one reason for it being like it is (there may of course be more…)

Since a POST has to send something to the server, the server needs a way to determine how it should interpret that something. This means that we need to send a Content-type: header. I don’t see any way to specify content-type other than in the data that we pass in R4.

Potential users of the URL, AcornHTTP and AcornSSL modules are being let down by inadequate documentation. I’ve been feeling a desire to put something up on the Wiki, where other modules are documented, but the killer for me is that there would be some many holes and questions in what I would write.

Does any more complete and up to date documentation exist for any of them? Alternatively, if I do put something up that is clearly a work in progress, will there be anyone else able and willing to fill in the holes?

I mean… there’s a POST method, why doesn’t it just… do it?

Well, it does do it, but maybe not quite like you’d hoped. It doesn’t ignore the data you pass in R4, but it looks from a cursory glance as if it requires at least one header line to be passed to it in the R4 data block. Otherwise all your data end up POSTed in the header. Which is not how it’s supposed to work.

The HTTP docs do look a bit sparse. The information you were after about the R4 data block is actually hidden away in the URL API documentation (although it doesn’t specify what newline format should be used!)

https://gitlab.riscosopen.org/RiscOS/Sources/Networking/Fetchers/URL/-/blob/URL-0_58/Docs/APISpec#L229

I’ve been feeling a desire to put something up on the Wiki

I’ve made a start on documenting the URL_Fetcher module. Please feel free to join in!

And yet again, thanks to Jeffrey! Where would we be without you?

We ought to organise a weekly clap for Jeffrey…

Dave can you provide a link to the URL_Fetcher documentation in the wiki? I could not find it.

https://www.riscosopen.org/wiki/documentation/show/URL_Fetcher

I’ve clonked in a large part of the old Acorn specification for the URL_Fetcher module. I’ve made hardly any changes except for formatting.

I would be grateful for any help in checking the information for completeness and correctness of the bits that I’ve put up, bearing in mind that the specification document I’m copying from is nearly 22 years old!

I’ll probably put up the last SWI tomorrow, and see how I can handle the remaining information in the old spec.

Oh, wow. Is that it? I found POST didn’t work either.
I mean… there’s a POST method, why doesn’t it just… do it?

It is documented under URL_GetURL. I’ve copied a lot of the documentation over from the old document, so you should now find the information in the expected place (despite the SWI prefix being URL, the module is URL_Fetcher). However, the wording used may not make the requirement for a header – blank line – body structure in the R4 data as clear as it could be, so if anyone can think up a clearer explanation, then please edit it accordingly.

Referring to document https://gitlab.riscosopen.org/RiscOS/Sources/Networking/Fetchers/URL/-/blob/URL-0_58/Docs/APISpec#L229

If you look at the lines leading up to line 625, it appears to end prematurely. Does anyone know how the sentence should end, and if there should be anything else after it?

Argh. This is driving me nuts.

I’m trying to send data to a Vonets VAP11G.

Here’s what Firefox sends:

POST /goform/wirelessBrdgApcli HTTP/1.1
Host: vonets.cfg
User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:47.0) Gecko/20100101 Firefox/47.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-GB,en;q=0.7,en-US;q=0.3
Accept-Encoding: gzip, deflate
DNT: 1
Referer: hxxp://vonets.cfg/wireless/hotspotsSettingPage.asp?ssid=Livebox-XXXX&auth=WPA2-PSK&encry=AES&bssid=xx:xx:xx:xx:xx:xx
Content-Length: 309
Content-Type: text/plain;charset=UTF-8
Connection: keep-alive

apcli_ssid=Livebox-xxxx&apcli_mode=WPA2PSK&apcli_enc=AES&apcli_ishide=0&apcli_wpapsk=PASSWORD&apcli_issyn=0
&apcli_repeaterssid=VONETS_12345&ra_off=1&EnDynamicMatchPara=0&isDnsNeedChange=1&allow_motion_dect=0&dhcpEnableButton=0
&ApcliMatchMode=1&ApcliWhtCount=1&ApcliWhtMac0=xx:xx:xx:xx:xx:xx

The line at the end is all one line, it is split here to be visible hopefully without scrolling. I’ve also X’d out private details. Also changed “http” to “hxxp” because of Textile.

Now here’s the URL module POST version. It sets the referrer and the content type and the content itself, the module applies the rest.
Note that this is directed to 192.168.1.30 (my PC) so WireShark can capture what is actually being sent.

POST /goform/wirelessBrdgApcli HTTP/1.0
Connection: close
Host: 192.168.1.30
Referer: hxxp://vonets.cfg/wireless/hotspotsSettingPage.asp?ssid=Livebox-XXXX&auth=WPA2-PSK&encry=AES&bssid=xx:xx:xx:xx:xx:xx
Content-Type: text/plain;charset=UTF-8
content-length: 301
Cookie2: $Version="1"
Accept-Encoding: deflate, gzip
User-Agent: Acorn-HTTP/1.03
Accept: */*

apcli_ssid=Livebox-xxxx&apcli_mode=WPA2PSK&apcli_enc=AES&apcli_ishide=0&apcli_wpapsk=PASSWORD&apcli_issyn=0
&apcli_repeaterssid=VONETS_12345&ra_off=1&EnDynamicMatchPara=0&isDnsNeedChange=1&allow_motion_dect=0&dhcpEnableButton=0
&ApcliMatchMode=1&ApcliWhtCount=1&ApcliWhtMac0=xx:xx:xx:xx:xx:xx

Both look more or less the same to me. There are small protocol differences (HTTP 1.0 vs 1.1) but otherwise, the referrer and the payload should be the same thing.

Only… the server inside the Vonets keeps sending this back:

<html><head><title>Document Error: Data follows</title></head><body>SSID or BSSID is empty</body></html>

Grrr!

There are small protocol differences (HTTP 1.0 vs 1.1)

HTTP/1.0 vs HTTP/1.1 may look like a trivial difference, but it is highly likely that whoever wrote the Vonets software never bothered testing it with an HTTP/1.0 request. This could take it down all sorts of odd code pathways with unknown consequences.

Another oddity is that the “URL module” version specifies “Connection: close”. This differs from the Firefox request which specifies “Connection: keep-alive”. This may be a distinction without a difference, as the payload is small.

Is there any way of forcing an HTTP/1.1 connection?

If the payloads are identical, why are the content-lengths so different?

why are the content-lengths so different?

Probably the editing out of the real password – the Firefox one is bigger, yes?

never bothered testing it with an HTTP/1.0 request

It’s a browser I’m sniffing, not an app. Are there no browsers at all that use HTTP 1.0 these days?

I should add – logging in, retrieving information… all of that works (login is a form POST). It’s just this part that is throwing back an unexpected error.

If you change the order of the parameters so apcli_ssid isn’t first do you get a different error message?

It may be that the first parameter isn’t read properly – possibly early characters go missing. try putting the apcli_ssid parameter in twice or add some space to the beginning of the parameter list.

isn’t first do you get a different error message?

I’d already tried that, stuck in a “dummy_value=1” at the start. Same thing.
I’ve not tried repeating it though.

I’ll need to fiddle with Firefox to see if I can suppress the referrer to see if it’s trying to read the (B)SSID from the request, or if it is merely choking (for some reason) on the content.

This is just frustrating as apart from the headers (due to implementation), the payload looks the same as the browser generated version. I’m looking at hex dumps in WireShark and don’t see anything obvious (like too many newlines between the header and content).

I’ll need to fiddle with Firefox to see if I can suppress the referrer

Insomnia can be useful for customising requests (if you have a machine that can run it).

Rick: the “Host:” header refers to the server, i.e. the virtual host at (in your case) the Vonets end. HTTP 1 assumed a 1:1 equivalence between IP address and server, and the “Host:” header wasn’t in the spec or wasn’t required. In HTTP 1.1, it became a required line, recognising that more than one host could be on one IP address. So I think you need it to be “Host: vonets.cfg”. Whether that fixes the problem is, of course, another matter.

As for your capture method, you should be able to capture on RISC OS using Wiresalmon, and ship the capture file across the LAN to your Wireshark machine. In my case I use Moonfish to an Ubuntu machine.