MangaReader and AcornSSL

The MangaReader website seems to have reverted back to using SSL; however it isn’t working with AcornSSL.
It churns a few times, until a call to URL_Status reports that it cannot connect.

Are others seeing this too?

Try this URL: https://www.mangareader.net/hyakumanjou-labyrinth/1

Seemed to take me to something totally incomprehensible with jagged speech balloons and Japanese stuff.

I seem to have AcornSSL 1.06 available here.

What the implications of this are I leave to you to interpret.

Using AcornSSL v1.06 (03 Jun 2020) mbedTLS 2.16.7
and the HTTP_Client that was using for testing SSL, it responded with 384 bytes to the get … which did include the text ‘400 Bad Request’. One of those helpful errors which are designed to hinder! The request was simply
GET https://www.mangareader.net/hyakumanjou-labyrinth/1 HTTP/1.0
which looks what I would expect.

I also see ‘Server: cloudfare’ in the returned headers.

I also see ‘Server: cloudfare’ in the returned headers

That could be the origin of the problem.
The stated purpose of the cloudflare setup is to provide resilience for websites.
My experience is that bypassing the cloudflare service can often be faster and more stable. That’s Windows based browsing so there’s no RO quirk in the mix.

Seemed to take me to something totally incomprehensible

Using what? It works with NetSurf, but that doesn’t use AcornSSL/URL.

which did include the text ‘400 Bad Request’.

Ah, you got further than I did. I’ll need to check my AcornSSL is the latest (if only it had unique version numbers…).
Odd thing is, I got as far as that with SecureSockets.

I also see ‘Server: cloudfare’ in the returned headers.

Oh. Crap. Appeasing that bloody CDN is a major pain.

bypassing the cloudflare service can often be faster and more stable.

And how does one do that? https://shadowcrypt.net/tools/cloudflare tells me that 104.22.41.215 is protected by Cloudflare, while 185.87.25.235 is not.
Guess what annoying CDN answers the doorbell at the second IP, if given directly to the browser?

And how does one do that? https://shadowcrypt.net/tools/cloudflare tells me that 104.22.41.215 is protected by Cloudflare,

Ah, working with suppliers that use cloudflare and knowing from them the backdoor. I wanted to test and eliminate the cloudflare element since everyone was blaming internal “network” issues (as ever) and I wanted to test the whole chain via NHS net and also on a direct Internet connection.
I replicated the flakey behaviour on the direct and removed it when given direct access to the actual server.

Using wget with the —no-check-certificate option I managed to download an HTML file

Below the log:

Resolving www.mangareader.net… 104.22.41.215, 104.22.40.215, 172.67.28.55
Connecting to www.mangareader.net|104.22.41.215|:443… connected.
WARNING: cannot verify www.mangareader.net‘s certificate, issued by ’CN=Cloudflare Inc ECC CA-3,O=Cloudflare\\, Inc.,C=US’:
Unable to locally verify the issuer’s authority.
HTTP request sent, awaiting response… 200 OK
Length: unspecified [text/html]
Saving to: ‘RAM::RamDisc0.$.manga1’

Last-modified header invalid — time-stamp ignored.
2020-08-14 14:41:51 (829 KB/s) – ‘RAM::RamDisc0.$.manga1’ saved ³³⁹⁷⁰

Unable to locally verify the issuer’s authority.

Ah, so now there are two issues to deal with:

1. I’ll need to set up SNI or see if I can remember the magic incantation to tell AcornSSL not to validate the certificates.
   Simply ignoring might be easier, given the number of subdomains and the fact that security isn’t necessary (it’s been plain http for ages).
2. Faking headers sufficiently to trick CloudFlare into believing it’s talking to a real browser. NetSurf works, so I could just copy the headers from that, perhaps? The problem here is that the URL module fiddles with headers itself. Hmm…

Bloody Cloudflare…

BTW: Probably won’t happen in a hurry. Pi is off (still forecasting thunderstorms (until Monday), and although it’s a sun/clouds kind of day, that can change in a mere quarter hour). It’s also the end of my holiday. Back to work on Monday. Jeez, that went quickly…

Bloody Cloudflare…

Different wording, same sentiments as my experience…

I was thinking about this earlier today while clearing bramble chaff, and I realised… it isn’t my problem.

I don’t talk to AcornSSL, I talk to the URL fetcher.

So the question is – does the URL fetcher support SNI and/or “accept all certs”. If so, is there a flag I’m missing? If not, why not?

Looking further on, the chain of command is:

App → URL → HTTP → AcornSSL

It appears as if SNI is supported by the HTTP part of the fetcher (HTTP connect.c in the function opensock); but all I’m getting from my test code is an “Unable to connect to remote host” error.

Is there any way to get HTTP/AcornSSL to report why it failed?

I have tried to replicate the same HTTP headers as NetSurf (within the understanding of how URL/HTTP messes with the headers itself); so I don’t know why this one is failing.

Note a surprise the HTTP client test fails. What is this rubbish?

DIMbuffer% 4096
$buffer%="GET "+prefix$+"://"+host$+page$+" HTTP/1.0"+CHR$10+CHR$10
bufferlen%=LEN($buffer%)

I’ve replaced that with a horrid kludge to work around BASIC’s interpretation of CRs to mark string termination… ;-)

REM Request the page
DIMbuffer% 4096
crlf$ = CHR$(123)+CHR$(10) : REM Work around BASIC's use of <13>!
$buffer%  = "GET "+page$+" HTTP/1.1"+crlf$+"Host: "+host$+crlf$
$buffer% += "Accept: */*" + crlf$ + "Accept-Encoding: gzip" + crlf$
$buffer% += "Accept-Language: en, *;q=0.1" + crlf$ + "DNT: 1" + crlf$
$buffer% += "User-Agent: NetSurf/3.8 (RISC OS)"+crlf$+crlf$+CHR$(0)
bufferlen% = 0
REPEAT
  bufferlen% += 1
  IF buffer%?bufferlen% = 123 THEN buffer%?bufferlen% = 13
UNTIL buffer%?bufferlen% = 0

FOR l% = 0 TO bufferlen%
  PRINT CHR$(buffer%?l%);
NEXT

Using the latest CertData, and AcornSSL (1.06/2.16.7), the result is:

*HTTP_client
Got session handle 542343476
DNS record for <a href="http://www.mangareader.net">www.mangareader.net</a> is 104.22.41.215
Connected to host on port 443
GET / HTTP/1.1
Host: <a href="http://www.mangareader.net">www.mangareader.net</a>
Accept: */*
Accept-Encoding: gzip
Accept-Language: en, *;q=0.1
DNT: 1
User-Agent: NetSurf/3.8 (RISC OS)

Requesting page /..
Error block at data write:
Address  :     7 6 5 4     B A 9 8     F E D C     3 2 1 0     7 6 5 4     B A 9 8     F E D C     3 2 1 0 :            ASCII Data
20002A24 :    00813F27    646E6148    6B616873    72652065    20726F72    61747328    33206574    39352C30 : '?Ŵ.Handshake error (state 30,59
20002A44 :    20002932    6E756F66    756F0064    6E00646E    6620746F    646E756F    20796E00    61776E75 : 2). found.ound.not found.ny unwa
..
Failed writing 154 bytes; errno = 813F27 
*

Ah, there’s something in the dumped error block – Handshake error (state 30,592).

Perhaps I need to patch SNI into the thing now… hmm…

But this is low-level talking to AcornSSL, not via URL.