Printing

If you want headers as well, then it’s 1 + (2 << 8) to set the appropriate flags.

I thought I’d look up the flags in the docs, but URL_GetURL says that the flags are up to the method, and the AcornHTTP method isn’t documented – possibly because it’s only intended to be used by the URL module. But where else would one look up the flags for the HTTP method?

If someone can point me at documentation for the HTTP module, I’ll see about adding the relevent pages to the Wiki.

Matthew, I see you’ve put up a lot of docs – thank you!

If someone can point me at documentation for the HTTP module, I’ll see about adding the relevent pages to the Wiki.

There is no documentation other than the few details accompanying the source code.

If you want headers as well, then it’s 1 + (2 << 8) to set the appropriate flags.

This is terribly badly documented (i.e. not at all). I have been reading through the source and experimenting this afternoon, uncovering the crazy design decisions in this pair of modules.

You know where URL_GetURL says that R5 must be 2 if it is not being used to pass the length of the data block? Well, actually, if R0:1 is zero then R5 bits 0 to 7 will be treated as the “method dependent” value. If R0:1 is set then the “method dependent” value is obtained from R2 bits 8 to 15 as documented.

I can see no good reason for this! Why can they not be passed in R2 bits 8 to 15 all the time?

And as for the values, at present they are not “method dependent” as such, but protocol dependent, and for AcornHTTP these values currently apply:

0 = return body only
1 = return head only
2 = return both
(also any value from 3 to 255 will be treated as 2).

In the case of a redirection (301, 302…) it would be highly useful to extract the Location pointer from the header and send it as the content body.

I can see where you are coming from, Rick, but the alternative to your approach is to send “2” in the “method dependent” value (where in R2 or R5) and always handle the headers yourself. Up until I started playing with IPP, I was always making GET requests, so never had any body to send in my request, so I was always setting R5=2 as instructed. I had no idea you could opt to receive just the body.

Anotehr option would be to have a flag which would tell AcornHTTP just to process any redirection itself.

There’s another thing that’s starting to concern me. AFAICS any extra header lines, and the body, need to be contiguous in a buffer.

There is no mechanism that I can see for POSTing a very large request body, and this is clearly needed for printing. The interface between the URL Fetcher and the protocol modules does not currently facilitate this either. If the URL Fetcher module was enhanced, there would need to be a way for it to know which protocol modules supported the technique. I think a flag in URL_ProtocolRegister would be the sensible indicator.

Chunked transfer encoding, which I have previously mentioned as an example of HTTP/1.1 elements supported in the module, is only supported for the response body. It is automatically decoded and the client is presented with an unchunked response. I don’t even know if HTTP allows it for the request body.

The other major improvement I would like to see, which would also require the protocol module interface to be changed, would be support for non-global cookie storage. At present all cookies processed by AcornHTTP go into a single global cookie file and a single memory store. This is not only useless for security but also prevents the implementation of common functionality like “private browsing”.

I think we’re going to have to thrash out an updated specification for URL Fetcher and the protocol module interfaces. I would suggest starting another thread, perhaps in Code Review? I might have a go in the next few days if I have the energy.

and the AcornHTTP method isn’t documented

It very definitely is.
https://gitlab.riscosopen.org/RiscOS/Sources/Networking/Fetchers/HTTP/-/blob/master/Docs/SWIs

and always handle the headers yourself.

Why? The point of using something like the URL fetcher is to make things easier.

This is not only useless for security

Ooh, I’ll quietly back away from this one.

My personal policy is to allow all cookies, and any that aren’t explicitly whitelisted get summarily deleted twenty seconds later. It’s why I’m using an older Firefox, it can run the add-on that does this.

I’m not entirely sure I’d want to use a browser where I cannot say “periodically nuke these cookies”. Session cookies aren’t finely grained enough and online tracking is insidious to the point of being disturbed. You need to allow a lot to get several modern sites to load properly, but once that’s done everything it needed should be tossed. No if but or maybe. And f*** “legitimate interest”.

(on Chrome, when I use it on my other phone (as right now) I wipe site data (cache and cookies) daily…not great but better than nowt, I really ought to get Firefox running on this device)

This is terribly badly documented (i.e. not at all).

It’s definitely documented somewhere, but I can’t find it at the moment! [Edit: See Rick’s post above]

It was actually broken prior to 2018, when I submitted a fix to make it work in accordance with the docs, which is why I’m confident that the docs exist… somewhere :)

It’s definitely documented somewhere

I stand corrected. I have looked at that file in the last couple of weeks, but I was concentrating on the cookie SWIs then.

and always handle the headers yourself.

Why? The point of using something like the URL fetcher is to make things easier.

Returning the Location header instead of the content body when the client has asked only for the body might be a bit unexpected, though. When servers send a redirection with a Location response header, they often send a body as well, with HTML giving the user a redirection link.

I think a better API would be to have a flag that allows the client to ask the URL Fetcher module to handle any redirections automatically.

On cookie security, I wasn’t really thinking about whether the user wants to keep them, etc., but more about whether other running applications can (a) access them or (b) would get them added to their requests unexpectedly.

If you are against tracking cookies, then the problem with the current arrangements is that if I write an application which goes and gets a few things from web APIs, and accepts all the cookies I am offered into the in-memory store, then an application written by someone else might find those cookies being added to its requests. If you sign into the ROOL Forum on a web browser that uses URL_Fetcher (like Browse) then any other browser using URL_Fetcher running on the same machine would find itself signed in also.

Write an application to do online banking via a bank’s API, and other applications on the same machine can access the session cookies.

I’ve put the flags into the documentation of HTTP_GetData. It’s not the tidiest, but I can’t immediately think of a tidier way, and there’s only so much struggling with Textile that I can take in one session.

Maybe more tomorrow.

I’ve tweaked your new page a bit. I have also created a number of wiki pages to document the Protocol module interface which previously was only available via a plain text version

I am working from an HTML version of the same document, and it wasn’t lacking the tail end of URL_ParseURL 3 so I have fixed that!

There’s probably more I could add about the Service calls, as well as documentation specific to AcornHTTP.

Once it is all up, we’ll be in a better position to debate how to go foward with the major changes that are needed.

I’ve tweaked your new page a bit.

Thank you, Matthew. I wish I’d thought to do it like that!

There is no mechanism that I can see for POSTing a very large request body, and this is clearly needed for printing.

During the course of my work on IPP, I created a sprite from a 600dpi printout of an A4 page. The sprite ran to about 130MB. The significance of this information is only that a full A4 page containing an image that doesn’t lend itself to compression (think of a photo where no two adjacent pixels are exactly the same colour) would produce a bitstream of the same size, give or take a little bit. And that’s only 600dpi (the printer I keep interrogating can do 1200 by 4800 dpi!) and only A4. This means it is relatively easy to create a bitstream that cannot fit in RAM of some machines. The bitstream would have to be written to a temporary file.

So I think it will be vital to devise a method of dealing with a very large request body from a file.

There are some more things that worry me about IPP printing. First is that the dumper needs to know (or to be able to resolve) the IP address of the printer it should send to. It’s possible to find an address by searching for appropriate devices, but this can only work if the dumper can only see one compatible printer – 2 or more (which I suspect is not all that rare) and it won’t know which to send to.

I’d like to be able to pass that information through from the Printer Definition File, but I can’t see any mechanism to do that. It seems to be retained at a higher level. If the dumper could even know the name of the PDF, it could parse it itself… but the same limitation applies.

Also I can’t see that the dumper has access to the total number of pages in a print job – this is a field in the IPP header. Without that, every page would have to be sent as a separate job of one page.

If I’ve missed something, I’ll be happy to be told!

The plan that I had in mind (but never implemented) was to “print to sprite” in a manner akin to SPrinter, and that would be all the printer driver did. Once the print job was completed, the sprites would be translated into that the printer was expecting (at the time I was experimenting with URF images), and then would be batched up and sent, once the amount of data, number if pages, etc was known. As well as which printer to send to.

Writing a custom dumper, it would make more sense to output directly in printer ready form (this probably would have happened in time), but as far as I could determine it would still need to be done after the fact unless, as you suggest, each page is a separate job.

a few minutes later…

PWG5102.4 says that the total page count can be 0 if the total page count is not known when the file is created. Sigh of relief.

Well, the first stage of the custom IPP dumper is written. Still held together with duct tape and string of course. So far it only attempts 24-bit RGB (srgb-8), single sided, short edge first and all those sort of defaults, and it doesn’t yet attempt compression – but that’s only an optimisation. I can relatively easily add monochrome. I ought to be able to add Uniraster, since it appears that all the negotiation with the printer is the same for Airprint as for IPP, and the bitstream generation is only a small amount of the whole. But unless it integrates with the printing system, it will remain techies-only.

I used my PDumperIPP to generate a bitstream from a noddy Drawfile, and used ipptool to send it to my son’s printer. Out came a printout that resembled the original.

So far, so good.

But what I haven’t worked out yet is exactly what to send to the printer. ipptool’s log shows stuff under the “Print-Job:” heading that I wasn’t expecting, and I’m not sending. I thought it was as simple as sending the PWG-Raster file as the body of a POST request, but that doesn’t seem to work. I’ve not spent enough time yet looking at ipptool’s sources, nor the copious documentation from the PWG (jeepers, what a lot of standards documents). Maybe I need to send some stuff IPP-formatted, followed by the bitstream. Maybe I have to MIME-encode the two parts. Dunno yet.

It seems there has to be a small IPP preamble before the printable bitstream.

I used my PDumperIPP to generate a bitstream from a noddy Drawfile, and used ipptool to send it to my son’s printer. Out came a printout that resembled the original.

Excellent work/news Dave.

Out came a printout that resembled the original.

Whoo! This is a great development. :-)

If you can support URF bitmaps as well (AirPrint, uses IPP transport IIRC, just an older bitmap format) then this will allow RISC OS to be used with most (if not all) WiFi printers…which honestly is a far better situation than the mess of drivers we had to deal with in the ’80s and ’90s.

If you want somebody to test with a DeskJet 3630 inkjet (IPP+AirPrint) and a Samsung SL-M2022W laser (AirPrint), you know my email address.

Sounds fantastic, Dave.

I have a XEROX laser printer that works a treat with CUPS, so I’d be delighted to take part in any beta tests.

If you want somebody to test

I have a HP Deskjet 3637 inkjet (IIP-Everywhere/AirPrint1.4) and HP 254DW (Airprint 1.6) available for testing if required.

Another small step forward: I’ve sent my bitstream, using my code, and got a printout. So at least I’ve discovered the secret sauce that’s required to send jobs.

The code uses the URL module, but the challenge is to build the extra header lines, the preamble and the bitstream into a buffer to be sent. Particularly as the header lines include Content-Length, which states the length of all the stuff following.

Edit: original Drawfile and a photo of the result at https://davehigton.me.uk/Printing/Stage1.zip

The printer is something like 20 miles from me. That’s the wonder of the Internet, innit?

Thank you for the offers to test my dumper. It will be a while before I can generalise it enough to pass it out to others. The first obvious failing is that there isn’t a way to specify the printer’s IP address or DNS name. The second is that the Printer Definition File needs to be created with knowledge of the printer’s available resolutions. This is not unduly difficult, but it isn’t quite as usual, I think.

Anyway, I keep plodding on, and I will get stuff out…

Particularly as the header lines include Content-Length, which states the length of all the stuff following.

Doesn’t the fetcher sort out the content length for itself from the size of the data given?

The printer is something like 20 miles from me.

Well that is impressive. :-)

The first obvious failing is that there isn’t a way to specify the printer’s IP address or DNS name.

I wonder if it could be handled on a per-printer basis? For example, my LJ6 printer specifies to connect via RemotePrinterFS to an IP address. Perhaps the Connection settings in Printers could be used to provide something similar?
It’s not quite as flexible as on Android (“look, I found a printer, wanna dump this to it?”), but given that you just printed to a device in another town, one can’t argue. ;-)

The second is that the Printer Definition File needs to be created with knowledge of the printer’s available resolutions.

Hmmm… I wonder if there could be some mileage in something akin to my FindIPP that can scan a local net (or a specified IP address) and retrieve the relevant bits of information to auto-generate the PrDefFile?

Anyway, I keep plodding on

What you’ve done so far is bloody marvellous. I’d give you a big thumb up, but this is RISC OS and it doesn’t have built-in emoji. So, I dunno… =^. or something? ;)

I wonder if there could be some mileage in something akin to my FindIPP that can scan a local net (or a specified IP address) and retrieve the relevant bits of information to auto-generate the PrDefFile?

Yes. Does that make you a volunteer? :-)

Doesn’t the fetcher sort out the content length for itself from the size of the data given?

No, it doesn’t! It does rewrite the incoming Content-Length of a response in some circumstances.

When calling URL_GetURL you can give it a block of data to send, but that includes any additional headers. The URL Fetcher is designed to work with several different back-end protocols, but insufficient abstraction has been done, and the client therefore has to know rather a lot about the protocols to get any serious work done.

Yes. Does that make you a volunteer? :-)

… and if it does, it’s my turn to send you some code. I use the URI module, which makes the code much shorter – and it makes it dead easy to try http, then https if the first fails. Or vice versa.

There are some printers that won’t accept http (my son’s is one), and I believe there may be some older printers that won’t accept https, so I think it’s important to be able to try both.

Re. Content-Length: that’s a hangover from when I was trying to use the sockets API. I gave up because it’s too difficult to swap between http and https, whereas the URI module does it effortlessly. So thank you for the reminder!

But, if I pass the length of my data including headers, does the fetcher put in a correct Content-Length header for the body only? The content is binary, so AFAICS the Content-Length has to be correct.