Enhancements for the URL module

Some suggested enhancements for the URL module when fetching HTTP/HTTPS resources:

1. There’s a flag to request headers to be included. Add this to the documentation file. I had to wade through source to figure out 2<<8… :-/
2. In cases where the HTTP response expects to see additional information within the headers (like 301 → Location:), this information should be copied out to the data block passed in. Keep the “bytes read” as zero so as not to upset any existing code that might try reading data if it is flagged as being there, otherwise set the start of the data block to be a NULL terminated string (if it fits) with applicable data.
   This is important because in the case of a redirection, one needs to use the undocumented flags to get the full headers to try the request again to figure out where to go. It’d just be so much simpler if the URL module could return appropriate information.
3. Related to the above, a request flag that will permit the URL fetcher to actually follow redirections [it’s up to the implementation if it bothers to notify the callee of this, or just do it transparently]. This way, the callee can say “please fetch this” and have it “simply work”.
4. A command like “URLfetch” that will work like URLfetch $.blogpage hxxps://heyrick.eu/blog/ (as its always useful to have a command to fetch stuff).
   [hxxps to fool Textile ;-)]

I know all about #1! :)

The trick here is that the head/body stuff is specific to HTTP and doesn’t apply to the other protocols supported by the URL module. Therefore, it’s described in the HTTP docs rather than the URL docs.

Bits 8-15 of R2 are the type of data wanted (if R0:1 set, else in bits 0-31 of R5)
0: Body only
1: Head only
2: Both

With that said, I think there’s probably some logic in having a simpler document that explains how to use the module for common tasks, without getting bogged down in implementation detail and cross-references.

#2: It’s too early in the morning for me to parse this :)

#3: This on my to-do list for HTTPLib but I’m not there yet. Having it in the underlying module would certainly make things easier though.

#4: Agreed.