early demo of a RISC OS module to integrate the CLI with HTTP Protocol

For everyone interested,
I just released a video with a quick and early demo of a RISC OS module that extends the CLI commands to integrate with HTTP Protocol.

The inspiration comes from Gerph and Stuart Swales ideas shared during last ROUGOL meeting, in which I quickly coded an app using cURL, but cURL relies on UnixLib and so doesn’t fit the RISC OS Module approach. Hence I recoded it using AcornHTTP/AcornSSL etc…

This is partially for my interest in using RISC OS for IoT, but it can be used with WIMP applications as well and I am thinking of replicating excellent Chris Mahoney HTTPLib naming conventions to make the two works in similar ways (so they would look familiar etc.) thoughts?

link to article and video here:

https://paolozaino.wordpress.com/2020/12/19/risc-os-a-module-project-to-integrate-the-os-cli-with-http-protocol/

Neat!

Why should the command line interpreter need to be amended? Surely all you have done is to provide *commands for some internet modules?

@ Chris Hall

Thanks for your questions.

Why should the command line interpreter need to be amended?

Few reasons:

• help automation
  • allowing a TaskObey to also communicate with a CI/CD when building stuff
  • quickly push a notification to services like Slack after a build with a build result
• help software prototyping
  • Focus on the logic you want to build instead of surrounding details
• help less skilled developers to still be able to build more powerful applications
  • let’s empower everyone to code on RISC OS instead of an “elite”
• HTTP is everywhere and Linux already has curl to use it in every possible form and quickly, RISC OS can use cURL too, but a module would make things even faster ;)

There is more, but I believe the above are enough, if not please let me know, I am happy to answer any question :)

Surely all you have done is to provide *commands for some internet modules?

Actually no, have you watched the video with the demo?

The module presents a set of commands that are not a wrapper of existing SWIs in AcornHTTP and AcornSSL, for example a test command right now allows this:

* HTTPtoFile <url> <destinatio-file>

This will pull down the URL content into a file, you can also use RISC OS pipeFS for this for example or a network resource if you want all in a single command and no special changes if the url is HTTPS. A similar method will be available to upload (or POST content).

I am also adding support for authentication and token management.

Also, please don’t forget this is a Module and so has to be able to handle multiple requests at the same time and finally to handle all the requests and put not pressure on RISC OS kernel it uses queues to store the requests and serve them in a FIFO fashion, so also warrantee the order and potentially may work with Jeffrey SMP (this is planned, but not yet done) where a piece of code executing in a different core would make a request to FetchAURL and such request would be satisfied on RISC OS core and then notify the caller.

Hope this answer your last question (or correct your statement as it sounded like a statement as well), sorry if I did not understand your questions correctly.

That sounds great Paolo :-)

Also, please don’t forget this is a Module and so has to be able to handle multiple requests at the same time…

At the moment is it best to run from a task window like in your demo? Does it keep going if you press F12?

or a network resource if you want all in a single command

What did you mean by this?

where a piece of code executing in a different core would make a request to FetchAURL and such request would be satisfied on RISC OS core and then notify the caller.

Sounds like an interesting way of offloading network requests while the application that made the higher level request gets on with something else.

I can see how you could potentially use that to pull video content and push that to a location the video renderer can operate on it. Perhaps even throw the video at the GPU?

@ Julie

Thanks!

At the moment is it best to run from a task window like in your demo? Does it keep going if you press F12?

I am testing it without the TaskWindow mostly, so does not needs the taskwindow.

For the temporary exit from the WIMP (F12): so the module doesn’t requires the WIMP at all, I am planning to work it as a driver, so it may use IRQs or other possible solutions like callbacks to keep working in the background. However suspending the WIMP via F12 may need some thinking, for example:

• If a user just requested HTTPToFile method then sure it should be fine to keep working
• If a WIMP application is using HTTPKeepProcessing this is a unit of work and if the WIMP doesn’t keep calling it (for example during a NULL event) then it won’t progress. This is to integrate the module with WIMP application and so I think it’s ok to freeze it in case of F12.

Quick and dirty example of HTTPKeepProcessing:

HTTPKeepProcessing <token>

Which (when completed) should allow a WIMP task to request the module to progress in the operation it originally requested (for instance GET, POST, etc.).

However this is where I am at with my thoughts right now, if there are things I am not considering and I should please let me know :)

Please do not assume the final command set will be the one I am showing in these early examples, because I think it’s best I re-use Chris Mahoney naming conventions so that the module and HTTPLib appear to work in a similar way to the user and so they both are more familiar and easier to use. I think Chris did an excellent work on his HTTPLib, so no reason why I should implement something completely different in the eye of an end user.

or a network resource if you want all in a single command

What did you mean by this?

What I mean is you can store an HTTP content straight into (or fetch it from to be uploaded):

• pipeFS
• OmniClient shared folder
• Access/Access+ shared folder
• Other protocols depending on time and possibilities

Also, on the future requirements I am thinking to:

• Integrate it with SpackFS (create, open archives and either put content into them directly or fetch content from them and push it via HTTP)
• Add a JSON Parser module (external to the FetchAURL, aka in a separate module) to allow JSON parsing on the fly even via Obey/TaskObey/BBC BASIC/ASM/Whatever :)

However everything has to be small and in modules to avoid having to constantly load extra code like it would happen using cURL for example. When I’ll push on github my initial cURL integration effort of the previous attempt you’ll see what I mean, cURL is big even when I used latest cURL3 available for RISC OS and it’s still static linking so pretty heavy on a cheap SD card to be constantly called.

While FetchAURL is very small (so suitable for devices with small amount of memory) and stays in memory and so it’s very very fast at being accessed and obviously very simple to use, it should have many available commands in the end, but for now I need to make it as solid as possible and possibly deal with FTP protocol as well.

Hope this answers your questions and thanks for asking! :)

@ Steve Pampling

I can see how you could potentially use that to pull video content and push that to a location the video renderer can operate on it. Perhaps even throw the video at the GPU?

In theory yes the content can be delivered to whatever you want, in practice when we talk about streaming we talk mostly about UDP and Multicast which is not a domain of this module, however we could reuse the code (when finished and working fine) for another module that focuses on UDP based protocols.

What I am trying to point out mostly out of this work is the strengths of RISC OS compared to other OSes (obviously in the IoT field, but I do not see why they can’t be reused to improve the WIMP experience as well).

Believe it or not there is an impressive effort on the IoT market to make Linux smaller, lighter, cooperative… well RISC OSish if you want, so why not start from an OS that is already there.

Think this:

- uClib, musl Libc etc.. these are all attempts to make GNU Clib smaller and lighter weight
- Alpine Linux etc.. all attempts to make Linux suitable for low power and small devices

All stuff that is not yet even close to RISC OS size and requirements, so believe it or not RISC OS there has advantages, but yes it also needs some improvements like SMP support, newer network stack and a bit more security :D

Now the newer network stack is at a very good stage, Andrew showed it working on last ROUGOL meeting and it was awesome, thanks again to Andrew for that presentation!

Jeffrey did some miracle (as usual!) with the SMP library and he is still working to improve it

So we are not far away from where we should be, definitely much more close to it than the Desktop System for everyone¹.

¹ Which even if I support it as a user I totally realise it’s going to be a titanic work.

Paolo it looks very interesting.

I currently use wget for a lot of my apps that collect info from APIs which whilst it is downloading the info turns the computer into single tasking.

Would your module allow multitasking?

@ Kevin

Would your module allow multitasking?

Yes, the purpose of HTTPKeepProcessing is exactly this, to allow a WIMP application to keep pulling things down during a null event for example or any other event you want.

Also the other advantage you’ll have is that FetchAURL is in memory, so no SD access to load wget to execute a fetch (or cURL for that matter). This is a big thing on RISC OS because RISC OS doesn’t caches FileSystem operations (like Linux does for example), so there is never an advantage to keep running the same command from a slow SD over time.

And finally I/O operations freezes the WIMP, so by not using the SD you also make the overall experience better. I mean I am pretty sure that Gerph and Stuart knows one thing or two about RISC OS ;) so if they asked for it there are definitely advantages into the approach.

And finally I/O operations freezes the WIMP, so by not using the SD you also make the overall experience better

The major reason I keep suggesting the I/O is run on another core and offers the result to the OS on the WIMP core (for want of a better label)
Of course the twisty, loopy, spaghetti of the kernel/wimp/keyboard handling doesn’t help.

@Paolo

Yes, the purpose of HTTPKeepProcessing is exactly this, to allow a WIMP application to keep pulling things down during a null event for example or any other event you want.

Thanks I will look forward to playing with it in due time.