Blocking on input stream or interrupts

How can a RISC OS Module block until the input stream (as read by OS_ReadC) to has characters available to read or particular interrupt occurs? This should work inside and outside a TaskWindow, with and without file redirection etc.

The overall goal is to, on the Linux port of RISC OS, is the start a Linux program and feed characters from the RISC OS input stream (OS_ReadC etc) to program and the programs output to OS_WriteC.

A busy wait is easy, the challenge is to wait without using CPU time.

You can WFI in a loop to halt the core. You’ll probably want to thread it via RTSupport with another thread that checks your input source for data.

It depends of course on what you mean by “block”. If you really want to block all other code from executing, turn off all other IRQ except for your data source, use WFI in a loop and only drop out once you’ve handled the IRQ.

If it only needs to be a user mode process, give the module a start entry point, and use a *Command to enter the module using OS_Module 2. In the start code set up your environment handlers to allow exit on escape, and sit there reading characters. It will then block on the desktop until finished, but also play nicely if run in a TaskWindow.

If you want to know if there is a character available to read, then check the number of bytes in the input buffer.

If you want to yield to the rest of the system OS_UpCall 6 (TaskWindow_Sleep), 0 (or any address which has a non-0 value) – I believe that address 0 is assumed to be non-0 by the main user of this call, TaskWindow, although it could strictly be used by anyone else (threading solutions have used this in the past to indicate a thread yield).

If you want to read a character if there is one, and continue if there isn’t without blocking, use OS_Byte &81 (INKEY), with a 0 or small value (if you wish to combine yielding to the system and checking for input).

If you want to wait for a particular hardware interrupt, then I guess you’re going to need to have a module that claims the interrupt handler, and marks the interrupt as happening, and then wait for that interrupt in a separate mechanism. If you don’t mean a hardware interrupt, but just mean ‘some other external event’ then you’ll need to check for those events separately. If your event is a mouse click, then you can check the mouse buffer, or just try to read the most recent mouse event. If the event is socket data or transition, then a Socket_Select call is what you’ll need (which can itself be used as a yield point, because you can supply the timeout which will yield to the system, if you’ve used ioctl FIOSLEEPTW). Other events you might be able to trap separately, but can catch with a module that traps the event and sets a pollword. I suspect that from your intention you don’t actually mean an interrupt, but just that you want to perform the streaming between the two environments smoothly.

Amusingly (or not… it depends on how you look at it), I’m implementing something this is very similar at the moment. I shall explain what I’m doing, in the hope that this helps you:

The host process is executed in a different thread to the core system. This thread is communicated with using an object which arbitrates access to its output in a thread safe manner. The output from the process is buffered in chunks and able to be read from the object. The object can be sent data which will be given to the process as input. It may have a working directory that reflects the environment that your RISC OS system is working in. Or it might not. You might want the process to be interruptible when the user presses escape.

You may want to pass input, or you might not – there is a down side to passing input to the subprocess. In doing so, you may drain the RISC OS buffer to no effect. For example, consider the process you’re running is ‘sleep 10’. If you blindly pass input from the user to that process, you’ll basically be sinking that data, and the user’s typing in those 10 seconds will be lost. If, on the other than, you aren’t dealing with the input pass through to the subprocess, the user’s typing will remain in the buffer and then become the input to the next reader. You cannot know whether the process is waiting on input (actually, that may not be true; it might be possible to check the subprocesses state to see whether it’s become blocked due to SIGTTIN, but I have not investigated whether that’s even feasible).

So anyhow, that’s why passing input may be an option, rather than a requirement.

The output from the subprocess will need translating by encoding to make sure that it does the right thing – you don’t want to be outputting things from your host-side process which are not readable in the RISC OS-side. You know the RISC OS encoding (it’s the alphabet) and you can guess that the encoding of the host-side is UTF-8. It might not be, so let’s make it configurable.

Your input is the same – you know it’s in the RISC OS alphabet, and – again – you guess that the host side is UTF-8.

Your question was specifically about how you efficiently check for the state; well, my solution isn’t ideal but it involves sleeping to ensure that you give back CPU sufficiently. I slice up the events into 1cs chunks, processing a set of output from the process chunk stream, and set of input from the RISC OS side, and then sleep for a while. Escape checks are performed (if the process is interruptible), and we trigger any other events explicitly (in the RISC OS world this is callbacks and timers and other events that might be pending).

Let’s have some code, because talk is cheap…

    def run(self, cmd, cwd=False, ansiescapes=False, encoding='utf-8', env=None, shell=True,
            interruptible=False):
        """
        Run a host command and capture the output.

        @param cmd:     Host command to run
        @param cwd:     Whether to change working directory:
                            True - change to native working directory
                            False - don't change directory
                            str - change to directory as a RISC OS path
        @param ansiescapes: Whether to convert ANSI escapes in the output (True), leave the output as-is (False),
                            or strip the ANSI entirely ('strip')
        @param encoding:    Encoding for the output of git commands to turn into current locale
        @param env:     Environment to pass to the process, or None to use the current environment
        @param shell:   Whether to invoke through the shell
        @param interruptible:   Whether escape will interrupt the operation

        @return: returncode
        """
        (returncode, _) = self._run(cmd, cwd, ansiescapes, encoding, env, shell, interruptible, capture=False)
        return returncode


    def _run(self, cmd, cwd=False, ansiescapes=False, encoding='utf-8', env=None, shell=True,
             interruptible=False, pass_input=False, capture=False):
        """
        Internal implementation to run a command and capture the output.

        Additional parameters:

        @param capture: Whether we capture the output or write to the screen
        """
        pass_input = True

        fh = None
        if capture:
            fh = io.BytesIO()

        if ansiescapes:
            writer = WriteANSI(self.ro, fh=fh, strip=(ansiescapes == 'strip'))
        else:
            writer = WriteRaw(self.ro, fh=fh)

        if cwd is True:
            cwd = self._native_cwd()
        elif cwd is False:
            cwd = None
        else:
            cwd = self._native_cwd(cwd)

        # The maximum number of characters that we'll feed to the lower level before waiting
        max_input_per_loop = 100

        # The maximum number of output bytes we'll handle before breaking out of the loop
        max_output_per_loop = 1024 * 8

        with ThreadedStreamedInput(cmd, shell=shell, cwd=cwd, keep_stderr=True, env=env) as si:
            completed = False
            interrupted = False
            last_newline = True
            while not completed and not interrupted:
                if not si.is_running():
                    completed = True

                output_written = 0
                for chunk in si:
                    timestamp, data = chunk

                    # Convert the output WRT locale
                    data = self.convert_host_output(data, encoding=encoding)

                    # Write the output to the VDU
                    writer.write(data)
                    last_newline = data and data[-1] == '\n'

                    output_written += len(data)
                    if output_written > max_output_per_loop:
                        # No more output this loop
                        break

                    # Trigger any outstanding events from the Kernel (eg Ticker events, or escape)
                    if interruptible:
                        if self.ro.kernel.api.escape_check():
                            interrupted = True
                            break
                    else:
                        self.ro.kernel.trigger_events()

                if pass_input and self.ro.kernel.input.is_pending():
                    for _ in range(max_input_per_loop):
                        key = self.ro.kernel.api.inkey(0.01)
                        if key is None:
                            # Escape condition
                            if interruptible:
                                interrupted = True
                            else:
                                self.ro.kernel.api.escape_clear()
                            break

                        if key == -1:
                            # No input pending
                            break

                        try:
                            if key == 13:
                                # Return in RISC OS is CR, which is expected to be NewLine when
                                # supplied to host processes
                                key = 10

                            data = chr(key)

                            # Convert RISC OS key input to host input using the host encoding/RISC OS alphabet
                            data = self.convert_host_input(data, encoding=encoding)

                            si.send(data)
                        except IOError as exc:
                            # Almost certainly a broken pipe; we'll just ignore that.
                            #print("IOError: %r" % (exc,))
                            break
                else:
                    # Wait 1cs between checks for output
                    time.sleep(0.01)

                # Always trigger RISC OS events, even if we didn't do anything
                if interruptible:
                    # escape_check implicitly triggers events
                    if self.ro.kernel.api.escape_check():
                        interrupted = True
                        break
                else:
                    self.ro.kernel.trigger_events()

            if interrupted:
                si.stop()
            if not last_newline:
                # FIXME: Decide if this should actually be an option
                writer.write('\n')

            if capture:
                captured = fh.getvalue()
            else:
                captured = None
            return (si.returncode, captured)

The bits you’re asking about are essentially self.ro.kernel.input.is_pending() – which is effectively ‘check whether there’s anything in the input buffer’ – then an INKEY within a loop with a short timeout (1cs, because this will be my effective sleep and reading the keys, which avoids an explicit sleep).

The actual time.sleep() call would probably be implemented as an OS_UpCall 6 in a real RISC OS, and since you’re managing the host process on a separate thread you could just poke the memory you supplied as the pollword, and then the system would wake up quickly if you were running in a TaskWindow. In my case, I’ve not cared about such cases.

This has probably digressed further than you’re interested in, but given the similarity of what I’ve implemented to what you’re asking about, I hope it’s of some use.

If you want to know if there is a character available to read, then check the number of bytes in the input buffer.

TaskWindow provides no API to do this.

If you want to read a character if there is one, and continue if there isn’t without blocking, use OS_Byte &81 (INKEY), with a 0 or small value (if you wish to combine yielding to the system and checking for input).

As is, the kernel OS_Byte &81 (INKEY) can return indicating no key is available when the input buffer is not empty. I wrote a change that fixes this, without realising I was fixing my future problem. TaskWindow helpfully does not have a similar race condition.

If you want to yield to the rest of the system OS_UpCall 6 (TaskWindow_Sleep), 0 (or any address which has a non-0 value)

Also undocumented, OS_UpCall 6 will return if a new key is in the input buffer.

A loop could look like this :

_kernel_irqs_off();
while(...) {
  *pollword = 0;
  int i = inkey(0);
  if (i >= 0) write(tty, &i, 1);
  while(read(tty, &c, 1) != -EAGAIN)
    os_writec(c);
  os_upcall(6);
}

You may want to pass input, or you might not – there is a down side to passing input to the subprocess. In doing so, you may drain the RISC OS buffer to no effect. For example, consider the process you’re running is ‘sleep 10’. If you blindly pass input from the user to that process, you’ll basically be sinking that data, and the user’s typing in those 10 seconds will be lost

I could open the pipe or pseudo-tty as an EXEC file.

If you want to know if there is a character available to read, then check the number of bytes in the input buffer.
bq. TaskWindow provides no API to do this.

TaskWindow is irrelevant to the issue, thankfully ¹. The OS provides you with an interface to read whether there are bytes in the buffer, which you should use. TaskWindow hacks its way to make that happen, but You Don’t Care because you just use the standard interface.

The interface to read the number of bytes in the buffer resolves (eventually) to CnPV, which is handled by the OS, and by the TaskWindow within its environment.

A loop could look like this… (code)

IF you’re in user mode acting as a regular process, you should not be messing with IRQs at all. If you turn off IRQs there, then you’re going to block the system from doing anything else whilst you’re running. Outside the descktop, there’s nothing to turn the IRQs back on again within that while loop is there? Whilst IRQs are turned off, transient callbacks won’t happen either, so nothing else is going to make any progress during that operation.

I’ve just realised in reading your code that there’s a bug in mine, too – if you fill the child’s buffer, the write call will block. In my case, it might recover if the process exited, because I’m reading data off a separate thread, but in your case, you can produce a deadlock more easily because the child can fill your buffer after you’ve filled its buffer, and it’s then blocked waiting on you to read its data, and you’re blocked waiting on it to read your data. That might be a factor of your pseudo code, but there’s a definite bug in mine, for which it’s been handy to spot.

Regarding the use of it as an exec file, I considered a similar approach (and just using OS_ChangeRedirection actually) but decided that the complexity of inserting a separate filesystem for the exec source into the mix didn’t fill me with joy. You may feel differently :-)

¹ Unless it’s broken, in which case, it’s TW that’s at fault not anything else – and oh my is TaskWindow broken, just not in this case, I believe.

and remember to pass ctrl-F12 and friends back out.

you should not be messing with IRQs at all.

Given future multicore behaviour, one probably ought to stop messing with IRQs… but RISC OS has a long history of mucking around with IRQs at the drop of a hat…

Outside the desktop, there’s nothing to turn the IRQs back on again within that while loop is there?

The OS_Byte INKEY call re-enables interrupts. In the single tasking the call to OS_Upcall should be a call to Portable_Idle, I was thinking of making OS_Upcall 6 call Portable_Idle by default.

The OS provides you with an interface to read whether there are bytes in the buffer, which you should use.

I can read keyboard buffer or the serial buffer, but I don’t see any interface for checking the current input buffer.

The interface to read the number of bytes in the buffer resolves (eventually) to CnPV, which is handled by the OS, and by the TaskWindow within its environment.

The only action TaskWindow does with CnPV is to flush it’s input buffer when a request to flush buffer 0 is made. Fortunately there is no need for background code to flush the keyboard buffer.

However the subject of this thread is waiting for either character input or another event (for which a custom interrupt handler is installed).

you can produce a deadlock more easily because the child can fill your buffer after you’ve filled its buffer, and it’s then blocked waiting on you to read its data, and you’re blocked waiting on it to read your data. That might be a factor of your pseudo code,

Yes it is simplified

Corrected examples.

Non Taskwindow Code:

_kernel_irqs_off();
while(...) {
  *pollword = 0;
  int i = inkey(0);
  if (i >= 0) write(tty, &i, 1);
  while(read(tty, &c, 1) != -EAGAIN)
    os_writec(c);
  Portable_Idle();
}

Taskwindow code:

while(...) {
  *pollword = 0;
  int i = inkey(0);
  if (i >= 0) write(tty, &i, 1);
  while(read(tty, &c, 1) != -EAGAIN)
    os_writec(c);
  os_upcall(6, pollword);
}

Making OS_Upcall 6 call Portable_Idle would make those the same.

Given future multicore behaviour, one probably ought to stop messing with IRQs… but RISC OS has a long history of mucking around with IRQs at the drop of a hat…

Future multicore code ought to ensure OS_Upcall 6 returns when character enter the input buffer.

> Outside the desktop, there’s nothing to turn the IRQs back on again within that while loop is there?

The OS_Byte INKEY call re-enables interrupts. In the single tasking the call to OS_Upcall should be a call to Portable_Idle, I was thinking of making OS_Upcall 6 call Portable_Idle by default.

That certainly was my intention in the past; making that the default claimant’s operation is reasonably sensible.

> The OS provides you with an interface to read whether there are bytes in the buffer, which you should use.

I can read keyboard buffer or the serial buffer, but I don’t see any interface for checking the current input buffer.

OS_Byte 177 allows you to read which input stream is currently in use.

Seriously though, the serial input should just be dropped entirely ¹. The input system is complicated enough already. Anyone relying on it should be able to update their software, or run it on a BBC.

> The interface to read the number of bytes in the buffer resolves (eventually) to CnPV, which is handled by the OS, and by the TaskWindow within its environment.

The only action TaskWindow does with CnPV is to flush it’s input buffer when a request to flush buffer 0 is made. Fortunately there is no need for background code to flush the keyboard buffer.

Oh gosh. I’m sorry. I remembered that TaskWindow has a CnP handler, but it’s just bloody stupid if it doesn’t replace the count interface too. Chalk another one up to failings of TaskWindow.

Oh and you need to check the redirection handle as well. And Exec. Oh god, TaskWindow does its own function key expansion as well. I knew there was a reason I wanted to burn it in hell.

¹ https://share.gerph.org/s/S72Cr1aycGJF7gU

Ah TaskWindow .. it all looks quite from a distance… at night… with your eyes shut.