CObey

With one of my tests with Fortify, I did something very similar, which ran a single command application (sed) with some echos around it, and then exited. It gave an ‘abort on instruction fetch’ (prefetch abort) because it was trying to execute at the address used by the marker for a freed block. So I think you’re freeing the stack that you’re executing in.

You can see the recording and log here:

https://asciinema.org/a/hEHKZLbJesrG0DxhxnGYu1kqr

A little while later (and through a haze of coughs and high temperature which is no fun) I’ve implemented trace points to report what the system was doing up to the point at which code was executed.

The first case I gave was of a simple ‘echo’ which reported a double-free. So, setting a trace point on the `Fortify_Deallocate` function, I get the following debug output:

Supervisor

*cobey:obey echo
set_obeydir: @
handle 255
OS_Claiming...
process_lines
I am in directory @
unlink walk: current 07008af8 this 07008af8 item 07007708 next 00000000
current 07008af8
OS_Releasing...
 7003254: <TRACE> CObey:Fortify_Deallocate
            r0  = &07008af8, r1  = &00000001, r2  = &07001f28, r3  = &00000191
            r4  = &07008af8, r5  = &00000001, r6  = &07009af8, r7  = &00000000
            r8  = &00000000, r9  = &07009b20, r10 = &0410021c, r11 = &04107ee8
            r12 = &07002438, sp  = &04107ec0, lr  = &07001f0c, pc  = &07003254
            CPSR= &20000013 : SVC-32 ARM fi ae qvCzn
            SPSR= &40000013 : SVC-32 ARM fi ae qvcZn
          Locations:
            pc is DA 'Module area', module 'CObey': Function Fortify_Deallocate+&0
            lr is DA 'Module area', module 'CObey': Function close_obey+&dc
FINISH
 7003254: <TRACE> CObey:Fortify_Deallocate
            r0  = &07008af8, r1  = &00000001, r2  = &07001ad0, r3  = &00000132
            r4  = &070076dc, r5  = &070076f0, r6  = &00000000, r7  = &00000000
            r8  = &00000001, r9  = &00000000, r10 = &0410021c, r11 = &04107f78
            r12 = &000002a8, sp  = &04107f2c, lr  = &07001c5c, pc  = &07003254
            CPSR= &20000013 : SVC-32 ARM fi ae qvCzn
            SPSR= &40000013 : SVC-32 ARM fi ae qvcZn
          Locations:
            pc is DA 'Module area', module 'CObey': Function Fortify_Deallocate+&0
            lr is DA 'Module area', module 'CObey': Function main_command+&394

Fortify: Possible "free()" twice of (07008af8) was detected at c.main.306
[Warning: Possible "free()" twice of (07008af8)]
*

This shows that the `Fortify_Deallocate` function (called as `free` in the source) is indeed called twice, once within `close_obey`, and once from `main_command`. In both cases the r0 parameter is `&7008af8`, so I think that confirms my manual analysis.

The second case, with the `sed` command (just a command that installs handlers and executes to make the code take different paths), can be traced in the same way, and shows that the same free is problematic in this case, too:

*cobey:obey echosed
set_obeydir: @
handle 255
OS_Claiming...
process_lines
I am in directory @
UpCallV handler 256
find_exit: exit_handler 07001884 exit_handle 07008af8
service_newapplication_handler 42
find_exit: exit_handler 07001884 exit_handle 07008af8
CObey:Exit handler
process_lines
I'm back
unlink walk: current 07008af8 this 07008af8 item 07007708 next 00000000
current 07008af8
OS_Releasing...
 7003254: <TRACE> CObey:Fortify_Deallocate
            r0  = &07008af8, r1  = &00000001, r2  = &07001f28, r3  = &00000191
            r4  = &07008af8, r5  = &00000001, r6  = &07009af8, r7  = &00000000
            r8  = &00000000, r9  = &07009b20, r10 = &07008d18, r11 = &07009aa0
            r12 = &07002438, sp  = &07009a78, lr  = &07001f0c, pc  = &07003254
            CPSR= &20000010 : USR-32 ARM fi ae qvCzn
          Locations:
            pc is DA 'Module area', module 'CObey': Function Fortify_Deallocate+&0
            lr is DA 'Module area', module 'CObey': Function close_obey+&dc
Exception: Prefetch abort
  r0  = &a9a9a9a9, r1  = &a9a9a9a9, r2  = &00000000, r3  = &a9a9a9a9
  r4  = &00000191, r5  = &07001f28, r6  = &07008ac0, r7  = &00000001
  r8  = &07008aa8, r9  = &07007720, r10 = &07008d18, r11 = &a9a9a9a9
  r12 = &a9a9a9a9, sp  = &a9a9a9a9, lr  = &a9a9a9a9, pc  = &a9a9a9a8
  CPSR= &60000010 : USR-32 ARM fi ae qvCZn

Error: Internal error: Abort on instruction fetch at &a9a9a9a8 (Error number &80000001)

In this case we can see (again) that the address being freed is the same one as before, but this time we’re executing in USR mode, and we can see that the stack pointer is `&7009a78` and the block freed is `&7008af8`. The stack is 4K (allocated in `wk`), which is `&1000`, so we can see that, yeah, we’re executing in the USR mode stack that we’re about to free.

These reports probably don’t add much to the original manual analysis, but at least they confirm what I thought was the problem, and probably mean that reorganising (or deferring) the free of the workspace block will probably make it safer.

A little while later (and through a haze of coughs and high temperature which is no fun)

That’ll be the Covid 19 then; self isolate for 14 days, and if you die in the meantime, don’t bother the rest of us!

This seems to be government policy, so it must be right!

I recommend ibuprofen (available from your local supermarket – oh, sorry, you can’t go there!).

Whatever you do, don’t visit the doctor’s, or anywhere else, come to that.

You just have to curl-up and …

Is ‘Fortify’ still around?

Any examples of it’s use? Thanks

To be clear, I’m talking about Simon Bullen’s ‘Fortify’ header and C source files which I’ve used in a huge amount of RISC OS stuff, and is part of my libraries that I use (and therefore part of the build system – that’s why it’s really just a matter of using `FORTIFY = yes` in the makefile to make it work). Other things called Fortify exist, which is why I’m explaining that.

Many instances of it exist on the Internet, and just searching for “Header file for fortify.c” will find it used in many places. To actually use it you `#define FORTIFY` (or define that in your compiler invocation) and `#include “fortify.h”` in your source (and obviously build the fortify.c as well). It then replaces your use of malloc and free (and friends) with its own tracking versions which will then warn you when you do things oddly. Modern C libraries and compilers have options to do some of this for you, too.

I’m not sure of a canonical location for the source; it’s under a slightly different license to what you might be used to in that it declares what you cannot do with it, like distributing modified versions. Which is sad, because it’s a great little library, and there have been some cool things done with it (I built some nice hierarchical diagnostics from it back when I was at Picsel).

Because of that Artifex (who develop Ghostscript and muPDF) created an entirely distinct implementation, which follows the spirit of Fortify, called Memento. However, that has its own restriction that if you distribute things with it you’ve got to place them under the AGPL, which is an even more frustrating license. Fortunately, if you use it for development and don’t actually distribute anything built with it, you’re under no obligations.

Any examples of it’s use?

WinEd is one, which is how I’m now more familiar with it than I was a month or two back. Again, as already noted, it is not supplied with WinEd, and notes about it are in the !WinEd.Source.ReadMe/txt file – the sources expect it to be present, and you need to either add it yourself or remove the references in order to get WinEd to compile.

I found another bug.

Make two Obeyfiles A and B in the same directory.

A

Echo 1
Obey <Obey$Dir>.B
Echo 2

B

Echo 3
Quit
Type <Obey$Dir>.C
Echo 4

where C does not exist. Then Obey gives

1
3
File not found

whereas CObey gives

1
3
2

Edit: forget this one, not sure why I thought it was doing this, it’s working fine.

…and another one

A

Obey <Obey$Dir>.B
Echo You can’t see me

B

Run Resources:$.Apps.!Chars

When you quit chars, Obey does the final Echo, but CObey doesn’t.

Thanks for pointing out Fortify. To fix the stack problem I’ve used a few lines of assembler to do OS_Module Free and then OS_Exit, since there’s no way to call free(this) when this contains your stack.

Obey does the final Echo, but CObey doesn’t.

Obey gives up continuing on when I try accessing sys$returncode, which is inconvenient. I haven’t tried accessing it with X yet, it may help. One use of X I found is to allow continuing on if a an illegal string/numeric comparison is tried. eg testing is it a number or a string?

I’ve attempted to write a description of the Obey language.

After that discussion of the BASIC case, it turns out that the original Obey 0.40 doesn’t handle that properly anyway!

Try

A

Obey <Obey$Dir>.B
Echo Done A

B

Repeat Echo Resources:$
Echo Done B

and them from BASIC do *Obey A, you only get Done B, and then Obey gets in a tangle :-(

*RMKill Obey
Module Obey is already in use

You can see by doing *Vectors 1 that Obey didn’t unlink properly.

CObey is still work in progress and subject to further formal testing. Having ascertained with a softloaded module that it does not explode I have installed it in locally built Titanium and RPi ROMs and nothing untoward has happened.

It has been in use for about two weeks.

So far, so good ..

For anybody who didn’t see it and wants to try out the lastest, you can get it by following this link.

Good work!

I’ve made an experimental branch of CObey which adds two command line options to Obey:

• The -b option makes Obey print a backtrace when an error occurs
• The -t option makes Obey try to throwback using DDEUtils when an error occurs

Please download from here and give it a go. For the throwback you’ll need DDEUtils loaded, which you might not have, so do

*System:Modules.DDEUtils

before you start your editor.

You might find it useful to set

*Set Alias$@RunType_FEB Obey -b-t %*0

so that backtraces are enabled when you double-click on an obeyfile, but note that throwback will only work inside a taskwindow.

Let’s have a space in -b-t ;-)

sigh … that would be nice …

I thought about using OS_ReadArgs, but it doesn’t cope with command tails. But that wouldn’t solve the problem anyway. I don’t know how we could change the syntax with either breaking existing code or making it inconvenient to use arguments with an obey file.

arguments with an obey file.

I noticed the existing arguments are only relative to files, well I guess that is the only time “obey” is used.
I always avoided TaskObey files and used obey -c which also allows a file to be of type data. Good if you dont want it to run isolated, possibly setting/running things without proper control from the calling program.

A different useage I have is to have an Obey file (eg. call it sc) with simply
%*0
and running it like <path>.sc <starcmd>
Interestingly, when sc is in RAM, the current FS changes to RAM while it runs.
My unixlib dash shell can then access star commands, and earlier it wont find system alias’s either.
Just wondering how your -b-t could be implemented,
Edit: just using sc obey -v file works ok.
Does the -b only report file line numbers? will have to load your module
for a look.
I think X will work because it is part of the cmd being asked for.

Edit: C versions of star commands could be built into the likes of dash using unixlib instead of sclib and interact fully with pipes, variable indirection and scripts.

That reminds me, I should say the backtrace and throwback modes are inherited, so you can call an existing nest of obeys without modifying them.

A simple example to see what it does is make Obey1 and Obey2:

Obey1

Echo A
Echo B
Obey <Obey$Dir>.Obey2

Obey2

Echo C
Echo D
Echo E
Error Test

and then do

*Obey -b-t Obey1

The December version (non-backtrace) had a painful bug in which I’ve fixed now as well.

I noticed that the current (assembler) Obey 0.41 accepts an address

*Obey -m &...

which can’t be greater than &FFFFFFFF.

I’d like to use the ‘WIDE’ version of OS_ReadUnsigned to accept addresses up to &FFFFFFFFFFFFFFFF.
That would leave me with an int64_t, but then what to do?

I could conditionally compile

int64_t arg = ...;
#ifndef 64BIT_PTR
  if (arg > 0xFFFFFFFF)
    raise error;
#endif
address = (void *)arg;

or at run-time I could check, say with an extended OS_ValidateAddress

int64_t arg = ...;
int psr;
_swix(OS_ValidateAddress64, _INR(0,1) | _OUT(_FLAGS), lo(arg), hi(arg), &psr);
if (psr & _C)
  raise error;
address = (void *)arg;

or I could just truncate

int64_t arg = ...;
address = (void *)arg;

but I don’t like that because the address would be meaningless so there should be an error.

I noticed that the current (assembler) Obey 0.41 accepts an address

Sounds like an option to just drop. Unless the transfer of ownership can be done in a safe way, this would be dangerous as an operation – either because the referenced memory is leaked or the memory is at risk of being removed underneath the Obey module. If that’s been considered then that’s fine, but I can’t imagine a way that you could ensure that this was safe that avoided one of those two failure cases.

With OS 5.30 out of the way, wouldn’t it be time to replace the old assembler modules with Julie’s C conversions in the daily builds?

Seems to me to be a good idea.