Where to start kernel hacking?

Wasn’t someone giving away copies of the DDE to interested coders a while ago?

That was for BBC BASIC (aka ABC Compiler) only I think

I think there was an announcement some years ago that full DDE might be available to those who signed up to particular tasks.

Found it, from 2020, which feels like yesterday: https://www.riscosopen.org/news/articles/2020/02/21/new-desktop-development-environment-reaches-out

Are you a new developer wanting to work on the RISC OS source code? If the thought of buying a copy of the DDE is putting you off, perhaps we can help? ROOL are giving away a number of copies of the NutPi version of the DDE

I am not sure why people still insist on using CHatGPT for Assembly code, it has not been trained for it. It’s already a miracle that when it writes C doesn’t messes up too much with pointers…

Fun fact: it can eat raw x86 byte code and convert it to C++. But I doubt it will handle ARM without a tool, since it is not byte code. But as I mentioned several times already, it works best when you have well documented source code to fed it with, otherwise it gets confused what you want from it. It is not amazing at ARM machine code (despite it being the simplest and cleanest out there), since all the ARM tutorials are confusing, incomplete and targeted at “security experts”. I personally read that Aug 1986 ARM assembler reference manual, which appears to be the easiest source to read (the documentation at arm.com is just impenetrable, even to LLM).

Whilst you may be able to concoct an ObjAsm replacement, there’s still the matter of integrating that into the build system for RISC OS, for which you still need Norcroft C from the (modern) DDE. And its linker. And so on.

The problem is: I already need a public domain ARM assembler for my project. So why not create one compatible with the ObjAsm? There is not a single C file inside the Kernel. I do need a linker. But then again, I need a PD ARM linker for my Symta lisp system project to detach it from any dependencies onto the host C compiler. In other words, there are no other option but to do that. If I ever need to compiler Norcroft C, I think I can adapt my transpiler for it. It has some unusual calling conventions like __value_in_regs, but I think it can be solved with a trampoline, which I again need to study to properly transfer Symta’s FFI runtime to ARM.

Currently going through the encoding format (went to RPCEmu for reference).

Operation{cond}{S} Rd, Rm, Rn
data processing instruction format:
BitOffset/NBits: Name - Description
28/ 4: C  - condition. See CND_.* macros
26/ 2: T  - instruction class (always 0x0)
25/ 1: I  - IF 1, Rm is immediate
21/ 4: O  - operation code. OPC_.* macros
20/ 1: S  - set flags
16/ 4: Rn - 1st operand register
12/ 4: Rd - destination register
 0/12: Rm - 2nd operand register or immediate
            Register (I=0):
              0/4: Register
              4/1: ShiftSource
              5/2: ShiftType
                0 = logical left
                1 = logical right
                2 = arithmetic right (sign bit handled)
                3 = rotate right
              7/5: shift value (0 to 31)
                Immediate (ShiftSource=0)
                  7/5 = immediate (i.e. 7 for LSL#7)
                Register (ShiftSource=1)
                  7/1 = should be 0
                  8/4 = register
            Immediate (I=1):
              0/8: value
              8/4: rotate right amount
                   note: it is multiplied by 2
                   before use

////////////////////////
// Operation Codes
//Rd - destionation
//Rn - 1st operand
//Rm - 2nd operand

//Rd = Rn & Rm
#define OPC_AND 0x0

//Rd = Rn ^ Rm
#define OPC_EOR 0x1

//Rd = Rn - Rm
#define OPC_SUB 0x2

//Rd = Rm - Rn
#define OPC_RSB 0x3

//Rd = Rn + Rm
#define OPC_ADD 0x4

//Rd = Rn + Rm + C
#define OPC_ADC 0x5

//Rd = Rn - Rm - C
#define OPC_SBC 0x6

//Rd = Rm - Rn - C
#define OPC_RSC 0x7

//NZCV <- Rn & Rm
//S should be always 1
#define OPC_TST 0x8

//NZCV <- Rn ^ Rm 
//S should be always 1
#define OPC_TEQ 0x9

//NZCV <- Rn - Rm
//S should be always 1
#define OPC_CMP 0xA

//NZCV <- Rn + Rm (compare negated)
//S should be always 1
#define OPC_CMN 0xB

//Rd = Rn | Rm
#define OPC_ORR 0xC

//Rd = Rm (Rn is ignored)
#define OPC_MOV 0xD

//Rd = Rn & ~Rm (bit clear)
#define OPC_BIC 0xE

//Rd = ~Rm (Rn is ignored)
#define OPC_MVN 0xF

There is not a single C file inside the Kernel.

Did you have a look at the HAL?

There are many compiled C files in the ROM, mostly for modules.

Wasn’t someone giving away copies of the DDE to interested coders a while ago?

I think there was https://www.4corn.co.uk/articles/acornc5/

It could be useful for reference.

Did you have a look at the HAL?

I’m interested in getting the Kernel running first.

There are many compiled C files in the ROM, mostly for modules.

Can’t I take modules from the already compiled distribution?

I think there was https://www.4corn.co.uk/articles/acornc5/

Not very helpful, that’s an archaic DDE. Useful for building RISC OS 3.71 maybe, but definitely not RISC OS 5.

Can’t I take modules from the already compiled distribution?

No, because the C modules in the ROM are statically linked to the single copy of the SharedCLibrary, which would end up moving address if you changed the kernel that lives at the start of the image.

But if you want to instrument the DHCP module, that’s a different matter, that’s not ‘the kernel’ in RISC OS. You could build a RISC OS ROM with the DHCP module moved to the end of the image and just keep replacing that, perhaps. Though that, as a C module itself, would need linking against the SharedCLibrary each time. All the ASM modules are PIC, the C ones are not.

Allow me to point you at my blog – https://heyrick.eu/blog/index.php?diary=20230306 – about halfway down the page I asked for a simple routine to convert a string to lower case, using the CP1252 character set (that’s basically the Windows incarnation of ISO 8859/1, a traditional eight bit character set not unlike that used by RISC OS).

I particularly enjoyed its claim that a lowercase × symbol is ÷. (Edit: Which I now see you mentioned in the comments.)

@NancySadkov You might be able to manage something that overwrites the HAL, but not much.

YOU ARE IN A MAZE OF TWISTY LITTLE PASSAGES, ALL ALIKE

I’ve been stuck in them for the last couple of years!

Outline of the assembler, which compiles just the basic form of data processing opcodes (i.e. `movsne r0, r1`)
https://pastecode.io/s/7owwgg11

Macros apparently will be the dirtiest part.
But ObjAsm doesn’t appear to allow nested namespaces, so…
Still in the end shouldn’t be more than 4000 SLOC.

the C modules in the ROM are statically linked to the single copy of the SharedCLibrary, which would end up moving address if you changed the kernel that lives at the start of the image.

There is still one classic strategy: parasite hook into the working build, replacing functions one by one. HLE emulators do that and they used this trick to reverse engineer Dungeon Keeper and Diablo. The people initially reversing Might & Magic VI went directly with HexRays dumping, and the result was an explosion of bugs and crashes.

Allow me to point you at my blog – https://heyrick.eu/blog/index.php?diary=20230306 – about halfway down the page I asked for a simple routine to convert a string to lower case, using the CP1252 character set (that’s basically the Windows incarnation of ISO 8859/1, a traditional eight bit character set not unlike that used by RISC OS).

Latest GPT3.5 version writes correct code (if you’re lucky).
https://chat.openai.com/share/83c54477-6574-47b6-a524-806d17549df9

Macros apparently will be the dirtiest part.

Good luck with that. Macros are something ObjAsm excels at, and RISC OS uses them frequently.
There’s a nice macro kicking around that pushes registers on to the stack, and has a matching pull that pulls the correct registers back off the stack without you having to specify them.

There is still one classic strategy: parasite hook into the working build, replacing functions one by one.

?

Surely C modules start up in a specific way and make certain library calls to begin? If you can identify this, then you’ll have a known address to a known library function. From there you can may be able to derive the addresses of the library functions (an easy way to grab these would be to get the library to fill in the application jump table (or just steal one from a running app)).
Do the same on the destination machine and you then ought to have two sets of addresses to translate one to the other.
In theory. ;)

Latest GPT3.5 version writes correct code

Interesting. It gave me something different.
https://chat.openai.com/share/56db2425-b2cb-4522-8df7-9d734041ea64
And for some reason titled it in… Spanish?

if you’re lucky

Passed today playing with ChatGPT. It seems it is okay at fairly simple things, but messes up as complexity increases. Still, the broken code it gave me was enough to get me going to write something that worked.

https://heyrick.eu/blog/index.php?diary=20230920

Next, we would solve the multiplication. 2 * 4 = 14

Typo time :)
2 * 7 = 14 might be more accurate.

While I’m here, there’s something about `movsne r0, r1` that’s nudging a braincell, asking for activity.

Typo time :)

I never claimed to be good at maths. ;)
Fixed, thanks.

asking for activity.

An interesting little article on parsing ARM instructions by somebody not used to ARM:
https://www.labri.fr/perso/fleury/posts/programming/learning-arm-deep-down-dirty.html

I think the mistake people (including the above) tend to make is in thinking that all those instructions are different. MOVNES is really just MOV with the S option and the NE conditional. It’s better to break it apart like that, because if you’re going to parse them as separate instructions, there’s like a hundred instructions (figured pulled from my posterior quarters as life is too damned short to go count them), which have, what is it, 14 possible conditions? Plus multiple options (like S for all the maths ones, B for LDR/STR, the stack modes for LDM/STM), etc etc. You’d be looking at parsing something like 2,000 odd instructions. Yikes!

Easier way. Read the first three letters. That’s usually the instruction, though they broke that will UMULL, UMLAL and such but they are recognisable enough. Likewise special handling for B and BL.
Then follows the conditional (if not AL), and finally the options if any.
Etc.

A couple of small observations on the pastebin stuff:

• Some jurisdictions don’t have an actual legal concept of “public domain”. Simplest way to handle that is to release it on a very permissive licence like the bsd one.

• //RISC OS C ABI – we call it APCS, and yes, it’s always in lowercase.

And oddballs, such as RRX

Some jurisdictions don’t have an actual legal concept of “public domain”. Simplest way to handle that is to release it on a very permissive licence like the bsd one.

IIRC BSD cant be just thrown into an existing code base and you have to credit everyone involved.

A couple of small observations on the pastebin stuff:

The main error is that I tried to introduce an intermediate instruction format (the x86 experience), but with ARM the right thing is to map mnemonic to a prototype and then just or-in arguments, because ARM is so simple.

An interesting little article on parsing ARM instructions by somebody not used to ARM:
https://www.labri.fr/perso/fleury/posts/programming/learning-arm-deep-down-dirty.html

There’s also this series, written from a Windows perspective but a lot of it’s probably still relevant.

And oddballs, such as RRX

“I have no idea who would ever use RRX anyway.” – Part 3 of the series linked above :)

Interesting. It gave me something different.

That is because the Chat has the P parameter set probably. In the OpenAI GPT models (both in GPT 3 and in 4) there are some parameters a user can set to have it more creative vs more precise, in other words it’s possible to “configure” a model (it’s also possible to fine tune it by providing a fine-tuning dataset).

Given that a Chat may require some degree of “creativity”, most likely it’s pre-configured with P = 1.0 or similar (parameter values are float). This makes it more creative and also causes the effect of the model responding differently to the same question.

Which is why, it’s a terrible idea to use it for ASM or C/C++. There, “creativity” (mind, that is not the same meaning of human creativity), can cause problems.

There are models (or even using OpenAI GPT API instead of ChatGPT, where it’s possible to set all the parameters) that can do a better job at coding problems. Mind that the APIs generally are “context-less”, in other words they do not keep a context like a Chat format does, so prompts have to be extremely clear and descriptive.

As a side note, it’s totally feasible to develop models that can deal with ARM assembly or C in a very good way, but they tend to be more dedicated, while ChatGPT is an ambicious project which tries to develop a single model to be used in all scenarios.

There, “creativity” (mind, that is not the same meaning of human creativity), can cause problems.

Machine creativity = add random bits

Human creativity = add random bits passed through a filter (filter = will this kill me or do something bad? AND do I care?)

I think human filters can be quite a lot more complex than that! I suppose machine ones could be too. Maybe they’ll get there…in the end. Human designers of creative machines not creative enough? (Yet?)

First milestone.
https://pastecode.io/s/x77t7p26

It can compile the hello world from https://www.riscosopen.org/wiki/documentation/show/A%20BASIC%20guide%20to%20ObjAsm
Usage: ./nrm.exe hello.s hello,ff8

TODO:
1. implement rest of the opcodes and directives.
2. implement object and library formats.
3. implement a simple Norcroft compatible C compiler to go with it.

OS_WriteS * &01
OS_NewLine * &03
OS_Exit * &11

    AREA helloworld, CODE, READONLY

    ENTRY

    SWI OS_WriteS
    = "Hello, world!"
    DCB 0
    ALIGN
    SWI OS_NewLine
    MOV R0,#0
    LDR R1,abex
    MOV R2,#0
    SWI OS_Exit
abex
    DCD &58454241 ; "ABEX"

    END

First milestone.

Ah, good, you’ve made it CC0. Some jurisdictions (ahem, a chunk of Europe) don’t have a concept of releasing things as public domain as there isn’t “copyright” so much as “moral rights” and you can’t waive those.
Furthermore, I’m not a solicitor but I think under British law works can only become public domain by the expiration of copyright. I’m sure someone will correct me…

That being said, legal rubbish is legal rubbish so you might need more than a single line saying FREEDOM! (picture Mel Gibson here)

See here: https://wiki.creativecommons.org/wiki/CC0_FAQ#May_I_apply_CC0_to_computer_software.3F_If_so.2C_is_there_a_recommended_implementation.3F

Otherwise, pretty good going. 👍

You’ll have fun with the macro stuff. This’ll make your head spin: https://gitlab.riscosopen.org/RiscOS/Sources/Programmer/HdrSrc/-/blob/master/hdr/CPU/Generic32

2. implement object and library formats.

Libraries are the job of the librarian (LibFile in the DDE) to collect together all the object files into a “library”. It’s a separate program because any AOF can be used to create a library. It’s quite normal to mix bits in C with bits in assembler.

3. implement a simple Norcroft compatible C compiler to go with it.

Simple C is doable. Norcroft isn’t that unusual, so it might be possible to lightly bash an existing compiler for anything Norcroft specific.

Where it will matter, however, is in its optimisations. Norcroft is pretty clever in that respect.

Ah, good, you’ve made it CC0. Some jurisdictions (ahem, a chunk of Europe) don’t have a concept of releasing things as public domain as there isn’t “copyright” so much as “moral rights” and you can’t waive those.

CC0 is different from basic PD, since it is a license with explicit dedication:
`Creative Commons Zero (CC0) is for dedicating works to the public domain.<sup>45</sup> It works on two levels: as a waiver of a person’s rights to the work, and in case that is not effective, as an irrevocable, royalty-free and unconditional licence for anyone to use the work for any purpose. The rights waived include database rights, so CC0 is suitable for use with data.<sup>46</sup>`

it might be possible to lightly bash an existing compiler

As of now there is just Clang.
Unsure if it is easy to modify for APCS.

This’ll make your head spin:

I did a turning complete macro-processor with lexical scoping before, but it was single pass and had different constraints:
https://nancygold.itch.io/new-c-macroprocessor
Main difference is that it has only string variables, auto coerced to types, while ObjAsm has arithmetic, logical and string variables. ObjAsm doesn’t appear to have nested scopes, so not sure if this should be allowed.

Still unsure how to properly approach the macro expansion, since ObjAsm apparently allows to use forward resolved labels in its EQU expressions, so one has to implement macros declarations after their use. Assembler just ignores them during the first pass, assuming a single word opcode and then goes back during the second pass to compile it.