Getting ready to program in C with GCC on RISC OS

A major bonus of GCC is that it enables RISC OS programmers, for the first time, to use dynamically loaded libraries. However, apart from !GCC.docs.SharedLibs.!ReadMe, which I cannot say I fully understand, there is nothing to help programmers new to GCC to get going with dynamic loading. There are no examples and no tutorials, and nothing about how to compose appropriate makefiles. I, for one, would be very grateful for any help or guiding hands in this direction. Lee Noar has been very helpful to me, and patient, in the past, but I think it is too much to expect only him to shoulder the whole burden of educating what programmers there still are in the RISC OS community. In particular I would like to see examples showing me how to use the dlfcn library. Can anybody here help?

By way of a silly trivial example, I would like to see a C program, myfoo, that:
1. reads a character from stdin.
2. if the character is ‘x’ it loads a library called foolib.so containg a single function

void foo (void);

@ DavidS

as they are patched to link at run time it is a poor implementation.

I do not understand this remark. Linking at run time is what the word dynamic is used for. Linking at compile time is static linking, which we could always do, even with aof files. What is the beef? I suspect there is some language problem going on here. Could you be more specific about your reasons for saying it is a poor implementation? The elf format was designed for many purposes, one of which was the ability for linkage at run time. The aof format is inadequate for that.

The aof format is inadequate for that.

We wrote a dynamic loader for that in 1993 – used in Fireworkz. As I’m sure others did.

I think DavidS appears wedded to the idea of something akin to the BCPL global vector (or sets thereof)

We wrote a dynamic loader for that in 1993 – used in Fireworkz.

Was that called rink, or am I thinking of something different? I thought that had a problem with shared libraries.

@ Gavin

By way of a silly trivial example, I would like to see a C program, myfoo, that:
1. reads a character from stdin.
2. if the character is ‘x’ it loads a library called foolib.so containg a single function

void foo (void);
which when called prints “foo” to stdout.
3. calls foo( ); .
What do the files myfoo.c and foolib.so look like? What does the makefile to compile them look like? That is what I want to see by way of a tutorial for GCC.

So, GCC 4.7.4 on RISC OS (which at this point is the base right?) supports dlfcn library (dlfcn.h) which allows you to do exactly that, load plugins dynamically at runtime on demand.

If it’s something that multiple people want, I can write a tutorial on my blog on how to use it.

It’s also possible to create a counterpart for DDE that uses AOF, if that is needed.

@ DavidS

Just the concept of dynamic libraries (notice word link missing) that use a table call through (like Amiga, not as much like the global vector [there are differences]). No linking needed, simple stubs to use the libs, simple as can be and reliable. Only limit is it requires that the code be position independant (just like Modules require, as well as Util type programs).

Amiga had a funny way to use Libraries and that is because 90% of AmigaOS runs in user-space, there is absolutely no memory protection of any sort (and it cannot even being implemented if one wanted) and library calls required to push “offsets” in the 68K registers. So in general I would not use AmigaOS as a good example of how to implement dynamic libraries.

It was developed that way because the Amiga 1K and 500 were freaking slow (memory bandwidth issues) and so implementing the OS correctly would have made them even slower (CPU context overhead). Hence, they had no choice.

Atari TOS was luckier because it inherited BDOS from CPM/68K which indeed had a better architecture and the ST had not the memory bottleneck the Amiga had, plus the CPU was 1Mhz higher in clock speed, but you can see how TOS appears slower than AmigaOS because of that.

Ok I guess I need to repeat what have said before. We all know that Amiga OS was a non-protected OS and took advantage of that.

It’s not about “having read about Amiga OS” somewhere. it’s about understanding all the consequences of certain choices, let me explain to you using your own words:

That does not preclude using the basic concept of the libraries in an OS that has memory protection, just need data areas on a per task (if multitasking) basis.

How do you manage these areas? how does code stored in this area “talks” with your process?

In the most basic form, if it is an out-of-process approach then you’ll need (on RISC OS) have at the very minimum a Module that handles the allocation of memory for the plugin and also provides a way to interact with it.

If you chose to have a basic “jump” to a method offered by an external plugin, things may go wrong with the way RISC OS handles memory, for instance what if that plugin has to allocate memory? Where and how is it supposed to do so? Possibly again via the Module that manages them.

Are you starting to see the overhead of jumping in and out of the Management module? So, maybe one could force the Module to stay in user-space too. Still in this case who owns the memory that has been requested to be allocated?

I have a feeling that you are thinking in simple terms of some arbitrary assembly that do not request memory. If this is the case you can load binaries already in BBC BASIC, but again they will have very limited applicability.

So, to have more features you’ll need an infrastructure that can allow a plugin to act as a regular DLL or as a regular application. RISC OS (without the WIMP) is single task and so cannot offer help with that. So, the last easy chance one has is to implement it in the WIMP, run the plugins as Apps and use WIMP messages to make them communicating with each other, oh wait that is there already, so no need to code anything.

The library code would be run mostly in user mode if called from user mode, thus reducing context switching.

Not really, if you do not have support from the OS side to do this, you’ll need to use Modules, but you could force the module to run in user-space, but you still need to use SWI to access it.

If the library does something that requires switching to supervisor mode then a context switch would accure. Very simple concept, do not know why everyone ignores the obvious.

Probably because everyone is thinking first about memory management than an SWI call. Sure, an SWI call will save a return address, but who owns that memory? In other words, if you run this in the WIMP and you do not use a Module to access the library how can you ensure that the library memory page is always mapped correctly?

And I did realize that a better analogy to the concept of libraries at hand would be RISC OS Modules, just without the overhead of going through an SWI, and without the need for context switch when the call can be accomplished in user mode.

Modules have overhead, BUT you can simply use GCC with dlfcn and load the plugin/library within your own address space and that is exactly what you want. It’s fully dynamic, it has no overhead, it works on both RISC OS CLI and the WIMP and does not require a module to handle it.

If one is not a C developer then you can load binaries in BBC BASIC (I explained how to do it in another thread in this forum in the past), you can also load AIF, there is a trick to avoid their initialisation process and use them as "plugins in BBC BASIC.

There are also ways to create your own plugin management library for DDE (if one doesn’t want to use GCC), again the trick is to keep the dynamic loaded library within your address space, this makes it easier and makes it work with or without the WIMP. This is what Stuart is talking about btw.

Hope this is a bit more clear now.

Was that called rink, or am I thinking of something different? I thought that had a problem with shared libraries.

No, it was something different.

keep the dynamic loaded library within your address space

Oh yes!

If it’s something that multiple people want, I can write a tutorial on my blog on how to use it.

Yes please. Lua has a library loadlib that uses dlfcn to implement dynamic loading of shared C libraries, together with fallbacks for systems that do not have it (and #ifdef alternative sections for Windows systems). When it came to making RiscLua 600 everything compiled nicely with GCC 4.7.4 and I was able to make the riscos and lpeg libraries dynamically loadable, but I did not know how to write the GCC makefile lines to create them. So I bodged some Obeyfiles, such as

dir <Obey$Dir>
echo "-->"
gcc -std=gnu99 -fPIC -O2 -Wall -Wextra -Wl,-E -shared -mfpu=vfp -DRISCOS -o so/riscos sys.s riscoslib.c
echo "Finished making riscos"

So any help about dynamic loading will be useful, and will make for smaller binaries in RiscLua distributions.

@ Gavin

Yes please.

Sure, no problem and happy to help the Lua project! :)

@DavidS
Aren’t you supposed to be concentrating on your projects?

Regards.

@ Gavin
Just for you my friend, even faster than writing an article on my blog and with an example that involve math! :)

So first of all the source code:

#include <stdlib.h>
#include <stdio.h>
#include <dlfcn.h>

int main(void) {

  // Set a pointer to the Library Handler:

  void *plg_handler;
// Set a pointer to error (this is used to return errors)
char *error;
// Set the name of your plugin to load and link:
char *plugin = “SharedLibs:lib.abi-2/0.libm/1/0/0/so”;
// Dynamically load and link the library:
plg_handler = dlopen (plugin, RTLD_LAZY);
if (!plg_handler) {
fputs (dlerror(), stderr);
exit(1);
}
// Set a static API access Point pointer:
// You can improve this by adding a
double (*my_tan)(double);
my_tan = dlsym(plg_handler, “tan”);
if ((error = dlerror()) != NULL) {
fputs(error, stderr);
exit(1);
}
printf (“So let’s call a Plugin function: %f\n”, (*my_tan)(2.0));
// We’re done so let’s close and clean up:
dlclose(plg_handler);
my_tan = NULL;
return 0;
}

To compile this code make sure you add the following to your gcc line:

gcc blah blah -ldl

That’s all you need. And tested on RISC OS 5.28 with the old GCC 4.7.4, so any GCC you have should work fine.

Now to the cool stuff, first off look at the line:

char *plugin = "SharedLibs:lib.abi-2/0.libm/1/0/0/so";

As you can see I am loading a Dynamic Linkable Library (btw GCC folks looks like there is a typo on that library name!)

The library in question is the math library (in Gavin honour).

The library is NOT being linked in the usual form (aka when the executable is loaded. It instead is loaded an dlinked at runtime when this line is executed:

plg_handler = dlopen (plugin, RTLD_LAZY);

Now to access a function, in this simplified case I search for its symbol in the library using the function dsym, BUT you can add your own Plugin layer there and ensure your plugins always have an init function that you call.

You could pass a struct to that init function to map all functions in your plugin or whatever you need to become aware of what methods it offers and even how to call them.

The whole calling process is typical C function pointers, so nothing extra terrestrial.

This approach is really fast, no SWI involved and so you lose no performance from the library being linked dynamically. The library resides within your Process address space and so it’s totally fine with RISC OS WIMP as well as with the traditional CLI. Works also in the TaskWindow.

And so it solves the issue completely for who uses GCC.

If you run the executable it will indeed calculate the tan for the number.

Hope this helps you man and let’s get together for the math libraries work, I am totally excited with your long numbers lib, I defo need to find the time to work with it! Thanks for that work!

P.S. Sorry for the weird code formatting, pre and code tags do not seems to like my code! XD

This is extremely abusive of C
[…]
making sure it still compiles and runs correct

What’s abusive is that you’re trying to pass this off as a program you have apparently tested.

• You’re defining InitSpr(void) and elsewhere trying to pass a value to it. This isn’t a warning, this is an error.
• What happens, when plotting the sprite, if R6 and R7 aren’t zero? You’re only setting R0-R5, and as far as I know, swi(x) doesn’t set unused registers to zero.
• You’re trying to return a pointer as an int (in InitSpr). This isn’t a warning, this is an error.
• If a function doesn’t return a value, don’t declare it as doing so, and then not bother explicitly returning at all (ClearSpr).

After fixing the bugs, and marvelling at the horribleness of the for() constructs, the program works, drawing a set of rectangles that looks like a top-down view of the roof of a house.
Though I note that there’s a blue line running down the left side, and had to patch your calculation of spr.base because you were backing up too far and overwriting important sprite information. I’m surprised the OS managed to render the sprite at all, given the offset-to-sprite-data word was one of the things overwritten.
[how do I know? I added _swi(OS_SpriteOp,_INR(0,2),12+256,spr.ar,"$._dspr"); to the start of CleanUp() and looked at the data]

Also this shows one thing that I do not care for in C, it is very slow

Let’s see how it behaves with one written properly, huh?

After all, it turned this:

int hline(long x, long y, long xe)
{
  long *fb, cnt, offset;
  if (y > 0 && y < spr.y && x < spr.x)
  {
    if (xe < 0) return 0;
    if (xe > spr.x) xe = spr.x-1; if (x < 0) x=0;
    offset = y * spr.x + x; fb = spr.base;
    for (cnt = xe - x; cnt--; fb[ offset++ ]);
  }
  return 0;
}

into this:

hline
        CMP      a2,#0
        LDRGT    a4,[pc, #L000194-.-8]
        LDRGT    ip,[sl,#-|_Mod$Reloc$Off|]
        ADDGT    a4,ip,a4
        LDRGT    ip,[a4,#4]
        CMPGT    ip,a2
        BLE      |L0002cc.J4.hline|
        LDR      a2,[a4]
        CMP      a1,a2
        BGE      |L0002cc.J4.hline|
        CMP      a3,#0
        BLT      |L0002cc.J4.hline|
        CMP      a3,a2
        SUBGT    a3,a2,#1
        CMP      a1,#0
        MOVLT    a1,#0
        SUB      a2,a3,a1
|L0002c0.J15.hline|
        MOVS     a1,a2
        SUB      a2,a2,#1
        BNE      |L0002c0.J15.hline|
|L0002cc.J4.hline|
        MOV      a1,#0

        MOV      pc,lr

So, it seems to me that the compiler has done a pretty good job here, optimising out a lot of the rubbish; and amusingly exposing the faulty code – namely, if it’s supposed to be drawing a line, shouldn’t it be writing something somewhere? There’s no STR in the assembler. I’m going to go out on a limb and suggest that the fb[offset++] is supposed to have an equals sign after it, followed by something to write?

Lucky this function isn’t used, then, isn’t it?

Please stop trying to “abuse” C. You’re nowhere near as clever as you seem to think you are.

Looks like the offset still needs tweaked. Or maybe the code is writing too much. If it was written clearly, one could look to see what’s actually going wrong. But since it’s an exercise in masochism, I’m just doing the least possible to get a functioning sprite out of the code.

If a library requires task specific data (as most will) it will have an allocated area of memory per task for said data.

Where? In the task? In the library? Who owns it, the task or the library? And what happens when everything piles up as “starting at &8000”?

When the Amiga runs in a flat memory map, different processes will have different addresses. This isn’t the case under RISC OS where (typical user mode) applications are physically paged in to start at &8000 and are paged out (and thus not a part of the logical memory map) when a different application is paged in and running.
Tricks that work on a flat memory map may not apply to a properly paged setup.

As far as the C code of the application is concerned it is calling the functions directly, as it is just calling through a table directly, tha base address of that table being the base address in the library for the function table.

The reason it’s done the way it’s done in CLib is in order to have the quickest way of calling individual library functions while having LR point to where to go back to and not corrupting other registers.
If you’re indirecting through a pointer to a table, how would you preserve LR and not muck up any registers?

That is to have the stubs pass out the pointer in a register that is otherwise unused when calling the library.

There aren’t any “otherwise unused” registers in the APCS.
R0-R3 (a1-a4) are corruptable, and are specified for passing registers to a function, and (R0) receiving a result back.
R4-R8 (v1-v5) are for use by the program. The compiler will likely place here frequently used values that are retained across function calls (like loop counters and such) to prevent them being continually reloaded.
R9 (sb) is either Static Base or v6. The DDE compiler uses it as v6.
R10 (sl) is either Stack Limit or v7. The DDE compiler uses it as Stack Limit.
R11 (fp) is the Frame Pointer.
R12 (ip) is an Interim Pointer or a scratch register. It has a specific function at the start of functions, it can be used as a temporary register afterwards. The C compiler tends to use it when loading data from the stack.
R13 (sp) is the Stack Pointer.
R14 (lr) is the Link Register.
R15 (pc) is what the processor is currently looking at (girls! donuts! maplin catalogue!).

I live on earth aka Tera Terra, third plaet planet out from a yellow white star, nice place, please come visit.

Yes, the sun is white. You think it is yellow because you are looking at it through a blue atmosphere.

Oh, and given that a great many of the inhabitants of this planet seem willing to slaughter each other “because their imaginary sky fairy told them to”, I’d suggest that there are likely much nicer places to visit.

Oh, and for the love of the aforementioned sky fairy, if you’re going to visit and abduct something claim a souvenir, please do it to a crash test dummy “football” player during the superbowl. Not some random cow (again) in Buttf***, Nebraska.

and specify what features are available to the current system.

Like, say, OS_ReadSysInfo ?

@Pablo

Thanks for the example. It will help me to understand how loadlib.c works in the Lua sources. I note that I had the line

MYLIBS= -ldl

An important aspect of big number handling is to cache as much as you can. Have a look at this article to see what I mean.

@ Gavin

It translates this to an actual filepath to load from by means of a template, which can be read by the Lua interpreter from an environment variable: LUAPATH for Lua libraries or LUACPATH for C libraries. The templates are lists of filepaths containing the symbol ? which is replaced by the library’s name.

Hummm I am not as familiar as you are with Lua internals, I only used it in integration with my C or C++ code as a scripting language to configure my solutions. However, I remember that Lua exposed mechanism to load “C Packages” allows to use paths:

local my_object = "ZLuaLibs:so.featchaurl" -- I use to put my code in a !ZLuaLibs that expose a RISC OS path and in there I put my "so" (remember "so" are compiled in GCC via -Wl,s etc...)
local FetchAURL = loadlib(my_object, "HttpToBuff")

It’s been a while I haven’t been playing with it, but it should work, tomorrow I’ll give it a try. You should be able to use cURL with this approach IIRC in Lua.

So, what I think you should put in the predefined path is the stub file (not the binary), but again I had no time to test this, so consider it just a thought for now.

One of the big advantages of Lua over BBC BASIC is its capacity of loading C written libraries which not only makes it faster, but that also makes it capable of reaching low level features without the complexity of maintaining ARM ASM in a BBC BASIC code. I know you know this Gavin, I just thought this was a good opportunity to mention it for the rest of the community (for who has no idea). Lua has the same simple syntax as BASIC has, but it offer a way better approach to handle Libraries and also support binary libraries which truly makes it a great tool to have on RISC OS, so once again thanks for all the hard work on the port!

P.S.:

So require takes as argument just the library’s name.

I see, that kind of match my previous comments, I think you ARE talking about the stubs, isn’t “require” used to load Lua code?

Had a quick look and this should be “loadlib” implementation, which indeed uses dlopen:

static void *ll_load (lua_State *L, const char *path) {
  void *lib = dlopen(path, RTLD_NOW);
  if (lib == NULL) lua_pushstring(L, dlerror());
  return lib;
}

path is definitely passed to it, so (without more testing) is seems to confirm what I thought above.

However this is NOT the “require” implementation, that one follows later on in the lua loadlib.c file.

However, given that dlopen depends on the value of the define LUA_DL_DLOPEN which depends on LUA_USE_DLOPEN which depends on:

#if defined(LUA_USE_LINUX)

#define LUA_USE_POSIX

#define LUA_USE_DLOPEN          /* needs an extra library: -ldl /

#define LUA_USE_READLINE        / needs some extra libraries */

#endif

There is a chance that your lua build may NOT be using dlopen. So you should tell me at this point how you’ve configured your build. Yes you added the -ldl to your MYLIBS, but did you also enable LUA_USE_DLOPEN?

but did you also enable LUA_USE_DLOPEN?

Yes, in risclua.h which gets pulled in by

#ifdef RISCOS
#include "risclua.h"
#endif

@ Gavin

I am enheartened by your tracking this down, and I will try and get it to work for the latest RiscLua. Many thanks.

No problem at all and (as always) thank you for the Lua port effort, can’t wait to test my modules on latest RiscLua.

In one of my (way too many) projects, I am trying to use Lua to replace Acorn FrontEnd. This requires exactly what you’re trying to fix.

Basically with LuaFrontEnd one can simply compile his/her C (and C++) as a regular /so and then use LuaFrontEnd to control it in the WIMP. This will obviously give people a way more elastic approach to do WIMP programming. For instance (but I still have to decide this yet), we could implement a YAML layer to define the WIMP behaviour for LuaFrontEnd.

Given that GCC allows to build /so from C, C++ and GAS (and ASASM which supports ObjAsm syntax) there are a ton of backend languages usable with LuaFrontEnd and also there can be more. For example one of my M/Forth implementation for RISC OS runs as an embeddable Library, so this means it could be used with LuaFrontEnd. If I’ll ever manage to finish my OOP BASIC it also can run as an embeddable library so it will be another candidate.

If you do not fear GitHub/Git I can post early code of LuaFrontEnd on the ROC GitHub and you can either have a look, or help me or even take over the project if you would like, i do not mind :)

@Paolo

I am sorry to report failure. I am back at the same impasse that I had with RiscLua700. In more detail, I decided to make a new version, temporarily called RiscLua86, without the riscos and lpeg modules statically linked. In fact just like RiscLua600 but with Lua 5.4.3 instead ofLua 5.4.2. I compiled with GCC 4.7.4-rel5-1. No trouble compiling lua and luac, or with compiling the modules riscos and lpeg. But they would not link. Initially, in rlua86.!Boot I did

set LUA_PATH_5_4 rlua86:lib.?,?
set LUA_CPATH_5_4 rlua86:solib.?,?
set UnixEnv$lua$nonametrans ""
set UnixEnv$luac$nonametrans ""

lpeg = require "lpeg"
print (lpeg.version)

print (package.path)
print (package.cpath)

rlua86:lib.?,?
/rlua86:solib/?,?

@ Gavin

I am sorry to report failure.

Sorry to hear that, I did some debugging (it took me a while ’cause I had to rebuild RiscLua).

So few things needs to be done to enable dynamic load:

in risclua.h make sure you have added:

#define LUA_USE_DLOPEN

#define LUA_DL_DLOPEN

And obviously, in your Makefile, in MYLIBS add (as you did) the -ldl.

Then, if you want to load a library from an arbitrary location you need to:

1) create a Lua library file and place it in !RiscLua.lib, the file content should be something like:

libpath = “SDFS::HD4.$.myCLib/so” — please note the /so at the end of the file name, it’s required!

mylib = assert(package.loadlib(libpath, “luaopen_myClib”) — between the double quotes you need to specify the init function (obviously)

I have called mine: myClib

Finally write your Lua program that uses require as follow:

#! lua

require "myCLib"

blah blah

Here my !RiscLua is loading my test library. If you want I can push all my changes in github so you can have a look at them.

Now for your lpeg, if you follow my instructions then yes all you have to do is run a lua script that contains:

#! lua

lpeg = require "lpeg"

print (lpeg.version())

To me it’s showing 1.0.0

What I need to see are the sources to dlfcn.

To look at the GCC and GCCSDK sources have a look here: https://www.riscos.info/downloads/gccsdk/gcc-4.7.4-release-5/

Hope this helps and again it appear to be working here.

P.S. I started to add some function into my module to do some more deeper test, however RiscLua 84 seems to be having some issues (maybe passing the Symbol to Dsym?) I get this error:

[!RiscLua84:bin].lua543: Text relocation of data symbol '' found:
 [my so full and correct path].myCLib/so (offset 0xAC0)

This is a linker error and it simply means that my library has been market as having text relocations, so need to remember the bits it wants when compiling my library.

I am too tired now to do more debug, so I’ll start again tomorrow…

Thanks for the tip about #define LUA_DL_DLOPEN and about the so suffix being mandatory. That has improved things so that when I try print (package.clib) I am getting the right answer. It is now finding the module. But it is saying it cannot resolve the symbols it finds in it.

I am not sure how to write the makefile so that it is compiling the modules as well as lua and luac. At the moment I have separate Obey files to create the modules.

Paolo, I have written my makefile but evidently not correctly as I am getting

gcc: error: riscoslib.o: No such file or directory
gcc: error: sys.o: No such file or directory

@ Gavin

But it is saying it cannot resolve the symbols it finds in it.

On it, give me a couple of hours, I already found a missing bit, but when added it’s causing an issue with the GCC posix, so I am digging this now.

Paolo, I have written my makefile but evidently not correctly as I am getting

Is there any chance that I could email it to you for your critique? Just at this moment PlusNet’s webmail is refusing to load, so I cannot check whether you are in my address book, but I suspect not. Do you have my email address?

You can always email me, no problem at all. My email is in one of the libraries I ported/packaged for RISC OS in my RiscPkg repository, or you can just send me a quick message via my blog Contact Me form, I’ll receive your email via that and answer you so you get mine. This avoid sharing email addresses on the forum that only lead us to get loads of unwanted spam.

my blog URL: https://paolozaino.wordpress.com and it works fine in !NetSurf in case you may be a RISC OS purist ;)