Debug symbols for ROM images

I’m fed up of manually tracking down which line of source corresponds to a specific address within ROM images. So I’ve started work on modifying the build system to allow it to produce a set of debug symbols for each ROM image that’s built. By modifying the common makefiles it’s possible to get most components to produce debug symbols when they’re built. I’ve also written a tool to parse a ROM build log and load all the symbol files into memory, to allow easy translation from address to source function/line (actually this tool is just a modified version of profanal). However before I can go much further and get everything properly integrated into the build system we need to come to agreement on a few things:

Location and format of symbols files

At the moment there are two types of symbols file produced – GPA files from assembler components (via the ToGPA tool), and plain old symbols files for C components (via Norcroft’s -Symbols option). For C I was hoping to make use of the -g switch to get proper debugging info, but there’s a long-standing bug/“feature” in Norcroft which means that enabling the -g switch causes substantially different code to be produced, making it useless for getting debugging information for production/non-debug builds of components.

My current makefile modifications result in the symbols file being output in the same directory as the linked component binary, with _gpa suffix for GPA files and _sym suffix for symbols files. This makes it easy to keep track of which symbols belong with which binary. However it goes against the convention (with debug builds of assembler components) for the GPA file to be located in a ‘gpa’ folder. The system I’m using also goes against the way which the NetBSD USB drivers output their symbols – they place their symbols files in the ‘rm’ directory (even though the linked ROM module goes in ‘linked’ – I’m guessing this discrepancy is due to the fact softload versions of the module will go in ‘rm’, and the symbols files were first enabled to allow easy debugging of the softloads?), with a ‘sym’ prefix. They also output map files to the ‘maps’ directory.

Although the above-mentioned methods for outputting symbols files are already established, I think my method (or something similar to it) is better in the long run. Firstly, as mentioned above, it makes it easy to keep track of which symbols belong to which binary. Secondly it also allows the symbols file to easily be stored alongside the module when the module is copied to the Install folder prior to ROM linking. Of course a better approach might be to switch to binary formats which allow the debug symbols to be stored in the same file as the binary, but that would be significantly more work (and run the risk of the symbols unintentionally making it into the final ROM image).

So what are peoples thoughts on this? I guess what I’m proposing is to completely drop the use of the ‘gpa’ folder for assembler components, and make sure that any component that produces debug symbols during its build places those symbols alongside the binary, with a suitable suffix to indicate their type.

However while I can see this working OK for modules (whether being built for ROM or softload), I’m not too sure how it would work for applications. A quick look at my disc build tree suggests that the !RunImage files tend to get placed straight into the Resources folder, which then gets copied over to the empty application directory in the Install folder. Storing debug symbols in the Resources folder could make things even more cluttered, and could easily result in the symbols making it into the disc image if someone does a recursive copy instead of copying each file individually.

Control over when debug symbols are produced

Unless anyone has any objections, I’m planning on making it so that the symbols will always be produced.

The address → source tool

As stated above, at the moment this is just a modified version of my profanal tool. The tool allows you to easily load up the symbols for an entire ROM image and lookup addresses. However I can’t decide whether to stick with profanal (making it more than just for analysing profiling results) or create a new tool which is dedicated to dealing with ROM symbols. E.g. I often find myself having to manually unwind stack traces, so having a tool which can take in a stack dump (or dumps plural if different processor modes are in play) and output a stack trace would be greatly beneficial. Some of this is pretty easy (e.g. looking for APCS stack frames, following the IRQsema chain, detecting SWI despatches, etc.), but other bits will be trickier (unwinding assembler stack frames, making sense of tail-recursive calls, avoiding red herrings from partially overwritten stack frames, etc.). So the tool is unlikely to be perfect, but at the least it should be able to point out all the obvious function calls and leave it to the user to work out which sequence of calls makes sense.

Anyone have their own thoughts on what things they’d like to see from ROM debug symbols, or how the symbols should be handled by the build system?

My own thoughts are rather simpler. Why not create a module whose purpose is solely to record the build date, build options/switches used, and a data table which records module/component and address so that a simple ‘*’ command will (so far as the user is concerned) give a meaningful response to a command ‘* HELP BUILD’ such as ‘This ROM was built on 13th June 2012 and is for the OMAP4 port’ but when interrogated further (in a way that does not need to be advertised) something like ‘*HELP BUILD SDFS’ would provide information such as ‘SDFS resides at address FF000000 to FF000100’ or ‘RISCOS_BCM2835.$.Sources.SDFS.Thisfile compiled to address FF00000’. All the assembly process then needs to do is to add to a table of CVS source file name and assembly address, using label names until the assembly address is known later during the build.

I confess I don’t really know what I’m talking about here, but the GCCSDK folks have symbols output in DWARF format in GCC, and are planning it for asasm. The aim is to allow remote debugging by tools like gdb (eg potentially over serial or JTAG). How much would that be aligned (or not) with what you’re trying to do?

Storing debug symbols in the Resources folder could make things even more cluttered, and could easily result in the symbols making it into the disc image if someone does a recursive copy instead of copying each file individually.

This seems inevitable to me. You’d need to fix a lot of install rules (ROM install and disc install). Especially if:

I’m planning on making it so that the symbols will always be produced.

Mind you, having the symbols file(s) inside an application directory might not be such a bad thing, especially if an “More information” button in an error box could output some analysis or there was some way to dump the same to file for the user to post back to ROOL. Or something like that.

I confess I don’t really know what I’m talking about here, but the GCCSDK folks have symbols output in DWARF format in GCC, and are planning it for asasm. The aim is to allow remote debugging by tools like gdb (eg potentially over serial or JTAG). How much would that be aligned (or not) with what you’re trying to do?

Remote debugging isn’t something I’m aiming for at the moment, but it would certainly be a nice feature to have.

Although Norcroft can produce ELF’s with debugging information, it doesn’t seem to be in DWARF format. Instead it appears to be in ASDTF format (the same as used in AOF/AIF files), stored in a section with the name “DEBUG_AREA”. Which means it’s pretty useless, since you’re highly unlikely to find a tool which supports it (except perhaps ARM’s tools, and even then I think you’d have to rename the section to “.asd”).

Of course, if we switched to building using GCC it would be a lot easier to get DWARF format debug symbols, at least on a per-component basis.

I’m planning on making it so that the symbols will always be produced.

Wouldn’t that make everything slow down massively? When assembling the kernel a good fraction of the time is it doing the ToGPA step.

But the general idea is great. When looking at JTAG with the Beagleboard xM, using IAR as the debugger and a JLink, you could get function level debugging (not line level) by making a fake ‘.s’ file with the function names spaced out by the same amount as the corresponding code – in essence using IAR to make a symbol file. When in the debugger this pile of symbols is overlaid on top of the ROM code in the GUI, which is very useful.

When assembling the kernel a good fraction of the time is it doing the ToGPA step.

Not for me! objasm takes 22 seconds to assemble the code. Linking (whether debug or non-debug) takes under 1 second. ToGPA takes 3 seconds. That’s not much at all, especially since the ToGPA will only ever be done if the kernel/module is re-linked.

Although adding the debug symbols will obviously slow things down a bit, I’m working under the assumption that the slight increase in build times will be more than offset by the significant decrease in time needed to find out where crashes have occurred. After all, you’ll only be kicking yourself when you find a crash and then realise that you don’t have the debug symbols built.

That’s not much at all, especially since the ToGPA will only ever be done if the kernel/module is re-linked.

Oh – if it’s only when relinking then that’s fine.

Yesterday I checked in my first set of changes. The AAsmModule and CModule makefile fragments will produce the debug symbols during ROM builds, which means most modules will now be producing symbols. However there are a fair number of modules which aren’t using the standard fragments which I have yet to update (including the kernel & HALs). At the moment disc-based components don’t produce any symbols at all, so there shouldn’t be any issues with symbol files finding their way into disc images.

I’ve also released a new version of fiqprof/profanal. Apart from now working on the Pi, and working on OS’s with high processor vectors, profanal now has a ‘loadrom’ command which can be used to load the symbols for an entire ROM image. E.g. ‘loadrom <build$dir> aUVZ00-00’. The ‘info’ command can then be used to easily lookup addresses.