BASIC compiler

So I spent a pleasant evening yesterday thinking about how a BASIC compiler might work. I’m not planning on writing one, just a thought exercise really. It is pretty much a given that EVAL can’t be implemented as it has the ability to do some quite horrific things.
But it seems the more you dig, the more BASIC can be (ab)used in all sorts of awful ways. Consider – multiple DEFFN statements with only one return point after them all. Or how about a function that can return an integer or a string, the return type chosen at run time. If you consider old style code that uses line number references, you can get creative (read scary) code. And it seems BASIC is quite happy to accept GOSUB FALSE as that uses two bytes while GOSUB 0 uses four.

While I’m here – I’ve offered a bunch of times to look to modernising ABC (the BASIC compiler we do have) with some of the features added to the language since…the ’90s.
ROOL, hello?

Can you make (me) a summary of possible ideas?
(and others too).

Nota: it’s perfectly possible to implement eval in compiler. The only big problem is interaction with code already compiled, that supposes to maintain an array of references of variables.

While I’m here – I’ve offered a bunch of times to look to modernising ABC (the BASIC compiler we do have) with some of the features added to the language since…the ’90s.
ROOL, hello?

Try email. They don’t have much time to check the forums.

Can you make (me) a summary of possible ideas?
(and others too).

If I were to write a BASIC compiler:

• I’d be tempted to write it as a frontend to llvm/clang. No point writing a compiler backend when someone’s already written one for you, and generating opcodes for a virtual machine will be easier than (e.g.) converting BASIC to C. Of course there is the slight downside that we don’t have a stable port of llvm for RISC OS!
• Code structure is the most important thing – getting that right first will be more important than things like trying to (e.g.) reproduce the behaviour and limitations of BASIC strings.
• Rather than viewing the source as a series of functions and procedures it probably makes the most sense to view it as BASIC does – a series of numbered lines, with functions and procedures mainly just being a slightly more programmer-friendly method of flow control than GOSUB and GOTO.
• Make sure you respect the fact that LOCAL variables aren’t actually local, all they do is save and restore a local copy of the global variable. Generated code should just use global variables. In a way, it might make sense to view the program as if it were designed to run on a machine with a large number of registers, so that each variable can be allocated to a register. The optimisation steps in the compiler can take care of mapping those virtual registers to real registers.
• Once you’ve parsed the source and built the flow graphs it should be possible to perform many optimisations, e.g. eliminating line numbers if there aren’t any variable-based GOTO or GOSUB statements (or if possible, computing what possible line numbers those statements could take you to), or optimising variable accesses so that values are only written back to memory when necessary, or examining functions to determine what the possible return types are.

Make sure you respect the fact that LOCAL variables aren’t actually local, all they do is save and restore a local copy of the global variable.

Something at which ABC fails miserably. I had a programme that had a rather long procedure which used LOCAL variables so that it used a local environment (e.g. width% and height%, variables used elsewhere with their global values) and, itself, called functions like FNarea (rather than FNarea(height%,wdith%)) everywhere that was needed. In ABC the result, inside the procedure gave the global area not the local area. Most confusing.

What I had done was to take a chunk of code (complicated, but used in several places) and convert it to FNcomplicatedcalc so that I only had to edit it once. DOesn’t work in ABC.

Some thoughts. Read an old discussion in this forum first: https://www.riscosopen.org/forum/forums/11/topics/3890

Then decide if you want to produce a BBC BASIC compiler that tries to mimic all the hilarious stuff possible with the interpreter, or if you start with a sane new BASIC dialect that closely resembles the good stuff of BBC BASIC and other BASIC dialects (after all, we do want composite types, visibility rules, proper strings with UTF8 support, pointer types, memory management way beyond DIM…).

If you decide to go the first route, just stop and give up. It will all end in tears.

If you decide to go the second route, you could do worse than trying a google translate on one of my blog posts: http://riscosblog.huber-net.de/2016/02/risc-os-projekte-ein-kompetenter-basic-compiler/

Do not try to reinvent the wheel. You don’t want to write a compiler backend – trust me. Just implement a frontend for GCC or LLVM. If you have the perfect frontend, you can still decide to write your own backend later (i.e. never).

(after all, we do want composite types, visibility rules, proper strings with UTF8 support, pointer types, memory management way beyond DIM…).

If you want that kind of functionality, I’d say you’d be better off forgetting about BASIC and porting a compiler or interpreter for a modern language instead (Python, Java, C#, Go, etc.)

If you decide to go the first route, just stop and give up. It will all end in tears.

It was just something I was thinking about. I don’t plan to create yet another attempt at a BASIC compiler.
If I was to do anything, I’d modify ABC’s code to use more modern facilities and support some of the more recent changes to BASIC.
Rolling my own? Life’s too short.

You don’t want to write a compiler backend – trust me.

My current understanding of “compiler” is akin to “read instruction, emit some code that does the same thing”.
I know the Norcroft compiler parses the input, turns it into some sort of intermediate code, does all sorts of freaky analysis on it, then emits some code. I did read an ebook on compiler design, but it quickly got really complicated. So no, it’s not something I plan to do. Please don’t think that. :-)

(after all, we do want composite types,

Structures would be lovely. Imagine WIMP blocks when we can use block%.something instead of all those !## offsets.

visibility rules,

Aren’t all variables global unless declared LOCAL in which case they exist only for that function (and any function called from the LOCAL function)?

test% = 5
PROCmeh
PRINT test%
END

DEFPROCmeh
  LOCAL test%
  test% = 2
  PRINT test%
  PROCwhatever
ENDPROC

DEFPROCwhatever
  PRINT test%
ENDPROC

When run, it would print 2 (PROCmeh), 2 (PROCwhatever, inherited from PROCmeh), 5 (original global variable).

proper strings with UTF8 support,

I’m guessing by UTF-8 support you mean LEN returning the character count instead of the byte count, MID$ (etc) working with characters instead of bytes, and ASC/CHR$ coping with codes outside of the byte wide range? That would be nice, but given how much support RISC OS has for Unicode….. don’t get me started…again… :-(

pointer types,

Isn’t block%!offset% just a really clumsy way to handle pointers? We can even do $(block%+offset%) to fudge strings (although BASIC and the rest of the world use different concepts of string termination).

memory management way beyond DIM…).

DIM isn’t bad, it just needs to be able to:

• Cope with self-modifying the WimpSlot if the DIM expands beyond the current allocation.
• Specifically FREE a DIM’d bit of memory, including tidy-up to release the memory. It shouldn’t matter moving the other DIM blocks as you’d be referring to them by variable name instead of absolute address, right? RIGHT? ;-)
• Because people can be weird, a mechanism (SysVar?) to disable the garbage collection.
• The ability to REDIM without doing stupid things like trashing the data already in the array (unless REDIMing to a smaller size, in which case the end data can be discarded).

Well you probably know already what I would say upon this topic :).

I must say that I am a bit puzzled by this obsession with BASIC. Yes, it is there in the ROM ready to be used, warts and all. Of course there is the nostalgia factor too.

Do not try to reinvent the wheel.

But some people find it fun to think about reinventing the wheel, or so I imagine.

If you want that kind of functionality, I’d say you’d be better off forgetting about BASIC and porting a compiler or interpreter for a modern language instead (Python, Java, C#, Go, etc.).

Indeed, but I fear that there are now very few RISC OS users with the time or inclination to learn anything new :(.

If you want that kind of functionality, I’d say you’d be better off forgetting about BASIC and porting a compiler or interpreter for a modern language instead (Python, Java, C#, Go, etc.)

I used to think that way, too.

However, the fact that so many RISC OS users/developers still use BBC BASIC even if a compiled language like C or C++ would be a much better fit (or an alternative interpreted language like Lua or Python or Perl, all of which are or were available) has convinced me that “use another language” just does not work in RISC OS world.

A competent BASIC compiler would be a much more natural upgrade path for your average BBC BASIC coder than any alternative, modern language.

There are things that C provides that one wishes BASIC could do:

• structures with named elements
• printf style string handling
• the usefulness of the ? : construct.
• DIM was fine in the single tasking world of the eighties. Now? Now it is a ridiculous limitation that imposes on people the need to either waste a lot of space with unnecessary DIMs or design their own flex like memory management.

But then there are things that BASIC does better:

• building composite strings is a lot easier in BASIC (you are probably supposed to sprintf to a temporary string and strcat that to the end of the one you want, although with pointer abuse you can directly sprintf it)
• BASIC routines are more consistent with parameter placement (some parts of standard library feel like “we just made this up at the time”)
• the standard library tries to offer a lot of functions, but it has some puzzling omissions – for instance I can create a uniquely named temporary file but there is no way to list files. Granted, BASIC doesn’t offer this either, but then it doesn’t pretend to be everything that C tries to be. Every platform offers additional routines to do such things, but “poof!” there goes this so called platform independence. We now have similar C in the same way that true BBC BASIC and BBfW are similar but not the same.
• Pointers. Probably the biggest headache of the language for many people.
• A complete and utter inability to manage or enforce any sort of bounds checks. While the supposed elite often say that you’re supposed to know what you are doing in C, the sheer number of fundamentally important privilege escalations in software and systems resulting from something overrunning buffers and trashing stack, workspace, or other parts of the program prove that reality is quite different. I’d rather choose to lose a few cycles every memory access, but standard C does not offer such an option.
• Error handling. I’ve used BASIC since the mid eighties and C since the mid nineties and I still think BASIC has a better handle on dealing with errors. C’s approach looks like it never matured much beyond what was typical in the era K&R was devised. And as for the awful safety net that is longjmp…….

In short, tl;dr version: every language has anachronisms and quirks. I highlight C compared to BASIC but I’d imagine it would be the same for any other language, just a different set of list entries. If a programmer likes how BASIC works, be it familiarity or habit, who are we to say “use something else” when they wish for BASIC to offer some more modern facilities?

It is worth noting how many snippets of code and examples of things are given in BASIC. While the language isn’t really suitable for big complicated applications, it is the language of choice for smaller things and being built into the machine means that it is available to everybody equally.
It is also worth noting that Amcog’s most recent game, the Fervour++, was written in BASIC. Entirely in BASIC.

While the language isn’t really suitable for big complicated applications, it is the language of choice for smaller things and being built into the machine means that it is available to everybody equally.

I can see one downside to having a readily available (free/built in) compiler for BASIC:
All those little snippet tools etc that people produce that “just work” when moved onto a new hardware set may well fall over in a heap on a compiled release and have no readily available “source” so the user can just fix it.
Almost a case of needing to enforce a “thou shalt include the source” rule.

If a programmer likes how BASIC works, be it familiarity or habit, who are we to say “use something else” when they wish for BASIC to offer some more modern facilities?

In most cases, “we”¹ are the ones that they are expecting to implement it.

Who are they to say that we should spend hundreds of hours working on extending BASIC, as opposed to porting compilers/interpreters for languages that are more modern and popular?

¹ I.e. anyone who opposes the idea

Swift anyone??

A competent BASIC compiler would be a much more natural upgrade path for your average BBC BASIC coder than any alternative, modern language.

Steffen: I think you are right. The difficulty, as the ABC compiler probably demonstrates, is that BBC BASIC appears to lack any kind of formal definition, whether for syntax or operational semantics, beyond its implementation. That makes it rather hard to test the correctness of a compiler.

It is worth noting how many snippets of code and examples of things are given in BASIC.

But only those things that are already expressible in BASIC. I do not think BASIC is very convenient as pseudocode because it lacks all sorts of programming concepts: higher order functions, for example.

Pointers. Probably the biggest headache of the language ( C ) for many people.

One of the things that I like about Lua is that it does not have pointers, and yet it can still express, using tables, all the data structures for whose definition C uses them, and just as easily. Bounds-checking headaches cannot arise because there is no explicit allocation of memory in Lua. Lua uses hash tables at the lowest level, rather than the notion of array. In retrospect, the Achilles heel of the traditional imperative languages was basing all data-structuring on arrays. This has so many limiting consequences, not least upon how programmers think – I believe that E.W.Dijkstra’s rude remarks about BASIC were actually intended to address this point, and could more fairly have embraced many other languages.

BASIC was born before Pascal, which taught the importance of not confounding types, and before Modula, which taught the importance of information-hiding. BBC Basic is full of wonderful facilities: for I/O, for accessing the OS, for graphics and for sound. But they are all thrust higgledy piggledy into one bag. Keeping separate facilities in separate libraries, as more modern languages do, helps keep the language simpler – an important consideration when choosing a language for its educational possibilities.

When I was using the ABC Compiler, many years ago – the main reason I used it was that I wanted compiled code that couldn’t be altered (easily).
Not everything should be open source.
From what I remember (in the arm2/3 days) compiled code wasn’t loads faster than interpreted BASIC.
It’s one of the things that people always seemed to point out.
But at the time, before I started into C programming, it always seemed to me compiled code “seemed” more professional.
Now having seen other peoples BASIC code and my C code, that clearly isn’t always the case, there is a lot of beautiful BASIC out there – but it’s the impression that some people have, right or wrong.

Trying to ABC !Xeroid (for fun) is interesting, it gives different errors nearly every time, from Abort on Data Transfer, unknown SWI errors on line numbers (where there clearly isn’t)

So I’d support a bounty to update it, it’s a useful tool and possibly as was suggested a nice way for people to start learning about compiled code.

Not everything should be open source.

Well, that is one way of ensuring that the software will not survive future hardware upgrades :(. How much enthusiastic effort (AppBasic, … ) has been lost to future generations because authors have BasCrunched or compiled their work; deliberately concealed the sources in case, oh horror, it is pinched. It is not as if they were losing any money, the RISC OS market being what it is.

Do not get me wrong. I am all for software authors being recompensed for their efforts, and I have been happy to pay up for some excellent, and excellently maintained software. But in the present circumstances I think we should not forget about future-proofing as much as possible. In fact I would go so far as to say that authors of compiled or crunched code should make accessible the sources, together with detailed notes on the implementation, the algorithms, the to-do lists, the platforms used, etc. Plagiarism is the greatest compliment, so encourage it. The hide-your-poker-hand attitude is disastrous for the future. Isn’t it?

Sensitive readers should avert their eyes at this point. It is confession time. I have never asked for money for any of my software efforts. That is partly because I am not entirely confident in my ability adequately to package, distribute and maintain my efforts. These are important aspects of software provision that are sometimes overlooked, being less exciting than actual coding. They are what the money should pay for, when it is paid, IMHO. And if money is not being paid, and you want others to use your work, it would help to document it as well as possible.

The RISC OS community is mostly old. I am 77. Software authors need to think about what happens to their software, their websites, their products, when they are gone.

Well, that is one way of ensuring that the software will not survive future hardware upgrades :(. How much enthusiastic effort (AppBasic, … ) has been lost to future generations because authors have BasCrunched or compiled their work;

Which was my point.

In fact I would go so far as to say that authors of compiled or crunched code should make accessible the sources, together with detailed notes on the implementation, the algorithms, the to-do lists, the platforms used, etc. Plagiarism is the greatest compliment, so encourage it.

If you’re giving it away you aren’t losing income if someone produces a modified, possibly better, version and puts that out for public use so why not make it available.

I would agree that people need to think about their software’s future. But that doesn’t mean it has to be open-source necessarily.
The reasons for most software being abandoned is down to the users.
To say everything should be open-source is naive at the least.

There are plenty of examples of people putting in lots of effort in both software and hardware and other people taking the piss, that’s in RISC OS and outside of RISC OS.
Both recently and historically.

The first thing to be fixed in RISC OS is to end the pointless politics.

I do like the Castle licence that allows delayed release, that’s really nice.
But the source needs to be held somewhere…

I would agree that people need to think about their software’s future. But that doesn’t mean it has to be open-source necessarily.
The reasons for most software being abandoned is down to the users.

Lack of users is probably more accurate, even people who do stuff for free tend to like to see it used regularly. For those charging then the price of a fish n’ chip supper isn’t much recompense and there’s probably more to had elsewhere so RO loses the author.

To say everything should be open-source is naive at the least.

I don’t think anyone suggested that. I certainly didn’t.
Anyway, if all we’re talking about as the use of a compiler is the obscurity of the code content then I think the discussion is in a pretty poor state.
Obscuring the source on paid for software is valid, I can’t see that obscuring the source of free software has any real merit.

One application I’ve written, albeit for Windows, I distribute for free but I’m not releasing the source code because it uses code (or rather, my translation of C code into Pascal) from a commercial product (with the publisher’s knowledge and OK).

Going back to the subject, I’d like to see a package on RISC OS like Delphi or Visual Basic – where you can create the forms (windows) and attach code (in Pascal or Basic) to buttons and events. Then you don’t need to worry about the Wimp_Polls, or window handling, as this is dealt with by the compiler. But, aren’t these two written using their own dialect of Pascal and Basic?

(OK, I know that there is bound to be a package already available for RISC OS just like this, and I just haven’t done my research!!!)

Cheers,

Gerald.

@Steve – Agreed.

There is DrWimp not exactly what you were talking about, but it looks a good start.

It’s avaliable for download via PackMan I notice as well

One application I’ve written, albeit for Windows, I distribute for free but I’m not releasing the source code because it uses code… …from a commercial product

That, I think everyone would agree, is a valid reason for obscuring the code and not releasing source.

Have you looked at http://appbasic.jettons.co.uk ?