BASIC development

I am starting this topic because Rick Murray said:

We’re all shooting hints like crazy to Steve to implement the nifty stuff from Basalt directly into BASIC; but so far he has resisted, partially due to the level of compatibility …

I do not think you read what I said carefully. Nor do I think “We’re all” is likely to be accurate. You might also read what Justin Fletcher has said in his rambles in the sections about BASIC and about the Toolbox. In any case, I am not ready to work on the BASIC module itself for additional reasons.

However, I would be pleased to discuss:

• whether the way Basalt works is a suitable way to extend BASIC
• whether the small limitations Basalt puts on programs is acceptable
• whether the calling overhead of alternative keywords is significant
  – note that Basalt does not interfere with calling of standard keywords
• how the syntax might be improved within the constraints of an external module
• how Basalt and library-based development systems are not compatible
• whether the *BasaltInit¹ at the start of a program is a barrier to its use
• whether the dependence of Basalt on other external modules for certain features is ok
• whether is would be worth pursuing a modular approach this type of extension

and any other matters that concern BASIC programmers.

¹ BASICExtend anyone? ;-)

and…

Hello, I’m lurking.

And no, I don’t think ROM BASIC should be Basalt.

whether the ABC compiler will be updated

Or, as an alternative, set on fire and stamped on? Just a suggestion.

And no, I don’t think ROM BASIC should be Basalt.

Nor do I.

Or, as an alternative, set on fire and stamped on?

I would have been milder, but my recollection from a long time ago is that the special requirements and limited gains of ABC are not worth the effort. I am sure there are devotees who will point out my error, but this topic was not really intended to go that way. ;-)

I agree, the ABC compiler fails the first test: it cannot handle interpreted BASIC syntax. It is a separate language.

RISC BASIC was a true compiler of ARM BASIC and would execute/compile standard interpreted BASIC syntax (with the obvious exceptions of ‘EVAL’).

Nor do I think “We’re all” is likely to be accurate.

Hyperbole, my friend. ;-)

Point is, there are already extensions to BASIC which are good and used by people, but for the various reasons these things have not been integrated into the BASIC module.

I was not saying that you should put Basalt into BASIC (we’ve already covered this), but more using it as an example of why things have not been changed – as a suggestion that DavidS consider the implications carefully of changing BASIC to have more modern features.

You know, BASIC is perhaps the only code on RISC OS that spans the entire machine family; and well written code (nod to GJH) will run on everything from a BBC Micro to a Pi. I should clarify – the same well written code.

In that respect, BASIC is a little bit special.

Don’t have experience with ABC.
It’s a shame the sources aren’t in the archive though, I’ve always thought it’d be fun to look at a compiler (a simplistic compiler may dedicate a standard akin to the APCS and stack/unstack registers all over the place, but a more sophisticated one will pay attention to what registers are used where – but upon what basis? That sort of nerdy thing interests me!).

I agree, the ABC compiler fails the first test: it cannot handle interpreted BASIC syntax.

It can’t even handle COLOUR r,g,b.

I agree, the ABC compiler fails the first test: it cannot handle interpreted BASIC syntax. It is a separate language. bq.

My experience is that there’s only a very limited set of instructions that the ABC compiler doesn’t understand, such as EVAL and something like i%()=0 (I never used COLOUR r,g,b, though). Speed gains heavily depend on what sort of code has to be compiled, but while a factor of 2-3 is normal, I’ve also seen a factor of 40 (!) in two of my programs. I really would be lost if I couldn’t use the compiler anymore.

My experience is that there’s only a very limited set of instructions that the ABC compiler doesn’t understand,

It does not handle LOCAL variables correctly, keeping them in scope only within the procedure in which they are defined. Procedure and function calls from that procedure use the global variable of the same name rather than the local value. They are treated as PRIVATE variables to that procedure rather than being pushed onto the stack.

I really would be lost if I couldn’t use the [ABC] compiler anymore.

You will always have the compiler, but any extensions would have to be written directly for it. It only shares its source with interpreted BASIC where it touches and otherwise it is quite separate.

Has no-one anything to say about BASIC itself?

Any mention of ABC is guaranteed to raise the hackles in some quarters (Zap and StrongEd?). I used ABC a lot in years gone by, and found it very useful for giving a speed increase (as Paul mentions sometimes a lot more than the 3 times often quoted) in large programs. Writing code within the compiler specification means there is no problem with e.g. local variables, using sensible names, and passing variables to functions as needed.

I certainly found it invaluble for some projects.

Is anybody actually developing ABC? Maybe there is something to be said for putting up the sources so it can be poked around with? I don’t expect anything other than BASIC itself will support EVAL, however some things could be added/fixed.

As for BASIC itself, Steve, it’s an interesting bind. BASIC perhaps needs some modifications such as the ability to use a named struct instead of seemingly random poking in memory (how many WIMP programs are full of block%!40 and the like?) ¹; but we must be really careful to not break BASIC and we’re stuck with the idea of “that’s cool but will it run on an A3000?”.

One step forward + one step back = … ;-)

¹ Yes, I know, I know, but you were asking about BASIC.

Re Basic enhancements: I have been running an updated Basic module here for months without any problems, even though it is softloaded in PreDesk. I accept that if the module is killed or replaced, then problems may happen if there are any Basic programs running. I strongly support any moves to enable a standard Basic to be available for ALL machines, either in new RO5 roms or as a softload.

One facility in the original BBC Micro Basic was that the ‘Mistake’ error could be trapped, so it was possible to add keywords to Basic. This ability was removed when changed to a RISC OS Module, but it would be a very useful way to be able to ‘plug in’ enhancements for testing or actual use if a proper facility was added. Obviously this would not cover all possible additions, only the extra keywords.

However, I think that some enhancements should be added to Basic itself, where they are of general benefit and use.

Because this topic now includes ABC I am going to take the opportunity to expand a little about compiled and interpreted BASIC. Nothing is meant to criticise those who want to use ABC for their own good reasons.

I paid real money for ABC more than 20 years ago, before I really knew how to program in assembler. I did quite a lot of experimenting with it then, and yesterday I pulled out the manual to remind me of that time. There are many more constraints than I remembered, and very many more than Basalt imposes.

Undoubtedly, some programs will benefit greatly from compilation. Of note is the CASE structure, particularly the jump table version. CASE is the Achilles heel of BBC BASIC. So many programs are written without regard for how it works, with all the action code included between WHEN clauses rather than being encapsulated in separate PROCs ¹. CASE is used often within loops, which magnifies the effect. So compilation is valuable here ².

However, there are instances where a compiled program is much slower. ABC does not support whole array operations, so A()=B()+C() would have to be written as, possibly nested, FOR…NEXT loops. Even when compiled this is not nearly as fast as optimised assembler. I seem to remember problems with floating point operations, too.

This brings me to my main point. A suitable keyword using optimised assember is likely to be faster than a routine written in BASIC and then compiled. As an example take Basalt’s SWAP$, which can be used to change case. Even the most efficent BASIC routine will involve several statements and a loop ³. The compiler will have to contrive this using generalised methods for variables etc, and a BASIC function could never be made as flexible as that keyword without a mountain more code.

Taken to an extreme, Basalt also has a POLL keyword, used in conjunction with FNs registered against Wimp or Toolbox events. That one keyword can replace pages of BASIC code, including CASE structures.

So, if you identify commonly used routines in BASIC that are slowing programs and then write keywords to do the job, you may actually have more benefit than from a compiler ⁴. I learned this philosophy from the author of BetaBASIC for the Spectrum and later for the SamCoupe, Andy Wright, and he had it from earlier commentators. It can be challenged in many ways, of course, but it is still a guiding principle for me.

BTW Nothing here questions the advantages of compiling a language designed to be compiled.

¹ I have seen plenty of CASE structures extending over several screensful.
² A good while back there was a discussion in c.s.a.p about improving CASE and it remains a to do for Basalt. I know that nemo produced his own version of a pre-calculated CASE. Can users of ABC identify any other particular areas where compilation produces large speed increases? They might also be targets for improving interpreted BASIC.
³ The odds on a programmer using an efficient method for this appear to be slim judging by posts on the newsgroups over the last 20 years. {bitter or what?}
⁴ I have not included the use of crunching. I suspect that this might have many of the speed benefits of compiling, but I have no real knowledge.

My poll handling code tends to be a big CASE structure. Is CASE really that bad?! [I call handler functions where there is lot of code, but short things are dealt with right away; also use CASE for wimp message dispatch…]

I’d have thought the advantages of compiling would have shown up most in a program that does a lot of calculation.
Akin to VB’s p-code vs native code option, there isn’t that much difference in a program that makes heavy use of the desktop and just fires short handlers for the incoming events. A program that does lot of calculation, on the other hand, has something a good compiler can speed up. Consider a sprite editor applying an effect to an image or doing an error diffused resize.
Oh, and while there is always the option to hand optimise chunks of code in assembler, for some assembler is scary and complicated, which is, I guess, the ideal target demographic for a product such as ABC.

I have been running an updated Basic module here for months without any problems, even though it is softloaded in PreDesk.

I have been running soft-loaded BASIC v1.20, modified by ExtBASICasm, for ages. But, I think I know what I am doing and I am prepared for the consequences. The real question is what the general user will make of having to soft-load versions of BASIC (plural for updates).

One facility in the original BBC Micro Basic was that the ‘Mistake’ error could be trapped, so it was possible to add keywords to Basic.

I once asked Sophie Wilson whether there was any way to use something like that from BASIC V and I think the gist of her reply was that the error handling had been completely changed to allow for the use of international BASICTrans. However,it is noteworthy that the aforementioned ExtBASICasm modifies the BASIC module to provide vectors from quite a number of errors so that it can provide all the new assembler features. I do not think it would be a case of just one vector now.

What I am more interested to know is why such a feature would be acceptable, whereas there seems to be resistance to the way Basalt provides an equivalent hook by modifying the BASIC program itself. There would be hardly any difference in the overhead, if anything towards Baalt because new keywords are tokenised rather than text and there is no error to handle..

However, I think that some enhancements should be added to Basic itself, where they are of general benefit and use.

Once you add any enhancement with a view to making it universally available in a softloaded module you make it necessary for all users to follow. I do not oppose this future, as I said in my much earlier summary, but I am aware of the problems.

Is CASE really that bad?

I think so. Its just the extra time it takes to look line by line to each WHEN clause, even though a CRUNCHed program becomes optimised for this. Single lines following WHEN are not really a problem.

I’d have thought the advantages of compiling would have shown up most in a program that does a lot of calculation.

I agree. On the other hand ABC always uses the FPE (real FPU anyone?). It can do single, double or extended precision, but I seem to recall that this is not particularly fast compared to the 5-byte routines within BASIC itself. My recollection is hazy here and I stand to be corrected. Integers are undoubtedly quicker and I expect the cumulative effect of interpretation over a complicated set of calculations is also significant.

the ability to use a named struct instead of seemingly random poking in memory (how many WIMP programs are full of block%!40 and the like?)

The topic of Structs returns again and again. There have been a number of implementations over the years, either modifications of BASIC itself, which have not found their way into the wild and I have not seen, or BASIC libraries. I think all of the latter I have seen have involved some assembler. It is also possible to make a good stab at structs using just indirection, but the namespace gets very crowded and the [sub]struct syntax is neatest when done from right to left, which is a bit counter-intuitive.

It is probable that all our active developers using BASIC are using libraries, either published or their own. All those indirections for the Wimp are already written and do not need to be thought about. Or they might be using the Toolbox.
Just how important is the availability of structures? I designed a system of my own a year back and published it as a library . That was the fun part, but I have always intended to implement it directly into Basalt, where it would be somewhat neater. I have had no comments about it. Would you care to take a look?

I think structures are the most critical thing missing from BASIC.

A key part of programming is using the right data structures. Data structures require language support, particularly typed structures – so you can’t write a string into a float. If you don’t have easy means of using structures, you’re limited in the algorithms you can use because the structures get clumsy very very quickly. Structures are also a key to writing modular code. BASIC is quite a nice ‘scripting language’ (to use the modern term) but for this.

The other thing that goes alongside structures is dynamic memory allocation (including freeing and resizing) – DIM in BASIC is pretty braindead (eg DIM in a procedure is a great way to memory leak).

What does BBCB4Windows do relating to structures, and would that be applicable to Basalt?

The real question is what the general user will make of having to soft-load versions of BASIC (plural for updates).

This didn’t seem to be a great hardship with SharedCLib. There will, no doubt, be a period of upheaval as things are done in which case plenty of releases may occur. However, it is worth noting that just because a version is available, world+kitten does not need to update. So long as backwards compatibility is maintained (duh!) and the hot new features are not required by the programs that users use (or write), then there’s no reason why they can’t “be aware” but skip installing the latest version. I, myself, am not running the latest versions of 5.19. I update periodically, but not daily.
Plus, there’s a danger in using bleeding-edge builds, the general user will probably not want to do that. Full releases will likely not be that often. After all, look how many hundreds of versions of RISC OS 5.19 have followed 5.18. How many hundreds more do you think there will be before ROOL proudly hands 5.20 to the waiting world?

What I am more interested to know is why such a feature would be acceptable, whereas there seems to be resistance to the way Basalt provides an equivalent hook by modifying the BASIC program itself.

Speaking for my own interpretation – I wonder if it is to do with dependencies? Whereas you might have to *RMEnsure the correct version of BASIC, and rest with the hope that future versions of BASIC will be automatically compatible; with Basalt there are now two things to keep track of plus the concern of “will this Basalt module work with that new version of BASIC”?
To be honest, I think it’s something of an artificial concern given the amount of rubbish most useful programs check for (this module, that module, this OS resource, that OS resource). If you can keep track of several modules, adding another shouldn’t be a hardship.

Question is, though, if App #1 has loaded an older version of Basalt, can App #2 load a newer one of top of it? Does Basalt contain any state data or is it replaceable during runtime?

Once you add any enhancement with a view to making it universally available in a softloaded module you make it necessary for all users to follow.

Users don’t need to use new functions – I wonder how many make full use of BASIC’s array handling – and … as for backwards compatibility, it isn’t as good as should be.

RPCEmu with recentish RO5:

>COLOUR ON 255,0,255
>CLS

It works.
However, RedSquirrel with RISC OS 3.70:

>COLOUR ON 255,0,255
Unknown or missing variable
>

…

Though, should this stop us from looking to ways to extend BASIC on newer machines? Back in the days of Acorn User, it was quite common for *Info programs to specify RiscPC only because most were cool graphical demos and wibbly-wobbly things and most needed more than the 64-colours-plus-tints offered by the original VIDC hardware.
I think so long as a softloadable version of BASIC is available for the later machines (say, Iyonix and later¹), that could be a cut-off point for “new features” vs “old features”; knowing that if you wish to support hardware prior, you need to “do it the old way”. Not ideal, but it isn’t as if this has never happened before.

¹ I suspect how far back a softload of BASIC can actually support will be a purely technical decision – can I softload RO5’s BASIC into a 26 bit world? Are there flags in the BASIC module to build a 26 bit version?

I agree. On the other hand ABC always uses the FPE

That’s all we had way back when. However these days with not one but several different FP co-pros on the silicon, it doesn’t make a lot of sense to use a slow emulated FP.

but I seem to recall that this is not particularly fast compared to the 5-byte routines within BASIC itself.

I’d have thought a specially crafted routine in ARM code would work about as fast as… a specially crafted routine in ARM code. ;-) You need hardware FP for there to be any difference to speak of here.

I expect the cumulative effect of interpretation over a complicated set of calculations is also significant.

I had a program that manipulated 256 records in a structure. When written in BASIC, removing the first and shifting all the rest up a position took about 40 seconds². It would have been quicker to GBPB the tail end of the array to disc and reload it. Perhaps not an optimal algorithm, but the exact same code rewritten in C and compiled did the job in about half a second. So when you are farting around with big wodges of data, you can see benefits. Each program is different.

² On an A5000, clocking something in the order of 25MHz. 25MHz is a 40th of 1GHz, so by a rough estimate it should take a mere second on a Beagle. ;-)

The topic of Structs returns again and again.

I am writing a program in C. I don’t want to give anything away until I have something to release, so I will just say that the API has “zones” that are controlled by data in structs, and the elements themselves are structs which contain substructs for flags and state.
Therefore – thingy[thisone].state.something = TRUE is a hell of a lot better than trying to use direct indirection block!(this+state) or something, or heavy abuse of multidimensional arrays.

In my opinion, what is needed is a veneer so we can DIM a bit of memory and then refer to parts of it by name rather than offset addresses. During development (with a lot of code already using the struct), I realised that I needed an extra element in the state substruct (processing now to save a headache later). So I modified the appropriate header file and recompiled. Imagine the pain in BASIC if using indirection, everything that refers to the struct would need to be checked and amended. This might be as simple as tweaking some offsets, or it might require extensive reworking.

I point out Wimp calls because these are perhaps the first place a BASIC coder will come across heavy use of indirection for getting a job done. As you say, everybody ought to have (or have access to) a library by now, given the age of RISC OS! This only glosses over the lack of struct support in BASIC, it doesn’t remedy it.

I’ll take a look at your library later; though I may not be the best person to give an opinion as I’m a polyglot. If something is going to use a lot of data structures and it looks “hard” (subjective!) in BASIC, I’d just write it in C instead. ;-)

Just a quick question:

Before a stucture can be used a prototype must be defined with FNcDEF; 
Person=(FNcDEF{first$,second$,birth%}) 

Definitions can include previously defined prototypes, which can be
recursive: 
Book=(FNcDEF{author{person},title$[24],pub!})

Defined as “Person”, referenced as “person”. Typo, or are the elements not case sensitive?

What does BBCB4Windows do relating to structures,

Sort of what we’d be looking for, if the docs are any indication…

http://www.bbcbasic.co.uk/bbcwin/manual/bbcwin2.html#structures

I think structures are the most critical thing missing from BASIC.

Would you use them if they were available? Do you do that sort of programming in BASIC? Do you like the general outline of structures that I implemented in that library?

The other thing … is dynamic memory allocation

This was the first thing in Basalt over 10 years ago; it is the foundation of a lot of what Basalt does. Even so, it is not that difficult to do in BASIC code, or using available support modules. It only requires the will.

As a matter of principle, Basalt does not replace BASIC’s DIM, but provides an alternative syntax, DIM HIMEM. I am often tempted to break that principle, though. ;-)

What does BBCB4Windows do relating to structures, and would that be applicable to Basalt?

Not really. There are lots of barriers to copying it directly.

@Rick Thank you for your thoughtful comments. I covered many of them in my original summary. Have a look at what I said about using DIM LOCAL.

This didn’t seem to be a great hardship with SharedCLib.

Not in the distant past, but remember the 32-bit SCL. I do not mean the internecine war, but the problem some people seemed to have getting it installed. Without the overarching control of Acorn things are different and we are dealing with a split user base.

As for RO 5.19, no-one would write a BigProgram using any features not in 5.18, surely? And who would write a BigProgram using the highly desirable features of the Toolbox in RO 6 for general release? Someone might write or update a program for release using the latest version of BASIC and then find their users annoyed at having to keep reinstalling the BASIC module. I don’t know. Can anyone use the ROOL versions other than with RO 5 anyway?

Question is, though, if App #1 has loaded an older version of Basalt, can App #2 load a newer one of top of it? Does Basalt contain any state data or is it replaceable during runtime?

Yes. It is designed that way specifically to overcome the problems I have been going on about. To be completely transparent, if a program is in a POLL loop when the module changes, there is an error, but this can be trapped and the POLL recalled without the program user noticing. I could work on that. ;-)

backwards compatibility, it isn’t as good as should be.

The examples you gave are for single tasking; you would not use them in the Wimp.

[FPE] I’d have thought a specially crafted routine in ARM code would work about as fast as… a specially crafted routine in ARM code. ;-)

Yet BASIC VI is much slower here than BASIC V. I wrote a pet project of the hyperbolic functions: SINH etc. Initially I could only do this by invoking the FPE and I was amazed to find that my keyword was about 10 times slower than defining a BASIC function to do the same. Eventually, I found a way to use the necessary routines within the BASIC module and the keyword is just about faster, and it does the bounds checking that would slow the BASIC function further.

You need hardware FP for there to be any difference to speak of here.

I have not investigated far, but I think BASIC VI is faster than BASIC V on the new processors.

what is needed is a veneer so we can DIM a bit of memory and then refer to parts of it by name rather than offset addresses

That is the model I have adopted.

Defined as “Person”, referenced as “person”. Typo, or are the elements not case sensitive?

Typo.

Not in the distant past, but remember the 32-bit SCL. I do not mean the internecine war, but the problem some people seemed to have getting it installed. Without the overarching control of Acorn things are different and we are dealing with a split user base.

? I just took the installation stuff and threw it at !SysMerge…until I figured out what SysMerge does at which point I just copied it in by hand. What sort of problems were people having?

Remember also that the RPi release has introduced the concept of packages. Perhaps things such as core-component updates (if not a part of the OS itself) could be handled by way of packages and a package manager; and reverting to the likes of SysMerge for older machines.

As for RO 5.19, no-one would write a BigProgram using any features not in 5.18, surely?

I was thinking of a program written for 5.xx and 3.xx.

And who would write a BigProgram using the highly desirable features of the Toolbox in RO 6 for general release?

I prefer to pretend that RISC OS 6 doesn’t exist. If nothing else, it being v6 would surely mess up the version number checking!? Will we, here, have to skip to version 7 if we make a large incremental update? Maybe we should adopt Firefox’s insane nonsense, we’d be on version 27 by now, surely…

Someone might write or update a program for release using the latest version of BASIC and then find their users annoyed at having to keep reinstalling the BASIC module.

As I said, once the core modifications have been made and tested, it will surely settle down. So the number of coders using bleeding-edge versions of BASIC in general-release programs is likely to be a rather small figure.

Can anyone use the ROOL versions other than with RO 5 anyway?

There’s an important question. Can anybody with the BASIC sources build a 26 bit version of it? [my DDE is Castle-era, probably too old to do it correctly]

The examples you gave are for single tasking; you would not use them in the Wimp.

The examples I gave were highlighting changes in BASIC which were not made backwardly compatible.
If we are talking Wimp programs, there are many changes that are not backwardly compatible (R02→3 WimpMessage filters, pollwords; RO3→3.5 better error boxes, fonts as standard; RO3.x→ nested Wimp; RO3.x→5 start up Unicode, a lot of visual stuff, and more); but none of this is BASIC’s fault or problem.

Yet BASIC VI is much slower here than BASIC V.

Well, isn’t there a lot more work to do to support a deeper level of accuracy?

Shouldn’t new features in RO5 be supplied for older systems in modules?