BASIC Tokeniser Bug?

Having been studying the binary content of tokenised BASIC files recently, I’ve come across something that seems a bit odd with the tokeniser:

*BASIC
ARM BBC BASIC V version 1.54 (C) Acorn 1989

Starting with 651516 bytes free

>10IF I% = 0 GOTO 11
>11GOTO 12
>12IF J% = 1 GOTO 13 ELSE GOTO 11
>13END
>LIST
   10IF I% = 0 GOTO 11
   11GOTO 12
   12IF J% = 1 GOTO 13 ELSE GOTO 11
   13END
>RENUMBER
>LIST
   10IF I% = 0 GOTO 11
   20GOTO 30
   30IF J% = 1 GOTO 13 ELSE GOTO 20
   40END
>

The line numbers following the conditional GOTOs (lines 10 and 30 in the final LIST) don’t seem to be getting turned into binary constants (and therefore don’t get updated on a RENUMBER). The GOTO following ELSE, and the stand-alone one, do.

This doesn’t seem correct. Given that it’s been many years since I last used a GOTO, have I misunderstood how much RENUMBER can do?

Isn’t the ‘THEN’ mandatory before a GOTO?

Isn’t the ‘THEN’ mandatory before a GOTO?

IF I% = 0 THEN GOTO 11 does appear to tokenise correctly with the ‘11’ recognised as a line number constant, but I can’t see any mention of this restriction in the BASIC Reference Manual.

The (fairly numerous) examples of the above which have been on my hard disc have presumably been functioning OK over the years, too. Also, on checking, it seems that the original files that I collected from around my various discs seem to have these constants in binary form. It’s only after they have been through a LOAD → TEXTSAVE → TEXTLOAD → SAVE process¹ on a recent version of RISC OS 5’s BASIC that the values end up in textual form. So either the originals that I checked were not tokenised in BASIC, or BASIC’s behaviour has changed at some point.

¹ For those not following csap, I’m writing a platform-independent BASIC V tokeniser². Given the, er, “relaxed” nature of the BASIC spec, I’m testing it by collecting all the BASIC files that are on my disc(s) — mine, third-party apps and bits of the OS — and turning them into text using TEXTSAVE. The output of the tokeniser on all these files is then compared to that from using BASIC’s TEXTLOAD. So far, the above curiosity is the only discrepancy between the two that I can’t explain in around 3,500 different files.

² Tokenize, from http://www.stevefryatt.org.uk/software/build/ — it’s still a work in progress, but feedback is welcome.

OS5.21 (26-Feb-14) Raspberry Pi :-

*BASIC
ARM BBC BASIC V version 1.57 (C) Acorn 1989

Starting with 2093308 bytes free

>10 IF I%=0 GOTO 11
>11 PRINT "hello"
>LIST
   10 IF I%=0 GOTO 11
   11 PRINT "hello"
>REN.
>LIST
   10 IF I%=0 GOTO 11
   20 PRINT "hello"
>

As Steve finds a THEN evades the issue.

OTOH see below.

OS4.39 VRPC iMac :-

*basic
BASIC V version 1.29 � Pace Micro Technology plc

Starting with 651516 bytes free

>10 IF I%=0 GOTO 11
>11 P."hello"
>LIST
   10 IF I%=0 GOTO 11
   11 PRINT"hello"
>REN.
>LIST
   10 IF I%=0 GOTO 20
   20 PRINT"hello"
>

OS4.02 RPCEmu iMac :-

*BASIC
ARM BBC BASIC V version 1.20 (C) Acorn 1989

Starting with 651516 bytes free

>10 IF I% = 0 GOTO 11
>11 PRINT "hello"
>LIST
   10 IF I% = 0 GOTO 11
   11 PRINT "hello"
>REN.
>LIST
   10 IF I% = 0 GOTO 20
   20 PRINT "hello"
>

Looking at the source for the tokeniser, I suspect the problem is the bit of code marked as “KJB fix for PRINT#TRACE,2 or BPUT#TRACE,32, plus later fix for IF F% THEN 30”.

If there’s no THEN to return to Left mode, the tokeniser is already in Right mode when it hits the GOTO and so the fix drops the Constant Due flag before reaching the line number.

In contrast, BASIC on 4.02 gets IF I%=0 GOTO 20 correct but treats the ‘2’ in BPUT#TRACE,2 as a line number — which RISC OS 5’s BASIC doesn’t, due to the above fix.