BASIC tokens

Writing a detokeniser that is capable of processing BASIC V binaries… Can anyone point me to a current token table?

BASIC reference manual. PRM documentation

Or write a program to generate all the possible tokens, one per line, set its type as BASIC, and look what you get.

Writing a detokeniser that is capable of processing BASIC V binaries… Can anyone point me to a current token table?

If it’s to be used on RISC OS, could you not just use the one built into the BASIC ROM?

Otherwise, the ultimate reference must be the BASIC source code

Use BASIC itself to do the detokenisation. CALL TOKENADDR

Use BASIC itself to do the detokenisation. CALL TOKENADDR

Exactly… and if the question is related to this then also use BASIC for the tokenisation. The two processes are surprisingly non-trivial, so leaving it up to BASIC itself is the only sane option.

If you can’t do that (eg. if you’re trying to write a cross-platform tool), then please seriously consider using Tokenize (also here) to do the tokenising. It was put on RISCOS.info long before I was releasing other source code sensibly in the hope that no-one else would need to reinvent this particular wheel.

If there are problems with doing that, then I’m open to discussions about what would be needed to overcome them. Tokenize was necessary for the cross-platform requirement (you can’t call BASIC’s routines when you’re not running on RISC OS), but now it exists, the last thing we need is to have to maintain a third tool to do the same job. :-)

For the tokens try my StrongHelp BASIC manual

There was a series of blog postings by Matt Godbolt doing a Python detokenizer and how he found out about the details: https://xania.org/200711/bbc-basic-v-format

Then, there is “bascat” by Mark Wooding and Matthew Wilcox, written in C: https://git.distorted.org.uk/~mdw/bascat

I also did one, but this is as yet unreleased (and in Java).

Some info on the tokenized format here: http://www.tofla.iconbar.com/tofla/bas/018/index.htm

Tokenise looks pretty much like what I’m after… I’d thought about using BASICs own internal routines, but want to have the facility to process BASIC tokens independently of the host so as to allow detokenisation of current BASIC on older hosts with older versions of BASIC in ROM. If that makes any sense…

allow detokenisation of current BASIC on older hosts with older versions of BASIC in ROM.

There have been no new tokens since BASIC V v1.05 so you are safe to use BASIC to detokenise on any version.¹

The simplest detokenising is a Command (Exec) file:

*BASIC
LOAD "MyProg"
TEXTSAVE "MyProgText"

This ensures that spaces will be inserted where necessary so that the program is tokenisable – see Steve F and the assymetry of these operations.

If you want to you can use the TOKENADDR routine outside BASIC because its address is in the system variable Edit$Detokenise.

¹ that bold statement is is a hostage to fortune ;-)

There have been no new tokens since BASIC V v1.05 so you are safe to use BASIC to detokenise on any version.

When reversing the process, however, the way that QUIT was tokenised changed, so that needs to be watched when re-tokenising because it’s different between RISC OS 5 and RISC OS 4 or Six.

There might be other stuff from before Acorn put the sources into CVS; I didn’t test against RISC OS 3.x when developing Tokenize.

There’s also VDUP. in the inline assembler to watch out for, in a similar way to MOVEQ has a keyword as a substring.
No new tokens were created, but the way the tokeniser in BASIC 1.68 works was modified so as not to end up with VDUPRINT in the middle of your carefully crafted assembler.