BASIC allows redimensioning of a local array inside a FOR loop

The IOMD build of BASIC V allows a local array – i.e. one declared as LOCAL a%() – to be redimensioned, provided that the action takes place inside a FOR loop.

Sample code:

PROCtest
END

DEF PROCtest
LOCAL a%()
FOR i%=1 TO 5
PRINT i%
DIM a%(10)
NEXT
ENDPROC

This will fail on the third pass.

Remove the LOCAL a%() line and the program will fail on the second pass.

RPCEmu 0.9.3a for Mac, RISC OS 5.24 (BASIC 1.75) and RISC OS 5.29 (BASIC 1.84).

The BCM2835 build does not exhibit this bug. (EDIT: Yes it does. See next post.)

For the discussion on Stardot that prompted his post, see https://stardot.org.uk/forums/viewtopic.php?f=29&t=23021

I’ve worked out why I was getting different results on the Pi: I had retyped the program rather than copy it from machine to machine, and on the Pi I had used “NEXT i%” instead of “NEXT”. Change the line to “NEXT” and it fails on the third pass.

That merely prompts a follow-up question: why does “NEXT i%” fail in this case?

The failure when i%=3 is after…
LOCAL a%() :defines the variable, but undimensioned.
FOR i% = 1 :creates a FOR entry on the stack
DIM a%(10) :dimensions the array, and allocates storage for it in the FOR entry on the BASIC stack.
NEXT :reverts the stack back to the initial FOR.
FOR i% = 2 :the a%() has vanished from the stack
DIM a%(10) :dimensions the array, and allocates storage for it NOT on the stack.
NEXT :reverts the stack back to the initial FOR.
FOR i% = 3
DIM a%(10) :tries to dimension the array, but fails because it is already dimensioned.

Which is just as Richard suggested in his initial Stardot post.

If the DIM is moved to just before the FOR, it runs without errors, which seems a more logical place to put it.

The failure when i%=1 and NEXT i% is used is because the NEXT cannot find the expected FOR entry on the stack as the array gets in the way.

As tested using
RISC OS 5.29 (11 Jul 2021) Titanium
BBC BASIC V 1.84 (23 Jun 2021)

It is possible to cause a one-off memory leak in slightly less contrived circumstances.

PROClocal
PRINT a%(1)
END

DEF PROClocal
LOCAL a%()
PROCsetup
PROCsetup
ENDPROC

DEF PROCsetup
DIM a%(1000)
ENDPROC

This will fail at the PRINT a%(1) line, illustrating that the array is no longer accessible. Inspecting the value of END at various points in the program will show that the BASIC heap increases substantially during the second call to PROCsetup, and the memory is not released when PROClocal exits.

The memory leak is one-off (you only lose more memory if you create another array in the same fashion) so this bug may fall into the “not worth fixing” category.

Tested using BASIC 1.81 on RISC OS 5.28 (Raspberry Pi 4).

The memory leak is one-off (you only lose more memory if you create another array in the same fashion) so this bug may fall into the “not worth fixing” category.

I think this is an equivalent memory leak to:

PROClocal
PRINT a%
END

DEF PROClocal
LOCAL a%
PROCsetup
PROCsetup
ENDPROC

DEF PROCsetup
DIM a% 1000
ENDPROC

where the pointers to the dimmed blocks are lost.

This is more obvious, and I think it comes into the category of a programmer’s error.

BASIC heap increases substantially during the second call to PROCsetup

Because the LOCAL version of the array from the first call of PROCsetup (which was on the stack) was removed when the stack was reduced at the end of PROCsetup.

the memory is not released when PROClocal exits.

Because the second PROCsetup allocated a normal array (not local) so went in the heap, and is there for evermore.

It just serves to show that re-allocating arrays has to be done with care, and I agree with Steve that it comes into the category of a programmer’s error.

Memory or Arrays allocated to LOCAL veriables vanishes at the end of the PROC they are allocated in when the stack is reduced.

it comes into the category of a programmer’s error

Sure, but BASIC isn’t exactly helping here, is it? It’s obvious in the stated examples what is going wrong, but in a real-life situation the programmer could spend a lot of time debugging before discovering that the array had been dimensioned twice.

At the very least we need a modification to the BBC BASIC Reference Manual. Perhaps an admonition of the form “The corresponding DIM statement must appear in the same procedure/function as the LOCAL statement, and there should be as few intervening structures between the two statements as possible”.

Memory or Arrays allocated to LOCAL veriables vanishes at the end of the PROC they are allocated in when the stack is reduced.

I had a look with *ReportTrace/Mem and saw this happening, but I still don’t understand why. I can see that the allocation from DIM in PROCsetup is let go from the stack at ENDPROC (setup), but I would have thought there’s something still on the stack frame for PROClocal ‘representing’ the LOCAL a%(). If we change it to

PROClocal
  LOCAL a%()
  PROCsetup
  DIM a%(1000)
ENDPROC

I’d’ve expected it to say array cannot be redimensioned, rather than heap allocating the DIM. How is it that PROCsetup’s ENDPROC causes the PROClocal stack frame to forget about the LOCAL a%()?

That merely prompts a follow-up question: why does “NEXT i%” fail in this case?

I am not sure where I got this from, but my StrongHelp BASIC manual has had this for a couple of decades:

“However, if this is done in the FOR structure, the NEXT keyword cannot then be followed by a loop variable”

So it has been know about for a long time. ;-)

Martin explains it.

At the very least we need a modification to the BBC BASIC Reference Manual.

The manual says: “When you declare a local arrary, the LOCAL statement must be followed by a DIM statement to dimension the local array.”

Is that not sufficient. Perhaps emphasis on “must be followed” is required.
A reading of Arrays passed by reference" is useful, too.

The scope of normal local variables is any routine deeper than the LOCAL statement. However, arrays are referenced and really do need to be passed as parameters. Do that in the example and all is well. Is this clear enough in the BASIC manual? Probably not.

If this case is intended, rather that a bug, then I agree that the BASIC manual needs clarification about the restriction.

How is it that PROCsetup’s ENDPROC causes the PROClocal stack frame to forget about the LOCAL a%()?

Because a%() is defined as LOCAL in PROClocal, so is contained in the PROClocal stack. The array a%() is DIMmed in PROCsetup, so is contained in the PROCsetup stack. So at the PROCsetup ENDPROC, the stack containing the array is discarded, but a%() is still defined as LOCAL – although it appears now unusable for a LOCAL array to be defined.

My theory is…
After the LOCAL a%(), the ‘l-value’ of a%() is 1.
After the DIM a%(), the ‘l-value’ of a%() is a pointer to the array data, in the stack (temporary).
After PROCsetup, the ‘l-value’ of a%() is 0 (zero).
After the second DIM a%(), the ‘l-value’ of a%() is a pointer to the array data, in the heap (permanent), which seems wrong.

If after the first PROCsetup, if the ‘l-value’ of a%() is set back to 1 (with great care!), the second DIM a%() allocates the array data on the stack, which is what I would expect.

I have so far been unable to find where in the source the ‘l-value’ of a%() is reset on ENDPROC, or why there is a difference between one and zero (most checks seem to be <16).

I’d’ve expected it to say array cannot be redimensioned

I think I would, too. What documentation there is about how arrays work is under CALL, which is perhaps not the most obvious place to look. I have managed to get some strange problems like this in other circumstances, when re-using an array variable, eg a%(), at different points in a program. They probably have the same cause. The problems go away when arrays are passed as parameters.

Martin is doing a sterling job tracking this down, so I will leave it to him. ;-)

most checks seem to be <16

That is what I would expect, as it is documented. I wonder if there is another sanity check that the array pointer is in the heap, ie <FSA, and if not, it is also treated as undimensioned. This would be so for a local array that has been released at ENDPROC.

The problems go away when arrays are passed as parameters.

But this doesn’t cover every use case.

It’s perfectly reasonable to have a LOCAL array that can be different sizes on each call to the function or procedure in which it exists, for example. Which causes no problems at all as long as the array disappears when the function or procedure returns.

It’s perfectly reasonable to have a LOCAL array that can be different sizes on each call to the function or procedure in which it exists

Of course, but that says nothing about using such an array at deeper levels than where it was made local.

Of course, but that says nothing about using such an array at deeper levels than where it was made local.

Using them at deeper levels is no problem, as long as you don’t go and redimension one you’ve already dimensioned at whatever level. Even dimensioning them at lower levels is okay, as long as you only do it once. What I’m failing to understand is why it’s not a bug if BASIC fails to tell you you’re trying to redimension one.

I think I have found the cause of some of these issues, and I may have a fix.

Here is a summary of my understanding of the processing involved…

When an array is DIMmed, array storage is allocated for a list of sizes of dimensions, terminated by 0, followed by the total number of elements, then the values of each element. This storage is normally in the Heap, but for LOCAL arrays will be in the Stack.

The array name and it’s l-value are allocated on the Heap, and the l-value is the address of the array storage.

If an array variable is made LOCAL in a PROC (or FN), the previous l-value is stored (for restore when the PROC ends), and the l-value is set to one, indicating undimensioned.

When an array is DIMmed and the variable name already exists, it is an error – unless it is a Local array. If l-value is one (undimensioned) then array storage is allocated on the Stack, the l-value set to point to it, and the array is added to a list of LOCAL arrays.
New arrays are allocated on the Heap.

When the PROC ends, stack entries for the PROC are discarded, including the LOCAL array data. When this happens, the list of Local arrays is checked, and any discarded have their l-values reset to “undim it” … but it is set to zero not one.

Thus, if the array is DIMmed again while the variable is still LOCAL, it is allocated on the Heap and NOT the stack (because the l-value is zero not one). This I believe is a bug.

That may appear to ‘work’, but is a memory leak, and if the array is DIMmed again while still LOCAL it will fail because the (global) array already exists on the Heap.

This explains the error seen in the program in the first post in this thread.

My possible fix is to set the l-value back to one when an array is lost from the stack, and the original test case runs without errors.

However, nothing in the BASIC code is simple and straightforward (to me, anyway). There are other places which change an array l-value, sometimes to zero, and I am unsure what other effects there may be. More checking is required.