Reading lots of files

I know that RISC OS only provides a limited number of file handles. Sensible code closes one file before it opens another. Is there a danger that out-of-order instruction-scheduling on some ARM chips will fail to close files in time? The reason I ask is that Jim Nagel uses some software that I wrote for him some years ago, and he appears to have encountered a strange bug. The software involves reading hundreds of small files. A couple of years ago I changed from using the Norcroft C compiler to GCC. The software still works fine on my Raspberry Pi but Jim has been finding that it does not work on his ArmX6 or his Armini, crashing after 76 files have been read.

I can provide a lot more detail, but I wondered if anybody else had encountered problems involving reading large numbers of small files?

On my various RasPis I’ve copied things like the RO source tree, and GCC around the place many times without issue. Not sure if that’s helpful, but they consist of many small files. So drag and drop copying doesn’t experience the issue.

Is there a danger that out-of-order instruction-scheduling on some ARM chips will fail to close files in time?

Unless your code is reliant on voodoo like Meltdown or Spectre, no. For regular programs an out-of-order CPU will look and behave just the same as an in-order one, i.e. as far as the program is concerned the instructions will appear to execute in order.

Possibly slightly clearer to say that wherever it could make any difference, they actually DO execute in order. I believe that to be true, no?

The reason I ask is that Jim Nagel uses some software that I wrote for him some years ago,

and

crashing after 76 files have been read.

Probably a red herring but exactly how antique is the code? 77 file limit??

reading large numbers of small files?

I think the ultimate test of that would surely be building RISC OS itself? There’s gotta be a couple of thousand files involved in the end…

77 file limit??

I’m not sure why it would work on one modern system and not another, but I’m with Steve here – that’s a very suspicious coincidence…

RISC OS … isn’t appropriate for discarding the other hand represents most shop and banking details of the original spec for companies to understand that you can bake at the end of the original French booklet and the waterproofing membrane to the compiler will see how much more beneficial for any inconvenience farmers are asking about checking multiple cores without stalling over issues such as the manufacturer to be honest is that this method of construction actually got something like that for you.

Sorry. The little bar above the swipe type “keyboard” on my phone provides up to three suggestions based upon the previous word, so I just picked the one in the middle each time. The worrying thing is that it’s a bit like the recent spams, it almost (but not quite) makes sense.

Now back to our scheduled programming … RISC OS supports up to 255 file handles (counting down from 255). Some are held open by DeviceFS, some by FontManager, but this still leaves around 230-240 handles. If an application fails to correctly close them, it would eventually go wrong… But that figure of 76 bugs me too…

Is your program enumerating files? How is memory allocated? My (previous) Manga cache we are directory with over 3,000 files. Clearly a program that allocates “enough for 77 files” will fail… As would a program that allocates dynamically but only assumes the filenames will be ten characters in length.
If it is old code, check these are not assumptions being made within the code…

Though, sadly, dynamically allocating memory for holding file details is rather more painful than it should be thanks to no more arbitrary limits and no way of knowing how much memory is required in advance. I think now we’ll need to be looking at building a linked list on the fly or something.

That said, in my own programs I tend to take the option of faulting or ignoring (as applicable) and path/filename over 236 bytes. If it can’t fit into a Wimp_Message (and there’s a pretty good chance the filer will choke), then such a file will cause all sorts of problems actually being used. Sorry, but until the Wimp is rewritten with shiny new big Wimp messages and UTF-8 and all, the implicit limit on file names is 236 bytes…

.. no way of knowing how much memory is required in advance.

It sounds as if you are in need of a language with automatic memory management ;).

I don’t have access to Jim’s computers, nor have I been able to test the latest RiscLua (6.3, from two years ago) on an Armini or ArmX6. He has sent me jpegs of error dumps, which show me something but not enough to make a diagnosis.

It did 76 of the single scripts (totalling about 46K) but stopped after a few lines of the 77th script. In other words, the last 8 single scripts (totalling about 8.5K) were incomplete or missing.

I still wonder whether the Lua-Convtext glitch smells like some sort of memory-trampling. That would perhaps account for the unpredictability — working the first time but crashing the next, or seeming to work but producing an incomplete output, etc.

No problems with earlier versions.

Rebooted Armini. Now *lua says 5.70.
Doubleclick Gather+clone — it completes with no problem.

This makes me suspect that it has something to do with !SharedLibs, because RiscLua 5.70 was compiled with the Acorn C compiler, but RiscLua 6.3 with GCC. Probably version cross-pollution somewhere.

Maybe the ADFS format of the disc supports only 77 files per directory?

(I guess that is very unlikely, …)

I guess that is very unlikely

Pretty much out of the question – I can’t see R-Comp reintroducing the 77 file limit into the OS build on the ARMx6, however the application code or a linked library might have an obscure bit of crufty old code that is assuming no more than 77 files.
We can guess from the failure happening on an ARMini that it isn’t a deprecated processor instruction like SWP

Does your program call os.execute? I had a lua program that seemed to crash randomly, and I’ve discovered that os.execute causes it.

repeat
  os.execute("Pointer")
until false

I find that the above loop crashes after running for about a minute. On RISC OS 3/4 it takes out the whole system, forcing me to reboot. It doesn’t crash in older versions of RiscLua.

#include <stdlib.h>
int main()
{
    while (1)
        system("pointer");
}

Same loop in C. This also crashes after about a minute if I compile it with GCC.