No. of files in a directory

Is there an upper limit on the number of files in a directory?

I had a directory of 330000 sprite files and I was running a nested series of Obey files calling up !InterGIF to convert them into GIFs and I got bored with waiting after an hour or so and did a CTRL-Break. ALT-Break would just stop the particular call to !InterGIF and then continue with the next one.

The directory concerned reported ‘broken directory’ after restarting and although !DiscKnight fixed the disc and reported it OK, the directory was still broken.

Should I try to ensure that there is less than half a million files in any one directory? Converting 330000 files with a different directory for the output files seems to be OK.

You may say why do I need so many files in a single directory?

One example is that I have half a dozen draw files showing the railway lines that I have traced in England, Scotland and Wales. I render these and capture 125×125 pixel sprite files showing a 1km square with a transparent background and the railway line traced on it. It takes 97405 non-blank map tiles to cover Great Britain. I batch convert the sprites to GIFs and then concatenate them into a single pair of files and render the requested GIF using a Perl script.

A little more explanation is given here

I render these and capture 125×125 pixel sprite files showing a 1km square with a transparent background and the railway line traced on it. It takes 97405 non-blank map tiles to cover Great Britain.

Model things on the OS mapping. Their electronic versions split things on OS major grid squares and then sub-divide

Not sure I follow. I need to be able to plot individual railway lines knowing the ‘line code’ of each – for example GW001 Paddington to Old Oak Common.

When I am producing the map tiles themselves, I organise them as map tiles as 147.030/jpg so that each sub directory is a vertical slice 1km wide. I can therefore upload them without having a huge number of files in a single directory. In the case of these overlay tiles, the tiles are uploaded in a single data file (plus a file listing the filenames inside it and the start and end address of each) so it’s easier to collect them in a single directory before processing them. If I use !InterGIF on Virtual Risc PC, the large number of files in the directory does not slow it up. Also in RISC OS it is only slow when you ask it to display the contents of the directory on screen. RISC OS can count the the files in the directory quickly.

Are the tiles on an aligned¹ grid?
If so you could then maybe have a directory for each row. I realise you’d have to change the program somewhat.
I think you will find processing will speed up by more than a factor of ten if you can do this.

¹ I realise the OS use a staggered grid which would make things even more difficult.

The tiles are on a grid aligned with the OSGB36 Ordnance Survey National Grid so each grid line is a great circle. I have a directory for each vertical slice (which does speed things up if I am producing map tiles as the end result). However it only seems to slow up markedly if you add files other than in alphabetical order (or have a directory display open as you are adding files?) and I happen to be adding files in alphabetical order as I produce them.

Filecore E+ definitely has a limit, I think it is usually around 80,000. There used to be a detailed explanation about this limit available, but I can’t find it anymore.

However, if this limit is reached, you should get the dreaded “directory full” error, not a broken directory as a result.