Filecore bug

If I use the programme below:

   10REM >FCTest
   20REM Generate 1-byte files
   30FOR i%=0 TO 255
   40  P=OPENOUT("OG"+RIGHT$("000"+STR$(i%),3)+"/gif")
   50  BPUT#P,i%
   60  CLOSE#P
   70NEXT
   80:
   90DIM done%(855,455)
  100done%()=0
  110i%=0
  120REPEAT
  130  j%=RND(855)
  140  k%=RND(455)
  150  IF done%(j%,k%)=0 THEN
  160    OSCLI("Copy OG"+RIGHT$("000"+STR$(RND(255)),3)+"/gif 
o1.OverGrid"+RIGHT$("000"+STR$(j%),3)+RIGHT$("000"+STR$(k%),3)+"/gif 
~c~v")
  170    i%+=1
  180    PRINT "Copied ";i%;" files             ";:VDU 13
  190    done%(j%,k%)=TRUE
  200  ENDIF
  210UNTIL0
  220END

then after 3hr30min on my ARMX6 (with FileCore 3.67 (2-Oct-2015)) some 80619 1-byte 18-character filename files have been copied, in a non-optimum order, and I get ‘broken directory’ – the remainder of the disc is good and DiscKnight cannot fix the problem. It needs a reformat to solve it. Bug reported by e-mail. This error is repeatable and I now have two such broken directories on this disc. The error does not necessarily occur at exactly 80619 files copied.

I suspect the problem has gone unnoticed because in a filecore to filecore copy operation, the files are presented in alphabetical order, so that no directory entry shuffling is required. The filer uses a non optimum shell sort (when an insertion sort would be much more efficient) after each and every individual file copy operation. The ArmSort module can sort 100000 strings in about 0.4 seconds on an ARMX6 using a flash sort. In a NAS to filecore copy they are produced in a seemingly random, but repeatable order during a copy.

Quoting from an earlier post by Jeffrey: ‘marutan.net has copies of 1309,207/FS and 1309,208/FS’ and the latter says the filename limit for 19 character filenames is:

For names of up to 19 characters, we can have up to 87380 directory entries

Different errors occur if you try to write similar files to a SparkFS archive.

RAMfs (which would be faster and not generate a permanent disc fault) only allows 11310 1-byte 18-character filename files (making 11567 files in total on a 125Mbyte RAM disc). Increasing the RAM disc to 128 Mbytes and only 7951 files could be copied, fewer files because the granularity (?) LFAU (?) worsens with increasing disc size. Hence the error cannot be demonstrated using a RAM disc. Writing to an SD card or to a pen drive is too slow for my patience.

and DiscKnight cannot fix the problem.

Whoa. Is the directory completely scrambled then? I’m surprised DK doesn’t have an option to simply hack the offending directory out of the map and try to recreate the fragments left behind as lost files.

Perhaps the lesson to take away from this is that 77 files was a rather (arbitrarily?) low directory limitation, but 80K is… Uh… A startling number of files. And RISC OS managed 15K more files than FAT32 can store in a single directory. ;-)

There is a hard fix and an easy “fix”. The easy “fix” is to clamp at 65K entries. Harder is to delve into the innards of FileCore and see what is happening. What I find strange is that it takes 80K files to trash things. Why not 60K or 15K? Looking at the older (E/F format) directory information, it seems a fairly simple structure. Your files have 18 character names. 19 with terminator? Assume rounded to 20 to word align. There follows four words of information. Let’s assume five words in case the disc address is 64 bit. That means we ought to get by with 40 bytes per entry (depending on how long filenames are actually stored, I may be well off the mark!). For 80619 files, 1612400 bytes, or 1.53MiB. There doesn’t seem anything special about that number to me, so my calculations are probably way off. While you say it is not necessarily 80K files, I am guessing it is around that amount. So the first thing I’d ask is “what changes at that point?”. Does the directory entry stray over some sort of structural boundary?
Have you tried 36 character filenames? It might be interesting to know if this failed around the 80K count, or at ~40K. It might also be worth keeping an eye on disc free space while making the tests, to see how space is being consumed for the directory entries.

tl;dr: I know nothing about FileCore that isn’t written in PRM2 (!) but I find this intriguing…

Whoa. Is the directory completely scrambled then?

Who knows? It cannot be opened, giving the error ‘directory broken’. DiscKnight says disc is good. You can rename the directory but it gives an error and leaves the directory name ambiguous, DiscKnight can restore the directory name to the original name again but still broken.

The directory buffer is the place where FileCore keeps the directory currently being worked on
when modifying it and can be up to 4096k in length. The directory info contains file attributes for each file (load, exec addresses etc. occupying 28 bytes) in the same order as the names, immediately followed by each leafname plus terminator, [padded to a word boundary if necessary], in alphabetical order.

to see how space is being consumed for the directory entries

The disc is 250Gbyte in size so approx 256k to 512k per 1-byte-long file so that 80,000 files would use up 20Gbytes to 40Gbytes.

The disc is 250Gbyte in size so approx 256k to 512k per 1-byte-long file so that 80,000 files would use up 20Gbytes to 40Gbytes.

Just out of interest, what’s your total used space? Could your broken directory be crossing the 128 GB mark?

The first broken directory may have crossed the boundary (but I think it did not) but the second did, leaving 75Gbytes free.

I have repeated the test on an almost empty 250Gbyte disc and after 80619 files have been copied, I get ‘broken directory’ again. QED.