Space used on disc

I have to create a directory and create 85,114 files into it, all small, of an average size of about 300 bytes (the output from InterGIF).

First attempt was on my ARMX6 with 9 Gbytes free on my 112 Gbyte SATA drive. After 31,538 files (9,196,268 bytes) had been created it was full. I tried the RAM disc (128Mbyte) and that filled up after just over 7000 files had been created. Is there a formula to identify the likely space usage of small files on a disc? Something like ((name length DIV 2^n)+1)*2^n + ((file length DIV 2^m)+1)*2^m where n and m can be determined from a call to FileCore?

It seems to indicate that the minimum block size on the 112Gbyte drive is 256k (effectively multiplying the file sizes of these small files by 1000) and so to create 85000 files occupying about 30Mbytes would require 30 Gbytes free. The RAM disc is a little better (block size 16k) only needing just over 1Gbyte to create 30Mbytes of these small files.

I also had a strange error – broken directory – while trying to create these files in the same directory as the source sprites (85,114 files occupying 1,522,814,427 bytes). I fixed it with DiscKnight, which now reports the disc is good but the directory is still unopenable reporting ‘broken directory’ (which must be wasting tens of Gbytes as it filled up with small files which I can’t now delete).

Would be nice to be able to see ‘space occupied on disc’ in a count operation and as an extra parameter in filing system calls so that I can find where all my storage space has gone! I have a 112 Gbyte disc containing 261,161 files totalling 40,714,089,533 bytes and reporting only 9 Gbytes free. Even allowing for a few tens of Gbytes in the broken directory I’d like to be able to find which directory of small files to delete!

There once was a cool utility called “RealCount” by Sergio Monesi, but not only is it not 32bit compatible, it is also not E+ compatible.

I have to create a directory and create 85,114 files into it, all small, of an average size of about 300 bytes (the output from InterGIF).

First attempt was …

Is there any mileage in creating these small files to an image filing system such as an ArcFS archive with no compression?

Would that get round the block size limitation?

Yes, I did think of that but speed is an issue so I didn’t try that. A SparkFS archive on the RAM disk is worth a try though.

85,000 files in one directory may well not be possible or be unreliable, but could also be very slow to access. We have one directory that has 1,700 file 80MB total. It is slow to access locally and over a network very slow.
n.b. There is one oddity. If over the network you select the 1,700 file directory and do a count it is reasonable quick. If you go into the directory and select all and then do a count it is about 100 times slower!
Doing the count locally also has a time penalty for ‘selected all files’ over counting the directory, but it doesn’t seem as significant.

That is normal. It isn’t possible to sensibly map every sector of drives that size, so the drive is split into larger logical sections. Tiny files in a large drive = much wastage.

Can you find a 256MiB USB key and format that? Smaller size should have smaller allocation unit…

https://en.m.wikipedia.org/wiki/Data_cluster

If I open the directory locally, it is slow to respond (a minute or two) but once it is displaying I can do a Select All/count and it is quick. If I don’t open it, a count is extremely quick. If I copy a file into it with it open it is alarmingly slow (it stops after every file to work out the contents afresh) but if I copy a file into it while it is closed, it is fairly quick. I have 118000 files in one directory but that is on a CD where the block size is more helpful.

Anyway the SparkFS zip archive (no compression) on the RAM disc seems to have no problem with speed – 12% complete after 10mins converting 85000 sprites into gifs – so we’ll see how it gets on with block size. Many thanks for the suggestion.

Note: didn’t work – not much improvement over RAMfs with no iamge filing system.

A more RISC OS specific explanation: http://www.chiark.greenend.org.uk/~theom/riscos/docs/ultimate/a252efmt.txt

Very full explanation, thanks. I have finally solved the problem of getting the job to work in sensible time (67 mins rather than 354 mins on an ARMX6 to create the 84,738 sprites and then 36 mins rather than about 20 hours to convert them to gifs using !InterGIF) by using Virtual Risc PC on a fast PC with an SSD hard disc.