Space used on disc
Chris Hall (132) 3554 posts |
I have to create a directory and create 85,114 files into it, all small, of an average size of about 300 bytes (the output from InterGIF). First attempt was on my ARMX6 with 9 Gbytes free on my 112 Gbyte SATA drive. After 31,538 files (9,196,268 bytes) had been created it was full. I tried the RAM disc (128Mbyte) and that filled up after just over 7000 files had been created. Is there a formula to identify the likely space usage of small files on a disc? Something like ((name length DIV 2^n)+1)*2^n + ((file length DIV 2^m)+1)*2^m where n and m can be determined from a call to FileCore? It seems to indicate that the minimum block size on the 112Gbyte drive is 256k (effectively multiplying the file sizes of these small files by 1000) and so to create 85000 files occupying about 30Mbytes would require 30 Gbytes free. The RAM disc is a little better (block size 16k) only needing just over 1Gbyte to create 30Mbytes of these small files. I also had a strange error – broken directory – while trying to create these files in the same directory as the source sprites (85,114 files occupying 1,522,814,427 bytes). I fixed it with DiscKnight, which now reports the disc is good but the directory is still unopenable reporting ‘broken directory’ (which must be wasting tens of Gbytes as it filled up with small files which I can’t now delete). Would be nice to be able to see ‘space occupied on disc’ in a count operation and as an extra parameter in filing system calls so that I can find where all my storage space has gone! I have a 112 Gbyte disc containing 261,161 files totalling 40,714,089,533 bytes and reporting only 9 Gbytes free. Even allowing for a few tens of Gbytes in the broken directory I’d like to be able to find which directory of small files to delete! |
Steffen Huber (91) 1953 posts |
There once was a cool utility called “RealCount” by Sergio Monesi, but not only is it not 32bit compatible, it is also not E+ compatible. |
John Williams (567) 768 posts |
Is there any mileage in creating these small files to an image filing system such as an ArcFS archive with no compression? Would that get round the block size limitation? |
Chris Hall (132) 3554 posts |
Yes, I did think of that but speed is an issue so I didn’t try that. A SparkFS archive on the RAM disk is worth a try though. |
Chris Evans (457) 1614 posts |
85,000 files in one directory may well not be possible or be unreliable, but could also be very slow to access. We have one directory that has 1,700 file 80MB total. It is slow to access locally and over a network very slow. |
Rick Murray (539) 13840 posts |
That is normal. It isn’t possible to sensibly map every sector of drives that size, so the drive is split into larger logical sections. Tiny files in a large drive = much wastage. Can you find a 256MiB USB key and format that? Smaller size should have smaller allocation unit… |
Chris Hall (132) 3554 posts |
If I open the directory locally, it is slow to respond (a minute or two) but once it is displaying I can do a Select All/count and it is quick. If I don’t open it, a count is extremely quick. If I copy a file into it with it open it is alarmingly slow (it stops after every file to work out the contents afresh) but if I copy a file into it while it is closed, it is fairly quick. I have 118000 files in one directory but that is on a CD where the block size is more helpful. Anyway the SparkFS zip archive (no compression) on the RAM disc seems to have no problem with speed – 12% complete after 10mins converting 85000 sprites into gifs – so we’ll see how it gets on with block size. Many thanks for the suggestion. Note: didn’t work – not much improvement over RAMfs with no iamge filing system. |
Rick Murray (539) 13840 posts |
A more RISC OS specific explanation: http://www.chiark.greenend.org.uk/~theom/riscos/docs/ultimate/a252efmt.txt |
Chris Hall (132) 3554 posts |
Very full explanation, thanks. I have finally solved the problem of getting the job to work in sensible time (67 mins rather than 354 mins on an ARMX6 to create the 84,738 sprites and then 36 mins rather than about 20 hours to convert them to gifs using !InterGIF) by using Virtual Risc PC on a fast PC with an SSD hard disc. |