HD, RAM, etc

Having been experimenting with recording audio using my ARMiniX and Iyonix prompts me to ask about some of the current ‘limits’ on file storage, etc. Note I’m specifically interested in ‘Filecore’ format storage here. Not something like Fat using Fat32Fs.

Firstly, what is the current situation (e.g. for RO 5.21) wrt the sizes of hard discs that can reasonably be used? I’m thinking here of the reality that conventional HDs as small as 200GB are getting rare and the norm is in the TB range these days. Can we use discs bigger than around, say, 250GB OK? What’s the practical limit for reliable storage and use? Note I’m thinking here of storing large files. So I expect typical files to be 100’s of MB each. Not lots of relatively small files.

Secondly, what’s the situation now wrt the largest file size that can be used? I’ve been assuming that for safety I need to keep to below 1 or at most 2 GB per file, which is actually fine. But curious to know the state of play.

Thirdly, can something be done about the size limit on RAM: disc? At present I can’t easily set more than 128MB and I think from earlier postings there is a higher limit caused by other problems. This is a shame on modern machine that might have best part of a GB of ‘free’ ram.

Extending the above into an area I know nothing at all about… :-)

Herebelow I’m thinking of a ‘remote’ net-accessed device which would be some non-filecore format like ext4, say.

Fourthly, what limits of the above kinds are there on using something like a NAS or net-accessed storage from a modern RO system? Wondering here of the possibility of recording directly to – or storing on – sets of large files as above. Is it OK to store / access files that are, say, about 1GB each on a NAS that has a storage capacity in the TB range? Or are there problems with this using something like an ARMiniX/PandaRO running 5.12?

Thanks,

Jim

Using network attached storage has the disadvantage of loading up the USB bus. Using SD card storage uses the SD card interface and leaves the USB bus unchanged. With a fast SD card you can get 20Mbytes/sec.

Using network attached storage has the disadvantage of loading up the USB bus. Using SD card storage uses the SD card interface and leaves the USB bus unchanged. With a fast SD card you can get 20Mbytes/sec.

Alas, in practice it isn’t that simple. Recording 96k/24 stereo means saving over half a megabyte per second. That sounds easy given quoted speeds like 20MB/sec. But I’ve been doing this and observing the timing when I save a chunk once per second every second for some mins. The reality is that – even singletasking (no wimp polling, etc) – the delays for some saves can be tens of centiseconds whilst most just take a few centisec. The problem, I think, is that every now and then a card or reader does some ‘re-arranging the deckchairs’. So you need a large buffer (1 sec for my tests) to reduce the risk that even a rate as ‘low’ as 0.5MB/sec won’t have problems.

And I’m wary of regularly writing large amounts of data to an internal SD card that is also my main ‘HD’ because of the added ‘wear’ and risk of ‘Sudden Death’. Seems much safer to write to some other device. Note that 200 MB is only just over 5 mins at 96k/24. Given this, USB seems unavoidable for modern hardware.

BTW I’m avoiding Fat here because Fat32Fs seems to do a ‘flush’ every now and then, and that cripples the above as it causes a loss of data because it takes so long that even a 1 sec buffer will often over-run. Shame as it would be convenient in other ways. Maybe I should ask Jeff D. about this problem…

Jim

Been doing some more tests here. I’ve experimented with using ‘create’ to produce a blank file of the max required size, then trim it back after the recording ends. This helps Fat32Fs a bit for an external CF card in an adaptor. But not enough to allow reliable 96k/24 stereo recording on my ARMiniX. And if anything the change makes recording to some Filecore devices (USB SD card in a reader and the main SD HD) slightly more prone to ‘occasional delays’ that mean buffer over-runs.

48k/24 stereo seems OK for both, either using the create-trim approach or simply building the file as you go as usual. So I guess the problem here is that some kind of internal ‘deckchair shuffling’ in the hardware takes too long every now and then.

I’m getting a USB SSD Filecore format, so when that comes I’ll compare it with an external USB ‘spinning rust’ HD. But this keeps making me feel that being able to save lots to RAM may be the best idea. The problem, of course, being that has size limits which are smaller than something like a typical SD card, etc.

Jim

I should point out that many hard disc recording systems record to a multitude of internal files (it makes editing easier) rather than one long one.

I should point out that many hard disc recording systems record to a multitude of internal files (it makes editing easier) rather than one long one.

Yes. I’m planning to impliment that at some stage. However the dedicated recorders I’ve used/use all do this to files with sizes typically in the range from hundreds of MB to a couple of GB per file. e.g. my Tascam HD-P2 lets the user chose a value in this range on a ‘per project’ basis. However the delays I’ve been getting show up with files a small as around 10 – 20 MB. That is a very short recording at 96k/24, so not a very practical split size for the purpose.

Been doing more tests today. Jeff D. suggested I use ‘create’ first to produce blank files of the default max length, then write to them and trim them back (OS_Args 3) when done if their duration means their content is shorter.

That helps Fat32Fs. But regardless, I still get occasional delays with SD / CF cards, be they accessed via SDFS, SCSI, or fat32fs. The delays are the order of tens of secs to a min apart and are long enough that for 96k24 in 1 sec ‘chunks’ the buffer over-runs on some occasions.

i.e. usually each save process and save only takes a few centiseconds. But every now and then I get a much bigger delay – up to about a second or more when trying to save about 0.5 MB. The details vary unpredictably from run to run. Even if I delete the previous file recorded and repeat the process. All this ‘single tasking’ with no wimp apps running.

Afraid I don’t know why these delays occur, but I have the impression they are something about the devices in question.

If I record to ramdisc there is no delay worth mentioning, and these occasional ‘long delays’ don’t occur. But with a limit of 128MB on ramdisc I can’t record there for very long. Ideally need a device with some GB of space to handle long 96k/24 recordings. (FWIW With my HD-P2 I use 8GB CF cards.)

I’ve been wondering about ‘disc’ bufferring. Using *status on my ARMiniX doesn’t list buffer sizes for any of ADFS/SDFS/SCSIFS, although it shows a SCSIFSDirCache size. !Boot → discs shows ADFS buffer size. Which I guess refers to SDFS. But again I’ve no idea if altering that would help at all. Long time since I’ve done anything in that area. Not since using SCSI devices on my olde RiscPCs mumble years ago.

What is causing the delays? They aren’t as bad for 48k/24 recordings so they seem to be affected by the amount of data being shovelled in each 1-sec-worth of audio. Again my guess is that the SD/CF cards want to do some kind of internal ‘reshuffle and tidy up’ every now and then. But the CF cards work fine in my HD-P2 (which records 192k/24 fine to them, no problems) and in video cameras, etc if they are the better types.

Jim

Just to add one point.

I’ve just re-tried using a conventional HD I use. Spinning rust formatted Filecore 160GB IDE with an adaptor in a USB box with its own PSU. I only rarely use this for tests of this kind as it is one of my main data/file backups. But decided to use it for one more test.

I made a 96k/24 recording for 5 mins. The time offset value thoughout was typically 6 or 7 cs per save 1-sec grab/process/save cycle. The max offset for any save during the time was 8 cs. i.e. near perfect. No sign of any occasional ‘long delays’ of the kinds I get with the SD cards via any interface, etc.

So this does look like a problem due to the way these cards operate, I guess. Why they are OK with Linux, I dunno. I presume that simply has better bufferring and caching arrangements than Filecore?

I’ve bought an SSD ‘HD’ with USB adaptor. When that comes I’ll compare it with these results.

Jim

This is a reminder that a multicore CPU could have one core dedicated to filing systems and take the load almost completely away from the application core.

This is a reminder that a multicore CPU could have one core dedicated to filing systems and take the load almost completely away from the application core.

I suspect it’s more the lack of buffering within the SD file system and the fact it doesn’t do background transfers that’s not helping Jim here. Mind you, I’ve no idea how he’s writing the data; in what size lumps is he doing this and what type of RAM buffering strategy is he implementing within his own code? Also, is he using C’s file operations (which tend to have buffering to some extent with the CLibrary) or going directly via SWI calls?

Writing to SD cards is best done in big lumps because of the way the underlying flash memory works; it always has to erase relatively large blocks and write the whole block, even if you only wanted to write a single byte…

FWIW I’m currently simply using fwrite() for each chunk. I’ve tried other methodsm, and also using *create first, but these changes don’t seem to affect the appearance of the occasional ‘long delays’.

In contrast I’ve now tried an SSD (sata) with a USB adaptor, Filecore formatted. This works fine with negigible delays. Similar to a spinning rust HD (ide) with its adaptor.

e.g. Made a 16 min 96k/24 recording to the SSD. The max delay during that was 7 centisec. No sign of any occasional ‘larger delays’.

AIUI now SDFS lacks some of the buffering, etc, that older ADFS provided. But I think the real problem with the SD cards is in what you suggest. Something about the way they internally work. So without a lot of external bufferring, I guess the delays will show.

Whatever, since the program works fine with SSD and HD I’ve put a new version up for people to try out if they wish. cf c.s.a.a. and the USB audio thread here for more info… :-)

Jim

Whilst it isn’t in any roadmap I understand that adding Cacheing/buffering to SDFS & SCSIFS is planned, but it isn’t going to happen any time soon. Maybe a suitable project for a bounty?

FWIW I’ve just been testing playback from Fat format SD cards over USB using Fat32Fs. This is very fast for files that have been written unfragmented. No sign of the ‘occasional delays’ that cripple recording to the cards. And much faster than FileCore format cards of the same type, etc. Alas, I find that Fat32fs does crash at times for such USB devices.

I’ve now also experimented with memphis for this and it is fast, as you’d expect.

Jim

Alas, I find that Fat32fs does crash at times for such USB devices.

That’s a shame. I’ll definitely keep looking to see whether there’s anything I can do to reduce the issues. Just don’t hold you breath ‘cos I’m at a bit of a loss at the moment.

Jeff

@Jeff

I use Fat32fs by default on four machines – Iyo, BB, PB, RPi – and on the last three the main storage is 120 GB SSDs in usb enclosures. I cannot say I have had any real problems with Fat32fs. What niggles I do get occasionally might very well be due to the usb stack, or powered hub inadequacies.

When the SSD is Fat32 formatted it is much quicker than when filecore formatted, I would guess a factor of at least three.

I suspect that it something ‘odd’ about my ARMiniX that may make Fat32Fs prone to crash. Discussed this with Jeff and others before. Alas, I’m still no nearer to finding out the reason! :-/ Just means I can only use it sparingly and with fingers crossed.

Fat32Fs works fine on my Iyonix. Was using it this morning to copy files from some Fat SD cards to my FileCore SSD.

WRT SSD (FileCore) I’m puzzled by that too at times. Found another puzzle today. Note this us using it via USB (‘SCSI’) and an externally powered USB hub and a USB-SATA adaptor.

If I have the SSD plugged in to my USB before I load the USB Audio modules it works for playback/recording. But the time delays for playback are much bigger than if I connect it after I’ve installed the modules.

If I connect it after module installation I get about a 75 cs time for each 1 sec block loaded for 96k/24 data. This is slow, but doesn’t impede replay.

If I connect it before module installation I get about double the delay. 150 cs for each sec of data! Cripples replay by causing many pauses.

Why these long delays simply for reading data? And why does the value double like that if the SSD is connected beforehand? It means I can’t leave the SSD connected all the time. Delays of the order of 1 sec to load just about a quarter of a MB of data seem rather large.

Could this all be something odd about the USB controllers?

Jim

As you were… :-)

I think I’ve found a ‘cause’ at one level for the ‘double the delay’ I reported above!

It occurred to me that I was using a 4-way externally powered USB hub connected to one of the rear USB sockets on my ARMiniX. I know these are problematic as the machine won’t boot up if my keyboard is plugged into them – even via an externally powered hub.

So I’ve just tried having the SSD connected via a 4-way hub into one of the front USB sockets.

Now it doesn’t matter if the SSD is connected before or after I load the USB audio modules. The delays are the same. The delay is still 75 cs for each 1 sec chunk of 96k/24 audio which seems weirdly long. But works OK and doesn’t jump to well over 100 simply because the SSD was connected from bootup.

This increases my feeling that various problems I get are due to something not quite OK about the rear USB sockets on my ARMiniX. Which are, of course, on the PandaBoard!

In effect, I’m approaching the point where I can’t use the rear sockets for various purposes.

When I get a chance I’ll test using Fat32Fs using only the front sockets. Maybe it will then be reliable… Afraid that at present I’m having to do other things as priority so only doing ARMiniX audio tests, etc, when I can.

Jim