Hard drives on Iyonix

As I understand it, the Iyonix has a hardware limit of 128GB for fast hard drive access, and currently a software limit of 256GB (which might change).

Is this correct?

What are the implications of using a disks between 128 and 256GB?

(Obviously any access of the drive past 128GB will be slow, but where does ADFS place the data? ie would a 160GB drive start to perform badly once it has 128GB of data, or would it be much sooner?)

Presumably once we have partition support, then it will be simple, have a slow partition and a fast one.

(if partition support is added before the 256GB limit is fixed, would the limit be per partition, or apply to the drive as a whole?).

Yes, beyond 2^28 sectors = 128 GiB, it is necessary to use LBA48 ATA commands to access the data, and there’s a bug in the ATA controller in the south bridge chip used in the Iyonix that means UDMA doesn’t work for those commands. ADFS drops down to using LBA28 commands to access earlier parts of a disc, and these are accelerated properly.

The other limit comes from the FileCore / ADFS SectorDiscOp API where a 32-bit word is split between 3 bits (for drive number) and 29 bits for sector offset within the drive. A replacement API has been designed which not only increases the available sector offsets but also the number of drives per filing system. However, internally both FileCore and ADFS still use 32-bit numbers, so you’re still limited in practice.

FileCore doesn’t make any special allowances for part of a disc being slower to access than another – this would meant complicated changes to allocation strategies for the sake of one hardware peculiarity which is relatively transient compared to FileCore itself. However, I think you’ll find you start hitting slow accesses earlier than you expect and certainly before you have 128 GiB of data on the disc – this is because the disc map and root directory are stored in the zone halfway through the disc – a deliberate choice to reduce the average head seek time.

To keep data transfer times down, I’d recommend formatting large discs to 128 GiB and “wasting” the remainder. If and when partition support is added, any later partitions will be slow to access – partitions are a higher level concept than the ATA command set, which deals in sector offsets into the physical disc.

Thanks for the explanation.

From what you say I work out that my 160GB drive (formatted when the machine was new) will have started to slow down by the time it contains 96GB. (assuming it fills up evenly either side of the center at 80GB, 48 GB will fit above and the same below).

I think the only logical solution would be partitions and a default option on the partition utility that prevents partitions crossing the boundary.

I don’t see any great advantage in extending the 256GB limit (for ADFS partitions) while there is a 2GB limit on file size. Presumably fixing both of those would be a huge job, (and would there be inefficiencies in storage that woud need to be dealt with too?) so would a port of an existing filesystem be more sensible, (when the time comes)?

They’re actually quite separate problems. ADFS doesn’t know anything about files, it only deals in accessing floppies and hard discs as effectively very large byte arrays. It’s FileCore that has the 2 GiB file limit, and that’s probably down to some careless signed/unsigned coding and/or reuse of bit 31 (I’ve never investigated exactly why there’s this problem). More importantly, FileSwitch has no concept of >32-bit file pointers, so just porting another filesystem isn’t going to help. The fundamental OS file API would need extending. There are all sorts of knock-on effects here – for example, what would BASIC’s PTR# do for a 5 GiB file? (Remember that BASIC doesn’t have 64-bit integer types.)

This whole area is something I’d love to tackle if I had the time. The whole filing system part of the codebase is notoriously complicated and requires a lot of care and a solid knowledge of assembler programming to work on – Acorn tended only to trust their very best programmers to be let loose on it! Still, with patience and a methodical eye, I don’t think either problem is insurmountable.

One more subtle change that would be needed would be to make changes to allow the LFAU to stay small – because of the way things work, as discs get bigger, the range of available LFAUs get bigger too, and I suspect that with multi-terabyte drives you’ll find that most of the disc space gets wasted due to the coarseness of allocation.

The fundamental OS file API would need extending.

Would it not be better to replace it? (additionally with the new API being non blocking.)

For example, what would BASIC’s PTR# do for a 5 GiB file? (Remember that BASIC doesn’t have 64-bit integer types.)

Old programs would continue to use the current API, with the current limits.

Filesystems would either be based on the new or current system, and require a translation layer to provide access for programs using the other system.

(Hopefully ‘new api > current system’ would be a module that could be used on other versions of RO, allowing programs to be written using just the new API).

Files that are too big could be treated as a folder containing parts, (as CDVDBurn saves big ISOs).

In your example BASIC would see a folder containing 2 2GB and 1 1GB files.

I’m not a big fan of completely replacing the API. It’s a lazy option for the filesystem developer, because it pushes the burden of supporting two different APIs onto all application authors, and there is a much larger volume of code in applications as a whole than there is in the filesystems. We have a big enough problem as it is with ARM changing fundamentals of the CPU architecture every few years without giving application authors that sort of headache too.

The number of API calls at the FileSwitch HighFSI level that would need new extended versions is relatively limited:

• OS_File 5, 7, 11, 13, 15, 17, 20, 21, 22, 23
• OS_Args 1, 2, 3, 4, 6
• OS_GBPB 1, 3, 10, 11, 12
• OS_FSControl 28, 38 (49 is already 64-bitted as 55)

Many applications only ever do operations that load or save whole blocks of memory to a file, and so won’t actually use these calls and wouldn’t need changing at all (at least until the memory address space gets bigger than 32 bits, which doesn’t bear thinking about!)

Almost by definition, any application that works on a > 4GiB file does so by treating it as a stream – opening it, and repeatedly reading or writing a block of n bytes to or from it, often not actually caring what the file pointer actually is. For this sort of application, it would be a nuisance to handle your split-file solution, especially in cases where you’re writing the file and extend it from just less than 4 GiB to just more than 4 GiB. I’d hate to see us saddled with such a kludge of an API.

(You’ll also notice I keep going on about 4 GiB – remember, 2 GiB is only a current limitation of FileCore, and non-FileCore filesystems such as NFS have been able to handle files between 2 and 4 GiB for many years.)

You also raised the issue of non-blocking calls. I think this is a red herring. For one thing, calls are only likely to take long enough to warrant blocking if they’re doing data read/write operations, not accessing the file pointer. And for another, there’s nothing in the current API that stops you from implementing a non-blocking feature in any filesystem. The one thing I’d suggest would be the use of a standardised “call would block” error number across all filesystems so application code wouldn’t have to care about which filesystem a file resided on in order to determine if an operation has to be tried later. It would have to be an opt-in feature for each application, in order to be compatible with existing software that wouldn’t understand the meaning of the error, but following the example of DeviceFS’s “sleep” option, the obvious way to enable such a feature would be by use of a FileSwitch special field.