Enhancing CDFaker to support CD audio

(specifically, the UG_Text file)

Gah! MSCDEX was a memory I was trying to forget… ;-)

A bit of googling for the fabled application notes also turned up this post by Steffen, with the application notes mentioned being available from your local friendly Chris’s Acorns

I’ve used the documents on Chris’s Acorns to correct or clarify some of the SWI’s, although the docs only cover up to CDFS 2.28. From the OS source it appears CD_GetAudioParams / CD_SetAudioParams and possibly a few other SWI’s were added in CDFS 2.31.

Steffen’s post refers to the StrongHelp OS manual, but I’m not sure where to download it, the link provided no longer responds. As it appears there’s no official documentation on the audio parameter SWI’s, I’m going to ignore them on the presumption its unlikely any software will have been written that uses them. RISCOS 3.71 has CDFS 2.29 so 2.31 was possibly either an OS patch or part of the next major OS release, that didn’t see the light of day. RISCOS 3.80 jumps to CDFS 3.02.

Steffen’s post refers to the StrongHelp OS manual, but I’m not sure where to download it, the link provided no longer exists

It looks like riscos.info is down completely – but AFAIK that should still be the place to get it (following the death of the official StrongHelp manuals site a few years ago)

As it appears there’s no official documentation on the audio parameter SWI’s, I’m going to ignore them on the presumption its unlikely any software will have been written that uses them.

I think it should be possible to reconstruct some degree of useful documentation. Mark Watson of EESOX appears to be responsible for adding the definitions to CDFSDriver, implementing support for the calls within CDFSSoftATAPI, and updating CDFSFiler. E.g. CD_GetAudioParms and CD_SetAudioParms are used to control the volume level.

I have audio playback working via SharedSound using both AudioCD and *Play. DigitalCD doesn’t work, but I suspect that’s an issue in DigitalCD.

Could someone please explain the differences between the three SharedSound modes:

1. Immediate – I guess this is identical to Voice handlers and is called in IRQ when a fill is required. In this mode, won’t calling the SharedSounds fill routines be an issue as it relies on R14
2. CallBack – Done via OS_AddCallBack? This mode is next to useless as holding CTRL-SHIFT when scrolling prevents callbacks (on RO3.71, it possibly behaves differently on other OS versions)
3. Process – I couldn’t get any audio with this mode, it’s possibly not supported on RO3.71

I believe I need to use Immediate for the fill code and have a separate OS_CallEvery routine for the CD read-ahead. This will be called in IRQ, which poses a problem with disc access as FileCore won’t like it. What’s the best method for issuing the OS_GBPB? If the CallEvery routine issues a CallBack to the OS_GBPB code and sets R12 to 1 on exit, will the CallBack definitely occur immediately?

The “definitive” doc may help you out a bit. (In particular the Implementation section describes the different modes)

https://www.riscosopen.org/viewer/view/bsd/RiscOS/Sources/Audio/SharedSnd/Docs/Definitive%2Cfaf

Immediate – I guess this is identical to Voice handlers and is called in IRQ when a fill is required. In this mode, won’t calling the SharedSounds fill routines be an issue as it relies on R14

Yes it’s called in IRQ mode – but IRQs are disabled. The SharedSound docs seem to warn against enabling IRQs, but the 3.5 PRMs say you can enable IRQs from within linear handlers, so I’d be tempted to go with what the PRMs say if you’re worried about interrupt latency.

Process – I couldn’t get any audio with this mode, it’s possibly not supported on RO3.71

That wouldn’t surprise me.

I believe I need to use Immediate for the fill code and have a separate OS_CallEvery routine for the CD read-ahead.

I don’t think there’s any need for a separate OS_CallEvery routine. You could probably just register the callback from within the fill code whenever it spots that it’s running out of data.

What’s the best method for issuing the OS_GBPB? If the CallEvery routine issues a CallBack to the OS_GBPB code and sets R12 to 1 on exit, will the CallBack definitely occur immediately?

Yes, you’ll have to do the FS read from within a callback. There’s no way of ensuring it’s triggered immediately, but for the games you’re interested in I doubt they’re doing anything that would delay callbacks by any significant amount. And I’m not sure what you’re talking about with setting R12 to 1 on exit!

I don’t think there’s any need for a separate OS_CallEvery routine. You could probably just register the callback from within the fill code whenever it spots that it’s running out of data.

I tried that, the callbacks weren’t occurring in time. I was hoping RISCOS might force an immediate callback if an IRQ routine exits with R12 set.

There’s no way of ensuring it’s triggered immediately, but for the games you’re interested in I doubt they’re doing anything that would delay callbacks by any significant amount.

I haven’t attempted it under a game yet, as I’ve not managed to get it to reliably read-ahead via callback under the desktop or command line. As I mentioned in my previous post, holding CTRL-SHIFT whilst scrolling prevents callbacks, as does most OS activity. The desktop redrawing for example stops callbacks long enough for SharedSound callback fills to not occur in time. The fill is reliable in Immediate mode, however the read-ahead callbacks don’t occur in time, or at all in some situations.

At the moment it’s buffering 64 frames (~0.85 sec / 147KB) and read-ahead occurs when 23 frames have played (~0.3 sec / 62KB) to reduce the number of callbacks/sec. I’ve purposely kept the buffer below 1 second to reduce the RMA footprint (DA’s are out as they’re not available on RO3.11.)

I had the same issue with callbacks and FileCore in ADFFS and eventually had to code my own method of callback. I was hoping to avoid doing that for this, but I don’t believe there’s any reliable way of streaming from disc under RISCOS 3.x as it stands.

Hmm, I’d probably just use a bigger buffer (and not support CD audio on older OS versions), rather than try and implement an alternate callback system.

Software CD audio on physical 3.11 systems would be a pain for many reasons (mixing with the 8bit sound system, small hard disc sizes, slow CPUs), so probably isn’t worth supporting. And for emulated systems, the best way of handling things would be for the emulator to include CD support.

My target is 3.50 and later, that said, I’ve avoided actively breaking support for 3.11 – ignoring the filesystem file size restrictions of course, which are a showstopper for all of the CD images in question.

What I might do is attempt to allocate a large DA (445 frames / ~6 sec / ~1 MB) and fallback to a small buffer in the RMA if that fails, it doesn’t solve the underlying problem but might hide it provided the game isn’t running elevated. I suspect all the games in question are C based, so elevation shouldn’t be an issue.

A large (read-ahead) cache however has its own issues, namely how quickly to populate it and what size chunks to read at each callback. Too large a chunk will stall the machine and not make for a pleasant experience and too small a chunk will result in near constant disk access. I suppose I should add configurable cache/chunk size and a threshold trigger, so it doesn’t start populating until the cache falls below X%.

I suppose I should add configurable cache/chunk size and a threshold trigger, so it doesn’t start populating until the cache falls below X%.

Yes, that sounds sensible.

• Buffer size
• Minimum read amount (i.e. how frequently to read)
• Maximum read amount (to avoid a apps/games stuttering if callbacks have been blocked for too long)

If the buffer is nearly empty, causing the maximum read amount to be reached, then to avoid stuttering there’d have to be a delay to avoid the callback being triggered again immediately. E.g. waiting until the minimum read amount has been consumed again.

It doesn’t appear you can use OS_AddCallBack from within a SharedSound Immediate fill, it caused random errors and eventually crashed the machine, so I’ve fallen back to using OS_CallEvery to perform the read-ahead. This has solved the CTRL-SHIFT issue, although I do need to test it on a RiscPC.

I now need to look into solving two major issues:

1. MODE 1 tracks need to perform error correction as unlike ISO, the data isn’t error corrected in the BIN file
2. Sound_LinearHandler needs trapping and redirecting to SharedSound_InstallHandler / SharedSound_RemoveHandler

I’ve establish MODE 1 uses Reed Solomon, but figuring out how its applied to a CD and then coding it is a mammoth task. Having compared the contents of a BIN to the original CD, there’s certainly a lot of error correction applied to data read from it.

Sound_LinearHandler probably needs trapping by SharedSound, is the source code available? Alternatively, I could add a shim to CDFaker. What’s the best way of taking over the SWI, wasn’t there an SWI added for replacing individual SWIs? Was this backported to RO3.x?

It doesn’t appear you can use OS_AddCallBack from within a SharedSound Immediate fill, it caused random errors and eventually crashed the machine

Did you remember to preserve R14_svc? I don’t think it’s guaranteed that anything will have done it for you. And on that subject, I’m not sure if SharedSound guarantees that you’ll be called in IRQ mode – the Sound_LinearHandler docs say you can be called in IRQ or SVC (although I think current implementations only use IRQ mode)

Sound_LinearHandler probably needs trapping by SharedSound, is the source code available?

Yes.

SoundDMA implements Sound_LinearHandler – but annoyingly we have three different versions in use (Sound0, Sound0HAL, Sound0Trid).

https://www.riscosopen.org/viewer/view/castle/RiscOS/Sources/HWSupport/Sound/

https://www.riscosopen.org/viewer/view/bsd/RiscOS/Sources/Audio/SharedSnd/

I’ve establish MODE 1 uses Reed Solomon, but figuring out how its applied to a CD and then coding it is a mammoth task.

Yes, it does look like there’ll be a bit of a learning curve, at least. This is why you want to support WAV files for the audio ;-)

Although, using WAV files might not help you much if you’re ripping the discs on RISC OS. The SCSI command that’s used by CD_ReadAudio has a flag to enable error correction, but CDFSSoftATAPI doesn’t set that flag (possibly because it’s crusty old code), and neither does CDFSSoftSCSI (I ran into trouble with recovering from SCSI command errors for drives which don’t support the flag. I should really take another look at that at some point…)

What’s the best way of taking over the SWI, wasn’t there an SWI added for replacing individual SWIs? Was this backported to RO3.x?

I don’t think there’s any way of taking over arbitrary SWIs, other than intercepting the SWI processor vector.

I think the SWIs you’re thinking of are OS_ClaimOSSWI & OS_ReleaseOSSWI, which are implemented by Select (and are for claiming kernel SWIs). And possibly OS_ClaimSWI / OS_ReleaseSWI, which are possibly some kind of DDE relic.

While I remember – can you make a note to check that the L+R channels are correct? RISC OS places the left channel in the upper 16 bits, but WAV and (I believe) CD place it in the lower 16 bits.

SharedSound’s buffer fill code has a flag for stereo reversal, but the code to handle it hasn’t been implemented. Unlikely to be an issue if you’re performing error correction on the data since you’ll be re-processing the data anyway, but it’s something to watch out for if you were going to implement your own 16bit WAV loader/player.

I don’t think there’s any way of taking over arbitrary SWIs, other than intercepting the SWI processor vector.

The last time I wanted to hijack some SWIs, what I did was ugly:

• Kill the original module
• Load the original module into some workspace
• Change the SWI number and base name in the loaded copy of the original module
• OS_Module 11 to push the amended module into RMA and initialise it
• Load my replacement module that gives the same API as the original – what it does is process the bits it needs to process, otherwise it just passes everything through to the original module

What I did was replace the IIC module to redirect all IIC accesses to a parallel port interface except accesses to the CMOS RAM chip, which went to the IIC via IOMD as usual.

As you can imagine, this method can work fine with some things, and would be near enough impossible with others – for instance I’m not sure one can sensibly replace FileCore in a booted machine…

annoyingly we have three different versions in use (Sound0, Sound0HAL, Sound0Trid).

On further reflection, I think I’ll use ADFFS to do the translation for testing, which will also allow this to work back to RO3.5

This is why you want to support WAV files for the audio ;-)

Audio isn’t the issue, it doesn’t have error correction, it’s MODE 1 / MODE 2 data that needs correction. At some point I’ll go back and add WAV/MP3/FLAC/AC3 etc support, but not at the minute as I have too many other projects on the go. This was supposed to be a quick win to allow games with CD Audio to be imaged on a PC.

possibly OS_ClaimSWI / OS_ReleaseSWI, which are possibly some kind of DDE relic.

That’s what I was thinking of, but obviously no use in this case.

can you make a note to check that the L+R channels are correct?

I’ve not found any details on the order of the audio data on a CD, if it is reversed compared to RISCOS, I’ll set the SharedSound stereo reversal flag – at some point it might work. I have a BBC audio test CD somewhere, so will image that and test.

This is why you want to support WAV files for the audio ;-)

Audio isn’t the issue, it doesn’t have error correction

Technically it does, but if you’ve only got the 2352 byte sectors then I don’t think there’s anything you can do to detect/correct the remaining errors.

it’s MODE 1 / MODE 2 data that needs correction.

ECMA-130 is probably the best place to start, since it goes into detail on the structure of the sectors.

if it is reversed compared to RISCOS, I’ll set the SharedSound stereo reversal flag – at some point it might work.

Fine by me. It should be pretty straightforward to implement support for it.

I’ve establish MODE 1 uses Reed Solomon, but figuring out how its applied to a CD and then coding it is a mammoth task. Having compared the contents of a BIN to the original CD, there’s certainly a lot of error correction applied to data read from it.

I am not sure what you are trying to achieve. Why use raw data for Mode1/Mode2 data? Just save every track with just the user data.

If you are really serious, you would need to implement full subcode support – a lot of work for no gain.

Reed Solomon is never relevant, you cannot access the CD’s data on such a low level. Same for EFM.

I am not sure what you are trying to achieve.

I’m adding CUE and BIN file support to CDFaker.

Why use raw data for Mode1/Mode2 data? Just save every track with just the user data.

You can use ISO for the data track if you want.

If you are really serious, you would need to implement full subcode support – a lot of work for no gain.

Are you referring to MODE2 sub code channels? As far as I can tell, CDFS doesn’t support sub code channels. CD_ReadSubChannel rather confusingly (assuming the documentation is correct) returns an amalgamation of info about the current head location, which I have implemented.

Reed Solomon is never relevant, you cannot access the CD’s data on such a low level.

BIN contains the error detection code (4 bytes) and error correction code (276 bytes.) The data needs to be error checked and corrected, otherwise its likely to contain corrupt data.

I’m adding CUE and BIN file support to CDFaker.

Yes, but why? It is surely an interesting coding project, but somehow I fail to see its use. BIN/CUE was invented to give full control over track structure and subcode stuff for Audio CDs. Most of the finer details are not accessible via CDFS anyway.

Are you referring to MODE2 sub code channels?

No, I am referring to CD subcode, both in TOC and data area. I don’t even understand what you mean by “Mode 2 sub code channel”. I am not aware of anything specific wrt Mode 2 blocks.

As far as I can tell, CDFS doesn’t support sub code channels. CD_ReadSubChannel rather confusingly (assuming the documentation is correct) returns an amalgamation of info about the current head location, which I have implemented.

There might of course be bugs in current softloadable drivers implementations, but CD_ReadSubChannel just provides a minimal abstraction over CD P/Q subchannel content. What exactly do you find confusing?

BIN contains the error detection code (4 bytes) and error correction code (276 bytes.) The data needs to be error checked and corrected, otherwise its likely to contain corrupt data.

ECC/EDC in Mode 1/Mode 2 Form 1 is not Reed Solomon. Reed Solomon is used a lot earlier in the process and is handled transparently by the drive’s hardware/firmware.

And if your hardware is OK, the data is extremely unlikely to be corrupted.

See here https://github.com/claunia/edccchk for an EDC/ECC checker.

Yes, but why?

To support mixed mode CD’s, please refer to the OP where I listed some potential options.

No, I am referring to CD subcode, both in TOC and data area.

Misunderstanding on my part, you’re referring to Q channel data in the frames. I believe only CCD/IMG/SUB (CloneCD) support this, but it’s not really necessary for mixed-mode support as BIN/CUE provides everything required. CD_ReadSubChannel has been coded to return the correct values based on the CUE contents, the only thing I haven’t coded is the pre-gap as its not required for games. The TOC I believe is stored in the lead-in header so can possibly be read if it’s not in the Q channel – I’ve not investigated. If there’s a CD player app for RISCOS that reads the TOC, I’ll test it.

ECC/EDC in Mode 1/Mode 2 Form 1 is not Reed Solomon. Reed Solomon is used a lot earlier in the process and is handled transparently by the drive’s hardware/firmware.

I must have misunderstood the document, it states “…the P-parity and Q-parity data provide a double interleaved Reed-Soloman code within the CD-ROM data”, I read that to mean the P/Q-parity data in the ECC were generated with Reed-Soloman. Thanks for the link to the EDC/ECC checker, I now need to figure out how to also do correction.

And if your hardware is OK, the data is extremely unlikely to be corrupted.

I imaged HOMM2 in BIN and ISO and then compared the sector data between the two for differences. The BIN had runs of bit errors in a few sectors, which going by their position in the image coincided with some surface marks on the CD. Given the age of the CD’s, I’m expecting most BIN images to require some error correction, so I need to add this before it’s going to be useful for CD-ROM data.

EDIT: I’ve found this PDF which states the ECC contains:

172 bytes – P parity (26,24) Reed Solomon codes
104 bytes – Q parity (45,43) Reed Solomon codes

The ECMA-130 PDF also covers encoding the level 2 Reed Solomon code in Annex A.

If I understand correctly, the F-frames have the 1st level of Reed Solomon correction and the sectors contain a 2nd level for CD-ROM. CD Audio has the 1st level, but no 2nd level.

If I understand correctly, the F-frames have the 1st level of Reed Solomon correction and the sectors contain a 2nd level for CD-ROM. CD Audio has the 1st level, but no 2nd level.

You are correct, I was wrong. I never before heard of Yellow Book ECC being based on Reed Solomon, but it seems to be the case. It is of course 20 years ago since I knew basically everything about that CD stuff, but I am sure I only ever heard of Reed Solomon wrt CIRC.

To support mixed mode CD’s, please refer to the OP where I listed some potential options.

The simplest option would be to invent a new, much simpler format – a TOC file and the tracks. I can’t imagine a RISC OS game using clever stuff like subcode access or invalid EDC/ECC or manipulated block headers for protection reasons – mainly because CDFS does not give access to such low level tricks, you could not even rely on CD_ReadAudio to work!

So you end up with three track types: mode 1/mode 2 form 1, mode 2 form 2 and audio. No implementation for EDC/ECC or subcode or whatever needed. You just handle the user data and construct the responses for TOC requests.

After lots of head scratching and debugging, I have HOMM2 playing CD music under emulation; it doesn’t work on physical however as it expects the audio CD to be in drive 0. I can’t find the code that selects the CD, but its possibly hardcoded to drive 0.

Ankh uses CDFS_ConvertDriveToDevice which fails under emulation as its returning invalid device info for drive 0. I’m testing under Red Squirrel which may be the cause.

In short, I need to redirect CD device 0 to the emulated CD if an image is mounted, but with so little documentation I’m not sure how to do it. Would intercepting CDFS_ConvertDriveToDevice and CD_Identify and changing the drive they return info for suffice?

See here https://github.com/claunia/edccchk for an EDC/ECC checker

edccchk turned out to be rather useful, I ran all the BIN’s through it and reimaged any with errors until I got a clean image. That saves me figuring out how to implement RS error correction.

The simplest option would be to invent a new, much simpler format

BIN/CUE do the job, but are wasteful and require two files. If I do implement a bespoke image format, I’d look at compressing the sectors with ZLib and strip out the headers/EDC/ECC in data tracks, I’d also include everything in one file.

EDIT: Ankh has a bug in it’s CD code, it doesn’t shift the values correctly when filling the CDFS Control Block so will always error when loaded. After correcting that issue the game played CD music correctly. Wondering how on earth it got released in that state, it appears it probably came with a patch floppy which corrects the bug.

How does CDFSDriver associate a soft driver with a particular drive? I’ve been staring at the source code for days and not really got anywhere, is it something to do with CD_Identify as it appears to be the only SWI where the soft driver can associate a drive type to a SCSI device.

If I understand the source correctly, there’s one soft driver associated with each drive type and all CDROM drives get a SCSI ID irrespective of being SCSI devices?

There are a couple of serious problems with the CD driver stack.

• CDFS doesn’t properly support drive detection/hot-plugging (you need to configure the number of drives for it to look for)
• The CD_ SWI API is based around SCSI device IDs, which means non-SCSI drives need to have SCSI IDs assigned to them, and there’s no guarantee that the assigned ID won’t conflict with a hot-plugged SCSI CD drive (which would then prevent that hot-plugged drive from registering)

CDFS has a loop where it iterates through all possible SCSI device IDs, calling CD_Identify to see if there’s a drive there. It’ll do this whenever you try accessing a CDFS drive which hasn’t had a SCSI ID associated with it yet; drive assignment is performed the first time the drive is accessed (possibly it also gets a manual kick during boot, but if that fails, as would be the case with CDFaker or USB drives, then it will try again when you try accessing the drive). Note that the enumeration order the loop uses has changed a bit over the years (e.g. a few years ago I changed it to start from card 3 and work down to card 0, to reduce the risk of ATAPI/CDFaker claiming an ID which will later conflict with a hot-plugged USB CD)

CD_Identify will issue a SCSI INQUIRY command via SCSI_Op, and then pass the result (regardless of success or failure) around all of the registered drivers to see which one responds to it first. https://www.riscosopen.org/viewer/view/castle/RiscOS/Sources/HWSupport/CD/CDFSDriver/s/Identify?rev=4.4#l55

Non-SCSI drivers like ATAPI & CDFaker just listen out for the first failed INQUIRY op and then hijack that SCSI ID. https://www.riscosopen.org/viewer/view/castle/RiscOS/Sources/HWSupport/CD/ATAPI/s/DriverCode?rev=4.19#l2706

The exact purpose of the “drive type” number isn’t clear, but the UG_Text document gives me the feeling that the main purpose was to allow the PC emulator to work correctly; presumably the MSCDEX driver needed to know the type so that it would know which SCSI commands to issue (I’m guessing it would have gone via a SCSI or ATAPI layer within the emulator, and then straight to the SCSI_Op SWI on the RISC OS side). I think there might have been a master list of drive type numbers somewhere, but I can’t seem to find it at the moment (either way, both CDFSSoftATAPI and CDFSSoftSCSI return type 0)

I think there might have been a master list of drive type numbers somewhere, but I can’t seem to find it at the moment

Ah, yes, the old CD player app has an interesting function, and a list of drive types near the top of the file. But it doesn’t actually use the type itself, it only fetches the type so that it can then pass it into the other CD_ calls (I’d forgotten that the drive type was part of the control block!)

Which makes me think that maybe the drive type only exists so that individual drivers don’t have to maintain their own table mapping SCSI IDs to drive types; they identify it once, return that info to the caller, and then require the caller to pass that same value back in so that each individual call knows what SCSI commands it should be using for the drive.

There are a couple of serious problems with the CD driver stack.

Yes, I did notice the potential flaw in the reliance on SCSI ID’s and lack of hot plugging. It probably wouldn’t have been so bad if there was room to use illegal SCSI ID’s, as it stands hot plugging a SCSI device might require a reinit of the CDFS stack if there’s a conflict.

In my attempts to take over the 1st CD-ROM, I changed CDFaker to identify all physical CD’s on init and then force and respond to CD_Identify when an image is mounted, but CDFSDriver continued to use the original driver associated with the CD. I then attempted to RMReInit CDFSDriver as it forces a rescan of the soft drivers, but that didn’t work either.

Essentially I need to hot plug between the actual soft driver and CDFaker, ideally without reinitialising the CDFS module stack every time an image is mounted/dismounted. One option I might look at is to intercept CD_Register and CD_Identify with a “hotplug driver”, which can then direct all CD_ SWI’s to the appropriate soft driver depending on CDFaker having an image mounted. Another option is to alter the drive number that CDFS_ConvertDriveToDevice recieves when an image is mounted, which is somewhat less intrusive, but does mean CDFaker is still taking a non responsive SCSI ID.

What do you think the best strategy is for claiming a SCSI ID when there’s no physical CD, given the chance a SCSI drive might be hot plugged? Should I simply give the user the option to set a specific SCSI ID to use? CDFaker currently supports up to 8 fake drives, HOMM2 needs at least 2 to install, unless I implement a way to swap images when the filesystem takes control of the machine with an “insert this CD” prompt, multiple SCSI ID’s for each fake drive are required.