Enhancing CDFaker to support CD audio

I’m proposing to add music playback to CDFaker, for games that use CD tracks for music. This however raises a few questions, which I’d appreciate some input on:

How many games play CD tracks?

I believe Might & Magic plays audio tracks, are there any others?

What format should the CD images use?

BIN/CUE is the widest used standard, but possibly isn’t optimal as it also contains the sector headers and audio would be 44.1k 16bit stereo (176KB/sec) which a lot of machines either don’t support, or it might create an I/O bottleneck trying to stream and play them.

Assuming BIN is used, the audio would need buffering to avoid stalling the machine doing constant I/O. How far ahead should it buffer and how often should it add to the buffer?

If BIN isn’t used, what other options are there?

ISO with separate audio tracks is one option, these could be either lossless (FLAC?) or lossy (MP3?) using an external player. Alternatively there’s existing proprietary single file solutions, Daemon Tools for example, although finding documentation on these might be an issue.

There’s also the option of designing a new bespoke single file image format, for example, using ZLib to deflate each sector individually to reduce the file size, but not impact random access times, and include pre-processed audio tracks in the file.

References:

CUE structure
BIN/CDROM sector structure
MDS structure
ECMA-130

CD based Games with CD Audio

F10016 Ankh: The Tales of Mystery (1998) (R-Comp Interactive) – played in-game
F10193 Heroes of Might and Magic 2: The Succession Wars (2000) (R-Comp Interactive) – played in-game
F10194 Heroes of Might and Magic 2: The Price of Loyalty (2000) (R-Comp Interactive) – played in-game
F10295 Quake Resurrection (2000) (R-Comp Interactive)
F10190 Towers of Darkness – Heretic, Hexen & Beyond (1999) (R-Comp Interactive)

CD based Games without Audio

F10628 Desktop Repton Issue Four (2004) (Alligata Software)
F10626 Krisalis Gold Collection (2001) (R-Comp Interactive)
F10413 Nevryon & Technodream Uncovered (2000) (ProAction)
F10528 Superior Collection, The (2000) (ProAction)
F10473 Superior Collection v2, The (2001) (ProAction)
F10360 Syndicate+ (1998) (R-Comp Interactive)
F10627 TBA Classics (2000) (TBA Software)
F10365 TEK 1608 (2002) (R-Comp Interactive)
F10386 Wizard Apprentice (1997) (The Datafile)

CD Audio state unknown

F10568 Dark Future (1999) (Grasshopper Software)
F10111 Descent (1998) (R-Comp Interactive)
F10112 Descent 2: Counterstrike (2002) (R-Comp Interactive)
F10658 Descent I and II (1998) (R-Comp Interactive)
F10113 Destiny (1999) (Acorn User)
F10119 Doom+ (1998) (R-Comp Interactive)
F10120 Doom II () ()
F10537 Doom Trilogy (1998) (R-Comp Interactive)
F10535 Dune II – Battle for Arrakis [CD version] (1996) (Eclipse)
F10533 Simon the Sorcerer [CD version] (1995) (GamesWare)

I believe Might & Magic plays audio tracks, are there any others?

Yes, IIRC HOMM 2 used CD audio. I’d expect RISC OS ports of Quake to use it too, although most modern versions of the engine have builtin support for alternate music formats (FLAC/MP3/Ogg/etc.)

Maybe some educational titles also used it?

44.1k 16bit stereo (176KB/sec) which a lot of machines either don’t support

Unless you’re aiming for Arc compatibility, resampling the audio should be trivial, especially since SharedSound provides you with a buffer fill routine which can do the resampling for you.

The algorithm SharedSound currently uses isn’t fantastic, but if more people make use of it (and complain about any deficiencies) then there’ll be a higher chance of improvements being made, which would then obviously benefit everyone.

or it might create an I/O bottleneck trying to stream and play them.

If you’re aiming for RiscPC compatibility, I’d wager that the CPU load of dealing with FLAC or MP3 would be much higher than the cost of waiting for raw 44.1kHz data to stream from the hard disc.

On more modern machines, it’s probably only the ones using USB for storage which are likely to suffer any latency issues – SD or SATA should be plenty fast (although you also have a faster CPU so FLAC/MP3 isn’t so expensive).

How far ahead should it buffer and how often should it add to the buffer?

Whenever I’ve done streaming audio (from CD/DVD) I’ve generally used a simple double-buffering system, each buffer holding one second of data.

With HD/flash storage you could get by with a lot less; maybe a circular buffer of one or two seconds, and stream in a new 64KB chunk whenever you have >= 64KB space available? (SCSIFS will split transfers into 64KB chunks in order to avoid problems with dodgy USB devices, so going above that will suffer extra overheads on some machines). Although now I think about it, the default USB mass storage DeviceFS buffer size is 32KB, so that might be a more sensible value. Find a couple of crappy USB devices and experiment!

There’s also an alternative option of preloading the entire track when you receive the seek request. Although for pre-RISC OS 5 the memory management might be a bit tricky (DA logical size limits; on RISC OS 5 you can just bypass those using a PMP)

There’s also the option of designing a new bespoke single file image format, for example, using ZLib to deflate each sector individually to reduce the file size, but not impact random access times, and include pre-processed audio tracks in the file.

Unless you have very specific requirements (e.g. Arc compatibility), I don’t think there’s any point reinventing the wheel.

Unless you’re aiming for Arc compatibility

RO3.71 support, yes, as I’ll be testing with HOMM. I’ve already created BIN/CUE images of it and extended CDFaker to support BIN files. I do need to submit a request for BIN and CUE filetypes, as I don’t believe there are any allocated for them currently.

SharedSound provides you with a buffer fill routine which can do the resampling

I’ll use that for playback then.

DeviceFS buffer size is 32KB, so that might be a more sensible value

32KB chunks seem sensible, with a few seconds read-ahead.

I don’t think there’s any point reinventing the wheel.

Quite, my point was around the size of BIN images. Heroes of Might and Magic 2: The Succession Wars is 763MB if the 1st disc is BIN/CUE and the 2nd is ISO. Heroes of Might and Magic 2: The Price of Loyalty is 1.2GB for both discs.

ISZ compresses each chunk using ZLib (a chunk being 1 or more sectors), the spec doesn’t directly support audio, but probably could do with an accompanying CUE file and inclusion of the audio sectors.

Some size comparisons of HOMM disc 1 using different methods:
BIN/CUE: 729MB
MDF/MDS: 584MB
ISO: 650MB
DEFLATED BIN: 608MB
DEFLATED MDF: 462MB

MDF/MDX/ISZ/NRG and most other CD dump formats are proprietary and require licensing, so can be ruled out. So it’s BIN/CUE or design a more optimal storage format. The problem however is that I’m not certain you can read raw sector data from a CDROM using CDFS, so imaging would require conversion of BIN/CUE.

I’ll stick with BIN/CUE for the time being.

Unless you’re aiming for Arc compatibility

RO3.71 support, yes

I was thinking of actual Archimedes machines (and early RiscPCs too, I guess) – lack of 16bit sound hardware would make the playback a lot more complicated.

I do need to submit a request for BIN and CUE filetypes, as I don’t believe there are any allocated for them currently.

BIN is just “raw binary data”, so Data would seem like a suitable RISC OS filetype. CUE could easily be Text, but I guess using a dedicated file type won’t hurt.

Quite, my point was around the size of BIN images.

Ah, I was thinking you were more concerned about the performance.

I guess for older machines, and for people hosting the files, smaller files would be desirable.

A quick skim through wikipedia suggests that there are a couple of useful features of CUE files:

1. You can specify multiple data files
2. You can specify the format of each data file

So you could have an ISO data track followed by several WAV or MP3 audio tracks. This could also give you the option of hosting the images in different formats – one common download containing just the ISO, and a couple of other downloads containing the cue sheets and music in different formats.

The problem however is that I’m not certain you can read raw sector data from a CDROM using CDFS, so imaging would require conversion of BIN/CUE.

Yeah, I’m not sure about that myself. But since BIN audio data isn’t a necessity, unless you’re creating 1:1 copies of the CD there’s no reason why you can’t just use CD_ReadAudio and write to WAV/MP3/whatever output.

The Windows Quake CD has the Nine inch Nails audio tracks.
The Ankh CD also comes with audio tracks.

I was thinking of actual Archimedes machines

I didn’t think CDFS existed until the RiscPC.

A quick skim through wikipedia suggests that there are a couple of useful features of CUE files

Yes, which could also be considered a drawback as you have to support untold random file types! I’m only planning on supporting audio within the BIN image, music files could be covered by external players provided they have an associated Alias$@RunType_XXX.

Quake CD has the Nine inch Nails audio tracks.
The Ankh CD also comes with audio tracks.

Wasn’t aware of that, thanks.

Yes, which could also be considered a drawback as you have to support untold random file types!

Well the official spec only lists a handful of types, so as long as you support those you shouldn’t run into any serious issues.

music files could be covered by external players provided they have an associated Alias$RunType_XXX.

Offloading support for formats to external players makes sense, but it’s a bit complicated than just running the file, since you also need to support stopping playback, seeking to specific timestamps, etc.

Considering the number of different music player modules and frontends that exist, I’m a bit surprised we don’t have any kind of generic (audio) media playback interface yet. I.e. a service call you can throw a filename / data blob at, and get back a handle/identifier you can use to control playback.

Considering the number of different music player modules and frontends that exist, I’m a bit surprised we don’t have any kind of generic (audio) media playback interface yet. I.e. a service call you can throw a filename / data blob at, and get back a handle/identifier you can use to control playback.

Is that the sort of thing that could allow RISC OS NetSurf to support the HTML 5 audio tag?

Yes, I believe so.

Considering the number of different music player modules and frontends that exist, I’m a bit surprised we don’t have any kind of generic (audio) media playback interface yet. I.e. a service call you can throw a filename / data blob at, and get back a handle/identifier you can use to control playback.

Indeed, but like the proposed Joystick API it’s not something that would get widely used, so there’s been no demand to implement one. A service call could possibly work, who’s going to spec it, code it and then modify mainstream players to support it? I’m not even sure which players support WAV, MP3, FLAC etc.

I didn’t think CDFS existed until the RiscPC.

I had CDFS on my A5000. I think it was a part of the Simtec IDE podule and not the Morley SCSI because I had a silly little 2x drive (2X=epic fast back then!) and I can remember that SCSI stuff was super expensive.
On the A5000 it barely worked. Some data CDs worked, many didn’t. Quite a few played as audio tracks (ouch). Things were much better on the RiscPC, it took a while for the technology to mature to the point where it…worked… :-)
[you may recall the who-the-hell-knows-what-kind-of-interface CD-ROM drives that used to plug into PC soundcards]

A service call could possibly work, who’s going to spec it, code it and then modify mainstream players to support it?

Specing the interface is easy, as long as you keep it simple. E.g. the first version probably only needs to support files as the data source. However I would say that it’s probably worth having a module which coordinates everything – i.e. an application would call a Media_Play SWI, and then the Media module would ask the various backends how the file should be played. By having an extra layer between the applications and the player modules this will allow us to more easily cope with any API changes in the future, by having the Media module translate between the different versions.

André Timmermans should have a good idea of what’s required for detecting audio formats and controlling players, since DigitalCD supports many different formats

There’s also David O’Shea’s !Amp, which used a plugin system to interact with the player modules. !Amp used Wimp messages to communicate with the plugins, so we won’t be able to use its plugin spec directly, but with a few adjustments you could probably come up with something which would be single-tasking friendly.

Also, (as shown by !Amp), there’s not necessarily any need to modify player modules to support the spec. The first implementation could just be a helper module that knows how to talk to a few of the popular player modules.

By having an extra layer between the applications and the player modules this will allow us to more easily cope with any API changes in the future,

Not just that; more importantly you can use the abstraction so that the application doesn’t actually need to know what our how the audio data in question is going to be played, just that it can or cannot be handled. This also makes it extensible – if Ogg is not supported today, it could be tomorrow without huge rewrites of the interface module.

Ideally we’d want to be in a situation where the interface can broadcast a ServiceCall asking what playback support is available and the playback modules themselves would respond with their capabilities. In this way, nothing needs changed to support format xyzzy – simply having the appropriate player loaded and responding would cause it to be added to the list of known types.
It’s okay if the same module replies multiple times, I think DiscSample deals with a number of different formats.
Don’t forget to have a flags word to allow the format to be described, perhaps in more detail than would be initially supported? Flags for things like “lossless”, “streaming”, “exact seek”, and “can have track title”.

I didn’t think CDFS existed until the RiscPC

IIRC CDFS was in most SCSI podules. I certainly remember finding out that Eesox CDFAST let me play Replay moves from a CD shared over SHAREFS without glitching on a A3020s!

It appears both CDFS and SharedSound are available on RO3.11 so BIN/CUE based CD audio will in theory be possible on legacy machines.

I finished coding the CUE parser today, with some restrictions:

1. Only one FILE entry is supported, this being the associated BIN and it must be type BINARY. MOTOROLA, AIFF, WAV, MP3 are unsupported
2. I’ve not found the sector layout of some of the more obscure standards TRACK supports (CDG, CDI), but do have AUDIO, MODE1 and MODE2 covered
3. Only INDEX 01 is supported as CDFS doesn’t appear to support subindexes
4. All CUE entries except FILE, TRACK and INDEX are ignored
5. The file specified in FILE must be in the same directory as the CUE, so can’t contain a directory path

I’ve successfully mounted a BIN of both MOMM2 releases, so the next step is adding Audio playback. I’ll start by testing playback using a raw CD Audio player if such a thing exists, before I try to figure out how to use SharedSound. SharedSound does present an issue for games, as it doesn’t mix in the 8bit sound system or provide 8bit to 16bit conversion. I’ll probably resolve this issue by modifying ADFFS, as the games in question are all 26bit.

before I try to figure out how to use SharedSound

SharedSound is pretty easy. See here for an example WAV player I wrote some time ago, although if I were to look at it now I’m sure I’d spot various bad practices which should be avoided.

SharedSound does present an issue for games, as it doesn’t mix in the 8bit sound system or provide 8bit to 16bit conversion.

On systems with 16bit audio, SharedSound installs itself via Sound_LinearHandler, which is provided by SoundDMA. SoundDMA is responsible for 8bit to 16bit conversion, and it also choreographs mixing the two streams (it doesn’t do any mixing itself, but it will tell the linear handler whether mixing is required). It’s only systems which lack 16bit audio where you’ll be in for major problems with audio mixing.

However, for systems with 16bit audio, you may run into the issue that some games want to use 16bit audio themselves, and install their own linear handler, disabling SharedSound (SharedSound didn’t exist when 3.5 was released, so using the linear handler was pretty much the only option available). So you might need to trap that and have a shim that adapts them to use SharedSound instead (potentially with some logic to ensure they mix into the existing buffer correctly).

See here for an example WAV player I wrote some time ago

Thanks.

Is the original Acorn documentation for CDFS available anywhere? The SWI’s documentation on the Wiki is missing some key information.

Made some progress today, DigitalCD recognises the audio tracks in a mounted BIN/CUE and attempts to play them. It seems to only play half a second or so before skipping to the next track though and I’m not sure why.

I’ve taken a guess that the CD_ReadAudio documentation means frames where it states Length of section to grab, but it might also be seconds or centiseconds.

There’s also some inconsistency in the order and SWI number of some of the SWI’s which had me confused for a while. CD_ReadAudio seems to be a lower number on RO3.71’s CDFS than it is in the source code for RO5, I’m not sure what’s going on there. Once I have it working correctly on RO3.71 I’ll see if it works under RO5.

I’ve taken a guess that the CD_ReadAudio documentation means frames where it states Length of section to grab, but it might also be seconds or centiseconds.

The length is the number of blocks (same as CD_ReadData); I’ll correct the wiki.

There’s also some inconsistency in the order and SWI number of some of the SWI’s which had me confused for a while. CD_ReadAudio seems to be a lower number on RO3.71’s CDFS than it is in the source code for RO5, I’m not sure what’s going on there. Once I have it working correctly on RO3.71 I’ll see if it works under RO5.

The only change to the header since 1996 is the addition of the CD_SCSIUserOp, so it would be interesting if that was the case. Are you sure you’re not confused by the numbering? &41240 + 38 = &41266.

Made some progress today, DigitalCD recognises the audio tracks in a mounted BIN/CUE and attempts to play them. It seems to only play half a second or so before skipping to the next track though and I’m not sure why.

Probably the classical “!50 frames bug” in the CD_XXXDriver used by your machine. I suggest you read the CD troubleshooting section in the DigitalCD help.

Are you sure you’re not confused by the numbering? &41240 + 38 = &41266

It’s the SWI value of passed to the driver. For those not familiar with CD drivers CDFS passes the SWI# called to the driver in R11, the driver then acts accordingly.

The driver is seeing R11=31 for CD_ReadAudio, but as you state CD_ReadAudio should be 38. For CD_Identify R11=30, but on the face of it should be 35 (&41240 + 35 = &41263). I’m hoping the restricted Acorn Developer SWI document might shed some light on the discrepancy.

Probably the classical “150 frames bug”

Setting that flag makes no difference. What’s the “150 frames bug”? Audio starts at exactly 150 frames on an standard Audio CD, is it related to that? The HOMM CD image I’m using for testing is a mixed MODE CD, being tested on an emulated RiscPC under CDFS 2.29

I tried another player (found via random Google search, didn’t save the link unfortunately), which played the CD perfectly so its possibly specific to DigitalCD. Thinking it might be a combination of factors, possibly related to register corruption or how DigitalCD determines the track has finished and CDFaker not returning correct values, I dumped DigitalCD and looked through the code to see what it does after reading the first frames of the track. I couldn’t find any references to CD_ReadAudio, so am somewhat at a loss as to what the cause of the issue is.

What triggers the skip to the next track? How does it know it’s read the whole track? I’d expect it to call CD_ReadAudio until it hits the LBA of the next track.

EDIT: If I attempt to quit DigitalCD after clicking play, but sit at the “There is unsaved data…” prompt, the CD plays correctly.

The driver is seeing R11=31 for CD_ReadAudio, but as you state CD_ReadAudio should be 38. For CD_Identify R11=30, but on the face of it should be 35 (&41240 + 35 = &41263).

Ah, yes. That seems to be intentional, although I’m not sure why.

The hashes header lists the different reason codes the backend drivers are called with.

I’m hoping the restricted Acorn Developer SWI document might shed some light on the discrepancy.

Steffen Huber sent me a copy of Acorn’s CD ROM driver development kit/sample when I was working on CDFSSoftSCSI. I can’t remember if it was particularly useful, but when I get home I’ll have another look and check with ROOL to see if there are any blockers to releasing it publicly.

The hashes header lists the different reason codes the backend drivers are called with.

I was looking at the CDFS hashes header, didn’t realise there was a different one for CDFSDriver. CDFS Technical Details stating R11 is “SWI number called – 1” is what got me, I mistakenly thought that meant it was the SWI#!

I’ll have another look and check with ROOL to see if there are any blockers to releasing it publicly.

Thanks, I’ll need it to add CD_AudioControl / CD_ReadAudioParams / CD_SetAudioParams as they’re not documented on the Wiki.

EDIT: The other query I’m hoping it will clear up is track numbers, do they start from 0 or 1. CD_EnquireTrack implies they start at 1 as 0 is used to specify a range.

FYI if you run across a broken link or two it’s because I’ve moved some of the CDFSDriver backend documentation to the new CDFSDriver Technical Details page (as well as added a list of the driver call reason codes). It wouldn’t surprise me if more moving/renaming is needed to make that area of the wiki make complete sense, but at least there’s now some basic separation between the frontend interface details vs. backend interface.

I’m hoping the restricted Acorn Developer SWI document might shed some light on the discrepancy.

Steffen Huber sent me a copy of Acorn’s CD ROM driver development kit/sample when I was working on CDFSSoftSCSI. I can’t remember if it was particularly useful, but when I get home I’ll have another look and check with ROOL to see if there are any blockers to releasing it publicly.

It looks like the most useful bit is already in CVS – https://www.riscosopen.org/viewer/view/castle/RiscOS/Sources/HWSupport/CD/CDFSDriver/Doc/DevKit/ (specifically, the UG_Text file)

The only other thing the development kit came with was a very bare-bones driver for SCSI Toshiba drives, which isn’t likely to be very useful considering you now have the sources to CDFSDriver, CDFSSoftATAPI, and CDFSSoftSCSI available as reference (although CDFSSoftSCSI is post-Acorn so might not entirely obey the spec!)

A bit of googling for the fabled application notes also turned up this post by Steffen, with the application notes mentioned being available from your local friendly Chris’s Acorns