Sound for Squeak/Scratch

It’s been while since the first versions of Squeak went out into the world of RISC OS and I still haven’t made the sound output work. I’d like to fix that and make Scratch more interesting – it seems like RISC OS might be getting some more interest as part of that project.

I have no idea at all about sound on RISC OS. I have no more idea about sound on OS X or linux or Windows, but there is working code for those three platforms to help work out what is expected. I’m hoping to find someone with an interest in sound that can help to get this done.

Please; think of the children….

My ability to help is severely limited due to my knowledge of Scratch’s capabilities being zero. If you want to synthesise waveforms on-the-fly Beeb-style, that’s obviously going to take a bit more effort. Ditto if you want to play music. Do you want to be able to play more than one sample at once? Which sample formats do you wish to support – raw data, Wave, MP3?

Fair point, Martin – but I didn’t want to write a huge post describing what is needed without having a clue if anyone might be interested in helping. Too much detail might bore people, too little might not attract them; where ego make the balance?

The requirement is to be able to feed arrays of samples generated by Squeak on the fly to the OS sound system and have the noises come out. I’ve attempted to make sense of the PRM sections on sound several times and I just don’t seem to get it. Clearly some other people have been able to generate sound data and play it or we wouldn’t be able to play movies and mp3s etc. I hope some of that code might turn out to be an easy way to solve my problem.

Anyway, the Squeak sound system low-level code is loosely based on how the old sound system worked on Macs circa 1995. Since Squeak running on OSX still produces all the sound people want I imagine the system hasn’t changed dramatically. For unix there are sound systems based on NAS, ALSA, OSS and for Solaris, then for windows there is w32 and direct-sound versions and the original Mac version, which gives an interesting spread and demonstrates that it is possible to adapt from Squeaks interface to a range of OS facilities. If anyone understand one of the already working varieties of sound support as well as RISC OS, it would probably make life much easier! Maybe someone that has ported an mp3 playing app from linux or windows might already have all the code I need?

The code assumes some sort of buffer that it can write samples to and that can be queried for the amount of room left empty and available for more samples. It expects a way to start and stop the playing of the contents of the buffer. It expects a way to query and set the volume. AIUI the Mac system has a pair of buffers that alternate, one being read from by the sound system and the other available for applications to write to, which seems fairly sensible to me.

There is also a set of code that wants to handle recording sounds into a provided buffer, set the record volume, start and stop recording , and query the sample rate of the recording system. There’s the same range of already existing examples to learn from.

All the code is available, the working Squeak system (minus sound!) is available and all help is most welcome.

Some info on the RISC OS sound system can be found here:

http://www.acornarcade.com/articles/Building_the_Dream_2_-_The_RISC_OS_Sound_System/index1209.html

Written by none other than Jeffrey Lee.

You could also try using DigitalRenderer, which is in turn used by UnixLib to provide sound support. The module seems to have a clean SWI interface.

EDIT: Steffen seems to have beaten me to it…

Ouch. Yeah, you’re going to want to write your own custom sound handler, then – and that’s one steep learning curve.

(I thought Scratch was supposed to be a beginners’ language? That all sounds incredibly low-level! I was expecting something more along the lines of a function call which played a sample once and then stopped, for which there are modules available to help you.)

Anyway, the main problem is, I have a reasonably good idea of how to operate the RISC OS sound system, and relevant experience in doing so, but absolutely no experience in either Squeak or Scratch. I’ve also never attempted to write a player in anything other than a BASIC/assembler combination, and in particular I’ve never programmed for RISC OS in C (from what I’ve seen, some of the APIs are distinctly nasty).

What I can do, if you’re having trouble understanding the PRMs (and the sound system has changed drastically since the PRMs anyway) is point you at an article I found very helpful back when I first started experimenting with playing sounds, here – written by Jeffrey Lee before he got famous.

You should note that there are actually three RISC OS sound systems. The original one, dating from the Archimedes VIDC, uses 8-bit logarithmic samples, on up to 8 hardware channels (simultaneous noises) at once – this is what *ChannelVoice, Sound_Configure, Sound_Control etc. use. The in-between one is Sound_LinearHandler, which has two channels of 16-bit linear samples – support for this was optional on most Acorn machines, and you could buy a 16-bit sound card to fit to your RiscPC (and earlier machines?) which would allow this system to work.

Both of these are now deprecated, and the one which the article covers is SharedSound, a module originally by Expressive Software Projects, but now available in the public ROOL CVS and in ROM builds since RO5.18. This also depends on the presence of 16-bit sound hardware, but I don’t think any computer since the RiscPC or possibly A7000 has supported the 8-bit system natively – although the original API is still emulated for compatibility. SharedSound allows multiple claimants to the sound system to play different sounds simultaneously, and to connect and disconnect in any order.

You could try implementing the other two systems for backwards compatibility later on, but I advise you to stick with SharedSound – amply explained with lots of example code in the article – as the only system which is really important here is the Raspberry Pi, which will definitely support it.

OK, excellent information.

The SharedSound module seems like it might be an excellent base to build on and I really don’t care at all about backward compatibility for ancient systems. They’re simply not up to running Squeak decently anyway. In particular all I’m worried about for now is Pi, since that is where theScratch action is. I’m hearing from people that even the current state of support for Scratch on my barely resurrected Squeak makes it easier to use than the same Scratch image running on a Squeak VM under Raspbian. I can’t judge since I’ve only got RISC OS on my Pi and haven’t run an ARM linux since the days of the Corel NetWinder around ’99. So getting some sound support working for them would be nice. Might gather some more RISC OS fans, too.

(I thought Scratch was supposed to be a beginners’ language? That all sounds incredibly low-level! I was expecting something more along the lines of a function call which played a sample once and then stopped, for which there are modules available to help you.)

Scratch is a teaching language, as you thought. It’s written in Squeak Smalltalk, The sound support in the Squeak system is supposed to be fairly close to what you imagined – generate a sound (all sorts of clever code does that for you, including a surprisingly tolerable speech system), dump the bytes to the sound system and play it. Loop as appropriate. Sounds might be as simple as a tone or as complex as a multi-part harmony that the user adds to in realtime to play along.

The bit I work on (since 1984) is the Virtual Machine and very low-level object engine. I wrote a moderately well known book section on the subject which might appeal to you as a way to beat insomnia – (http://www.rowledge.org/resources/tim‘s-Home-page/Squeak/OE-Tour.pdf ). It’s that low-level stuff that handles all the graphics, execution, memory, input, timing, sounds, network connections etc so that the higher level code has a completely portable view that works the same on every machine that runs it. You can take an image file (basically a memory dump), copy it from a windows pc to a Mac, run it a while, save it, copy it to a Pi and run it and the state will be consistent across all of them. Only the OS capabilities and the patience of nerds like me limits what each machine can do. On fast machines with high end graphics capabilities you can do shared world immersive environments with voice and video comms to create, well all sorts of things.

Anyway, I’ll read the suggested articles and return with new wisdumb.

You can take an image file (basically a memory dump), copy it from a windows pc to a Mac, run it a while, save it, copy it to a Pi and run it and the state will be consistent across all of them.

Whoo. I bet that’s fun on a differently-ordered processor. I sort-of wrote a 6502 emulator that compiled on ARM and x86, and the save-state dump was a pain to load – you couldn’t even rely upon structs having the same layout…

Wow I am surprised at difuculties going between ARM and x86, they are both little endian.

Endian issues are no problem to solve; there’s nothing difficult about swapping all the relevant bytes around. We have a flag word at the beginning of the file that makes it possible to detect if it was saved on a different endian system that you are starting up up on (until very recently that flag value was ‘6502’ :-) ) and the code just trots across the entire range of memory involved and swaps bytes. Then it makes sure to fiddle a little with some items that aren’t supposed to be swapped, or perhaps need some other forking around. Then you find the appropriate object, load up the state variables of the virtual machine and carry on as if it hadn’t stopped. Simple.

Sound, now that’s a different level of problem. I’ve read the referred article, followed a lot of links, downloaded a bunch of modules and code to study and it almost makes some sense. I just have to work up the nerve to start cutting code and test <bang!> whoops, debug <boom!> and rewrite it <crash!>.

This is about the time at which you discover that debugging a sound driver is almost impossible. Your toolbox is largely limited to one simple test:

1. Run program.
2. Did your computer freeze or otherwise bugger itself into an unrecoverable state?
3. If Yes: Your code is bugged. Stare hard at assembly until no longer bugged.
4. If No: Congratulations, your program is complete!

Debugging sound drivers is a bit tricky, but it’s not entirely impossible.

If you’re writing a sound driver which is running in application space, then the key thing is making sure you’ve got an atexit handler (or similar) which can deregister the driver if the app crashes. Assuming you’re dealing with C, I’ve found that different approaches are needed depending on whether you’re linking to unixlib or the shared C library – see the arcem sources for a fairly bullet-proof way of registering the required error handlers.

If your driver is running in a module then your best defence is to add a flag to the drivers buffer fill routine. Set the flag on entry and clear it on exit. If the flag is set on entry, then it probably means the last time the driver was called it crashed – so to avoid crashing again, exit immediately without clearing the flag.

That way you’ll only get one crash, instead of 100 crashes per second as the OS keeps trying to run your dead/broken buffer fill code.

If you’re writing a sound driver which is running in application space

…then you’re doing it wrong. (Fixed it for you)

Update: it looks like modules already exist to do what I need. John Duffell wrote SharedSoundBuffer and StreamManager about 10 years ago

OK, I’m trying to make sense of all the interesting little complications of this sound system and, with the hope that people won’t get bored silly and ignore the thread, I’m going to put my notes here along with questions. Maybe with some luck we can build a thread with long term value for anyone else trying to do RISC OS sounds. Besides, I find that trying to explain something to other people is a good way to end up understanding it myself, since I have to think hard.

The first thing I’ll try is explaining the capabilities needed for Squeak’s sound process, because I sort-of understand that; I will note that it is not very well documented. I’ll ignore the sound recording stuff for now since I have no idea if we can do that in RISC OS. Please tell me if we can.

Squeak Sound

Squeak’s sound system was originally written in 1995 to provide sound output for the system then running on a 100MHz PPC 601 powerMac. It hasn’t changed a lot. Given the very poor performance of that ancient machine there should be no problem with getting any post-StrongARM RISC OS machine to handle the workload; in fact back around then (late ’96) Squeak ran faster on my RPC-SA than it did on the powerMac 9500s.
The basic process is -

• setup the sounds system for sample rate, stereo flag, etc
• repeat
  • is there space in the buffer?
    • if so, write some samples to it
    • if not, wait a moment

There are also functions intended to stop the playing, change the volume, etc. Nothing very sophisticated is used at this level; all the sample creation, mixing of multiple voices, reverb, whatever is done in Squeak code running in another thread¹. The nearest to sneaky I can see is the expectation of being able to set independent left and right volume levels.

The expectation is that we can write about 1/8th sec long sample buffers for 22050 sample/sec 16bit per side stereo sounds (i.e. 2646 32-bit words). The Squeak code is perfectly happy to deal with the low-level code having larger or smaller buffers but I imagine larger is better for general performance? Nominally the sample rate should be settable but I don’t see much use of anything other than 22050 in use and likewise the stereo flag is always ‘true’.

Old Mac sound system

This is my understanding of how the old Mac ‘Sound Manager’ used to do things. It seems relatively sensible to me.

It creates a pair of local buffers to use just like double-buffering a screen, within a buffer structure that keeps records of the stereo flags, sample rate, buffer size etc. This structure is in application space, which I think we can’t do – it will need to be in a module space.
To query the available space (per above) it checks if one or other buffer is empty (i.e. has been used up) and if so returns the buffer size.
To write some samples it first makes sure one or other buffer is empty (you can’t rely completely on no-one having broken the Squeak code and this prevent buffer over-runs, an import feature of Smalltalk) and then

• makes sure the number of samples to write is the lesser of the number requested and the number of buffer spaces available, agin to aoid over-runs
• copies the sample words from the Squeak buffer object memory to the C code local buffer, transforming as needed – mostly if the stereo flag is ‘false, averaging the left & right samples. This is where one would do any bit-flipping if the Squeak values didn’t correspond with the sound system format.
• flags the buffer as being full
• returns the number of samples actually written, so the Squeak code knows what to do next.

Then the callbacks from the Mac sound manager happily run around in circles to

• decide which buffer was read from
• flag that it is now empty (and signal the Squeak system that there is empty space to fill up)
• fill the empty buffer with ‘silence’ in case it doesn’t get new samples in time for when it is being read by the sound system

I have to assume that the sound manager deals with deciding which buffer to play on the basis of some of the flags set. All in all this seems to make some sense and I can start to see a good mapping to how the SharedSound module appears to work.

Other sound systems

I’m not going to spend time studying the half-dozen unix and windows sound systems. I’m not a masochist. If anyone is interested and actually already knows about the unix ALSA/OSS/NAS/SUN, or the win32 systems, I can provide the code as an example.

What I Discovered About The SharedSound Module by tim rowledge aged 13 ¾. OK, 53 and a little teeny bit.

In his nice article on AcornArcade, Jeffrey Lee pointed out that there is a StrongHelp manual for SharedSound. This turns out to be a very nicely informative manual (hint to other authors) that doesn’t just list the SWIs but actually (gasp!) explains the system a bit. It even includes some example code!

If we can trust this documentation there is almost everything needed already in place. When handling the interrupt/callback one of the registers is set to point to a pointer to a pair of very useful routines that can copy your sample bytes (hopefully including doing any mixing and sample rater fixing) and fill up any unused space with silence. Thus the handler can be as conceptually simple as

• work out address for start and end of your buffer area
• call helpful fill routine
• maybe call helpful silence routine
• update your buffer pointers
• work out what value(s) to return to your main code and how

The left/right volume levels mentioned above appear to be handled in the helpful fill code but we’ll see how well that works when testing. All in all this is looking like it might even work.

A proposal for a module to make it simpler to makes noises

Since SharedSound needs to be able to call buffer fill code as an interrupt or callback we can’t (so far as I can see, feel free to tell me otherwise) keep the sound buffer for an application in application memory space; we can’t rely on it being paged in at any particular point. So I propose a (hopefully simple) module that will

• handle the initialisation, errors and shutdown of the interface to SharedSound
• provide & manage the buffers that SharedSound will read from
• provide such buffers for multiple clients so that either several applications can play sound or a single application can play several sounds or any mix of those two.
• pass user requested sample rates, stereo volume values etc to SharedSound
• provide only 16bit sample, stereo sound; applications can handle any stereo doubling etc
• provide some form of feedback to the applications that more samples are wanted

Questions:-

• where is the source for the SharedSound module? I found what seems like it in the ROOL CVS tree under bsd/RiscOS/Sources/Audio/SharedSnd but so far as I can tell it’s very old source and not for the v1.11 that *Help SharedSound reports.
• what methods of notifying applications do people like?
  • is this what the wimp poll word is for? I never did find an explanation of what its use is.
• how many clients would it be sensible to support concurrently?
  • I’d prefer to make it arbitrary if that is practical within the bounds of where I am allowed to allocate memory for buffers
  • nevertheless, I guess that 255 would be plenty in practice
  • does anyone know how many clients SharedSound can handle?

SAVE TO CHECK FORMATTING AND GET SLEEP

¹ I guess at this point I should mention that Squeak implements its own threads within it ² . The sound process is one such thread and various other threads would be busy building the sound samples ready to pass to the sound process loop. This means that it is possible to implement the ‘wait a moment’ above as ‘wait for a semaphore to be signalled’ rather than any sort of busy-wait.

² And has done since around 1970, when unix was a glint in the eye of Dennis Ritchie et al

OK, I’m trying to make sense of all the interesting little complications of this sound system and, with the hope that people won’t get bored silly and ignore the thread, I’m going to put my notes here along with questions. Maybe with some luck we can build a thread with long term value for anyone else trying to do RISC OS sounds.

Or even put a summary into part of the FAQ…

Excellent point; or even assemble it all as a document, put it somewhere with other doc and point to it from the FAQ. Whatever it takes to have a clear explanation somewhere!

What I’d like to end up with – code wise – is a nice, simple module that other people can use as well as me. The DRenderer module might possibly be that but it’s just confusing me so far…

what methods of notifying applications do people like?
is this what the wimp poll word is for? I never did find an explanation of what its use is.

Yes, that’s what the poll word is for. As far as I remember (haven’t used this for ages) the application will tell the module the address of its poll word (in non paged memory), and when something happens (eg a buffer needs filling) the module writes a non zero value ot the poll word. The application is calling Wimp_Poll with bit 13 in the mask clear and thus gets paged in with event 13. The app clears the poll word and does whatever is required.

If the app sets bit 23 in the poll mask, the pollword event is delivered with higher priority than messages and redraw requests.

Of course, if the module is also a Wimp task, it can communicate with an application by sending normal Wimp messages. I’m not sure which method would be most appropriate for this instance.

I think for app-to-app communication, wimp messages are the best bet.

PollWords are more intended for back-ends to communicate with some sort of UI front-end. For example, my MoreKeys application. There is a module that intercepts specific key presses (Ctrl + Alt + key). Upon this occurring, a word is set which will trigger the front-end to pick up the data. I have a SWI to clear the pollword/data block (I feel quite strongly about apps messing around in the RMA – if it was up to me, I would force the RMA to be read only from user mode apps).

However, I do not think that approach is sensible for the majority of app-to-app communication. For those, Wimp_SendMessage is your best bet – plus there is the ability to have replies, acknowledgments, and so on.

where is the source for the SharedSound module? I found what seems like it in the ROOL CVS tree under bsd/RiscOS/Sources/Audio/SharedSnd but so far as I can tell it’s very old source and not for the v1.11 that *Help SharedSound reports.

That’s the correct source code. The source code probably looks old because it’s an old module, much like the rest of the OS! If you look at the latest version of the VersionNum file you can see that it’s the sources for v1.11 (ignore the CVS revision identifiers, either look at VersionNum to work out what version the sources are or look at the CVS tags – e.g. SharedSnd-1_11 for 1.11)

Thanks Rik & Rick (ah the joys of multiple people with similar names. I used to work at a company where we had 3 Richards. Fortunately they were used to going variously by Rick, Rich and Dick, so we dodged the problem ) for that info. Sounds like a poll word of some sort is what I’ll have to use here.

Jeffrey, glad to know I was looking in the right place if not through the right set of spectacles…

Well this is interesting; while searching for any code that might illustrate how SharedSound could be best used I discovered John Duffell’s SharedSoundBuffer (and later his related StreamManager).
So far as I can tell it is exactly what I was in need of. However, the waters are bit muddied by what seems to be incomplete and confusing documentation combined with different versions being scattered across the web. On what seems to be John’s site there is an old and non-32bit package including sources. On the site for the RDPClient app there is a zip package with 32bit clean modules for SharedSoundBuffer and what seems to be some of the functionality factored out to StreamManager. The doc included looks great until you notice that the SWI base for both modules is shown as the same number… Sigh.
The good news is that the source for the RDPClient is decently written and appears to show a lot of what I need. And I don’t seem to be faced with building a moderately complex module of my own, even though I got about half-way through writing it.

There really ought to be a better way of finding out stuff like this with less effort. I’d certainly propose that modules like this ought to be at least referred to in doc on a site like this, if not included in the RISC OS package.

Sounds come out of the speaker!
Good grief.
What’s more, they’re almost the right sounds at almost the right speed. I have more work to do to get it actually correct, but so far sharedsoundbuffer/streammanager is Doing It Right.
Does anyone know about this pair of modules? The doc is annoyingly almost-right, which means wrong in important places.

Woohoo! Decent sounds coming out of the speaker. I am amazed.

Now how about sound in?

Shouldn’t be a problem learning to use Scratch; it’s an attempt to allow a Lego type experience that produces running code. You assemble elements, set values, attach actions. Some fairly complex programs can be produce but it is clearly not anything like typing out cryptic words in emacs – and it’s carefully designed not to be.

Squeak is a dialect of Smalltalk-80; possibly the most widely spread. It’s a direct descendant of the code that the folk at Xerox PARC wrote the original WIMP in. Everything is an object, there’s none of the half-hearted crap that makes java so utterly nasty. You never have to worry about memory allocation or freeing. As my old CEO used to explain it “ask, don’t grope” is the rule – unlike C-type languages you do not have pointers to memory locations that you use to rape and pillage your way through memory, stealing and corrupting bits. You politely send a message to an object to ask it to do something for you and get back an answer.

Quick, horribly over-simplified example:-
You are working on a city traffic simulation. There is already a load of code that simulates vehicles and their behaviour – parking, moving off, turning, stopping at traffic lights, etc. Your job is to add Taxis. A plausible way to do that is to create a subclass of ‘automobile’ – let’s be inventive and call it ‘taxi’ – that adds any new abilities for taxis and changes any behaviour that differs. So, you add picking up a passenger as a new ability, along with charging fares, providing annoying irrelevant political opinions and choice of fake cockney accent. But you also need to over-ride the ‘stop at traffic lights’ so that the taxi can just ignore red lights.

There’s a lot of educational and tutorial material on the web, though I’d be hard pressed to suggest any good stuff because I’ve been doing this for 30 years and really can’t remember what it’s like to not know it!

unlike C-type languages you do not have pointers to memory locations that you use to rape and pillage your way through memory, stealing and corrupting bits.

As an person not afraid of delving into assembler, I quite understand the C approach – even if it seems to you more akin to raping and pillaging. C is hybrid between an abstracted assembler crossed with some of the useful stuff of a high level language. C is, essentially, a sort of code for people who want to get close to the machine without having to actually learn machine code. While it would be nice to have some sort of built in protections (array overruns, etc), the danger is the sort of thing that is prone to being sanitised may be the sort of thing that works best. A good example is pointers-to-functions (and calling said function by its pointer and replacing the pointer during runtime). That’s something very prone to blow up in a person’s face – but if they understand what is going on, it can be a powerful ally.

I poked around the SVN repo and noted with some amusement that Squeak’s VM is written in C.
This is exactly what I mean – you cannot compare C and Squeak; both have their places. C is what makes Squeak work (except the Java port for Android, but Java is sort of bastardised C dontcha think? ☺ ); and likewise C is not a safe language to inflict upon beginners.

FWIW, the my uncle has written a few books on C++ and OO principles. I read one that he gave me way-back-when cover to cover and I just don’t see what all the fuss is. Constructors, friends, horrible horrible punctuation abuse… there seems to be nothing much in C++ that you can’t go in plain old C if you take the time to think about your implementation. Possibly the only real difference is to define a set of functions and then create multiple “instances” of this thing. Though, to be fair, this isn’t something I have found a need for in C. I suspect that C++’s ability to attach functions to objects sort of required this to happen, where I prefer to think of things as being either data (variables, arrays) or code (functions)…not a mixture all piled in together.
My liking of assembler potentially dictates my preferences for programming languages. Some safety is a nice addition, however bloodymindedly refusing to allow you to assign one sort of variable (say a float) to another (say, integer) without special rituals is annoying. Worse yet is stuffing it all together into this heinous can-be-everything (VBs “variant”, anything in php…) which is extremely disturbing to a balanced geek karma.
I’m afraid that it seems to me that the more abstract a language is, the more baggage it thrusts upon a person. For instance, VB which piles in around three and a half megabytes compressed worth of libraries and such. On my website I offer this as a separate download as it (the same thing) is required each and every time, but it is a magnitude larger than anything I have written using VB.

Hard to stay awake but too early to fall asleep. Damn. So I’m going to go watch… <rummage> “Shaolin Girl” or something. Looks sufficiently lots-of-action-but-mindless that I don’t think I’ll conk out just yet… ;-) If that doesn’t work, I have a Steven Segal (sp?) film which will be even more high-octane drivel but at least there won’t be subtitles to try to read
[t’was the “everything for a euro AND FIFTY PERCENT OFF!” at the supermarket; even a dumb Segal-beats-up-everybody-but-not-in-a-cool-Chuck-Norris-kind-of-way-because-he-takes-this-all-very-seriously film looks attractive when you can get three for the price of a coffee in a bar/café…]

Keep telling yourself that. The “C Programming Language” book itself describes C as a “mid-level language”. Personally, I don’t think that any language which supports pointers can reasonably call itself an HLL.

I will grant you that it attempts to abstract you from the hardware, though. This is more often a curse than a blessing, especially where such things as integer width, order of evaluation, etc. are concerned. I’d have said that even if I hadn’t spent a sizeable proportion of my Wednesday writing C to run on an Arduino, where all manner of unpleasantness is performed by the AVR libraries in order to work around C’s obsession with hardware abstraction and thus allowing you to actually do stuff like, say, service interrupts…

(And BBC BASIC, precisely because it’s so archaic and thus requires you to do lots of things for yourself, is possibly one of the lowest-level languages I’ve used…)