MidiMon 1.00, a USB MIDI app

would anyone like to try

Seems to work here except no timestamp.

At the top of c.module, you’ll see:

int  timestamp_type   = TIMESTAMP_MIDIBEAT; // default

Change that to be:

int  timestamp_type   = TIMESTAMP_FASTCLOCK; // test for JMB as MIDI seems to do this now?

Then, in c.midi for midi_init(), towards the bottom you’ll see the initial state reset where it resets:

timestamp_type = TIMESTAMP_MIDIBEAT;

Change that, also, to:

timestamp_type = TIMESTAMP_FASTCLOCK;

Then the timestamps will appear and work. ;)

Also in the midi init code, there’s another bug that I’ve just fixed just now. Setting bits 30 or 31 will turn the relevant option on, but clearing those bits didn’t turn it off.

So the relevant code is now:

      if (options & (1 << 30) )
      {
         // Bit 30 set: Flag that received Real Time Messages are to be passed to RX buffer.
         realtime_pass = TRUE;
      }
      else
      {
         realtime_pass = FALSE;
      }

      if (options & (1u << 31u) ) // specify UNsigned to avoid "signed constant overflow" warning
      {
         // Bit 31 set: Flag that received Real Time Messages are not to be treated specially.
         realtime_ignore = TRUE;
      }
      else
      {
         realtime_ignore = FALSE;
      }

(in other words, add the two “else” clauses)

I have an improved SimpleTest that now copes sensibly with SysEx data, as my keyboard spews some at startup, but this isn’t necessarily captured via USB if nothing is connected… so there’s a menu option to repeat it at will.

https://heyrick.eu/blog/files/simpletest2.zip

If you’re interested in what is received, it’s at https://heyrick.eu/blog/files/keyboardinit.txt but I suspect all of the SysEx stuff is Yamaha specific.

SYS “MIDI_FastClock”, -1 TO, a%

 PRINT a%

Try:

SYS "MIDI_FastClock", 1           : REM Turn on the fast clock
WAIT : WAIT : WAIT                : REM Waste some time 
SYS "MIDI_FastClock", -1 TO , a%  : REM Read the fast clock value
PRINT a%

On all current versions of the module except the one I made for you the other day, the default is to use the MIDI clock ticks (which is slightly broken, that’s a different issue), as the documents say that one should call the FastClock SWI to turn on the fast clock; however it appears (Peter’s module, Rhapsody’s expectation) that somewhere along the way this was changed to have FastClock be the default.

Note for testing Rhapsody… MIDI_Init will turn the fast clock off again, so you might want to pop FastClock,1 just after to ensure it’s always on.

@Rick
Just test (with Colin module version) before reading your post,
In Simpletest, I added the SYS line “MIDI_FastClock”, -1 TB, a%, and now I get the timestamp.

I will decode your post and do the tests…
I will recompile version 10 (June 24, 2023).
But I would like to know if Colin’s version of the module has any modifications on the USB side?

Yes changing the default timestamp worked colin2midi.zip.

Below are the changes – including yours – from your original source files that got my keyboard working.

I have one question having looked at the sources – probably been asked before if so just yawn and pat me on the head :-) Why the callbacks?

You are only processing 4-8 bytes per packet. Can’t you just handle them from the upcall vector instead of moving 38 bytes to the tickerV? The you would get realtime events at the midi vector.

The devicedead callback can’t that be done in the servicecall handler?

diff --git a/c/midi b/c/midi
index ea60c39..17b12ef 100755
--- a/c/midi
+++ b/c/midi
@@ -102,12 +102,20 @@ void midi_init(_kernel_swi_regs* ir)
          // Bit 30 set: Flag that received Real Time Messages are to be passed to RX buffer.
          realtime_pass = TRUE;
       }
+      else
+      {
+         realtime_pass = FALSE;
+      }
 
       if (options & (1u << 31u) ) // specify UNsigned to avoid "signed constant overflow" warning
       {
          // Bit 31 set: Flag that received Real Time Messages are not to be treated specially.
          realtime_ignore = TRUE;
       }
+      else
+      {
+         realtime_ignore = FALSE;
+      }
 
       if (options == 0)
       {
diff --git a/c/module b/c/module
index c1dfb86..0607d45 100755
--- a/c/module
+++ b/c/module
@@ -45,7 +45,7 @@ static int  device_connected = 0;      // address of new USB device
 static int  callback_none    = 0;      // number of times we've been in callback with NO data ready
 #endif
        int  callback_interval= 0;      // for *MIDIUSBDebug (etc), current callback delay
-       int  timestamp_type   = TIMESTAMP_MIDIBEAT; // default
+       int  timestamp_type   = TIMESTAMP_FASTCLOCK; // default
        int  midi_clock       = 0;      // Count of MIDI clock ticks
                                        // There is no Song Position Pointer, it is derived from MIDI clock
        int  realtime_pass    = FALSE;  // Now global
@@ -1227,6 +1227,7 @@ _kernel_oserror *module_veneer_ticker_handler(_kernel_swi_regs *ir, void *pw)
    int  port = 0;
    int  devno, cable, count;
    int  messtype = 0;
+   int expecting_sysex;
 
    UNUSED(pw);
    UNUSED(ir);
@@ -1274,18 +1275,26 @@ _kernel_oserror *module_veneer_ticker_handler(_kernel_swi_regs *ir, void *pw)
 #ifdef RXDEBUG
           if (size != 0) debug("%02X %02X %02X %02X\n",ptr[0],ptr[1],ptr[2],ptr[3]);
 #endif
-
          // Read the MIDI data block by block, catering for padding which may be present (yes, I know we specified nopadding but
          // we don't tend to get it (!) - parse out the pure MIDI from that and then work out when to poll for more.
-         while (size > 0)
+
+         /* The following expects 1 command per packet except for sysex commands */
+         expecting_sysex = 0;
+         while (size >= 4)
          {
             messtype  = ptr[0] & 0xF; // low message type, high midi cable number
+
+            // Can't just end on messtype 5,6 or 7 as messtype 4 is specified to mean 
+            // continue - whatever than means. Possibly multiple commands in 1 packet.
+            if (expecting_sysex && (messtype < 4 || messtype > 7)) break;
+            expecting_sysex = (messtype >= 4 && messtype <= 7);
+
             first = ptr[1];
             sec   = ptr[2];
             third = ptr[3];
             cable = (ptr[0] >> 4) & 3;  //          limit to 4 cables
             port = mididevice[devno].portno[cable];
-            count = usb_type_to_size[messtype & 15];
+            count = usb_type_to_size[messtype];
             ptr +=4 ; size -=4;
 
             if (messtype != 0 & port >= 0)
@@ -1313,7 +1322,8 @@ _kernel_oserror *module_veneer_ticker_handler(_kernel_swi_regs *ir, void *pw)
                }
                midi_rxinsert(port, count, first, sec, third, header.tick);
             }
-         } // while (size > 0)
+            if (!expecting_sysex) break;
+         } // while (size >= 4)
       } while (done == FALSE);
    }  // for (devno = 0

@Jean-Michel

@Lauren
What keyboard do you use?

I have two devices that work separately but I’m having trouble swapping between reliably:
1. A generic crappy USB MIDI Cable (Qinheng or whatever), which is connected to my window PC’s midiman midisport 2×2 and MIDI-OX.
2. An Arturia MicroFreak.

@Lauren

This will make you laugh. I wrote a line of basic to clear the buffers, got the system in a state where MidiMon wasn’t working, ran my line of basic, still didn’t work. Spent a while scratching head etc. Finaly realised I’d used,
SYS “MIDI_Init”,1
when it should have been
SYS “MIDI_Init”,2 (bit 1)
Ha Ha, you’ve not made the same mistake have you? If not I,m at a loss, as it springs into life when the buffers are cleared.

@Lauren
I cloned your repository to compile but despite your readme saying this

MidiMon uses the User Interface Toolbox and is packaged to be built with the ROOL DDE.

I note the sources don’t put the c and h files in their own folders so doesn’t compile with ROOL DDE – unless something has changed – is there a compiled version of your app?

Below are the changes – including yours – from your original source files that got my keyboard working.

Okay, I’ll have to try to work out what that all means to merge the changes in. ;)
Doesn’t look like too much to change.

What’s your family name, Colin? Be good to credit you properly…

probably been asked before if so just yawn and pat me on the head :-) Why the callbacks?

RISC OS’ fragility.

The devicedead callback can’t that be done in the servicecall handler?

Service Calls and such can happen when RISC OS is ‘threaded’, that means we’re butting in to do our work, because RISC OS wants to pass the service call around as it happens and not queue it for a better time.

As part of the device dead handling, we close the USB connection and such, and while we shouldn’t end up with any FileCore in use issues (as it’s a device, so shouldn’t get beyond FileSwitch), we either need to check everything to ensure it is properly reentrant, or we just fire off a callback and do whatever we like a few moments later when we know it’s safe. If the device is dead, it’s dead, a small delay between the service call and the callback isn’t going to make any difference.

Can’t you just handle them from the upcall vector instead of moving 38 bytes to the tickerV?

I didn’t write that bit of the code. However it associates the time at that point, so even if there’s a brief delay in getting the data processed, the timing will be correct and that’s what matters.

Upcalls, like Service Calls, happen when RISC OS is doing other stuff, so it’s best to do the processing in a callback to keep latency down. Which, granted, is probably less important on a 1GHz processor than an 8MHz one, but still… ;)

Okay, here’s the latest test build for playing with…

https://heyrick.eu/blog/files/midi_010c.zip 54½K

This archive includes:

• MIDI module
• Partial source (‘c’ and ‘cmhg’, the bits that have changed)
• SimpleTest (today’s better version)

I’ve also made a few tweaks so the MIDI beat clock should work now too.

Colin, as you can see at https://heyrick.eu/blog/files/keyboardinit.txt the SysEx stuff continues in subsequent packets. The spec is quite particular about stating that the packets are 32 bits in size. That said, I haven’t noted any issues with your changes. So if it works with more hardware than that’s a good step forward.

Thank you all for helping to improve RISC OS’ MIDI support.

The spec is quite particular about stating that the packets are 32 bits in size.

Yes the midi packet is specified as 32 bits but the USB packet isn’t. The max usbpacket size is 64 bytes – and for some reason my keyboards sends 38

As I understand it from the spec sysex is a sequence of a pair midi packets starting with command 4 and ending with 5,6,or 7 so as the normal midi spec shows the sysex commands coming in pairs I figured the could be sent in the same USB packet it is after all a single command in 2 midi packets. I was going to leave it at reading 1 or 2 usbpackets. Until I read that 4 could mean continue which implies extending a message further so I decided to cope with multiple sysex packets.

If only 4 bytes of a usb packet are ever used then why not make USB max packet size 4 bytes. As I see it there can only be 1 non sysex packet per USB packet because any value following it in the usb packet is a valid usb midi packet so you couldn’t tell when data ended and padding begins.

@Rick
just tested version module version 10c, it works, and the keyboard starts on the first try.
Lots of progress, thank you all, gotta go rehearse (Violin)!

Note:
Now there will be work on the Rhapsody side…

@Rick

Then again… If there can be more than 1 sysex packet in a usbpacket then you need to use sysex end (4,5 or 6) to determine the end of data and beginning of padding (which I didn’t do) as the word after the last sysex, which is padding, may be a random sysex command. As you are getting start and end in different usb packets you can’t look through the usb packet looking for end sysex packets. So I conclude you should change my mod to just read 4 bytes per usb packet. ie change

while (size >= 4)

to

if (size >= 4)

and remove the sysex stuff.

it works, and the keyboard starts on the first try.

Wahoo! 🤸🍾

and remove the sysex stuff.

They do make a simple thing difficult, don’t they? 😪

So I conclude you should change my mod to just read 4 bytes per usb packet

That’s not right.

A USB packet can be 64 bytes. Within this, we could technically have 16 MIDI commands. The first byte is always the packet information, followed by three bytes of MIDI data. In every USB packet there could be up to 48 bytes of MIDI information.

The original method that I used, which I’m sure others have fiddled with, is to look for a non-null byte and if one is found, read four bytes. Repeat until no more data.
This works because, SysEx aside, every valid MIDI packet has the non-zero initial byte ¹, and is exactly four bytes in length. So essentially I’m just skipping over unknown nulls (if they are present) because no valid command begins with a null, and every valid command is four bytes in length.

I think that’s the best approach to use. As we’re USB and that runs on upcalls and interrupts, we should always have valid commands, no partials to deal with.

there can only be 1 non sysex packet per USB packet because any value following it in the usb packet is a valid usb midi packet so you couldn’t tell when data ended and padding begins.

Like I said, the commands are always 32 bits in size, and the initial (CIN) byte is not zero, so there’s a way to tell.

Until I read that 4 could mean continue which implies extending a message further so I decided to cope with multiple sysex packets.

They are recycling the code.

If you look at the dump I linked to, you’ll see:

0     3      &F0, 67, 16    17965     System : SysEx start
0     3      &27, 48, 0     17965              SysEx data (continues)
0     3      &0, 8, 0       17965              SysEx data (continues)
0     2      &0, 247        17965              SysEx data (ends)

In this case it is receiving nine SysEx bytes, with an &F0 at one and and &F7 to terminate it.

This will likely appear in raw USB form as CIN 4 (SysEx start), 4 (continue), 4 (continue), 6 (ends with two bytes). I say likely, as the module strips the USB stuff to present regular MIDI, but I’d be surprised if it wasn’t done like that.

¹ There is a CIN = &0, but this is marked “Reserved for future extensions”, so we can ignore it.

Okay, here’s a compromise.

https://heyrick.eu/blog/files/midi_010d.zip

I’ve commented out the SysEx test, but it will still try to step through looking at each MIDI packet in turn, so should pick up on multiples in the raw USB data.

And giving it a quick test… holy crap… um…

Compare:

Before → https://heyrick.eu/blog/files/keyboardinit.txt

Now → https://heyrick.eu/blog/files/keyboardinit2.txt

Dunno what the hell all that startup initialisation stuff is doing (pitch bend!?), but there’s now easily twice as much.

So I think we’re on the right track here.

We’ll give this a bash over the weekend, and if it’s good (the response from Jean-Michel is encouraging!) then it can become a proper v0.11 release (so RMEnsure can do it’s thing!).

@Rick, Colin

Can I throw in my spanner here ?
Sysex’s:
My understanding is, Sysex MIDI event packets begin with 4,5,6,7.
4 = start or continuation, where continuations are the middle events with 3 data bytes, no top bits set.
5,6,7 = sysex ends with 1,2,3 byte(s).
I don’t think they are in pairs, you must have at least one event 4 and one of 5,6,7 but there can any number of 4’s in between to cater for any length of sysex.
The first event 4 will start with F0 then first 2 data bytes, there’s always at least 2 data bytes.
Any more event 4’s will just be 3 data bytes.
Event 5 is just the byte F7
Event 6 is (last data byte),F7
Event 7 is (last 2 bytes),F7
I don’t think this changes anything you’re doing but that’s how I think it’s meant to be formatted.

Rick, I’ve just seen your post above and the formatting agrees with this.

Rick, I’ve just seen your post above and the formatting agrees with this.

That’s all my understanding as well. You can tell the difference between a start and a continuation on the 4 by looking at whether the first MIDI byte is an &F0.

There is a CIN = &0, but this is marked “Reserved for future extensions”, so we can ignore it.

I don’t think so 0 is valid whether it is reserved or not. When they designed the interface they new that all command codes were valid and that all padding in a ‘midi’ packet is specified as 0 which means 0,0,0,0 is a valid midi command. Given that they new this they wouldn’t leave you guessing as to how the usb packet was packed with midi packets. Etherusb devices for example give details on how the ethernet packets are packed into the usb packet.

Now if normal packets could be packed into a usb packet you need to know how many there are or what value is used to signify the end. We know neither therefore we can only read one with confidence.

If we assume we can use sysex data (ends) to determine when to stop reading midi packets in the usb packet then you have to have specified that that usb packet will have the sys end or the usb packet is full of sysex data (continues) in which case the sysex data (ends) will appear in another packet. Your printout shows that you are receiving sys ends in different usb packets to the sys start and as you will have had enough room in the sys start usb packet we can conclude that the next 4 bytes after the sys start is padding not data and as such undefined.

As we can’t rely on the 4 bytes after the sys start being defined we can’t assume it is.

The situation I think is this. You have serial midi which is 31500 bits/sec iirc which is 4 bytes every millisec. You have class 1 usb delivering at best 1 packet every millisec It’s unlikely that midi is transfering solid data, more likely sporadic events so a slower serial midi throughput of 3bytes/ms translating to 4bytes/ms for usb seems workable.

Having 4 bytes per usbpacket simplifies things a lot – you only need to put 8 bytes (time and command) in your tickerv buffer and you always know at the tickerv end that the command is valid and there – makes it simpler still to process the command in the upcall.

Just my htoughts.

If you use reporter for debug output you can do a byte dump of the incoming usb packet – one of the reporter swis (use a 4byte wide dump). It would be interesting to see if any packet holds more than 4 bytes.

@Peter

Yes my ignorance was about how continue was used.

No problem, One thing I would add is that as a USB host we have to cater for a USB-DIN adapter which converts to 32.5k but we also have to cope with a USB device which doesn’t have that limitation, it only connects to a host and so may well send loads of data.

so may well send loads of data.

Yes it is able to but does the usb midi protocol say that it can – I don’t see anything explicit to say it can.

The Midi 1 spec only mentions 32.k DIN, I have ver 4.2.1 Feb 96, before the days of USB. The USB MIDI class makes no mention about it. I think the problem is that they don’t say it can or can’t so isn’t it safest to assume the worst case, 64 bytes per packet and might even be more than 1 packet in a frame.

Ah I think I have it.

The Class-Specific MS Bulk Data Endpoint Descriptor (page 26 of the spec) specifies the number of jacks associated with the endpoint. So each jack gets 1 midi frame per usb packet. Up to 16 jacks possible, 64 byte USB packet 4bytes per jack. My keyboard has 1 jack so I get 1 midi packet per usb packet

I think that has consequences for transmitting I think you’ll need to send 64 byte chunks with a midiframe for each jack in the endpoint. The 64 byte chunks need inserting into the output buffer in 1 transfer so that it isn’t split by the driver into 2 or more transfers. ie you put part in the buffer the driver says thanks I’ll deal with that so it sends part of what you want delivered in 1 packet

@Peter

This will make you laugh. I wrote a line of basic to clear the buffers, got the system in a state where MidiMon wasn’t working, ran my line of basic, still didn’t work. Spent a while scratching head etc. Finaly realised I’d used,
SYS “MIDI_Init”,1
when it should have been
SYS “MIDI_Init”,2 (bit 1)
Ha Ha, you’ve not made the same mistake have you? If not I,m at a loss, as it springs into life when the buffers are cleared.

SO. Here’s what happened. The function I was calling to test this was actually using MIDI_Init,0 because on the other module no bits set is ‘reset everything’ and this function was intended to do just that. But setting bit 1 indeed clears the receive buffers and I’m then successfully getting input from my Microfreak. So I should be able to work this out then.

@Colin

I note the sources don’t put the c and h files in their own folders so doesn’t compile with ROOL DDE – unless something has changed – is there a compiled version of your app?

Because I’ve been working entirely in RISC OS, I failed to notice that if you clone it from another platform it won’t translate the file extensions back, as it does when you clone it on RISC OS. Is there a solution that will make this work for both so I don’t have to change my workflow? But also to answer your question the 1.00 build is here:
https://github.com/laurenrad/MidiMon/releases/tag/1.00