How to do unpredictable USB input

OS_BGet is just OS_GBPB reading 1 byte

That doesn’t surprise me.

It appears that the padding is a bodge to stop OS_GBPB blocking.

That is now well known, yes.

If the padding is 0’s and your data doesn’t include 0 then I suppose you could scan the bufferfull you’ve read until a 0 is found.

That precludes unescaped binary protocols over serial. I found one of those in use on a multimeter with USB interface that I bought from Maplin a few years back. 0×00 is a byte that can legitimately be sent as part of the data.

I can’t see it being fixed without removal of the bodge and non blocking file reads and writes

They can be turned off in software, which I’ve done. It doesn’t help. If I ask for 64 bytes and receive 4, I have no way to know whether they were two transfers of 2 bytes each or a single transfer of 4 bytes.

or a non blocking DeviceFS_CallDevice block read

Now that is an excellent idea. That really would be a useful addition to the USB API.

Its easier to buy a serial port with a chip where you can read the uart buffer size :-)

It would have to have a USB interface, and we’d need to know the protocol. Do you know of any candidates?

Yes I’m comming late to this USB lark and I’m probably not adding anything new :-)

I’m playing with one of these at the moment USB – TTL UART adaptor. Windows drivers are available from silabs .com. There’s also a nice application note specifying the usb interface CP2102 USB commands. I bought it partly to try writing a USB driver thats when I came across your discussions.

I’ve just found the document showing the nopad special field. Its a long shot but you could try the nopad special field and an OS_GBPB request for 0 bytes. This may result a fetch from the device but return even if the fetch is 0 as you asked for 0 bytes. Then reading the buffer size will tell you if the fill routine fetched any bytes. Presumably nopad means only available bytes are put into the buffer.

Emptying the buffer would result in the next read being a new fetch.

Thanks for pointing out the information on the SiLabs device, Colin. I’m going to get one. I would much prefer to get one with a proper 9-way D connector on it – but I haven’t yet found one. They all seem to have TTL level I/O, and a subset of the pins at that. Not a problem for me, just something that ends up with an untidy result.

I don’t think your logic for calling with 0 bytes holds water, I’m afraid. There is still the problem that bytes fetched might be from one or more than one USB transfer. There is no watertight way to discern the difference from the data content.

Stap me. That multimeter I mentioned above? It has a CP210x chip.

I think my multimeter app might receive some attention from me over the next few days.

Re. the FTDI chip: I wonder if the secret is to interrogate the number of bytes in the buffer prior to doing an OS_GBPB, and, if the number of bytes returned by OS_GBPB is greater, treat the data as being from two USB transfers?

Something else for me to try.

I doubt that there is a queue of transfer requests more than one deep, i.e. I expect that the number of outstanding transfers can only ever be either zero or one.

I’ve been ferreting around the sources for a day or so now and this is what I’ve found:

1. nopad and noblock in the special fields work. I can see no reason why they wouldn’t. Edit: they do work but only if you set them via DeviceFS_CallDevice and OS_Args 9.

2. USB Fetches are initiated by every OS_GBPB call and are completed via a callback system.

3. When USB packets are received they are put into the buffer during the callback – if the buffer is big enough. This includes short fetched packets without padding if nopad is set.

4. With noblock set OS_GBPB will return with R3 showing the number of bytes to read. The USB fetch part of the read will be ignored if there is already one in progress. So, for example, with an empty buffer OS_GBPB will return showing it fetched no bytes but those bytes will turn up in the buffer when the USB fetch calls back. Say you wanted to fetch 4 bytes from an empty buffer:
a) The OSGBPB call tries to start a USB fetch and succeeds if one isn’t already in progress.
b) OS_GBPB returns with r3=4
c) You could keep calling OS_GBPB until the USB packet arrives in the buffer but that may initiate another 4 byte USB fetch when the bufferfill happens – because USB may not be in progress when OS_GBPB finally fetches the 4 bytes – but then again it may, in which case another fetch isn’t initiated. I’d play safe and wait until bytes arrive in the buffer before calling OS_GBPB again.

You always want to OS_GBPB endpoint maxsize bytes to read a USB fetch but if that returns say 2 bytes you can’t read that 2 bytes from the buffer with OS_GBPB without OSBGB trying to initiate a 2 byte USB fetch. So you need another way to retrieve the bytes from the buffer. DeviceFS_CallDevice Transferinfo will tell you the USB status so you can check if the fetch failed.

You can read/write some special field values with

OS_ARGS
R0=9
R1=filehandle
R2=Address of 2 word block

Block!0
Bit 31 – write value
Bit30- read value
Bits 16-23 – 0xff
Bits 0-15 – reason code

Block!4 – Reason code value

Reason code: 1 – set/read if stream will block
Reason code value: 0 – blocking, 1 – non blocking

Reason code : 4 – sleeping IO calls – Upcall 6
Reason code value: 0 – don’t sleep, 1 sleep

Reason code : 5 – timeout if sleeping
Reason code value: sleep timeout in cs, 0 – sleep forever,

Or use
block/noblock
sleep/nosleep
timeout n
in the special fields string

nopad can be in the special field or read/written with DeviceFS_CallDevice get/set options – which should be implemented as an OS_ARGS reason code.

re the cp2102 device I pointed out. I didn’t look for one with a normal connector and I didn’t find one with a quick look, but it does have 2 rows of marked holes for access to the other control pins if you are soldering iron inclined. Other cp2102 devices I looked at didn’t have this.

Sorry for the long post.

Is this reliably uniform for OMAP, BCM-Pi-thingy, and Iyonix?

Running the module, blockdriver and my LPC1114 app on the Iyonix and BBxM behave the same. At a detail level, it appears that I see the above-mentioned issue of flag bytes sometimes being treated as data more often on the BBxM, but that’s not yet a reliable observation – and in any case it’s probably dependent on what other devices are connected and what other apps are running.

“noblock” does not appear to work on the Iyonix with RO 5.18. I can’t remember if it works on the BBxM. Colin says that noblock works, so it is probably version-dependent.

The same module, blockdriver and application simply stiff the Raspberry Pi with the OS version I have – which is a good few weeks old now. The mouse cursor stops moving, and there is no response to Alt-Break, although the NumLock LED responds to the NumLock key.

Colin: I wonder if OS_GBPB could be given an extra argument, a flag to make it return only data already in the buffer, without performing another USB read, if data are there? That would allow us to know the boundary between two USB messages.

@Rick

Don’t know really.

The problem with OS_GBPB doing a fetch even if the data is available in the buffer is in fs_gbpb in s.FSystem . It could be fixed (so you don’t need to work around it) by checking the buffer first. Alternatively it could be fixed just for USB in USBmodule.

The OS_ARGS are part of DeviceFS

The padding is in read_cb (the read callback) in c.usbmodule

@Dave
yes it could.

Personally I think fs_gbpb should be changed so that it checked available bytes before initiating a new fetch. It doesn’t seem right if I ask for 4 bytes and 3 are in the buffer that os_gbpb asks for 4 bytes from the device. I can’t see it breaking any existing device. But if courage fails you can always add a flag.

You could work around it at the moment by making an OS_GBPB call reading an endpoint sized block then removing direct from the buffermanager when bytes become available.

Coincidentally, I’ve just received this from Si Labs:

http://www.silabs.com/Support%20Documents/TechnicalDocs/CP2102-9.pdf

May be useful?

Your problem with bbxm may be one of timing. Bulk endpoint reads are read at lowest priority so any interrupt, isochronous, or control commands are handles first and bulk uses what bandwidth is left.

It may be that solving the block read problem doesn’t solve your problem.

1. Every read starts with 2 status bytes
2. If the delay is too long another 2 are put in the FTDI buffer
3. You now read the FTDI and you get 2 status bytes, some data, 2 status bytes, some data

So unless you can guarantee a USB read before more status bytes are added to the FTDI buffer you need to find a way to recognise them or switch them off.

Yes thats the little devil. There’s an app note for it detailing the USB commands

I’ve been trying DeviceFS_CallDevice% &80000002, which is documented to return the buffer fill level in R3 and free space in R4. When the input buffer is empty, what it actually returns on my Iyonix (RO 5.18, USBDriver 0.55) is R3 and R4 both 8191. Presumably one of them is the buffer length and the other the free space?

Yes it appears the documentation is wrong it returns total buffer size in R3 and free bytes in r4.

Yes it appears the documentation is wrong it returns total buffer size in R3 and free bytes in r4.

That figures. Thanks for the confirmation. Now to try to remember where I got the docs from, to see if I can make the correction.

Last night I tried adding a counter to see how many times the bulk IN buffer was non-empty prior to calling OS_GBPB. I tried both DeviceFS_CallDevice &80000002 using R3 – R4, and OS_Args 2, to find the fill level. The counter never moved from zero.

However, using my LPC1114 app to repeatedly read lots of data, there were so many occurrences of multiple transfers in the buffer that the app was unuseable – but only on the BBxM. So I can confirm my observation that the Iyonix and BBxM behave differently to some extent.

I still don’t see any way to get single transfers, or to discover the dividing line between two transfers.

My last hope is to try another of the DeviceFS_CallDevice functions, to wait while a transfer is in progress and not request another until it’s complete. I have no idea whether it’s actually going to do what I need. Structured trial and error needed again…

I’m refering to this USB Doc unfortunately it is in the source tree

Whoa. It’s working. It’s working.

The secret is to use DeviceFS_CallDevice to determine whether a transfer is in progress, and to use the buffer functions to remove bytes from the buffer without kicking off another transfer.

I’ve just used it to receive 32 kiB of data, UUencoded plus protocol overhead, from an LPC1114 chip, at 115200 baud, on the BBxM. No resends.

So unpredictable USB bulk input can be done; it’s just a somewhat tortuous route.

So unpredictable USB bulk input can be done; it’s just a somewhat tortuous route.

Brilliant stuff! I’m going to have to go back and read all this stuff carefully to see if I can get MIDI running reliably and without the byte-stuffed bull or the Beagle fails.

I take it the open-a-file-then-OS_BGET-stuff route is dead in the water. ;-) Well, of course it is, programming wouldn’t be fun if it was that simple!

[edit: three times I said “stuff” in a two-short-paragraph message; this is bordering on being Buffy Speak !]

@Dave

So can you confirm that this is the situation. Assuming an endpoint packet size of 64 bytes:

1. Open the bulk IN endpoint with nopad, noblock in the special field
2. Call OS_GBPB to read 64 bytes
3. Wait until the usb transaction is complete
4. Empty buffer NOT using OS_GBPB
5. Go to 2.

If so, can you test something for me.

I don’t think the size fetched in OS_GBPB matters what matters is that you know that the transaction is complete before calling it again. By starting the next transaction after emptying the buffer you know where the next transaction starts.

So could you try out different fetch sizes even something like 8000 bytes. This could be important as fetching a packet at a time is slower – probably doesn’t matter for your application but it would be interesting to know if the transaction terminated after a short fetch.

Does something larger than the buffer work – say 16000 bytes. In that situation you could read from the buffer NOT using OS_GBPB while waiting for the transaction to complete. Your program is probably not the best to test this.

If it worked that way you should be able to fetch a stream of structured blocks of any size.

It may be the case that large transfers are seen by riscos as multiple smaller transactions – haven’t been able to determine that yet.

Edit. For completeness I should note that OS_BGET suffers the same problems as OS_GBPB so can’t be used to empty the buffer.

@Colin

The situation is almost as you describe; the only difference is that noblock doesn’t work for me; I have to get the same effect using calls to OS_Args to do ioctl functions. I think noblock must be dependent on the version of USBDriver.

As for the transfer sizes: I think we know the answer already. For the FTDI device, I always request 64 bytes because that’s the maximum number of bytes it will ever transfer; but most transfers are less than that, the most common number being 2, i.e. just the flags. After all, at the level of the bus, we always know when a transfer is complete because a short packet is sent – the length can be zero if the transferred data require an exact multiple of the maximum packet size.

Emptying the buffer without initiating another transfer is done by using the Buffer Manager functions. The buffer handle is returned by either the Return Handles or Return Handles 2 call to DeviceFS_CallDevice after opening the bulk in endpoint. The rest of the buffer parameters are obtained by calling Buffer_InternalInfo (PRM 5a-215 onwards).

One other thing to add: although yesterday evening’s code works fine on the Iyonix and BBxM, it still stiffs the Raspberry Pi when initialising the blockdriver. It’s not the most recent version of RISC OS, it’s from 2013 Feb 10 which is only a couple of months old.

noblock and the OS_Args equivalent both set the same flags in DeviceFS and are independant of the device so if noblock in the special string doesn’t work then that’s a bug. USB doesn’t use OS_Args – if it did the interface could have avoided exposing some internal handles.

If you say that 64 bytes is the maximum transfer because that is the size of the endpoint packet size then that may not be true. As I understand it what is important is the transaction size not the packet size and a transaction can be multiple packets. So if, for example the serial port chip had a 256 byte buffer and it was full you could fetch it in a single request – it would arrive in the buffer in endpoint packet sized chunks. If the documentation says its 64 bytes then that’s another matter.

If I’m right then it shouldn’t matter whether you request 3 or 3000 bytes it should still work. I was just curious.