How to do unpredictable USB input

Dave are you having problems with the data from an interrupt endpoint? The reason I ask is that I’ve discovered a few ‘traps’

1. Although the endpoint size may be maxpacket the data sent by the device may be less.
2. interrupt endpoints never pad so if noblock isn’t set OS_GBPB may need more than one interrupt to happen before it returns.
3. interrupt endpoint data is fetched at regular intervals in the background if the buffer is full you lose the interrupt data. As the buffer is being read in the foreground this is a likely occurence so the buffer size needs to be large to catch all the interrupt data.

As an example The CH340 serial device has an interrupt endpoint with a maxpacketsize=8. It returns a 4 byte status packet. If I set the buffer size to maxpacketsize (the minimum allowed and the ‘transfer’ size always used by interrupt endpoints – size8 in this case) reading 8 bytes I sometimes get 4 bytes, sometimes 8 bytes. if I set the buffer size to 64 -and read 64 bytes I’ve had up to 56 bytes returned.

Its obvious why this happens when you think about it in my case I’m reading the buffer in a taskwindow and a lot of interrupts can happen before I can read the data. So to catch all of the interrupt data you need to set a large enough buffer to buffer the interrupt data for you to read later.

Dave are you having problems with the data from an interrupt endpoint?

Not in my PTP/MTP app (my current area of interest or obsession, depending on your point of view!), but there is very little traffic through it, so it is about as low stress on the stack as you can get. It simply means that I wouldn’t know any such problem because it wouldn’t arise.

Under normal circumstances, the only traffic you get is one message when a new photo or video is taken, or one deleted.

Clearly the low traffic/stress level is not always the case with other types of device.

But thanks for the information. I’ll bear it in mind.

We must be open to the same problem again – if messages are queued, RISC OS throws away the message boundaries, leaving the programmer to rely on the structure and content of the messages to be able to reconstruct the boundaries.

Yes the ch340 is low traffic as well – only notifying changes. It can go for ages without any traffic. I discovered the large transfers when making and breaking the handshake lines by hand – which is subject to intermittant connections but it shows that a lot more data can be transmitted quickly.

Regarding transfer boundaries. I’ve come to the conclusion that you can’t do them from a foreground program because a short packet can always be filled up in the background. If it didn’t do this it would slow things down.

If I was writing a module, where the transfer callback can be acted on directly (it’s always paged in in a module), I still wouldn’t want to use DeviceFS I’d want to write using the NetBSD usbdi interface like the keyboard and mouse drivers do – and DeviceFS for that matter.

Foreground apps are different though some means of buffering is required and DeviceFS is as good as any.

leaving the programmer to rely on the structure and content of the messages to be able to reconstruct the boundaries.

And this is when you find out how will thought-out the protocol itself is…

[case in point – MP4 format video, it seems that if the file is not correctly finalised, it is pretty much junk data – why can’t you reconstruct frame by frame from the beginning? (like you usually can with AVI and some types of FLV)]

I don’t see structure in the input stream being a problem if each block contains a byte count.

hmmm. just had a thought. Maybe a special field option to include a bytecount for every transfer would help.

Maybe a special field option to include a bytecount for every transfer would help.

That sounds to me like a good idea.

Start a new thread in “Code Review” to review the idea.

I’m not ready yet. I’ve a few bugs to fix in the start_read, read_cb functions to get DeviceFS doing what it says on the tin and of course there’s the holy grail – why do you get IOERROR when trying to read 1 byte at a time.

Presumably the only device that requires this change that anyone knows about is FTDI usb-uart devices. Not sure what was wrong with the Ethernet devices – don’t ethernet packets have byte counts? You may be using Thomas Milius’s module by now but would the change have allowed you to use the FTDI device from a foreground program and more to the point, if I modified risc os with the change could you test it by writing a small program and show that it will be a worth while addition – I don’t have an ftdi device else I’d do it myself. If we can show it is a useful addition I can’t see it not being added. ROOL are reasonable aren’t they :-)

There’s only one real hurdle to overcome when adding it. Adding a byte count into the buffer before inserting the data may trigger a threshold upcall.

the holy grail – why do you get IOERROR when trying to read 1 byte at a time.

I think you get an IO Error when you get more bytes than you ask for. Ergo, if you only ask for 1 and the transfer is bigger (as it almost always is), you’re guaranteed to get an IO Error.

My solution to this, in the FTDI and PTP/MTP applications, has been to request lots of bytes, and then read them out of the buffer in dribs and drabs. But I would like to have a simpler interface – I don’t want to have to look whether a transfer is already in progress, whether there are bytes in the buffer, etc. It begins to make the Simtec interface look simple.

We need to look at what level of protection and recovery we want to cover against loss of sync – by which I mean you read what you think is the length, but actually it’s part of the data stream because you got lost. But I’m in danger of starting the review here.

I don’t see why we can’t review the idea at this stage. Implementation would come later, assuming we get consensus of what to implement.

Maybe a special field option to include a bytecount for every transfer would help.

Problem – when I request 256 bytes for MIDI data – I receive a number of MIDI packets with padding between. How, then, would this be handled with a byte count.

It sounds like what is needed is a buffer between the USB and DeviceFS; better perhaps than botching transfers so there is no zero data returned…

Questions in Code review

Dave. agree about IOERROR but it doesn’t always happen – I get it about 1 in every 5 times I read 1 byte.

Hmmm… I’ve been fixing bugs in usbmodule and testing the changes with your ptp program unfortunately when I fix one of the bugs ‘Getting data after GetNumObjects’ doesn’t return anything whereas leaving it in fetches some objects then fails.

The bug concerns the nopad flag. ie it should only be taken note of if there is a short transfer whereas at the moment it acts as though all transfers are short.

Don’t want you to do anything just having an Aaaargh!!! moment.

Good news is that whilst RISC OS 5.18 fails at the same point with IOERRORS, the bug fixed version just sits and waits with ‘transfer already in progress’. It may of course just mean that it’s not doing anything.

I’ve updated the app, which is now called MTP, because it supports some filetypes that are in MTP but not PTP and MTP is almost entirely a superset of PTP.

I have it working with a Samsung Galaxy S3, a Nikon Coolpix 3100 and an HP Photosmart M415. It doesn’t work with a Benq DC C800; as soon as the GetNumberOfObjects command goes through, the camera complains of “PRINTER ERROR”. Mind you, the USB protocol settings are “Computer” and “Printer”. Printer? What?

Anyway, this now crudely supports hierarchy. Please have a go; I think its chances of working are much more realistic now than ever before.

Because of the name change, note the new URL:

http://davehigton.me.uk/Progs/MTP.htm

Big improvement. With my fixed version of RISC OS I only get directories but with my RISC OS version without the nopad fix MTP seems to work properly.

My fixed version of RISC OS fails repeating the following.

Getting data after GetObjectHandles
Bulk in endpoint status (data) = 0
Read_bulk: tfrlen% = 560
Transfer in progress
FNread_bulk: 0 bytes waiting
FNread_bulk: Transfer Info returns -1; bytes received = 753
Read_bulk: r2% = 8E85C, buf% = 8E85C, r3% = 560, ext% = 0, tfrlen% = 560, r4% = 0; flags = 5
USB transfer failed: IO Error

does it mean anything that it says bytes received=753 when you only want 560?

Yippee! I found the problem with my code and MTP works correctly with my updated USBmodule now.

I’ve found a problem when downloading. mtplog.zip contains a logfile and a photo. When I download the photo it downloads with the correct size but the download process doesn’t complete. After downloading it I can’t fetch another photo – double clicking in the directory viewer does nothing. Downloading other jpegs works ok.

Yippee! I found the problem with my code and MTP works correctly with my updated USBmodule now.

:-)

mtplog.zip containes a logfile and a photo.

Logfile only, I’m afraid. I would like to see the last 12 bytes of the photo.

The logfile does show that the photo’s expected number of bytes were transferred (exactly that number), but the app waits for ever for the status transfer (12 bytes) that should follow the data. This is one of those things that seems to just happen with the RO5 USB stack, sometimes. It happens to me too. I’m sure (in a vague sense) that it has been mentioned in one of these fora before.

Yes I discovered the problem was intermittant and happened with other files so didn’t bother with it. It wasn’t corrupted. It matched the version downloaded via mass storage. note the log says 2409956 bytes but the file was only 2409944 bytes so presumably there was a 12 byte header. The file had been filetyped to a jpeg so presumably you do that when the download is complete as it has a data filetype while downloading.

presumably there was a 12 byte header.

That’s correct.

I was interested to see if the last 12 bytes were actually the expected status bytes. When it happens, I always look, but it has never happened. The chance was vanishingly small in this case because the number transferred was exactly the number expected.

So you’re confirming that you sometimes see this problem of intermittent packet loss?

I’ve written about it before. When I was working on the Bluetooth experimental stuff, I saw on the USB analyser that an input packet had been transferred, but was not apparently available to the calling application.

So you’re confirming that you sometimes see this problem of intermittent packet loss?

Don’t know the reason for the stall but I can confirm that sometimes I use it and nothing seems wrong for a while then all of a sudden it starts failing. I think it’s hard to blame packet loss when it always fails at the same point. control and bulk out transfers don’t seem to have problems. Directory fetch seems solid, it never stops in the middle of a download. If you are reading once and waiting for the reply it may be that the read didn’t happen.

After a brief glance at the OHCI spec. If the Bluetooth data is being sent to the host then the host requested it, when it was ready to, packetised it for an OUT endpoint and handled the storing of it for an IN endpoint all internal to the host device. The Host controller is fed requests to send and receive data in up to 8KB blocks. Transfers need to pass a CRC check – but you would have expected an error in that case. More reading to do methinks.

If you are starting a fetch after the download it may be worth trying a small time delay.

@Dave
!MTP is working with my usb2 Minolta 7D without errors on RasPi, gets so far on the Iyonix but generally errors out.
.MRW (minolta raw file) isn’t recognised (-1) but jpegs are downloadable.
Thanks

.MRW (minolta raw file) isn’t recognised (-1) but jpegs are downloadable.

The raw file is downloadable too, isn’t it? It should download OK but arrive as a data file.

MTP/PTP only recognise a few file types. PDF, for example, is not among them. I have to put in place a second level mechanism to offer a filetype based on the filename’s extension part. It’s on my to do list, but at this early stage there are higher priorities.

Thanks for the feedback!

I don’t know whether this is good news or not.

If I make an endpoint with ‘nopad’ set behave the same way as an endpoint without ‘nopad’ set – except that it doesn’t pad – then MTP stalls occasionally. If I leave it broke, if you don’t have ‘noblock’ set but have ‘nopad’ set it won’t return if the read gets split into more than 1 transfer.

The broke version of USB which works for MTP is obviously wrong in a DeviceFS context.

But at least we know where the problem is.

Colin, my friend: I’m sure you know what you’re talking about, but I haven’t a clue! :-)

But, let me tell you: I’m very glad you’re working on it.

With the work we’re doing, in our different ways, I’m certain that RISC OS will be much improved.

You clearly have an overview of the USB stack, and the various code files of which it is comprised. I, for one, would very much appreciate it if you would document it on the ROOL site. I was never able to form an overview by reading the code. Once somebody starts, it will be possible for others to add to it.

:-). I know while I’m writing it that it looks absolute rubbish but writing it down helps clarify things in my own head.

It does concern me that people need to ‘work around’ the DeviceFS interface and its bugs as fixing them causes programs to fail as is the case with MTP.

It amuses me that given my attempts to explain things comes out as absolute drivel (I freely admit that), how would my documenting it help.

All the bugs I’ve fixed so far I would consider nothing to do with USB. They are all in DeviceFS and the USB interface to DeviceFS. The standard BSD interface is alive and kicking under the weight of DeviceFS and is documented on the net.

I do know where the IOERROR is triggered though – getting nearer.

Ronald. On your iyonix does *USBdevices show your camera has the same bus number as the EHCI root hub – which you would expect for a USB 2 device. My camera is connected to the OHCI hub.