USB enpoint reads

At present all endpoint reads are presented to the user as a byte stream. Some applications need to know when individual transfers are complete and doing that from application space is a bit tricky.

So I propose a special field option ‘bytecount’ which will insert a bytecount for every insertion into the buffer that way if there have been multiple insertions before your program can read the buffer you can reconstruct each packet delivered.

Any thoughts? – other than it should be ‘endpoint reads’. The bottom of the edit page is wrong apparently I should worry about mistakes :-)

Problem – when I request 256 bytes for MIDI data – I receive a number of MIDI packets with padding between. How, then, would this be handled with a byte count.

It sounds like what is needed is a buffer between the USB and DeviceFS; better perhaps than botching transfers so there is no zero data returned…

If you set ‘nopad’ you won’t get any padding bytes inserted by usbmodule. If you need to know what each packet was you’d read the bytecount from the stream and then the relevent data.

a noblock type option not returning 0 would not allow you to multitask when nothing is received. You could use ‘sleep’ in a taskwindow, instead of ‘noblock’, but if ‘nopad’ is set it would only return when the requested number of bytes had been received. If the device sends a short packet you’d be waiting for the 2nd packet before it returned.

My solution to this, in the FTDI and PTP/MTP applications, has been to request lots of bytes, and then read them out of the buffer in dribs and drabs. But I would like to have a simpler interface – I don’t want to have to look whether a transfer is already in progress, whether there are bytes in the buffer, etc. It begins to make the Simtec interface look simple.

We need to look at what level of protection and recovery we want to cover against loss of sync – by which I mean you read what you think is the length, but actually it’s part of the data stream because you got lost.

re the ftdi and mtp applications these are 2 different problems. ftdi requires you to know when a packet is delivered and how big it is. This problem would be solved by the proposed change. You’d be able to read the bytecount then read the packet size.

The problem for the mtp application – well both really – is that you can’t read small numbers of bytes without running into the IOERROR error. If you could then you could then read the size bytes and then a packet of bytes.

Dave give us an idea of your ideal interface then we can see how far away we are.

Dave give us an idea of your ideal interface then we can see how far away we are.

Touché!

Turning what we’ve said on its head: the present interface would be fine if:

• we ask for a number of bytes, and it returns the minimum of that number and the remaining characters from one USB transfer;

• on return, it tells us how many more are available, and whether the transfer has ended;

• USB transfers are queued internally to the USB stack, maintaining message boundaries.

It’s only input that we have trouble with.

For speed, it would be good if transfers were requested automatically, as soon as the pipe is opened.

Open to the floor for better ideas!

I don’t understand why you are bothered about individual packets generally – as opposed to the ftdi device specifically. TCP sockets present ethernet packets as byte a stream and there’s no problem with that.

point 1/2. thats what os_gbpb does doesn’t it? If the input is a file either you are told how many bytes to expect (data is input with a bytecount) or you tell the other end how many bytes to send. So you ask for 10 bytes it gives you 6 you asked for 10 so there’s 4 more to come. If the input is a stream all you can do is keep asking for data and os_gbpb will tell you the bytes available at any moment in time won’t it?

point 3 If you want a stream of packets then that’s the same as putting a byte count for every packet received isn’t it? It effectively turns the input stream into a linked list of packets.

If you don’t want to poll for data an upcall handler on the buffer is the USB equivalent to the internet event.

Don’t you see things like that? Could it be simpler?

I realised it’s time to be more precise in my use of terms, so I went back to read some parts of the USB 2.0 specification. I got some surprises; my memory of what the specification says was wrong.

Section 5.8.3 (printed page number 53, PDF page 81) I found very illuminating, along with sections 5.3.2, 5.3.2.1 and 5.3.2.2 which define stream and message pipes.

Bulk pipes are stream pipes. A bulk transfer is complete when the endpoint does one of the following:

• Has transferred exactly the amount of data expected

• Transfers a packet with a payload size less than wMaxPacketSize or transfers a zero-length packet.

Note the last sentence of section 5.8.3: “If a data payload is received that is larger than expected, all pending bulk IRPs for that endpoint will be aborted/retired.” That may be the origin of the reason for “IO Error”.

So, if we don’t know in advance how many bytes will be coming in, perhaps we have to do one of:

• Request a transfer that is an exact multiple (from 1 upwards) of wMaxPacketSize for the device;

• Request a number bigger than the device could possibly give for the expected transfer.

I’m not bothered about individual packets in general. I have to stop talking about messages in the context of USB except in relation to control transfers, as only control pipes can be said to transfer messages.

FTDI may be a special case. I can’t quite get my head around how to treat it. Certainly the first two bytes of every packet (in the strict USB definition) are flags. It looks like we have to ask for one wMaxPacketSize transfer at a time, then we know the boundaries and therefore which bytes are the flags. (And if an input transfer results in 64 bytes, ask for another immediately, as the FTDI device’s buffer may be filling up with data that are arriving quickly.)

It is not possible, in the general sense, to tell the other end how many bytes to send. Some class protocols permit it (e.g. mass storage); others don’t (e.g. some PTP/MTP commands, and serial devices). It is not possible at bus level; there is no mechanism inherent in USB to do so.

My interpretation of it is this.

A stream consists of a sequence of IRP’s (Input output Request Packets). So you have

IRP1 – IRP2 – IRP3 and so on

travelling through the pipe. Each IRP consists of

Command packet – Data packets – Status packet

Its like an i2c transaction with a start-data-stop sequence but each byte in i2c is replaced by a packet in USB

The command packet contains the size of the transfer and the status packet marks the end.

Now section 5.8.3 says to me that whilst the command packet may request x bytes the device may deliver less than x bytes and when it does that marks the end of that IRP and the stream moves on to the next IRP. Should an IRP receive greater than x bytes then its an error and the stream of IRP’s are aborted.

It shouldn’t be the cause of IO ERROR because each IRP specifies how many bytes the device should send so there is no excuse for sending more.

OS_GBPB basically initiates an IRP the size requested though this will be split up if there isn’t enough room in the buffer.

The only situation that I see could work is this. If you have a buffer size=maxpacket use noblock,nopad and always read maxpacket you have the following situation.

1st read an IRP of maxpacket size is requested – nothing may be received
2nd read still waiting for input so nothing is received. But that size of the 2nd read request is queued.

In the mean time the data arrives making the freespace in the buffer<maxpacket. USBmodule then looks at the queued request and because there is <maxpacket freespace in the buffer the queued request is ignored. (If the situation was that the queued request wasn’t ignored you could get another IRP sent and received before getting round to reading the buffer).

3rd read you empty the buffer which is the contents of a single IRP

So OS_GBPB tells you how many bytes you received and the first 2 bytes are the status. Don’t try to read a byte at a time with os_gbpb because when the buffer empties the last read may queue a 1 byte IRP which you don’t want.

interrupts and isochronous work differently once started IRP’s for IN enpoints are sent metronomically at a specified interval if you don’t read them the buffer just fills up.