How to do unpredictable USB input

We should come up with a system that works,

Yes.

is documented correctly,

Yes.

and doesn’t break past code that can’tbe fixed(abandonware).

Maybe. As per your example re ignoring R3, we need also to understand that broken code is broken, no matter what its provenance (FilerAction, I’m looking at you). I have seen some absolutely awful bodges where the “system” was expected to be mutilated to support some important-but-ancient piece of crap that should have been rewritten long before…

As I see it the transaction can end early but the file isn’t at EOF. eg an endpoint has a 64 byte packet size you os_gbpb 93 bytes the 2nd packet delivered will complete the 93 bytes requested but that isn’t eof its the end of the transaction. If this were not the case streaming would require an open and close on every read.

I expect EOF to become true at the end of a USB transfer, which occurs when the device sends a packet of less than the maximum packet length (the last packet is of zero length if the transfer is an exact multiple of the maximum packet size). Remember that, although you can ask OS_GBPB to transfer a specific number of bytes, you can’t ask this of USB [unless a higher level protocol supplies a method]. In your example, if the device only had 93 bytes to transfer, it would send a 64 byte packet followed by a 29 byte packet. If the device had more bytes to transfer, the second packet would be longer than 29 bytes, in which case I would expect EOF to be false after OS_GBPB had given me the 93 bytes I asked for. What’s more, the device would continue to stream available bytes out when requested by the host, and only stop doing so when no more bytes are available – again, sending a less-than-max-packet as the last packet.

EOF = end of USB transfer. It’s important to understand exactly what a USB transfer is. It may or may not happen to be related to the number of bytes you request via OS_GBPB.

EOF doesn’t require a close followed by an open. If you do something to the device that makes more data available, EOF should go from true to false. You can then continue to request them through the same stream.

My interpretation of the USB spec (chapter5.3.2)

“A client can expect a variable-sized amount of data in an IRP. In this case, a short packet that does not
fill an IRP data buffer can be used simply as an in-band delimiter to indicate “end of unit of data.” The
IRP should be retired without error and the Host Controller should advance to the next IRP.”

which follows how I interpret castles’ implementation. But I’m doing a lot of interpreting, I’m no lawyer and thats one heavy read.

Anyway I managed to get some info from usbmodule regarding special fields and discovered the following.

When I open a device with

“devices#short;nopad;alternate=0;report=0;usbtimeout=5000;bulk;size=4096;endpoint=%d;interface=%d:%s”

The validation block passed to device_initialise is

('c.usbmodule'  func: 'device_initialise'  line: 2095) 
endpoint: 2
interface: 0
alternate: deaddead
report:    0
ep_type:   2
timeout:   deaddead
size:      1000
nopad:     deaddead
shortflag: 0

which is weird. I’ve changed the order in the special field and it makes no difference. I also changed nopad to padoff in the rom but it made no difference. deaddead is hex value decode special initialises the special field block to.

Just in case someone still messing on with this here’s what I’ve found.

Decoding of special strings in devicefs is broken – tell us something we don’t know.

The only way to get all options to work is to put them in your special field in the same order that they appear in the validation string. (You can get some options to work out of order but adding another option can make them fail).

This means that for example the mass storage class which puts the alternate option first never actually sets the alternate value passed in the string. If you set size after nopad nopad isn’t set. Similarly interface isn’t set if you set interface before endpoint.

This affects everything that uses devicefs.

Seems to me noblock always works correctly in the special string – but the flag which is set from the special field value is set wrong.

Seems to me noblock always works correctly in the special string – but the flag which is set from the special field value is set wrong.

So essentially it is broken, and the bit that works is also broken? WT[beep]!?

Not any more.

I recently restarted work on my PTP application. I got some response from my Samsung phone last night, which is a big step forward. But here’s the tale of woe. (All RISC OS 5.18 on the Iyonix.)

I discovered that sending a Set Configuration 1 to the device stopped all communication from my app. It appears that sending it causes RISC OS to perform a few control transactions. Removing my Set Configuration allowed communication to proceed further. Maybe at some point I’ll try again, introducing a wait after the Set Configuration to allow the other transactions to complete and be out of the way before my app’s next one.

I have successfully set the bulk IN endpoint to no padding and no blocking.

Many of the responses are of an unpredictable length. The length becomes known after the first 12 or 16 bytes come in. Logically, then, I would ask for 12 or 16 bytes, and, after reading them, ask for the rest, knowing exactly how many are coming. But every time the device transfers more bytes than I ask for, the stack generates an IO error. I have to ask for the highest number of bytes I could possibly expect. I’m sure it shouldn’t generate an IO error under those circumstances.

I ask for the number of objects. The phone returns a value of 93. I then ask for their handles. The phone returns 108. You’ve probably guessed it; when I see a value of 93 objects, I know exactly how many bytes to ask for. But the device returns more, so… IO error again.

I tried asking for a too-high number of bytes. The phone returns all the handles, but then returns the status as part of the data transaction. It just happens that 108 objects returns a number of bytes that is an exact multiple of the maximum packet size: 108 * 4 plus 16 equals 7 * 64. The analyser shows no sign of a zero length packet after the data transfer; the status just looks like part of the same transfer.

Gordon Bennett, this is hard.

I think what you need to do is set noblock,nopad,size=maxpacket and always fetch maxpacket. That way every fetch you make will be the result of 1 packet full of data sent by the device. Note the received bytes are put in the devicefs buffer in a packet by packet basis – not byte by byte.

Packets are only fetched if there is enough space in the buffer for 1maxpacket full. So if the buffer is empty an GBPB fetch will fetch a packet – specifying a fetch maxpacket should/could make the device send all the data it has and r3 will tell you how much this is.

That’s the theory however the problem is because of bugs a fetch can result in 2 packets being read if the first packet is only 1 byte. I hope to have a fix for this soon.

In meantime I’d try the above with size=maxpacket+2.

In meantime I’d try the above with size=maxpacket+2.

I guess you probably mean maxpacket+1, but I’ll try both.

If I ask for 64 bytes and it transfers 384 bytes, that will still cause an error, won’t it?

And I think that, if the buffer is only 64 bytes, it might take a long time to transfer a photo or video.

There may be another trick: set the buffer to 32768 bytes or so, and always ask for that many. More experimentation coming up.

No I mean maxpacket+2 anything less won’t fetch anything.

It fetches as many whole packets as there is space for in the buffer and 2 bytes are lost to bugs. Its not quite the same as the fixed version but worth a try

The problem with reading is that the last read size is queued so if you

1. read maxpacket and OS_GBPB returns 0 bytes ie nothing has been received yet.
2. read 1 byte which returns nothing but it queues the 1 byte read
3. read 2 bytes which returns nothing but it replaces the 1 byte queued read with a 2 byte queued read

When OSGBPB finally receives the data from step 1 it sets off another fetch (the one queued) if there is enough room in the buffer to do so ie >= 1 packetsize of space.

At this point you finally read data, queue another fetch but the previous queued fetch is already in progress.

The only way to stop the queued fetch happening is to have a buffer the same size as maxpacket.

If it is just a matter of fetching a file I would just fetch the whole file size. Buffer size shouldn’t matter

If fetching in blocks try making the buffersize maxpacket*n+2 and always read that amount

It must work somewhere Memorysticks use DeviceFS.

Are these all bulk endpoints you are dealling with?

No I mean maxpacket+2 anything less won’t fetch anything.

Interesting. Thanks. Although so far asking for maxpacket has worked. Maybe asking for maxpacket+2 will work better.

If fetching in blocks try making the buffersize maxpacket*n+2 and always read that amount

Worth a try, although some transactions are small (e.g. 12 bytes for status, 16 bytes for GetNumObjects, maybe only tens to a few hundred for GetHandles), whereas getting a photo or video is normally going to be megabytes.

Its buffersize that needs to be at least maxpacket+2. If the amount delivered by the device is asked for via the control endpoint then maybe if you ask for 12 via control and fetch for example 1000 via bulk IN the fetch is retired after you get the number requested ie keep trying to fetch 1000 until the bytes recieved = 12. Alternatively keep trying to fetch maxpacket

Earlier I meant to say make buffer size maxpacket*n+2 but read maxpacket*n

I have tried you ptp program but it hangs after opening a window. Disconnecting the camera aborts the hang.

Are these all bulk endpoints you are dealling with?

Yes. (There is also an interrupt IN endpoint, but that takes little or no part in ordinary proceedings. Not a problem – not yet, at least.)

I have tried you ptp program but it hangs after opening a window. Disconnecting the camera aborts the hang.

Yes, it’s not going to work on many cameras or phones right now. That’s why I’m working on it (banging my head against walls, largely) again. With some devices, unplugging it doesn’t even abort the hang.

When I have something more promising, I’ll update the version on my web site. There’s no point in doing so now while it’s work in progress with the wheels off. Right now I don’t think the version I’m working on will work with any device.

I’ve just updated my web site with version 0.10, which, for the first time, works with my Samsung Galaxy S3. There is a fighting chance it may work with other devices too.

http://davehigton.me.uk/Progs/PTP.htm

Doesn’t work but is an improvement. It no longer hangs and the device data window is filled in. here’s the log file ptplog.zip. The many repeats at the end are a result of me trying a refresh.

Thanks for tryng, Colin. I can see what’s wrong. I still haven’t sorted out the correct solution.

The basic problem is the number of bytes requested at line 3990 of the RunImage file. There are various quantities you can try, if you are willing:

512 (an exact multiple of the max. packet length);

16 + (num_objects% * 4);

28 + (num_objects% * 4);

32768.

In theory, 16 + (num_objects% + 4) will request exactly the correct number of data bytes; 28 + (num_objects% * 4) will request exactly the number of data bytes, plus the status response that comes after the data. In practice, I have found that when I ask my GS3 for the number of objects; it returns 93, but when I ask it to return their handles, it returns 108 handles. I still have to do a bit of work on that, because I was not sending the second and third arguments to both commands (the PIMA standard is ambiguous as the the meaning of “not using” arguments).

Your device definitely needs to send more than the 500 bytes requested, so the transfer ends with an “IO Error”. Since the app’s error handling is somewhere between minimal and non-existent, it gets stuck there. I will fix it, really I will.

It doesn’t get that far. If you look at the log I sent (at the top) there’s a sending command 1002 followed by sending command 1001 then before the log for the next time I ran the program it goes into an idle state.

On separate issue if you load Runimage into Edit and save it it tells you where you’ve missed 2 brackets

before the log for the next time I ran the program it goes into an idle state.

I’m not sure I am interpreting correctly what you wrote. At the moment the app does go into the Idle state after it’s got the device info; this is deliberate. Menu over the main window and select Refresh to get a listing of all objects that it can find. If you want the directory to be filled automatically on startup you can edit the !Run file. If this isn’t what you meant, then I’ll have to ask you to rephrase what you said.

If you run, quit and re-run, you may find the device gets upset. It’s safer to unplug the USB cable for a couple of seconds between runs. Again, this is something I’ll fix. There is no attempt to close a session on exit. Also, my GS3 needs to be plugged in for a few seconds before running !PTP, otherwise something goes wrong with the communication. Clearly that’s something else I’m going to have to fix. Right now, though, I don’t know what exactly goes wrong, and so I haven’t a clue how I’m going to fix it.

On separate issue if you load Runimage into Edit and save it it tells you where you’ve missed 2 brackets

Well, I never knew Edit did that! Very useful, thanks. Those unbalanced parentheses are now fixed, of course (but I’ve not updated the app on the web site).

Ok I hadn’t realised I needed to refresh to get anything.

It didn’t work properly but 16 + (num_objects% * 4) seemed to the best results. What happens is that it fetches so many objects then stops. Best I’ve had is 45% on the hourglass with the above setting but it varies with every run. I’ve updated ptplog.zip with the log.

Anything less than that value didn’t fetch anything. 32768 fetched nothing, 28+(num_objects% * 4) fetched some objects but stopped with what appeared generally to be a lower percentage. I tried a few other minor changes like 32+(num_objects% * 4) and 1000+(num_objects% * 4) but this didn’t help.

Thanks again for your feedback, Colin.

One thing that went wrong for you was something I’ve seen occasionally: a transfer finishes – and it must have finished properly in USB terms, because the last packet was shorter than the maximum packet length – but the stack, as interrogated by DeviceFS_CallDevice &80000006, thinks there is still a transfer in progress (the return value is consistently 0).

I’m currently looking at how to work round it.

I’ve had much more success this evening. I’ve been able to transfer all the photos off my S3 with !PTP.

Some more needs doing to generalise it; currently it’s hard coded to a Storage ID of &00020002, which is the S3’s micro SDHC card. I need to get it to read the storage IDs (which is implemented separately and known to work) and interrogate each of them. Asking for all objects over all stores doesn’t actually do what it says on the tin on the S3.

One neat feature is that taking a photo updates the listing automatically. (The notification comes over the interrupt IN endpoint.)

Oddly but explicably, taking a video also updates the listing, but videos are not visible via the Refresh function. This is probably because the S3’s video format is MP4, which is not recognised by PIMA 15740 (the PTP specification), whereas notifying the app of new objects is apparently not filtered by format.

I hope to put up a new version for people to try later this week.

I hope to put up a new version for people to try later this week

Looking forward to giving it a go.

Here are some examples of what I’m up against.

Ask my phone how many objects there are, of all types, in all storage IDs. It tells me “93”.

I ask it to return the handles of all objects, of all types, in all storage IDs. It returns 108 handles.

None of them is in the removable storage.

Ask my phone how many objects there are, of all types, in storage ID &00020002 (the micro SDHC card). It tells me “18”.

I ask it to return the handles of all objects, of all types, in storage ID &00020002. It returns 318 handles.

Some more good news:

I can now use my phone in MTP mode as well as PTP. Now I can see videos and MP3 files too.

I have the crude beginnings of a hierarchical display working. I do stress crude.

I can now use my phone in MTP mode as well as PTP.

No idea what PTP is, but my current phone (Xperia U) only does MTP so I’m definitely interested in something that will allow it to talk to RISC OS.

Keep us posted, right!? :-)