Bug inside DeviceFS possible

During examination of Raik Fischers UMTS transmission problems I was able to trace down one of his problems to my USBSDvEmu module which hasn’t changed since summer 2011.

USBSDvEmu emulates serial port behaviour on an USB port for Castle PPP module to allow UMTS communication (Serial Port will deliver data in case it arrives, but at USB you will need to pull possible data by query form the bus).

Since summer the PPP transfer sometimes blocked for no special reason. I was able to check it now on my machine because I had a really old and very expensive prepaid tarif before which I changed some weeks ago.

USBSDvEmu is relying internally on the upcall 9 (buffer empty) in which case it introduces a new USB OS_GBPB request to fetch more data. I was able to trace down that under certain circumstances when the buffer becomes empty no buffer empty upcall was received however.

I am really certain that the problem is not located inside my module even of course the old PPP module may corrupt something because it has been never compiled with a modern ARMv7 compatible compiler AFAIK.

Has anyone made similar expieriences on her/his BB xM or could renember about a change inside 5.19 made in summer 2012 which could have effected this area?

It must be something inside DeviceFS or the buffer handling. May be this has changed with one of the newer USB drivers but I don’t think that the USB driver has direct impact on this behaviour.

The last change to DeviceFS was in 2009, so its behaviour should be unchanged since summer.

It may or not be related, but…

I’ve been trying to get PTP to work with several cameras and camera phones. There are many cases where it just doesn’t seem to work.

One of the most bizarre cases was last night at the Southampton Acorn Users Group meeting, where someone brought a Nikon D60 to try. It’s a high speed device. It works beautifully when connected through a USB 1.1 hub (and therefore limited to full speed), but doesn’t work when connected at high speed.

You’d think that wasn’t possible, wouldn’t you? To work at full speed but not at high speed? There is no difference to the calls we have to make. It should just work. If it works at one speed, it should work at the other.

I can confirm Thomas statement.
The problem is the same as here: https://www.riscosopen.org/forum/forums/4/topics/1510?page=1 # posts-17952
Until about the end of April / beginning of May, there were no such problems.

I can also confirm with Sprow. DeviceFS is from 2009 but any things around are changed. Also the USB driver. Since my harddrive in xM is slower :-(

Another oddity I saw: earlier this year, when I was experimenting with Bluetooth, I noticed that I had got a reply from the Bluetooth dongle via its bulk IN endpoint, but RISC OS was resolutely not seeing it. I was using a hardware USB analyser, so the message definitely was transferred in over USB.

Now I’m missing bytes on an interrupt IN endpoint. The transfers occur 8 bytes at a time; I’m usually seeing the second 8 but not the first 8. I should explain that the message as a whole is 16 bytes, but the maximum packet length for the interrupt IN endpoint in the device is 8 bytes, so it has to be sent in two transfers. Occasionally I’ve seen all 16, but mostly the first 8 are not visible.

In case that DeviceFS or the Buffermanger has not changed it would think that it is a problem inside the OS (Upcalls lost). I don’t think that the USB driver even it still might be faulty is responsible for generating upcalls of a certain module.

Regarding Daves problem it must say that AFAIR USB 2.0 isn’t exactly the same as USB 1.1. Additionally parts of software are required to handle it and so of course these parts may be faulty.

Also thats with the 16 Bytes splitted interrupt transfer is such a thing. Yes it exists and it is allowed by the USB specification but I am not sure especally in BSD/LINUX sector whether all stack implementations are aware of this side effect. Over ten years ago as I wrote my own RISC OS USB stack I had also a look on the LINUX implementation and it was really poor. Some developers thought that USB was just another kind of a UNIX pipe and implemented the stacks accordingly in a simple way. Ok this was long time ago but perhaps some special effects are still not implemented properly unfortunately.

What happens when data arrive? Is there an interrupt? How do the bytes get into the buffer? Is it possible that some other event causes the arrived bytes to be missed? Is it possible that a second message comes in before a buffer has been cleared, and therefore overwrites the buffer, and the message arrival flag is already set, so the first one is not noticed?

Does data transfer on one endpoint interfere with data transfer on another, e.g. by only having a single “data ready” flag that applies to all endpoints?

More USB malfunction. (All recent reports are with Iyonix, RO 5.18.)

I’m sending a camera a GetStorageIDs command. I don’t know how many bytes will come back – it depends on how many stores the camera has. I know there will be at least 16, but I don’t know how many until I read the first 4. The particular camera I’m using (Nikon Coolpix 3100, which is a full speed device) returns 20 bytes (2 stores).

I ask for 16, and the stack gives me an I/O error.

I ask for 20, and it works as expected.

I ask for 64, and it returns 64, of which the first 20 are valid, and the remainder are all 0. You’d expect that the OS_GBPB would return R3 = 44 to say that 44 bytes were not transferred; but no.

Regarding your OS_GBPB: Yes it should return 44. I assume that you explicitely set the stream to nopad by something like this

nopad as parameter inside stream didn’t work for a long time (and perhaps isn’t still working).

Well, I’m back to one of the old problems: it seems I can only get an error free input transfer when I ask for exactly the correct number of bytes.

The problem, of course, is that there is no way to know how many bytes are about to come in, unless I read the first 4 – but if I ask for 4, it’s a wrong number, so the stack returns an I/O error. Worse, it is no longer possible to transfer any further bytes in on that pipe; it is broken until I close and re-open it.

Thomas: I have tried your code to remove padding. I’m not sure what the call should return; does 0 mean OK or error?

I’m banging my head against a brick wall with this accursed stack. I should stop.

Plan9 have been developing usb drivers recently.
A brief description of usage here:
http://9hal.ath.cx/magic/man2html/3/usb
One odd thing
“timeout n   Enable time–outs for the endpoint. Transfers are timed out by the kernel after n ms. This should not be used for control and isochronous endpoints, which are always timed out.”
What do they mean by “always timed out”?
I dont understand the timeout mechanism anyway :-(

I dont understand the timeout mechanism anyway :-(

There’s nothing to understand in RISC OS, because it doesn’t actually work anyway :-(

Thomas, can I check whether I’ve translated your code correctly, please?
This is what I’ve got:

REM Get DeviceFS stream handle
SYS DeviceFS_CallDevice, &80000003, usb_dev$, inpipe% TO r0%,,, r3%, stream_handle%
PROClog("Open inpipe: " + type$ + "; r3 = " + STR$(r3%) + ", stream handle = " + STR$~(stream_handle%))
REM Set not to pad
SYS DeviceFS_CallDevice, &80000008, usb_dev$, stream_handle%, 1, &FFFFFFFE TO r0%
PROClog("Returned value from call to stop padding: " + STR$~r0%)

The returned value in R0 from that second call is &80000008.

The USB input is still padded.

I would use &80000007 instead &80000003 and the required handle to pass to &80000008 will be returned inside register 5 instead register 4.

Sorry for answering so late but I did had much time over christmas.

Can anyone confirm that they have managed to prevent a bulk IN pipe from padding, on an Iyonix, with RISC OS 5.18, EVER?

I’ve tried the following:

DeviceFS_CallDevice% = &42744
usb_dev$ = "usb9"
SYS DeviceFS_CallDevice%, &80000007, usb_dev$, inpipe% TO r0%, r1%, r2%, r3%, r4%, stream_handle%
SYS DeviceFS_CallDevice%, &80000008, usb_dev$, stream_handle%, 1, &FFFFFFFE TO r0%

The device in question is USB9. inpipe% is the value returned from opening the stream, and is in the region of 250.

The pipe still buffers. I have never succeeded in preventing one from doing so.

I am calling OS_GBPB requesting 64 bytes to be transferred in via inpipe%. My USB analyser tells me that only 2 bytes are transferred on the USB each time; they have characteristic values of 01 60. OS_GBPB returns with R3 = 0, so my app has to assume that 64 bytes have been transferred, although 62 (usually) or 60 (occasionally) bytes have the value 00.

Can anyone confirm that they have managed to prevent a bulk IN pipe from padding, on an Iyonix, with RISC OS 5.18, EVER?

Sounds like the file I had with USBMIDI on the Pi. It shoved in loads of null bytes in blocks of 64; and looking at one of my dumps, it doesn’t seem to follow any pattern so I can’t necessarily read in groups of four bytes and discard the non-MIDI. No, I have to read byte by byte to look for the non-null. Hmmm…
At least that is a step up from the Beagle where all attempts to read failed.

I’ve borrowed an R.E.M. CD from the library and playing as I write this is “Everybody Hurts”. Coincidence?

[ and “The Sidewinder Sleeps Tonite” (sic!) is a nice sounding song that Neo/Matrix would appreciate with the numerous allusions to the POTS, although some of the alternative interpretations are…. um…. http://www.songfacts.com/detail.php?id=1737 ]

Further progress – in diagnosis, anyway.

Firstly, I discovered that the RISC OS sources contain an updated version of the user API documentation for the Castle USB stack. Now I know a bit more about what’s happening with these DeviceFS calls with function codes &80000007 and &80000008.

I was also able to verify that the “nopad” option when opening the endpoint does set the nopad flag. So does the recipe from Thomas Milius above.

The bad news is that, once nopad is set, the call to OS_GBPB 4 doesn’t return, so the machine hangs. It does this even when a transfer of 2 bytes is requested, and the USB analyser shows that two transfers, each of two bytes, occur.

One question Dave:

Which kind of device you are ttying to run? If reading about the two Bytes it sounds similar to the FTDI devices for which I wrote a driver.

First a remark:

The USB part seems to work with actual version of V5.19 for COMCentre and FTDI on my BB xM which is fine.

Regarding Dave blockings:
I am still uncertain about the block size of the pipes. Due to experience I am using a value of pipesize + 1 however. Ok not possible for 32768 as this seems to be the largest size for DeviceFS buffers.

I assume you shall have to set Non Blocking inside IO_CTRL. AFAIR in case you are eg. requesting 64 Bytes it will wait until it has receipt 64 Bytes for you want 64 Byte and not maximal 64 Byte :-(.

So you must do something similar as the following

ioctl_block⁰=0×80FF0001;
ioctl_block¹=0×00000001;
regs.r⁰=9;
regs.r¹=new_risc_os_ftdi→input.file_handle;
regs.r²=(int) ioctl_block;
if ((error_os=kernel_swi(OSArgs, &regs, &regs)) != NULL) {

where the file handle is that one returned by OS_Find!

The problem is that OS_Args 9 seems to be still undocumentated even after over 10 years since it has been developed.

Perhaps this helps.

Which kind of device you are ttying to run? If reading about the two Bytes it sounds similar to the FTDI devices for which I wrote a driver.

It is indeed an FTDI chip. Is your driver available to the public?

AFAIR in case you are eg. requesting 64 Bytes it will wait until it has receipt 64 Bytes for you want 64 Byte and not maximal 64 Byte :-(.

I requested 2 bytes. 2 bytes were transferred on each of 2 occasions, but still the call to OS_GBPB 4 did not return. I would have expected it to return as soon as the first 2-byte transfer completed.

The problem is that OS_Args 9 seems to be still undocumentated even after over 10 years since it has been developed.

Are you able to contribute more to its documentation?

The FTDI driver is BSD license. Its source resides inside the ROOL CVS Repository

https://www.riscosopen.org/viewer/view/bsd/RiscOS/Sources/ThirdParty/TMilius/HWSupport/InOutput/FTDI/

However as Steve Revill is too busy to setup an according page at ROOL until now I put a useable version on my homepage

http://www.thomas-milius.homepage.t-online.de/

under

http://www.thomas-milius.homepage.t-online.de/Download/Freeware/FTDI.zip

The distribution contains also a document covering usage of some (parameter setting for serial and parallel) but not all aspects of OS_Args 9.

AFAIR the blocking part is documentated somewhere inside the ROOL CVS sources. However It don’t renember exactly where. I shall have a look.

Ok. I think there is really no documentation for OS_Args 9. AFAIR I analyzed the source of DeviceFS to find it out. The related document “NonBlock” is located inside

https://www.riscosopen.org/viewer/view/castle/RiscOS/Sources/HWSupport/DeviceFS/Doc/

This lead to the piece of source code I showed above.

There is also a parameter called “noblock” as alternative. However as with “nopad” I am in doubt that it works under all circumstances.

Brilliant – I am now getting sensible results when I set the input stream not to pad and not to block.

Many, many thanks, Thomas!

What do the return values from OS_Args 9 mean? Is it possible to read the blocking status of a file stream?

Please have a look into my OS_Args 9 document supplied with my FTDI driver.
I don’t know whether read access works for the ioctl information. I can’t renember that I tried it. You shall have to try yourself.
The call would be very similar as the code above to set “noblock”. The differences should be:
Word 1 inside the ioctl_block don’t need to be filled. After the SWI it should contain the actual status. Its content after reading should be equal to the description inside the “NonBlock” document in my opinion.
Word 0 of the ioctl_block should get a 0×40FF0001 instead of a 0×80FF0001 to force reading instead writing if my documentation is ok.

To avoid misunderstandings:
In case that it works it will return whether “block” or “noblock” has been set. It will definitely not show whether the next OS_GBPB will block for no data is availble in the moment. However if choosing “noblock” OS_GBPB will return immediately in worst case with 0 Bytes of data read.

Please tell me about your results.