LanManFS losing directory contents after a while

I have failed to reproduce the missing files problem using LMcheck here with all variations I can think of. It would be very interesting if anyone who does see the problem could report if LMcheck also sees the problem or not, as this could be a useful clue.

Timing issues have been suggested as a possible cause, and I did wonder if the speed of the network could be a factor? Mine is a GB network & NAS … although neither my Iyonix or RPi3 can do justice to that.

I’ve had no luck repeating in the last few days. I connect via a wireless bridge (TP-WR710N). I’ve just been checking back every few hours as seems to work for Jeff. Perhaps trying too hard prevents it. Maybe the nas needs to sleep. Maybe somthing else on the network needs to change or maybe it’s an interaction with other software.
If it does indeed need a reinit of lanmanfs to get out of the problem, are there any global variables that could be involved? I’ll try to gather as much info as possible next time I see it, but currently that looks like it could be a while.

Would anyone who can repeat the truncated directory problem like to tryout this version of the module LanMan_timeout. The anti idleout interval seemed to be a long time in the original version so I’ve shortened it a lot hopefully it will fix the problem.

Timeout version seems to work just the same for me … with no problems! Any clues what circumstances provoked the long anti idleout to cause problems?

First impressions of the debug code suggests that the anti idleout code is transmitted quite often but on a little more digging I discovered that it is only actually transmitting a packet every 18mins when idle which seems a very long time to me so I’ve modified it to transmit every 2mins to see if that is the problem.

I can’t reproduce the problem here so I’m only guessing. It may be that timeouts are server software specific and that may be why it’s a problem for some people and not others. It may also explain why the directory enumeration initially works but then fails. In any case if someone can reproduce the problem with this new version then it’s unlikely to be a timeout issue.

Mounting 2 shares to the same server showed that the Anti IdleOut function was only happening on one of the shares. I’ve updated LanMan_timeout/zip to fix this.

Thanks Martin for spotting that.

Just in case anyone would like to try out a version with Anti IdleOut disabled to see if it makes the problem appear more easily I’ve made LanMan_notimeout/zip available.

If you have seen the missing files problem, can you remember if at the time there was more than one LanMan mount?
If so, was the problem was on the first one to be mounted?

Hi guys, sorry I haven’t been back on this topic. Have been down with a nasty fever until today, so am now trying to catch up on the various things that didn’t happen last week :( Thanks to Colin and Martin and everyone for looking into this.

Just wondering if anyone who has seen the truncated directory problem has done any testing of the modified module?

Colin, I have been testing the notimeout module. I’ve just sent you an email of some reporter reports.

With this module I find that it goes wrong a bit more regularly, usually in a day (tested 5 times), compared to a failure rate of between 3 hours and 5 days for the standard module (tested loads of times)

I was going now going to move onto the new timeout module to see if this fails more or less regularly

if at the time there was more than one LanMan mount?

No. Just one last time I had the missing file issue.

Currently, I’m running my PI against the NAS I previously had the issue on, using Colins debug LanManFS, and Iyonix against a share from Rasbian with the notimeout version.

Neither have shown the issue in the last few days, but, I have had a few aborts. Now, I had been ignoring them, not wanting to be distracted from the lanman issue, I think a couple might have been netserf, but when I checked in this evening, both had abort on data transfer on screen. I thought I’d look into it further. Both were in the same place in LanManFS. Oh dear. Now, given no one else seems to have an issue with this, perhaps it’s somthing common I’m running on both machines that causing it. But I’ve included the info below in case it’s useful.

Iyoinx abort

Address &2034D3B0 is at offset &0001A53C in module 'LanManFS'
*showregs
Register dump (stored at &2002D3B0) is:
R0  = 00000001 R1  = 201E0133 R2  = 201E0120 R3  = 2036D3EF
R4  = 2036D3EE R5  = 20164818 R6  = 002407E8 R7  = 002407E0
R8  = 201E0114 R9  = 20164818 R10 = FA20021C R11 = FA207A98
R12 = FA207A18 R13 = FA207A6C R14 = FC14F454 R15 = 2034D3B0
Mode SVC32 flags set: Nzcvqjggggeaift        PSR = 80000013
*memoryi pc-24 +28
2034D38C : ..Qã : E3510003 : CMP     R1,#3
2034D390 : .... : 1A000003 : BNE     &2034D3A4
2034D394 : .. á : E1A00004 : MOV     R0,R4
2034D398 : :.•â : E28F1F3A : ADR     R1,&2034D488
2034D39C : $<.ë : EB003C24 : BL      &2035C434
2034D3A0 : .@ۈ : E0804004 : ADD     R4,R0,R4
2034D3A4 : ...ê : EA00000B : B       &2034D3D8
2034D3A8 : . Õç : E7D52006 : LDRB    R2,[R5,R6]
2034D3AC : .. á : E1A00004 : MOV     R0,R4
2034D3B0 < 5.•â : E28F1F35 : ADR     R1,&2034D48C

Address &203A3F50 is at offset &0001A4BC in module 'LanManFS'
Register dump (stored at &2001B4D0) is:
R0  = 00000001 R1  = 20672833 R2  = 20672820 R3  = 203C4018
R4  = 203C4017 R5  = 2013C114 R6  = 00D56EEC R7  = 00D56EE0
R8  = 20672814 R9  = 2013C114 R10 = FA20021C R11 = FA207A98
R12 = FA207A18 R13 = FA207A6C R14 = FC178D60 R15 = 203A3F50
Mode SVC32 flags set: NzcvqjggggeAift        PSR = 80000113
203A3F2C : ..Qã : E3510003 : CMP     R1,#3
203A3F30 : .... : 1A000003 : BNE     &203A3F44
203A3F34 : .. á : E1A00004 : MOV     R0,R4
203A3F38 : :.•â : E28F1F3A : ADR     R1,&203A4028
203A3F3C : D<.ë : EB003C44 : BL      &203B3054
203A3F40 : .@ۈ : E0804004 : ADD     R4,R0,R4
203A3F44 : ...ê : EA00000B : B       &203A3F78
203A3F48 : . Õç : E7D52006 : LDRB    R2,[R5,R6]
203A3F4C : .. á : E1A00004 : MOV     R0,R4
203A3F50 < 5.•â : E28F1F35 : ADR     R1,&203A402C

Will, I’ve had two pi’s connected to my NAS, both pi’s have been running the notimeout LanMan module to see if the missing files problem appear more easily.

The only problems I’ve had is sometimes I’m getting ‘LanMan timeout’ error box (I assume caused by the anti idle out being disabled) and also sometimes a very long hourglass wait while it fetches info (I can guess that another module trying to access the NAS during this hourglass pause could cause trouble)

Anyway I’ve satisfied myself that the missing file problem is made slightly worse using this module (which was the aim of this module), and have now got both pi’s running the lanman timeout module. This time I will report any odd behaviour.

Will, any idea what the abort was? When you say that when you press cancel nothing bad happens is LanMan still running?

The address is in the debug code (DumpBuffer) and it looks like that is called in processing the response to LanMan requesting the status of a server name. If it happens again can you send any reporter output if you are able to access it.

I think the ‘notimeout’ version is causing confusion. It was only meant to see if it produced the problem any quicker. I’m more interested in the ‘timeout’ version. If anyone can reproduce the problem with that version just once then that will prove that the server timing out isn’t the problem.

‘Nothing bad’ was perhaps misleading. As in nothing I noticed to lanman. I noticed this morning though that the access icon was missing from the icon bar, so something knocked that out. I think I’ll clean image boot my iyonix and go from there, with the debug lanmanfs and get you more info if I get it again. I don’t want to be distracting with possibly unrelated issues.

This evening, I checked my Iyonix, with the intent of downloading the latest discimage to set up a clean boot, and found: Error: Internal error: abort on data transfer at &2034D3B0, again, clicked cancel, and the apps icon had now gone too.
From there, I opened a directory on the lanman mount anyway, and saw the 30 second hourglass Jeff had mentioned, and when the filer window opened, files were missing.
Unfortunatly it was still running the notimeout lanmanfs as I’d not reset for a few days now. Afraid I don’t have a log of what happened before the abort.

I’ve captured a report log of *cat, opening the filer, and running LMCheck (without recusion, just on the one folder). I’ve got a screenshot of the filer window, and the *cat output.
Cat showed least files, filer more, and LMcheck reported all that should be present. It seems this is affected by buffersize, but I don’t see why it’s so rare to go wrong.
To compare, I captured a screenshot of how it should look, and the report opening in filer from my pi pointing to the same mount.
This was infact the first time I’ve seen the issue on the Iyonix, and with it pointing to Samba on raspbian, rather than the Pi/NAS combo I first saw the issue on.
Log file etc. here

LMcheck reported all that should be present. It seems this is affected by buffersize, but I don’t see why it’s so rare to go wrong.

LMcheck before v0.03 used a GBPB buffer of 8000 bytes, but it now uses 512 bytes which I think is the same as the Filer, after I realised it could be a vital difference. Why it is so rare is still a mystery to me! I hope your latest log files will give some more clues to Colin.

Task manager has died now, not sure how long I’ll be able to get stuff out of it.
I just ran v0.03, I commented line 108 to stop it recursing down, it listed 12 files, filer shows 12 files and two dirs, so it seems consistent. I’ve added to the log zip above.

Ok I’ve made up LanManFS_01.zip which is the same as the ‘timeout’ version without the debug code – if you want to confirm that you are using it *help lanmanfs should end in ‘test-01’. If you or anyone would like to try this version I’d be interested if you had any problems at all.

This was infact the first time I’ve seen the issue on the Iyonix

The ‘notimeout’ version disables all ping code to keep the mount alive so if you are finding that that makes the bug appear more often then that is good.

It may be the debug code that is causing the aborts for some reason or it may be that the timeouts are causing more problems than just truncated directories. Either way there is a possibility that the ‘timeout’ version fixes the problem as I have had no reports of the file truncation problem with this version of the module and Jeff – who seems to be able to produce a truncated directory relatively frequently – reports no problems in 24hr so far.

All the directory listing failure reports I have – including yours – show that the server is replying to a request for more filenames with an empty list and an end of list marker so it’s not a case of the server sending the file list and LanManFS going wrong.

I don’t think that there is any need for more reports at the moment, I’m more interested in any truncated directory listing with either LanManFS_01 linked above or the ‘timeout’ version – which is the same with debug code and reporter output.

If you’re experiencing crashes, don’t forget that modern versions of RISC OS 5 can generate stack traces which can be useful for pinpointing the cause.

https://www.riscosopen.org/wiki/documentation/show/Debugger%20Exception%20Dumps

Thanks Colin, I’m now running test-01 on a clean !Boot on my Iyonix.
It took a week to get it to fail, but at least it showed it could happen on a wider range of setups.
It may be a while before I can conclusivly report on test-01, but I’ll run it over the next week or so and see.

don’t forget that modern versions of RISC OS 5 can generate stack traces

The current session was too far gone, so had to reboot. Noted for next time though, thanks.

I have now been running test-01 for 6 days continuously on my RPi3, without any problems. This was with two copies of LMtest v0.03, one listing listing a single directory with 981 files, the other listing 319 directories with 5935 files, at random intervals between 30 and 90 minutes.

Not sure if this proves anything, as I have never seen the problem here!

I’ve been running the iyonix all week, checking daily with no issues. test-01 has certainly eliminated the normal 30 second pause when comming back after it’s been idle for a while, so that’s good.

However, yesterday I found a quick simple way to force the problem to appear.
On my PI, I mount a share from Raspbian on another PI, open some sub dirs, all good. Now, close all the filer windows, then issue a service samba restart, wait a few seconds, then click the mount on the icon bar (this then has the 30 second delay), open some sub dirs, and files are missing.
But now, here’s the weird part, I mount another share from my NAS, and that too has files missing from sub directories straight away. All until Lanmanfs is reinit’d.
The process works equally well going the other way, by rebooting the NAS and then connecting to the other PI, it just takes ages for the nas to reboot.
It seems, the forced reconnect causes somthing to happen, that any new mount after that has missing files.
I was going to try to capture the network traffic on the connection, but I’m having trouble running wireshark on raspbian right now.

Ok I got linux, samba and wireshark up and running and can confirm that the process you describe does result in a truncated directory – yay. Initial findings show that the smb server is sending an End Of Search indicator early in response to a continuation request when enumerating the directory – as the LanMan debug output also shows. At least I can repeat the problem here now.

But this makes it look as if the server is at fault.