LanManFS losing directory contents after a while

I can’t reproduce the problem on windows – which is a shame as wireshark would show what is happening.

A few months ago when I was working on something I think I did have some success getting wiresalmon working to allow me to perform a capture on RISC OS – possibly all it needed was a recompile.

The only thing I can think of different on a NAS to windows is that NAS software tends to negotiate a larger buffer size so would transfer more filenames in one chunk.

Frequently the NAS units still support SMBv1 and as Jeffrey noted MS have been stealthly switching off SMBv1 in Windows 10 so the Windows 10 will be different to the Win 7 & 8 (use SMB2 and fallback to SMB1) or the pre-Win7 builds that were SMB1 by default.

I don’t decry the switch off as it pushes increased security on people, what I do think is bad is the stealth nature of the switch off which I believe should be done with a pop-up notification to tell you that you should do it and repeat the nag at regular intervals with no kill for the nag notification.

Ok,
This time LanMan lasted 20 hours before going wrong, last time lasted about 23 hours (not 11 hours, Senior moment!) , and again what cat lists and what the filer window displays (when its gone wrong) are not the same (but both are wrong).

After reseting the system I hit the filer system hard to see if I could make it go wrong quicker (just opening/closing lots of directories for a few minutes) but no luck, it wouldn’t go wrong, but after returning to the system after 15min or so a click to open a directory caused a long hour glass pause for about a minute and when the filer window opened it was wrong, does LanMan do any thing in an idle moment?

Later on I will reset the system and find out if a pause after a lot of usage makes it go wrong, and also as Rick has asked about other computers using the Nas I will test for this as well.

Also Colin asked about if its random files/folders not being displayed (to me it looked random), but this time I kept a “cat” record of directories and this shows that the missing files are not random, when comparing the two cat lists they are identical until the point when the wrong list ends (what order does cat list files?).

but this time I kept a “cat” record of directories and this shows that the missing files are not random, when comparing the two cat lists they are identical until the point when the wrong list ends (what order does cat list files?).

Cat depends on the order in which the underlying filesystem provides entries, usually they are provided in the order in which the directory entries are stored in the directory structure on the disk. On FileCore based disks (i.e. ADFS, SCSIFS, …) the directory entries are kept in sorted order, for FAT32 disks (DOSFS, Fat32FS) new entries are just added to the end of the list unless an entry which was marked as deleted can be reused so cat basically lists the entries in the order they are created on disk. For LanManFS it will depend on the server on the remote side but I guess they use the same principles.

The filer basically requests the directory list like cat but sorts the received list which makes it more difficult to understand what happens. He we now have a first hint: when things go wrong LanManFS returns truncated directory content lists. Now the long hourglass pause you mention could indicate a timeout of some kind. A network timeout maybe or some kind of corruption which prevents it from requesting the next parts of the directory listing.

That makes sense, *cat lists files in the order that they are sent from the server so a truncated directory indicates that the directory enumeration never completed. It may be easier to use

enumdir <pathname> vdu:

or

enumdir @ vdu:

to list the directory or current directory instead of *cat as you then get one filename per line making it easier to see differences.

Do you use !Reporter? If so you could try this debug version of LanmanFS which will output debug data to a reporter window.

If you are going to try the test module have you a folder that always shows the problem? If so could you:

Before loading Omni

1) run !reporter
2) Load the LanManFSD module
3) run Omni
4) connect to the share
5) clear the reporter window
6) in a taskwindow *enumdir <folderpath> vdu:
7) save the reporter window and directory listing
8) when the problem occurs repeat 5 – 7

You should now have 4 files, the reporter window and directory listing when there is no problem and the reporter window and directory listing when there is a problem.

Could you send me the files at the email address on this web page and I’ll see if it sheds any light on the subject.

Thanks Colin,
I will do this, it will be great if we can get to the cause of this problem.

Last night I tried various ways to make it go wrong quickly, but with no consistant success, the only thing that does seem consistent is that always go wrong after its had pause (10min or longer) of me trying to make it go wrong, but that could be chance.

the only thing that does seem consistent is that always go wrong after its had pause

There seems to be a timeout of some sort, after which something behaves differently in some way. Repeatedly poking it, even at 60 minute intervals, seems to me to be likely to prevent the problem from being observed.

The timeouts look like they could be different on different systems, which suggests that the timeout may be in the server.

Dunno. Just ideas.

It does an ‘Anti IdleOut’ callback every so often to keep the link alive.

I wonder what happens if after the problem occurs you try saving a file to the directory that is not being listed correctly. Does it error? Is it there next time you are able to read the directory?
Warning I suspect there is a possibility it might corrupt the directory so only do it on a test directory.

I have now run my LMcheck program on my Iyonix for 12 hours at a time, and on my RPi3 it is still running after 24 hours, all without a problem (yet). They are both using RO5.24 with LanManFS 2.61 and OS_GBPB to list 200+ directories and 33K unchanging files on my Synology NAS every hour. The Iyonix has been used, but the RPI is doing nothing significant.

Have there been any confirmed problems using a Synology? Or an Iyonix? Or OS_GBPB?

My program LMcheck is now available if anyone else wants to try it.

I am also now running Colins debug version of LanManFS on my Iyonix … but until I have a problem the output is no real use :-((

Martin

Your app doesn’t do the same as the filer when enumerating a directory. The filer just reads the directory it doesn’t read sub folders – don’t know if it makes any difference though.

LMcheck will indeed enumerate subfolders if there are any, because I wanted to be able to check as many folders as practical. In my case here it is reading many folders at the lowest level with lots of files and no sub-directories.

I can easily try using it on a single directory with just 900+ files, if you think it a test worth doing.

Until we discover a more reliable way of provoking the problem, debugging is difficult. It is playing hard to pin down … but will probably be bloody obvious when it is found!

That was interesting… I’ve been checking the folders I knew I had an issue with in the past trying to provoke the problem, with no luck. I then ran LMCheck (thanks Martin), and ran into the (still to be looked into) directory not found error, on enumerating files within a folder with too many spaces/special chars in the name. At that point, re opening the folders I’d just checked, there were files missing. 3 levels down from root, each folder only listed 14 items (all have more). Reinit lanmanfs and they are restored. Ran LMCheck again, this time the top folder only had 4 items. I could not traverse further ‘cos the folders I wanted weren’t there.
Reinit. again to restore, run LMCheck again, same directory not found, but no corruption of the listings.
Tried several more times, nothing.
I’ll have another mess about tomorrow to try to get a more consistent fail.

Edit: just realised I’m still on 29th April 5.25. I’ll get that up to date before doing more tests.
Edit edit: Still goes wrong on 22 Jul 5.25. Although still not failing in a consistent way

Did LMcheck and the Filer agree?

too many spaces/special chars in the name

What is too many?

LMcheck on my Pi is still running ok after 30+ hours, so I am starting to think that it is not going to provoke the error with those test conditions. Which is interesting, but not directly useful!

I didn’t check the output from lmcheck, it left the log file open when it died each time and never completed a run. Had to wait a minute between runs to open a different file. I think more than 2 is too many. Can’t check now but “a / b” will probably do it. I managed about 4 fails in an hour so hopefully I can test more tonight.

@Will: I had not realised LMcheck had failed. If you send me the error message I will improve the error handling!

My Pi has just passed 48 hours running LMcheck without a problem, so I have concluded that my conditions are not going to provoke the problem, and I have stopped it. If I can think what to change I will try again!

Basically, it knows you are looking.

More usefully, what does LMCheck actually do? It seems to me, from the above, that longish periods of idle may be a contributing factor? Maybe LMCheck checks too much, or too often?

LMcheck starts from a configured directory, and enumerates all entries within it using OS_GBPB,10. If it finds a directory, it enumerates that (recursively). All entries are listed to a timestamped file, and counted. After a configured time (I have been using 60 minutes) it does it again, raising an error if the counts have changed.

This seemed to match the reports of missing files in directory lists after a period of inactivity. However, using LMFS v1.61 to my Synology from my RPI it lasted 48+ hours without faltering. My Iyonix lasted 12 hours on two days. So somehow it is avoiding the problem.

But, as you say, it knows I am looking!

I have to agree with Rick, it knows were up to something!

I’ve been running Colin’s test module with !Reporter for over a day now and it still running ok…

It wasn’t really LMchecks fault. It got “Directory ‘LanMan::Volume_1.$.lt.LtankWeb.Laser Tank Ver 1/60 For RISC OS_files.Newer’ not found at line 70” due the case where lanmanfs presents the folder and content with spaces and / in the name, but then bails out saying not found when asked for the content.
However, right now running it as is gives me maybe a 1/50 chance of the issue then showing up just after, so while I try to pin exactly what’s up I’m not concerned that it errors. By handling it and moving on, it may not provoke the issue we’re looking for in the same way.

LMcheck obviously frightems the merde out of LanManFS and can now be used as a preventatif.

Bit like a patch, but perhaps more like a horse’s head on the pillow?

Ok I’m on my third or …

Lost count!

but perhaps more like a horse’s head on the pillow?

I think Schrödinger’s Cat, you think…The Godfather…?

<puzzled>

If it finds a directory, it enumerates that (recursively).

Maybe this is the problem? What happens if you pick a directory and watch only that?

This is proving more elusive than I thought; I’m really struggling to get a failure now.
Also in my haste/excitement my previous report of only 4 files was incorrect, I was looking in the wrong folder, sorry to muddy the water.
I’m also thinking that use of LMCheck is just coincidental to the failures as I now keep running it in the hope of a fail, I only ever see a fail after running it. I can also run it many many times without a fail, so that doesnt help much.

Anyway, some more detail from when it is in the fail state…

Failed listings

*cat LanMan::Volume_1.$.lt

Dir. LanMan::Volume_1.$.lt Option 00 (Off)

CSD  LanMan:“Unset”

Lib. LanMan:“Unset”

URD  LanMan:“Unset”

phonesys     WR/R

!ABC         D/

StrongDump   WR/R

sc2/png      WR/R

sc1/png      WR/R

LanManFS     WR/R

!SideDiff    D/

*cat LanMan::Volume_1.$.lt.tank

Dir. LanMan::Volume_1.$.lt.tank Option 00 (Off)

CSD  LanMan:“Unset”

Lib. LanMan:“Unset”

URD  LanMan:“Unset”

ObjectsHiRes    WR/R

Built           D/

testLev1        WR/R

LockOut         WR/R

testLevData/lvl WR/R

Objects         WR/R

fail4           WR/R

Should have been..

*cat LanMan::Volume_1.$.lt

Dir. LanMan::Volume_1.$.lt Option 00 (Off)

CSD  LanMan:“Unset”

Lib. LanMan:“Unset”

URD  LanMan:“Unset”

phonesys            WR/R

!ABC                D/

StrongDump          WR/R

sc2/png             WR/R

sc1/png             WR/R

LanManFS            WR/R

!SideDiff           D/

LaserTank 5/0/0/exe WR/R

LtankWeb            D/

!EGoat              D/

Tank                D/

Untitled1/jpg       WR/R

lvls                D/

!ARMalyser          D/

Programming         D/

test_level_data/lvl WR/R

!SID                D/

*cat LanMan::Volume_1.

Dir. LanMan::Volume_1.$.lt.tank Option 00 (Off)

CSD  LanMan:“Unset”

Lib. LanMan:“Unset”

URD  LanMan:“Unset”

ObjectsHiRes     WR/R

Built            D/

testLev1         WR/R

LockOut          WR/R

testLevData/lvl  WR/R

Objects          WR/R

fail4            WR/R

lockout2/lvl     WR/R

~LaserTank       D/

!RunImage/txtq   WR/R

proof            WR/R

fail6            WR/R

Tricks           DWR/R

HiRes            D/

!RunImage        WR/R

fail3            WR/R

tice             WR/R

ObjectsPreserve  WR/R

lockout3/lvl     WR/R

nsmail/zip       DWR/R

ltRequest        WR/R

!RunImage190/txt WR/R

Tricks/LVL       WR/R

tice1            WR/R

fail5            WR/R

Clearly the not displayed items are still accessable as I was able to *cat $.lt.tank which wasn’t visable in the $.lt listing, although at the time the filer was showing different, as below for $.lt
!ABC
!ARMalyser
!EGoat
!SideDiff
LanmanFS
LaserTank 5/0/0/exe
LtankWeb
lvls
phonesys
sc1/png
sc2/png
StrongDump
Tank
Untitled/jpg

likewise, in the fail state the lt.tank folder had, in the filer, 14 items, 7 of which showed in *cat and 7 not. And the rest just missing.

Another observation, when items are not displaying, new files and folders can be created, but don’t show up until a reinit of lanmanfs.