Write-through file caching
Jan Rinze (235) 368 posts |
Hi, With SCSIsoftUSB there is no file caching and certain operations are quite slow. Write-through file caching could be a real boost for things like compiling code or apps that re-use the same file for config or temporary data. It would not speed-up the writes but certainly speed up reads of frequently used files. Write-back policies for USB based storage could also include writeback intervals of centiseconds to ensure that files are written back before the device is removed. I thought it could be done as a module that overlays a new FS but i think that would be rather confusing. |
Jeffrey Lee (213) 6048 posts |
Funnily enough I was looking at some of the file caching/background op code last night. AIUI FileSwitch doesn’t support any kind of asynchronous/background file operations. But FileCore, which is used for all ADFS-like filesystems, does support background operations, including the kind of caching, read-ahead, and write-back that we’d be interested in. But it can only support background operations if the underlying hardware block driver supports them. I’m not quite sure how well the SCSI modules (SCSISwitch & SCSISoftUSB?) handle background operations – all I know is that when I was doing the optimisations, SCSISwitch was just sitting in a loop waiting for SCSISoftUSB to complete its (interrupt-driven) processing, which suggests that FileCore wasn’t asking for any background ops to be performed. I’ll try and have a further look tonight and see if I can work out what it’s doing and whether there’s any easy way of improving it. |
Chris Gransden (337) 1202 posts |
How about non-blocking IO or should that be in the ‘pipedream’ forum. :-) |
Jeffrey Lee (213) 6048 posts |
Probably best to put that in the pipedream forum for now :) It’s interesting to see how many times people have tried (and failed) to get threading working with RISC OS. There’s this odd branch of the Wimp (docs here), there’s a doc I saw somewhere which talks about plans to implement non-blocking file IO by switching out tasks when they block on file operations, then there’s been Acorn’s non-RISC OS attempts to get multithreaded OS’s working (ARX and Galileo?), and probably countless other things over the years (including third party stuff like Wimp2). |
Jeffrey Lee (213) 6048 posts |
I’ve now looked at the SCSI buffering a bit more, and:
Other things relating to the handling of background transfers in SCSIFS:
So the big question is, why is buffering disabled?
I have a feeling the only way we’ll find out why it’s disabled is for some brave soul to try enabling it (remembering to fix up the code to remember R1 after calling FileCore_Create) and see what happens! |
Jeffrey Lee (213) 6048 posts |
More info – I neglected to check the BlackLog The log looks like it covers two branches of the module – an Archimedes one (which began with v1.07 and then jumped version numbers to 1.14 when it gained buffering), and a RiscPC one (which began with 1.07, then went through versions 1.08 and 1.09). The VersionNum file in CVS was created with a version of 1.10, suggesting that it’s the RiscPC branch that we’ve got (and the changelog for our 1.14 has nothing whatsoever to do with buffering). So why wasn’t the Archimedes version used as the basis of the RiscPC one (the Medusa section of the log was started two months after Arc version 1.14), what happened to the code, and what was the debugging needed to get buffering working? Argh! |
Steffen Huber (91) 1948 posts |
Just thinking loud…if SCSISoftUSB is guaranteed to have exclusive access to a device, a read-cache could be easily implemented on the device driver layer (i.e. SCSISoftUSB itself). You would be working on block (sector) level, not file level. You would have to monitor the Read and Write low level commands. Of course, caching at FileSwitch layer would be preferable because it would instantly improve performance for all filing systems. |
Steffen Huber (91) 1948 posts |
Some things I remember of background ops: IIRC; Filecore only uses background ops when reshuffling files to deframgment them. I remember encountering many bugs in various SCSI implementations where background ops didn’t work properly all the time – you would usually only notice when your disc was filling up, so Filecore had to shuffle a lot. I also remember David Pilling’s TWAIN - it had a configuration option to turn background ops off, because so many SCSI cards would not properly support them. |
Trevor Johnson (329) 1645 posts |
Did anything ever become of Dominic Beesley’s disc cache proposal ? His website is still up, although it doesn’t mention this project. |
Jan Rinze (235) 368 posts |
Jeffrey wrote:
Which is what i have done and it seems to work. Although I have no idea what you mean by “remembering to fix up the code to remember R1 after calling FileCore_Create” all I did was remove the assert and set the DoBuffering to ‘T’ To make sure there is a buffer configured I have in my !Boot an Obeyfile that does: *configure scsifsbuffers 255 and I checked manually if it has been set, just to be sure. |
Jan Rinze (235) 368 posts |
Trevor wrote:
I had a reply from Dominic since i had thought the same thing :-) He replied that the sources are gone due to disc failure and he had abandoned the project. That being very unfortunate he also replied that the code was not well suited anyway. |
Jeffrey Lee (213) 6048 posts |
FileCore_Create returns two function pointers. In R1 it returns the pointer that should be called after a background floppy op completes, and in R2 it returns the pointer that should be called after a background HD op completes. At the moment the code only stores the value of R2, so it will probably fail badly if a background floppy op is requested. |
Jan Rinze (235) 368 posts |
Thanks for the info Jeffrey. Since I won’t be adding any floppy’s in the near future I will be safe on that one ;-) I am not sure if changing the SCSIFSbuffers actually changes it on the fly using *configure so I will hack in a CMOS default of 255 buffers and see if that changes anything. |
Jan Rinze (235) 368 posts |
USB discs are not by any chance considered ‘floppies’ from the point of view of the SCSIFS? Because forcing the buffers to 255 in the ROM seems to hang RISC OS. |
Jeffrey Lee (213) 6048 posts |
It wouldn’t surprise me if they’re treated as floppies. The USB drive numbers start at zero, don’t they? (Seriously – I can’t remember! |
Jan Rinze (235) 368 posts |
It looks like USB drives are treated as floppies and the FloppyCallback is not implemented. Since everything is in assembler this will become a daunting task to see how to implement it. Not my cup of tea unfortunately. It would be nice if we could set the buffering per USB disk so we can choose which one is the ‘root’ disk (where !Boot resides) and which ones will need to be treated as ‘hot pluggable’. |
Jan Rinze (235) 368 posts |
Is there still hope for a SD/MMC driver? If so we could use that as the ‘root’ disc and make sure it is allowed to buffer.. |
Jeffrey Lee (213) 6048 posts |
It didn’t look too difficult when I had a look last night – no more than a 5 minute job. Although I guess it could easily take longer if you’re not soo good with assembler :)
Yes. It’ll probably be the thing I look at next after the video driver. |
Jan Rinze (235) 368 posts |
hmm.. 5 mins? you did not by any chance take those 5 minutes to implement it, right? ;-) Wish I had an overview of the code like you have! For now I just fiddle with settings and such. |
Jeffrey Lee (213) 6048 posts |
I’ve fixed the floppy handling, and… it still doesn’t work. Reading the filecore docs in the PRM, and looking at the code, it looks like a fair amount of code will have to be modified to get background transfers to work. Regular foreground transfers just read N bytes starting from a particular disc address, but background transfers take a list of addresses to read from, and even have to deal with the foreground process extending the transfer list while the transfer is still in progress. The main complication with background transfers looks to be the way that SCSIFS converts the disc addresses – and the fact that SCSI_Op doesn’t support updateable transfer lists. Luckily support for updateable lists is optional, so it should just be a case of translating the list FileCore does give into one suitable for SCSI_Op (and potentially splitting it in two halves because FileCore can request for a certain number of bytes to be transferred as a foreground operation!). Plus of course I’m looking at the RISC OS 3 PRM, so I’ll have to try and track down all the 32bit/RISC OS 5 update notes to make sure that I’m using the RISC OS 5 version of the APIs and not the crusty old RISC OS 3 ones! |
Jan Rinze (235) 368 posts |
Jeffrey, I really appreciate the effort. For now I would like to say that anything more than the 5 minutes spent on it could wisely be spent on more useful tasks ;-) I am sure you like the challenge but I am also sure that we would love our main programmer to stay healthy and not be tempted to go overboard with random request of impatient Beagleboard owners like me. |