D4R issues

I’ve altered the code to use if >10MB then do a 20 shift if >1M do a 10 shift otherwise don’t shift,

By doing that, though, you are increasing the complexity of your code and reducing its efficiency: You are making two tests, instead of just one. Decide on a single total size above which a shift is necessary, and shift in that case, and only in that case.

Granted, the increase in complexity is only a very small one, but it’s something that is to be avoided if possible.

The reduced efficiency, though, may be less small – it might be small for a single download, but consider the situation when a user is using D4R to download a lot of files. Since these tests are being done when you’re calculating the percentages, that’s the amount of percentage calculations needed over the course of each download, times the number of downloads, times two.

I should really modify the slider as well to be finer than a percentage, as it’s not smooth at 0-100.

If you’re getting a reasonably consistent download speed (and the progress bar is being updated at regular intervals) it should be – but I wouldn’t bank on the download speed remaining consistent. Especially for larger files.

As for Ricks mentioning of windows time estimation (it’s much better under 2010) that’s fairly easy. Use a random number to decide how many weeks to increase or decrease the time by and do that again based on a random number.

You may think you’re joking, but I remember being appalled at the time estimation of a Vista computer belonging to a friend for writing some photographs to DVD. I can’t remember what units it used, but if you worked it out, the estimated time was indeed in a number of weeks. Luckily, however, it didn’t actually take that long!

i.e. I don’t want the slider to jump when it’s downloading smaller files (if possible).

That’s why I suggested that the total size part needs to be >100 for a percentage. 65536 will give a percentage, 64(K) will not.

I should really modify the slider as well to be finer than a percentage, as it’s not smooth at 0-100.

I guess multiplying by 200 or 400 (etc) would give you a finer grained result? One of my downloaders has the slider be 600 OS units wide. I’m happy with “jumping” 6 pixels on every %ge point because when things are slow people tend to be frustrated about things other than how smooth the display is. I’m actual fact, I only did the %ge calculation on each 16K block – spend more time receiving and less time worrying about visual fluff.

I suppose it further depends on whether the machine has and uses an FPU or emulates it.

There’s an FPU. I don’t think RISC OS does much with it.

But on a serious note, I need to add time estimation.

Please please calculate this using a heuristic method – work out from how much has been received and how long it has taken. It sounds obvious but it really seems like some software makes its guess based upon how fast the last <amount> took, leading to some utterly random estimations…

Do we really need every file transferred to be written into a displayed box that (hopefully) only displays the text for a time too short to read it properly or at all?

Yes. If they can’t get the “display pretty animation” right, imagine what a mess could result from having to first work out if the box is even necessary.
Logic would say to have a background timer to display the box if it takes too long. Well, logic might say a lot of things but I rather feel that the actual implementation would be closer to possessed windows that appear and disappear at random leading people to believe their computers are haunted (well, they are, that’s how they work, but that’s not what I mean).

Vince said what I’ve been hinting at – every unnecessary calculation wastes time. It might seem funny in a cooperative system where you can choose when to yield; however by stripping out the unnecessary you can concentrate more on the necessary…

Yes. If they can’t get the “display pretty animation” right, imagine what a mess could result from having to first work out if the box is even necessary.

I think you missed my point:

In RISC OS our configure for Filer allows two options

1. Display a box and show the file name of every file transferred “Verbose” (default)
2. Display nothing

A bit too binary for most users I would say. Would a simple, accurate count of files transferred not suffice?

In RISC OS our configure for Filer allows two options

Oh… We’re talking about RISC OS now. The part “The fact that without the swirly display the process would take so little time” indicated Windows, and I kind of assumed that it followed me.

Actually, we have three options. There’s the “Faster” choice.

Would a simple, accurate count of files transferred not suffice?

Not always. I usually copy with Verbose+Faster (I think that doesn’t bother to give a filename, just the amounts copied so far) but I switch back to non-Faster for a moment to see if this is going to be a TakeAges job, or not.

That said, FilerAction is doing a thankless job in unpleasant circumstances. Don’t expect bells and whistles.

Thanks for the feedback, again I’ve nothing to really comment – all common sense (I’m not saying I have any – but the comments seem logical)
At the moment optimising is fairly pointless. However as I’ve stated I need to support a huge number of downloads, at which point any optimising of any kind is likely to become essential.
At the moment the slider is set to one in a thousand, which still isn’t quite smooth. The slider is less than a thousand pixels across, so I need to sit down and work out a method of making it smooth and quick. I think using percentages isn’t the best method, there are much faster and smoother ways of calculating it.

One of the obvious problems is I can’t have 2000 “Filer” display windows open, so there will be a separate window – pretty much like a torrent download window which has all of the current downloads in it.
If you want a specific window, you will need to specifically ask for it.
One thing I should say, is that although it needs to support such a large number of downloads, that doesn’t mean they are downloading at the same time – queuing, pipelines and weighting should reduce 2000 to around 50 actual concurrent downloads.

As regards the time estimation, I wasn’t joking – I’ve seen Vista’s time estimation, my specific use of weeks was deliberate ;-)
I’m fairly sure the “standard” method now is to only do a calculation after a certain time span and as Rick suggested just take data downloaded within (for example) the last ten seconds and work the average speed out then divide that by the amount of data left. That’s just a guess and the way I’m thinking to do it.

Windows 2010 file copy is really pretty, I’m not sure of it’s actual usefulness but it looks pretty.

Please please calculate this using a heuristic method – work out from how much has been received and how long it has taken.

But that gives you a mean value of download speed to base a prediction on, which is rarely accurate and easily skewed. Median would be better, but impractical.

The best option, surely, is to take a regular point reading (either time or size based) and then base your ‘remaining’ prediction on an average of the last results, when is a tuneable number, dependant on whether your readings are time or size based. (ie. kb/s or s/kb)

The skew depends on the length of time it is based on. I was thinking of basing this on a 10 second slice.
If it’s less than 10 seconds, then it doesn’t really matter about giving a value at all.
Most downloads tend to stabilise after three seconds.
The problem with taking point readings, is they are massively variable – depending on hardware.
A combination of the lot would give a fairly accurate reading.
To give an idea, the data rate I use at the moment is calculated whenever the screen is refreshed.
i.e. after I update the display I store the time and set the amount downloaded to zero.
Then the next time the display updates I use the amount downloaded since the last update and the time passed to calculate the speed.
On my pandaboard this seems to be fairly static.

Just a small update, now I’ve got a machine again, I’m back on the case.
Increasing the socket buffer size to 32k does seem to have worked, on the MX6 the initial read gets 25k of data – reads after that are far lower (10k).
So I’m now downloading at 10-12MB/sec on the MX6, which is disappointing – so there is a lot more work for me.
The download speed fluctuates too much (between 10 and 12) so there is something funky going on somewhere.
Obviously getting to 80MB/sec is not going to happen, but getting above 15 would be a good step.
I’m looking forward to finding where the bottle neck is and seeing if I can work around it at all.
But on the MX6 the desktop stays really smooth, which is great.
Also there is no difference between saving to RAM or to disc, which is also good.

Thinking about this further, if I can get a socket buffer size set to be 256k then it would mean using a module would be a viable option (i.e. without using wimp_poll) as the centisecond timer would give up to 2.5MB/sec and the millisecond timer up to 256MB/sec – obviously they are all theoretical, a stable 10MB/sec just using wimp_poll_idle will be my next step.
It’s starting to get exciting again!

My theory, seems to have just been that :-(
Setting wimp_poll_idle killed the speed (down to 800Kb/sec)
So it appears the socket buffer size, whilst is vaguely useful, not too much.
32kbytes seems to be the largest value that gives any benefit – although even that is hardly worth it.
I assume that once any data is written to the socket buffer, no more can be written until the buffer is emptied again.
That’s “sub-optimal” to say the least.
If it could carry on until it’s full that would make sense.
So it looks like the best method is to have a wimp program to do the “downloading”, using a module just makes it more difficult and largely pointless.
Although I will provide a module at a later stage to interface with it via SWI’s, command line etc.

I’m going to try reading data of the socket, process it (i.e. decode headers, write to disc etc.) then attempt another read of the socket to read any further data – if that is sucessfull then loop, if there is no data to read give control back to the wimp.

Or something like that, I’ll have to give it some deeper thought….
Time for some James Pond or SR2000 now I think ;-)

Now that my stress levels have returned to an almost bearable level, I’ve managed to continue with this a little further.
I’ve managed to squash a nasty bug that was happening if the header data was returned with some file data.
It appears that apache and nginx work in completely different ways, which is not a surprise – but it’s interesting to see and it explains the slowaris bug.

I’ll need to rewrite the header code logic – which is now the third re-write of it, now that I have a hopefully, full understanding of how I want it to work.
But on my MX6 I’m only seeing download speeds of 5MB/sec, but the good news is that whilst doing that the OS is fully responsive.
I’m not quite sure why the speed has halved since I last tried, but not too bothered at this time. I have “upgraded” my router.

I’ve released a minor update for my program.
http://www.paymentlabs.com/riscos/qfa/d4r

It’s now at least usable, still alpha quality though – but it is functional and works without crashing (for me).
There are a lot of “what if’s” that will cause crashes I suspect.

The slow download speeds (5MB/sec) I am seeing I think are due to when my mx6 runs at 324MHz.

Next up is to clean up the primary buffer code ready for multiple downloads, which I’ve partially started – this stablised the download speed on my mx6 closer to 11MB/sec
I would be interested to hear what speeds are seen on the IGEPv5 or titanium, if anyone has the time/inclination.

My main interest of doing multiple downloads is to see if there is any speed difference at all downloading two files rather than one, i.e. will I get:
1) Two files downloading at 5.5MB/sec each
2) Two files downloading at 11MB/sec each
3) Somewhere in between.

Obviously this depends where the bottleneck is on my iMX6. If I get answer 2 or 3 – then I’ll look at implementing “chunked” downloading as an option earlier.
I remember downloaders used to give you the option of downloading from multiple mirrors at the same time. I guess with the introduction of broadband people stopped bothering as CDN’s were brought in.
But there are a lot of iso’s etc. where downloading is limited to 512KBytes/sec.

Sounds like what XP does – various download utilities were devised because breaking a file into pieces tended to run faster than a single piece, for some reason.
Please make your downloader capable of backing off if it gets weird errors. Quite a number of the free file hosts don’t like parallel downloads to non subscribers so the first connection works but others return 404 or 403 or the like. Don’t fail if a subsequent chunk throws an error…

Thanks Rick, will do. I was more thinking of having a “mirror search” option as the more cunning mirrors use ip based throttling.
I think there will need to be quite a few user configurable options as well for this.
There’s no point in “chunking” on say a 50k file for example.

I’ve done some more tweaking with the secondary buffer and now I’m getting up to 13MB/sec on an MX6 – although the problem I’m having is it’s using too little CPU so the clock speed keeps diving down, which then kills the wimp_polls and thus the download then dives to 3MB/sec. But that’s a good place to be I think, using too little CPU.

the problem I’m having is it’s using too little CPU so the clock speed keeps diving down

Adjust the low level CPU speed during the download? Portable_Speed2…
Then even if it drops the speed due to low CPU use the drop is minimised.

I’ll take a look at that, thanks.
Now I’m having a problem stabilising the speed – the problem seems to be due to disk writes “hogging” the I/O. (not sure if that is fixable)
If my last read attempt fails (due to no data) then it checks to see if the buffer is at the “low watermark” – if it is then it writes the data.
then it checks if a read is possible.
Obviously if the buffer gets full it is forcefully written to disk anyway.
256k for the low watermark and a buffer of 512k seems to work well.
At the moment I’m just using fwrite am I better using OS_File ?
FYI without the buffering the maximum speed seems to be 6-7MB/sec, with it it goes between 7-13MB/sec (I’d guess the average is around 12, without any definitive timings done yet)
Also I seem to have another bug crept in somewhere, when the download is quick I miss a bit of data…. GRRRR!
But hopefully the next version should be ready soon with the buffering much more efficient!

I’ve done a couple of minor tweaks, I’m now seeing 14MB/sec (sometimes) and fixed the downloading issues.
I definitely need to provide a configuration interface to use Portable_Speed2.
Buffer sizes have been greatly increased to provide much more stability in the download speed.
(Larger socket buffer, larger primary and secondary buffers)
I need to spend more “quality” time with the debug logs and produce a csv output so it can be analysed better.

I definitely need to provide a configuration interface to use Portable_Speed2.

So my off the cuff remark was useful. Good to know some of the grey cells work :)

I definitely need to provide a configuration interface to use Portable_Speed2.

There’s !CPUFreq and !CPUClock from !Store.

Yeah I use that at the moment, however that’s not really nice.
I was more thinking after Steve’s suggestion to up the speed if the download speed is actually affected by the CPU speed (which I can calculate), then increase/decrease the speed accordingly until it’s finished, then return it to the previous values.
Obviously it will be a configurable option.

I’ve done some more testing, found some more bugs and weirdness.
On my PC I can get 50MB/sec from my router, but from my MX6 I’m just getting a rate of around 9MB/sec (it still jumps up sometimes, I even saw 15MB/sec for a very brief moment!)
Yet as previously I can get much higher from my PC running Linux/apache.
So at least I’ve found something interesting I can debug that I can easily reproduce!

I’m guessing that I’ll get that weirdness sorted, fix some more bugs and leave it at a single download for the moment.
Possibly add the CPU scaling option in as well.
That way hopefully I can release a publicly usable, but definitely not great – 0.10 release.
In case anyone is wondering, internal TLS/SSL, DNS (and hopefully gzip) will be added before the 1.00 release – hopefully I can get moving quickly towards that.
[1.00 will be full HTTP 1.1 support, 2.00 will add another protocol (possibly imap), 3.00 not decided which yet – that’s a long way off!]

I’ve put up 0.08, which has a few bug fixes.
The main thing is that although multiple downloads aren’t supported (please don’t try!), you can at least keep it running and download one thing after another.
It’s getting close to a basic usable client now, ready for the 0.10 release – which will hopefully be largely usable (although nowhere near functionally complete)

I’ve found some interesting things out about the sockets on RISC OS, which maybe of interest.
The socket option to set the buffer size only works until the next read. Once you read the data the socket buffer goes back to it’s original size!
I found this out by noticing the downloads of my NAS were slow.
When I was testing it was down to 2-3MB/sec. When I took a look at the debug logs I noticed that all of the reads were small, with the exception of the very first one (which was 38k).
I also noticed I was also reading a Max of 38k but the sock buffer was set to 128k.
So I changed the read to be a max of 128k too.
This resulted in all the rest of the reads being small still, but the first read was around 52k.
At which point I remembered Rick mentioning that he couldn’t get the socket buffer to work…
Then I added the socket buffer option setting after every read.
The download speed went up to 9MB/sec and some of the read sizes went up to 78k.

The obvious reason for the actual slow speeds in the first place dawned on me though….
I stopped the music playing (MP3) and then the download speed was at the usual 13MB/sec
This time the reads were all much, much smaller around 10k each.

But at least now I can listen to music and get a reasonable download speed!

But it was interesting to learn! In case anyone else wonders.

The socket option to set the buffer size only works until the next read. Once you read the data the socket buffer goes back to it’s original size!

Did you actually read the size of the buffer after the read to prove this? I can’t see that anything changes the size of the receive buffer and setting the size of the recv buffer to the same value doesn’t do anything.