D4R issues

Just wondering if anyone knows if you can actually read data off the socket when an internet_event is called?
I can detect socket events and prevent socketwatch etc. from working, but on an internet event handler I’d swear that you can’t read any data.
I’ll need to learn DADebug to prove this one way or another, which is a good thing either way really.

After debugging and cleaning up my code and more investigation, I now realise why code that uses socketwatch is so dismal.
It’s seriously impressive that some of the code that uses it manages to download at anything other than bytes per second!

After some more debugging I’ve also found that you can either use FIONBIO (non blocking I/O) on the socket and use select, or use FIONREAD to detect if there is data on the socket and use blocking mode if you really want to.
Both seem to have the same effect (assuming you don’t attempt to read on a value < 1).
So I think I’m going to use non-blocking mode and use FIONREAD, as if it’s less than a certain value I can perform some time-slicing calculations before I read the socket.
Very minor optimisation, I admit as I’m getting both reading data at around 5840 kbytes/sec on a pandaboard.
Interestingly on a pi2 I get 9770 kbytes/sec running the exact same program, as against 5450 kbytes/sec on a Pi1 rev2
Both are running RC14 of the Pi ROM.
That’s not quite at the maximum speed on the Pi2, but it’s not bad at all.

I removed printf’s from my (benchmarking) program for null reads (produces quite a bit) and the Pi 1 then reached 7030 kbytes/sec
The Pi 2 however reached 11,010 kbytes/sec which is lovely, it’s pretty much at the maximum network speed for 100Mbit.
The Pandaboard was unchanged.

The tests were doing a http get off my PC at home downloading some random DVD iso image

At least this means I’ve got something to aim for! Get the same results (or better if that’s possible) multitasking in a module.
Which I don’t think it is, I think the maximum is going to be 1700kbytes/sec as that would be 17k (which seems to be the maximum amount you can read off a socket) * 100.
100 assumes I use TickerV, I wonder if there is anything else finer than TickerV.

Just a quick update to let you know work is progressing.
Toolbox does work fine under modules, it’s just a case of interpreting errors correctly.
Module errors are more FU somethings gone wrong.
All that was needed was a little break, a rethink and playing with other things RISC OS related for awhile to remove frustration caused by my own lack of knowledge.
I’ve now got Impression-X to play with as well now, as well as overlord to distract me when my frustration levels rise again!

I’ve managed to make some progress again.
I’m now downloading at 3MB/sec on my pandaboard.
It’s lower than the limit of 5MB/sec on the panda, but I’ve got some work to do on that next.
Then some functional changes and testing on the Pi2 to ensure it can get to 11MB/sec downloads.
Once that’s done, I would expect in around 2 weeks I’ll release an alpha preview.
The module part for the moment has been scrapped due to memory allocation issues I can’t work out.
But as the module will be just providing SWI/CLI interfaces – it’s not a massive amount of work to integrate that later.

The module part for the moment has been scrapped due to memory allocation issues I can’t work out.

What in specific?

I can use toolbox to load in the templates as long as they only use a tiny amount of memory.
As soon as it’s more than that the module barfs.
I’m assuming I need to load the toolbox and other stuff into wimp memory, but it’s being loaded into the RMA.
e.g. if I have an iconbar, and an menu with info and quit it’s OK.
However when I add an info window it crashes and if I add a choices item on the menu.
Specifically I think it’s the autocreate option.
So I’m thinking I have two options, one is to pass a memory block to toolbox to use (don’t know if you can)
Or manually create each window in a specified memory block (again not looked if this is possible)

I’m starting to get to grips with toolbox and oslib stuff now and slowly understanding how it actually works – rather than cut & paste guessing.

Oh, and I’m really enjoying Ergo Proxy ;-)

I’m assuming I need to load the toolbox and other stuff into wimp memory, but it’s being loaded into the RMA.

Surely the toolbox modules use their own private storage to keep details of the objects you create. You just need to keep copies of the window or menu handles, etc., which are returned when the object is created or autocreated.

@Chris – That was my assumption, but when it tries to create toolbox menus it barfs. I’m not sure of the actual fix I’ll need to do – but it’s to do with toolbox and memory allocation, or at least that’s my current theory. I suppose at the moment I don’t care enough to track it down, as I’m actually making good progress without a module.
Also it’s slightly disappointing that because the maximum “polling” (Vector etc.) within a module it’s way slower than using wimp poll. (100Hz I assume)
I have a tendancy of trying to fly before I can walk ;-)
I think I’m walking now though, just need to learn to run fast enough now for take off speed!

I’ve managed to get to 3.6 to 4.2MB/sec, so I’m hoping my last set of tweaks will bring this up to or close to a stable 5MB/sec – which is still poor, but that’s all it seems the poor panda can do.
FYI This is under the wimp, so multi-tasking.
The two main issues I’m seeing are some bottleneck on the pandaboard which limits the bandwidth on the NIC and the slow writes to the SD Card.
The CPU and wimp_poll can easily keep up with the data stream (its reading the data off the NIC at one frame at a time at the moment, which is impressive) – but wrong it seems! Closer analysis seems to indicate this is the pandaboard borking randomly when it reads slowly.

Thanks to a heads up from Rick, I’ve added some code to set the socket receive buffer size.
This seems to work, and doesn’t crash – however it doesn’t seem to affect the download speeds.
Also I’m not convinced it is actually setting the socket receive buffer size as the maximum reads are still around 17k.
That said, it’s difficult to say at the moment as there is “something” that is causing a bottleneck. At certain times, I’m reading around 2k (one frame) one after the other – yet the download speed is only at 3.5MB/sec. At other times I’m reading 15-17k one after another. The fluctuations are very interesting. Also I would like to work out why sometimes I get 4.2MB/sec and others only 3.5MB/sec, very interesting (from my point of view!)
Anyway, the rewrite is nearly finished – just a little bug to find (it’s not writing the last chunk to disk on certain downloads)
After thinking more, it’s probably my debugs that are causing the slow down to 3.5MB/sec
I’m going to try with the larger socket buffer and using poll_idle, I’m hoping that will hit the sweet spot!

I thought I’d put up a seriously alpha version, in-case anyone is interested
I’d recommend a reboot after using it, there are bugs, plenty of them!
If you save a link in netsurf as a text file to disk, then drag that file to QFA then it will bring up a save dialogue.
It doesn’t appear to save all of the file if it’s a large file.
If you quit without attempting a download it barfs. (that’s lazyness on my part, as this is a pure test program at the moment)
I can’t convert bytes to MBytes as I can’t seem to do any maths on the values, which is really strange! X/1024/1204 != X/1024/1024 apparently.
And when I do the conversion the percentage get’s messed up as well. I need to debug that, somehow :-(
The largest file it can handle is around 4GB (unsigned int)
But it shows off the download speeds at least.

I think a complete rewrite (again) is in order, I really don’t fancy debugging a memory leak – and at least I know how my code needs to be structured now.

I can’t convert bytes to MBytes as I can’t seem to do any maths on the values,

Why would you want to? https://www.riscosopen.org/wiki/documentation/show/OS_ConvertFileSize ;-)

which is really strange! X/1024/1204 != X/1024/1024 apparently.

No, it is right. 1204 isn’t the same as 1024. :-P

Seriously, why are you using divisions to change the magnitude of a file size? You could use “>> 20” (>>10 per 1024).

And when I do the conversion the percentage get’s messed up as well. I need to debug that, somehow :-(

Are you using 100 / {total} * {current} to calculate the percentage?
You should be using {current} * 100 / {total} instead.
Why?
Simple. Both methods work – however if we assume that the total file size is “66” (K, M, it doesn’t matter) and the current is “33”, the percentage is clearly 50% and entering this into a calculator will tell you that.
HOWEVER…
100 / 66 is 1.515151’
while:
33 * 100 is 3,300

The difference is the second method always creates larger numbers so can work with straight integers. The first method may require floats if it isn’t to lose data.

The largest file it can handle is around 4GB (unsigned int)

Can RISC OS itself cope with more?

For what it is worth, I did a test using the Norcroft/DDE compiler to see what it could optimise.

#include <stdio.h>

int main(void)
{
   int  meh = (17 * 1024 * 1024);

   printf("meh long  = %d\n", (meh / 1024 / 1024));
   printf("meh short = %d\n", (meh >> 20));

   return 0;
}

You would expect the compiler to spot the /1024/1024 and bitshift it, right?
Uh.
No.
Norcroft does better than that.

00008090 : .Ø-é : E92DD810 : STMFD   R13!,{R4,R11,R12,R14,PC} ; *** Offset not guaranteed
00008094 : .°Lâ : E24CB004 : ADRW    R11,-&00000004
00008098 : ..]á : E15D000A : CMP     R13,R10
0000809C : 6..» : BB000136 : BLLT    &0000857C          ; call: _kernel_unwind
000080A0 : .@ ã : E3A04011 : MOV     R4,#&11            ; =17
000080A4 : .. á : E1A01004 : MOV     R1,R4
000080A8 : ..â  : E28F0F05 : ADR     R0,&000080C4       ; -> string: "meh long  = %d."
000080AC : H..ë : EB000148 : BL      &000085D4          ; call: _kernel_alloc
000080B0 : .. á : E1A01004 : MOV     R1,R4
000080B4 : ..â  : E28F0F06 : ADR     R0,&000080D4       ; -> string: "meh short = %d."
000080B8 : E..ë : EB000145 : BL      &000085D4          ; call: _kernel_alloc
000080BC : .. ã : E3A00000 : MOV     R0,#0
000080C0 : .¨.é : E91BA810 : LDMDB   R11,{R4,R11,R13,PC}

(I think “_kernel_alloc” ought to be “_printf”; the older disassembler isn’t aware of the newer jumptable layout)

Very droll. Yes. Very.

For the lulz, I gave the same program to TurboC++ v1.01. Even switching to unsigned long, it just wasn’t going to insert the value into the “meh” variable until I casted every single part of the calculation. Pedantic piece of….

Anyway, enjoy this god-awful mess. It has been clever enough to spot that a division by 1024 can be replaced with a shift, but astonishingly it has to call a function to do this as the value is composed of two 16 bit halves so shr can’t be used. This code is using 80286 mode (hence the use of enter and leave) but 32 bit didn’t arrive until the 80386… The 80386 was introduced in ‘85 and the 80486 in ’89 and this compiler is dated 1990. Go figure.
Additionally, it hasn’t been clever enough to spot two identical shifts and to add them together. Rather literal reading of the input? Norcroft is so utterly in a different league!

_main	proc	near
	enter	4,0
   ;	{
   ;	   unsigned long  meh = 17ul * 1024ul * 1024ul;
	mov	word ptr [bp-4],0
	mov	word ptr [bp-2],272
   ;	   printf("meh long  = %lu\n", (meh / 1024 / 1024));
	mov	dx,word ptr [bp-2]
	mov	ax,word ptr [bp-4]
	mov	cl,10
	call	near ptr N_LXURSH@
	mov	cl,10
	call	near ptr N_LXURSH@
	push	dx
	push	ax
	push	offset DGROUP:s@
	call	near ptr _printf
	add	sp,6
   ;	   printf("meh short = %lu\n", (meh >> 20));
	mov	dx,word ptr [bp-2]
	mov	ax,word ptr [bp-4]
	mov	cl,20
	call	near ptr N_LXURSH@
	push	dx
	push	ax
	push	offset DGROUP:s@+17
	call	near ptr _printf
	add	sp,6
   ;	   return 0;
	xor	ax,ax
	jmp	short @1@50
@1@50:
   ;	}
	leave	
	ret	
_main	endp

27 instructions with 7 branches versus 14 instructions with 4 branches.
Plus, x86 is fugly as hell to look at. [so nerrr! etc]

Well, you’ll like my current attempt. I was wondering if it was an issue with doing maths on unsigned ints.
So for the “lulz” I sprintf’ed the unsigned int into a string, then used atof then did the calculation, then sprintf’ed that into a string for the icon.
I have a feeling my actual problem is some variable misuse on my part (i.e. I’ve got confused and used the wrong variable in one of my library functions)
It doesn’t really make sense, I have two strings “current icon value” and “previous icon value”.
If these are different I update the icon.
Now this works OK, but if I try and convert these to KBytes/Mbytes/Gigabytes then the total file size variable magically changes (which is a totally separate variable).
I’ve not had chance to debug it.

I think RISC OS has a 4GB file limit. I did think about handling any filesize by chopping the string up into 32 bit numbers. If I can think of a reason to do it I may do – possibly for streaming where you aren’t saving the file.

At the moment there is zero optimisation, as my focus is on memory/buffer logic.
But you’re entirely right about this!

Very interesting about the percentages, I never thought about it! As you guessed I use floats to calculate it – which isn’t good.

OS_ConvertFileSize looks lovely, the more I can whine about other peoples bad code the better ;-)
Mine is bad enough!

So for the “lulz” I sprintf’ed the unsigned int into a string, then used atof then did the calculation, then sprintf’ed that into a string for the icon.

Haha. Yeah. That’s one way I guess…

…oh bloody hell, you’re being serious, aren’t you?
<gulp!>

Now this works OK, but if I try and convert these to KBytes/Mbytes/Gigabytes then the total file size variable magically changes (which is a totally separate variable).
I’ve not had chance to debug it.

Look at how you are passing the variable. It sounds like you want to pass by value (your function will be using a copy of the file size value) but you are instead passing by reference (your function will be using a pointer to the file size value, in other words, the real thing).

I don’t experience this in my programs because I (almost) never modify any of the variables passed to a function. For example:

long div_by_kilo(long filesize)
{
   filesize = filesize << 20;

   return filesize;
}

versus:

long div_by_kilo(long filesize)
{
   long filetemp = filesize << 20;

   return filetemp;
}

I may be speaking out of my backside (I can’t test as I’m sitting in the sun in the garden ;-) ) but you might have a function accepting the file size variable by reference and then modifying it.

For instance:

long div_by_kilo(long *filesize)
{
   filesize = filesize << 20;  // will modify the REAL variable
}

and:

long div_by_kilo(long filesize)
{
   filesize = filesize << 20;  // will modify a COPY of the variable
                               // (and leave the original untouched)
}

The difference? A single solitary ‘*’ which instructs C to pass a pointer instead of a copy variable.

If I can think of a reason to do it I may do – possibly for streaming where you aren’t saving the file.

The problem here is that when you are streaming, the end position is potentially infinite. When are you going to stop listening to the service? Assuming you can know the future and know the instant you will turn the stream off (and ignoring the fact that the mere fact of knowing and telling your program will affect the time indicated), assuming you know this (and also believe in predestiny), then a percentage makes sense. Otherwise… a percentage of what?

OS_ConvertFileSize looks lovely, the more I can whine about other peoples bad code the better ;-)

Well, maybe it ought to handle GiB sizes these days? How would a DVD ISO image show up? Something like 4000 MBytes ?

A quick deviation while it is in my mind (nothing to do with RISC OS) – I used to use DVD-Rs to back up my data and while this was not a hardship when 8GiB to 20GiB harddiscs were the norm, it is rather less so now that storage is typically fractions of or multiples of terabytes and mere SD cards can be many times bigger than the capacity of a DVD.
What would be the ideal way of backing up things these days? Do people use some sort of high density tape? Is there such a thing as a Bluray writer? The way I currently deal with this is to save up my pennies and buy another 500GB (denary!) harddisc, but I don’t know the long term reliability of these so I dupe a lot of stuff. … plus, while all media fails, it is often possible to recover a lot of data from tape (snapping excepted) or optical by skipping over the bad parts. I have had three disc failures. One was my fault (dropped a screwdriver on it dead-centre while it was spun up, oops!), the other two were not my fault. The one that annoyed me the most was the FileStore E60S drive that just didn’t spin up one day when I turned it on. It was annoying because the stupid FileStore requires a specific type of Rodime harddisc with a specific marker in a write-once block on the disc and… ugh. That was part of the purpose behind the FileStore emulator I started writing (in VisualBasic (!) for the lulz, of course!), to identify exactly which bit of code was doing this and nuking it, because the “undocumented” configuration option to disable the checking didn’t seem to work – the FileStore simply refused to see anything else plugged in. And, then, the Beagle and the Pi came along and I decided that I didn’t really have the time or the inclination to continue with the FileStore in the absence of firmware source code. I have neither infinite time nor patience and that was pretty much a lost cause. Anyway, I’m not entirely sure I trust harddiscs as my experience is when they fail, it’s just kaput, the end, dead.
So… what would make a good backup system for the larger amounts of data these days?

The sprintf was just to prove 100% that it wasn’t the maths causing the issue – which it isn’t by the looks of it – so whilst I was being serious about doing it, it’s going to be removed as that would be a new level of insanity, even for me!
My functions just accept one parameter a struct. I’m coding sections in main, then once I know what I want I turn them into a function call.
At the moment these are just the download/header/buffer handlers, which require a ton of parameters.
I’ve just got around to doing the GUI side of things.
I agree it’s much nicer to return the value, rather than requiring a pointer on the input.

As regards backups, it depends on how much you need the data.
If you really want your data, then I’d recommend a blueray writer.
I’ve got a USB bluray writer, I backup my NAS every month or two.
At the moment I’ve got a single USB disc attached to my router, which I copy…er.. sometimes…onto my spare usb disc.
The discs I use are 25GB, I only backup the stuff I really need though,
so it easily fits onto one disc.
The more important stuff I sync to my online server, which then also gets ftp’ed to another server ;-)
It all depends on your level of paranoia!

A disc copy is not a backup, it’s just a copy – which is handy, but it’s not a backup that should be relied upon.

If you want to backup a lot more, then realistically your into enterprise kit…
LTO-6 does 2.5TB (uncompressed) on each tape.

LTO-4 would probably do, we see around 110MB/sec writing to tape which is nice.
They start at around a grand looking online.

For home office use, I’d probably recommend a NAS with mirrored disc (QNap or synology) and backup every month or so with a blu ray disc.
You can even sync the NAS drives if you want one at your house and another somewhere else.
A mirrored NAS can be picked up from around 200-400 GBP I think. (2 discs)
I’ve been looking at getting one recently, they look quite nice.

There’s also “cloud” storage, you can get free amounts of various providers.
Whether you want to do that is another matter.

So:

1) How much are you willing to spend
2) How much data do you actually need to backup (people always say everything, until they are presented with the cost)
3) How important is the data (i.e. can you reconstruct it, or recover it)
4) How often does your data change
5) How quickly would you need access to your data if you lost it.

For example if you have a 2 TB NAS with 1.5TB of videos, 0.4 TB of pictures and only 0.1 TB of data.
You may not care about the videos so much, as you already have the originals.
The pictures you may have copies of and you can do a backup every so often.
That leaves 0.1 TB of that how much is rubbish/replicated.

Data management is the key ;-)

I could go on, I must stop!

This is interesting….

2% 42948634*100/1969803264
whereas
0% 42951538*100/1969803264

It looks like I can’t multiply by 100 first as it rolls over the unsigned int size, so floats it is.

Anyway I got to the bottom of my issues
Ensuring correct casting in all places.
Don’t increment the amount read when we haven’t actually read any data!

If you really want to multiply by 100 first, then perform the division, consider shrinking the numbers in the first place.

Using your examples, with the percentages shown as integer results (I’m assuming you aren’t after higher precision):

( 42948634 / 1969803264 ) * 100 = 2% and
( 42951538 / 1969803264 ) * 100 = 2%

Consider using >> 10 on both the amount and total before calculating the percentage – i.e. calculate the percentage on kilobytes, rather than bytes:

( 41942 / 1923636 ) * 100 = 2% and
( 41944 / 1923636 ) * 100 = 2%

Now, Rick’s suggested method should be workable:

41942 * 100 / 1923636 and 41944 * 100 / 1923636 should both return 2%.

The reason this works is simply that before dividing by 1024, the original value was approximately 10x bigger than the shifted version will be once it’s been multiplied by 100. If the variable type you’re using can cope with the original value, it can cope with that.

If necessary, you could check the total size before deciding whether you need to apply the >> 10 to the values – so for a total size that’s small enough, it doesn’t happen.

The main problem is that I don’t really know what accuracy I need until I get the file size.
If it’s a small file then I’d need fine accuracy, but doing it by multiplying by a hundred first would be fine.
However if it’s a large file then the accuracy can be far less.
So I think I’ll use both methods depending on the file size.

The main problem is that I don’t really know what accuracy I need until I get the file size.

Remember, a percentage is a “per cent” – a part of a hundred. So any “total size” number over 100 is more than enough to obtain a valid percentage.

( 42948634 / 1969803264 ) * 100 = 2% and
( 42951538 / 1969803264 ) * 100 = 2%

PRINT ( (42951538 >> 20) / (1969803264 >> 20) ) * 100
2.12992545

I should point out that Vince’s method would not work with integer arithmetic, as the result of ( 42948634 / 1969803264 ) is 0.0218035145. Any percentage calculation that multiplies by 100 after the division will be working with numbers in the range of 0 to 0.99.

So…

PRINT ( (42951538 >> 20) * 100) / (1969803264 >> 20)
2.12992545

Same end result, but can work with straight integers. ;-)

In reality, I would recommend the shift being optional; if the file is over a megabyte then shift >>10. You are very unlikely to transfer files in the gigabyte realm on RISC OS (and anyway, >>10 then *100 is still within range of a 32 bit int) and the cut-off means that smaller sizes such as 64KiB use a value of 65 thousand (plenty of room) instead of 64 (less than 100 so potential loss of accuracy).

On the plus side, a user is less likely to notice a percentage going up erratically on a file that takes two seconds to download as opposed to one that takes two hours. ;-)

However if it’s a large file then the accuracy can be far less.

Hey – you would have to try to be as bad as the Windows “Time remaining” estimations.

My functions just accept one parameter a struct.

Ah-ha. You’re using pointers, so if you alter the contents of the struct, it will have global effect.

As regards backups, it depends on how much you need the data.

That is one of those great unanswerable questions. I would not like to lose the data, but if civil war breaks out or I suffer a major heart attack – then what’s on a harddisc will cease to be of great importance…

I’ve got a USB bluray writer, I backup my NAS every month or two.

Mmm, what do they hold, around 25GiB apiece? I’ll need to look up the prices of the writer and discs.
Is Bluray “just bluray”? I mean, there isn’t this +R and -R nonsense is there?

The discs I use are 25GB, I only backup the stuff I really need though,

Ah, so Bluray is 25GiB.

It all depends on your level of paranoia!

I would prefer a static optical media for the basic ability to stuff them into an envelope and stick them out in the barn – for the basic reason that it doesn’t make any sense to hold backups in the same place as the live copy. If a piece of the ISS should crash into my bedroom, that’ll be everything wiped out…
[I ought to make some backups at least of the source codes and put that in the barn; it’s one of those things I’ll get around to doing, but never do…]

A disc copy is not a backup, it’s just a copy – which is handy, but it’s not a backup that should be relied upon.

Ah, but a disc copy on two separate discs is more a backup than a single copy. ;-)

LTO-6 does 2.5TB (uncompressed) on each tape.

That looks nice. The price is more than a month’s pay, so… yeah… scribble that on the “when I win the lottery” PostIt.

There’s also “cloud” storage, you can get free amounts of various providers.

That’ll be a flat NO.
I won’t push personal data¹ to any cloud service hosted in the US or via a company based in the US – it’s a matter of principle.¹ It isn’t tin-foil-hat either. I have no illusions regarding France looking at stuff on French servers – I just don’t like the attitude of the Americans towards their supposed allies and citizens of these countries. Sure, the loss of my data is no big deal. I hope, one day, you can multiply me by a million (oh second thoughts, multiply somebody else by a million) and then maybe it might start to get noticed. I just wish the EU had the balls to demand that the EU’s own data protection laws were followed for EU citizens – as any service hosted in America or by an American categorically cannot provide this as they will be subject to The Patriot Act and the “All Your Data Are Mine” provision.

The other reason is purely technical – with 2mbit down and 0.5mbit up, cloud services are kinda pointless…

2) How much data do you actually need to backup (people always say everything, until they are presented with the cost)

Hehe… ;-) My personal data (source codes and such) can be backed up on an SD card. My photos might take two if I take out the rubbish and the near-dupes (you know, in order to get the right photo you just keep tapping the icon until there are a dozen potentials to choose from).

The harddisc content? Mostly animé and dorama and dead wet girls. There are DVDs of some things available, but being English/Chinese subtitled and fitting a 12 episode (~54 minute apiece) series on to 2 or 3 DVDs, I’m not sure it is entirely legitimate – been there, done that.

Data management is the key ;-)

I don’t have time for that! Just get another harddisc and draggy-droppy. ;-)

¹ I do make use of Google Docs (via Google Drive) but since the majority of that is rough copies for my blog (posted publicly) and user guide info (also eventually public), it is less of a concern than, say, scans of my X-ray and medical notes (for instance).

Mmm, what do they hold, around 25GiB apiece? I’ll need to look up the prices of the writer and discs.
Is Bluray “just bluray”? I mean, there isn’t this +R and -R nonsense is there?

Blu-Ray is Blu-Ray, but there are the R (write once) and RE (write and erase) variants. As well as the LTH and HTL variants. And the latest member of the family, the M-Disc, which claims an archival life of more than 1000 years.

And there are various capacities around – 25 GB, 50 GB, 100 GB. However, I would not recommend the 50 GB and 100 GB variants for long-term backup because of their multi-layer nature and the excessive prices. 100 GB media is not universally compatible, for the M-Disc you need special writers, but it is said to be compatible with all readers.

If it’s Japanese Horror, then I’d recommend definitely backing it up ;-)
The less data you have to backup, the quicker it is, so the more likely you are to actually back it up.
Or that’s my principle of work.
Of course at work I can stream onto quite a lot of tape drives/libraries at over 100MB/sec each drive from storage running between 200 and 800MB/sec.
But that’s more than a lifetimes pay!

This is a good example of what can happen
Apart from the horror story of an OS X server! (shudder)
There’s lots of variations on this too.

I use BD-R (write once) – they are fairly cheap for a 25 pack.

I knew – and agree with your viewpoint on cloud storage, I mentioned it in case anyone else read it.
Hosting in Germany is always good, due to their correct viewpoint on data.

I’ve added a little screenie from tonight’s work
There’s no real usable change apart from the display side working, so no update.
But once I fix the “chunk of data missing” from the downloads (which I think is a stupid oversight) I’m moving on to get multiple downloads working.
Thankfully I’ve worked out how not to have to do deal with memory fragmentation, which scared me to death for about 5 minutes.

I’ll need to revisit my structs once I’ve finished and clean up and optimise it all.
Linked lists or pointers to pointers….oh the wonders of C! I think it’s slightly cleaner with linked lists.
Reminds me of the days I was into multi-dimensional arrays (5-10 dimensions) I think LSD is probably lighter on your mind than those. (Don’t ask, I don’t want to remember)

I should point out that Vince’s method would not work with integer arithmetic, as the result of ( 42948634 / 1969803264 ) is 0.0218035145. Any percentage calculation that multiplies by 100 after the division will be working with numbers in the range of 0 to 0.99.

Quite so – I wasn’t necessarily suggesting he use that method of calculating the percentages. That’s just the method of calculation I normally use in real life (quite often in my line of work). I therefore instinctively used it to express the calculations and illustrate their results before and after the >>10

Also: Malcolm, as Rick points out, a percentage point is a percentage point is a percentage point. There are 100 integer percentage points between 0 and any size of file.

If you are just using it to display a progress bar, for example, then an integer result should be fine. (And given what D4R is, that’s what I’d guess you are doing – hence my assumption you weren’t interested in higher precision).

Plus, if you are working with integer values, as per Rick’s suggestion (and the reason for multiplying by 100 first, as well as the reason for bit shifting to avoid problems with numbers being too large) then you’re likely to be looking at integer results anyway.

I’ve altered the code to use if >10MB then do a 20 shift if >1M do a 10 shift otherwise don’t shift, that way there’s no loss in accuracy.
i.e. I don’t want the slider to jump when it’s downloading smaller files (if possible).
I should really modify the slider as well to be finer than a percentage, as it’s not smooth at 0-100.

I don’t how much of an optimisation that is, but it’s looks cleaner to me (no casting) and avoids the horrible use of floats – I suppose it further depends on whether the machine has and uses an FPU or emulates it.
So thanks for that, I really appreciate the advice.

As for Ricks mentioning of windows time estimation (it’s much better under 2010) that’s fairly easy. Use a random number to decide how many weeks to increase or decrease the time by and do that again based on a random number.
But on a serious note, I need to add time estimation. I’m downloading a fedora iso as a test, which if I have a local PC it’s not too bad, but I don’t at the moment and I’m getting around 512KBytes per second – so it would be nice to know when it will finish!

As for Ricks mentioning of windows time estimation (it’s much better under 2010) that’s fairly easy. Use a random number to decide how many weeks to increase or decrease the time by and do that again based on a random number.

The problem with MS time estimation is a result of the implementation of the display and active transfer process monitored.

A logical method would be to implement a file transfer process that called / updated a GUI object to display the progress (possibly with a swirly bit of paper or such like)

However what the numpties did was implement a swirly bit of paper display that calls a transfer process at intervals resulting in a transfer that is totally dependent on GUI update speed.
The fact that without the swirly display the process would take so little time that there wouldn’t be a displayed object for any meaningful human scale time seems to have escaped them.

Which reminds me: Do we really need every file transferred to be written into a displayed box that (hopefully) only displays the text for a time too short to read it properly or at all?