Crunching BASIC - again

This is a recurring topic, but it has arisen in a recent exchange and I want to solicit opinions.

There does not appear to be a BASIC cruncher in active development ¹, and I have long harboured an ambition to have Basalt crunch programs on loading.

Basalt already translates programs and the underlying mechanism is there, so I have written a utility in BASIC, Crunchie, as a way to work out and debug the necessary algorithms before attempting to replicate these in assembler.

This may be a useful as it stands, so I would be pleased if anyone would like to test it before I make it generally available. Let me know at steve@kappa.me.uk.

As it stands it reduces names, but it does not concatenate lines, so how important is that?

Are there any features that are otherwise important?

¹ Here are crunching programs that I know of:

BasCruncher 1.30 1989 BASIC
BasShrink 1.01 1991 BASIC
BasCrunch 1.02 1993 BASIC
BC 1.69 1995
BCrunch 0.01 2005
StrongBS 3.05 2009

The BASIC ones I only have in crunched form. BC is not 32-bit. BCrunch is Steve Revill’s, but does not appear to have been touched for a long time – Steve? StrongBS was made 32-bit in 2009, but has not been touched since. It does not have a command line option and I do find it rather intimidating for my purposes. ;-)

Are there others I have missed.

Edited for name.

My only contribution to this would be this: why replicate it in assembler? There’s absolutely nothing wrong with an app being written in BASIC, and for this particular purpose I’m sure BASIC would be plenty fast enough.

Are there any features that are otherwise important?

De-crunching?

If the reason for crunching is to keep secrets, it’s a bad thing! If it’s to save space, increase speed, it’s OK.

So much stuff has been effectively lost due to crunching for “security”.

So I would constrain use of such a cruncher to include source code, or incorporate an effective “uncruncher” if that were possible. Perhaps an obligatory “Undo” file? Can licencing do that?

Just my concerns, and yes, I have crunched myself!

BCrunch is Steve Fryatt’s

It is…?

why replicate it in assembler?

Only because that is how Basalt is written.

There’s absolutely nothing wrong with an app being written in BASIC, and for this particular purpose I’m sure BASIC would be plenty fast enough.

I agree. The techniques I have used depend on BASIC and can only be in code in Basalt because that operates in a BASIC enviornment.

So I would constrain use of such a cruncher to include source code, or incorporate an effective “uncruncher” if that were possible.

Surely you wouldn’t want to crunch source code? Perhaps crunch the released code, and make the source available separately for those who want it?

It is…?

Oops! I got my Steves tangled – Steve Revill, of course.

My apologies to all Steves involved. ;-(

crunch the released code, and make the source available separately for those who want it?

I think always include the source – no exceptions.

If the reason for crunching is to keep secrets, it’s a bad thing!

I entirely agree.

So I would constrain use of such a cruncher to include source code, or incorporate an effective “uncruncher” if that were possible.

The inclusion of the source in the crunched file is an interesting idea, now that memory and storage are hardly critical. I do not think a crunched file can be reversed in any meaningful sense.

The inclusion of the source in the crunched file is an interesting idea

That isn’t what I meant, of course, because I’m not that clever!

But it is very interesting!

If the crunched version, when it ran, ensured that an original version was present in the app directory, that would be great!

It would need to do some checks to ensure that no-one was trying to fool the system – perhaps a CRC check or something – but it would only need to write the original version once otherwise, and, after that, the full speed advantages of compression could be realised!

I think I’ve just accidentally generated a very good idea!

Whoopee!

If the reason for crunching is to keep secrets, it’s a bad thing! If it’s to save space, increase speed, it’s OK.

Hear, hear! I don’t bother to crunch my apps at all: I’ve no secrets, any gain in speed is small – where speed is that vital, I can use a bit of assembler embedded in the BASIC – and I don’t write enormous apps. (I’m not writing for A5000s or the like, or Risc PCs with 4MB memories – the data files I’m considering handling are generally too big for those.)

So much stuff has been effectively lost due to crunching for “security”.

I agree. But, if that was the author’s intention, then so be it.

So much stuff has been effectively lost due to crunching for “security”.

I agree. But, if that was the author’s intention, then so be it.

OK – but Steve’s version isn’t obliged to perpetuate that!

any gain in speed is small

Small, but not insignificant in larger programs.

It is much more significant if a program uses long, descriptive variable and routine names and is copiously commented, both highly recommendable.

It is much more significant if a program uses long, descriptive variable and routine names and is copiously commented, both highly recommendable.

Wasn’t there a discussion about the benefits of a particular variable being set which stripped much and optimised the programme loaded into memory, on the fly.
More to the point I think I recall that the default of of this being not set was changed to a default of being set.

Small, but not insignificant in larger programs.

Or small programs handling large amounts of data, resulting in massively repeated loops. This certainly used to be an issue with RiscPCs, but how often is it an issue with a Pi, or other newer machines? My !XP1Dr2SVG processes huge drawfiles into even huger SVGs – it’s not a big program, but it runs itself dizzy around loops handling its data – but it’s a hell of a big file that takes a whole second to process on a Pi. (Same file would have taken quite a few seconds on a RiscPC.)

I’m interested in Steve’s bit about optimization on the fly – anyone got a link to a description of this variable, and what exactly is going on there?

Wasn’t there a discussion about the benefits of a particular variable being set which stripped much and optimised the programme loaded into memory, on the fly.

That is BASIC$Crunch, and it has been mentioned here more than once, so a search should find it. It is not set by default, yet. However, that does nothing for long names, so crunching programs will have benefits anyway.

Crunching should be disallowed.

Crunching should be disallowed.

That’s pretty definite. I have to assume that you mean distribution of applications with only crunched files. If not, what is your objection?

There are a couple of ways to satisfy such objections.

You could have this in the !Run file:

*Crunchie <Obey$Dir>.!RunImage <Wimp$Scrap>
Run <Wimp$Scrap>

That would work with any command line cruncher.

Or, Crunchie could be written, in assembly, to be included as the first line of a program to do crunching in situ, rather as *BasaltInit does translation. This opens up further possibilities.

*Crunchie <Obey$Dir>.!RunImage <Wimp$Scrap>
Run <Wimp$Scrap>

I like that – if Crunchie is good enough!

None of the apps I’ve released into the wild contain any assembler – but would Crunchie cope with assembler routines that are assembled at run time? I only ask out of interest; it would of course be possible (in general…) to assemble them when you write them.

would Crunchie cope with assembler routines that are assembled at run time?

I have allowed for assembler mnemonics to be left alone, but not proof-tested this. I have seen a cruncher that does partial assembly – converting instructions to data words, but the ultimate cruncher would do the assembly and discard the BASIC that generated it. Crunchie is really quite simple, though.

I have to assume that you mean distribution of applications with only crunched files.

That’s it.

I don’t see that crunching confers any significant benefit. People talk about speed as if we were still in the 1980’s. All our modern platforms are fast enough when it comes to executing BASIC.

There is a very significant downside: readability, and from it the ability for others to comprehend what the app is doing, and how it’s doing it. Some of apps supplied with RISC OS could have been significant help in teaching users to become programmers; but no, they were crunched. Opportunity lost.

So, the overall score is: nothing worthwhile to gain, important stuff to be lost.

Please, people, don’t crunch your BASIC. Let other people learn from it.

I don’t see that crunching confers any significant benefit.

Repeated loops processing data is the most oft quoted excuse. I put that last word in italics as back when we used to crunch programs because our 8MHz processors needed all the help they could get.
But now?

Now I reckon a good algorithm and BASICVFP is capable of blowing away any assumed benefits of crunching…

People talk about speed as if we were still in the 1980’s.

Speed is like money. When you’re have it, you just want more.

You know why I upgraded to the Pi2? It wasn’t for the speed. It was for 4x the memory and more importantly lots more USB ports. I mean, RISC OS boots in like 15 seconds, there isn’t much difference between the Pi1 and Pi2 in that respect.
What did make a difference was a big enough RAMdisc to unpack the OS sources, taking a couple of hours down to a couple of minutes, and replacing a Class4 SD card with a Class10 one – a shocking difference.

So how fast a program actually runs can be bottlenecked in other places, meaning crunching or not is unlikely to make that big a difference, and if high speed was really really important…. You’d be writing it in C or something compiled to native code anyway. :-P

All our modern platforms are fast enough when it comes to executing BASIC.

I agree. GHz platforms, it’s insane!

There is a very significant downside: readability,

Indeed. I’ve been sent a program to look at (the one I quoted the assembly-as-a-string from a couple of weeks ago). The program has been heavily crunched and, sorry, it’s gibberish. I’m not really getting anywhere with making it 32 bit even though not that much is assembler, it keeps crashing.
Honestly? If I had some example files and a spec of the arcane formats it loads, it would be a quicker job to just rewrite the bloody thing…

Some of apps supplied with RISC OS could have been significant help in teaching users to become programmers; but no, they were crunched. Opportunity lost.

I’ve seen the code to Maestro. ;-)

That said, back in the day a good number of programs on the Acorn User / Micro User / Archimedes World cover discs were not crunched. How d’you think I learned to program RISC OS? ;-)

Please, people, don’t crunch your BASIC. Let other people learn from it.

Or a real life alternative: Please, people, don’t crunch your BASIC. In a decade when you’ve wandered off to create iOS apps, others can maintain your program if they still use it…
And, believe me, the list of ancient programs that people still use is huge…

Or a real life alternative: Please, people, don’t crunch your BASIC. In a decade when you’ve wandered off to create iOS apps, others can maintain your program if they still use it…

I feel that you and Dave might be missing the point…

The issue isn’t crunched BASIC. The issue is the availability of source code. Just having the uncrunched !RunImage file from an application isn’t going to help keep the software alive (for non-programmers, at least) after the author has wandered off to develop apps for iOS.

What’s required is that the applications are Open Source. That implies uncrunched BASIC¹, but also enables other people to update the code, and distribute those updates, after the original developer has lost interest.

Instead of demanding daft things like outlawing the crunching of BASIC, why not just encourage all non-commercial RISC OS software developers, in any language (BASIC, C, or whatever), to give serious consideration to making their software and all of their unique build tools Open Source? Aspiring developers can learn from studying other people’s C or Assembly Language (if those are their chosen fields) just as much as BASIC.

¹ …or an alternative. My BASIC apps are distributed CRUNCHed, but the source is available from the same page as the application download. Having had several conversations over the years when people have said “App X is really unfriendly to install, because it comes with a whole bunch of source code that I don’t understand”, I’d suggest that the best option is to keep the two well and truly separate.

I agree. In fact, I’ve finally done what I’ve been meaning to do for 18 months and have put the source to Seasonal on my site. As of this moment, all of my stuff is open source (not that anyone uses it, but that’s beside the point!)