Crunching BASIC - again

One of my occasional glances at the recent posts on the forum while still having a huge backlog of posts to read in my RSS client…

In all this discussion of whether or not people should crunch or not (or include uncrunched versions), there does appear to be a subtlety in Steve’s original post that nobody has spotted – and, indeed, Steve’s replies doesn’t highlight the point AFAICS. I’ve added some emphasis to a couple of words:

I have long harboured an ambition to have Basalt crunch programs on loading.

If I’m reading that correctly, Steve isn’t talking about a cruncher that allows someone to distribute crunched BASIC. Instead, he seems to be suggesting one that works with BASALT to crunch programs only when they are run. Crunchie, while it saves crunched programs, is only to test his algorithms.

Steve: Am I misunderstanding your comment?

Speed is like money. When you’re have it, you just want more.

When you have enough money and plenty to spare then more is just wasted.
I suspect the same thinking applies to most currently available applications on a modern RO machine.
There’s plenty of people using a multicore machine and massively under using the processor capabilities, although the redoubtable Jeffrey is trying to open up those extra cores.

I suspect most modern RO machines are i/o bound rather than CPU bound and unless the crunching magically speeds up the access to the data files then it’s probably irrelevant.

There does not appear to be a BASIC cruncher in active development

Is there much use for them these days? They were traditionally used by games companies to obfuscate code or where you wanted to either reduce the size of the program to fit on an A3xx or gain a slight increase in speed. The later two were probably relevant on ARM2/3 but not so much on modern hardware.

They were traditionally used by games companies to obfuscate code or where you wanted to either reduce the size of the program to fit on an A3xx or gain a slight increase in speed.

The simple CRUNCH removed spaces and sped up execution a little; a full cruncher significantly reduced variable and FN/PROC name lengths, and distributed them around the alphabet to minimise the lengths of the lookup lists. ISTR this being noticeable on a RiscPC.

I’ve no idea what the cumulative effect of the short names (less memory to step through during execution) and distributed initial characters actually is, or if it’s still relevant on modern machines.

Personally, I’ve always linked LIBRARY code in to the main file when releasing applications, so have always crunched at the same time. I did consider writing a full name cruncher as part of Tokenize, but concluded that the effort required would probably far exceed the benefits on modern systems.

Steve isn’t talking about a cruncher that allows someone to distribute crunched BASIC.

Fair point, but any competent BASIC coder would release:

• Load program
• It gets auto-crunched
• SAVE “Blahblah”

Voilà, a nicely crunched program to distribute. ;-)

a full cruncher significantly reduced variable and FN/PROC name lengths,

I believe bad-ass crunchers like StrongBS also did an amount of reworking of the program logic as well.

To give you an idea – here’s the ToDo list of the StrongBS update, should it ever actually happen ¹… https://sites.google.com/site/strongbsnow/to-do-list

¹ Last update nearly a decade ago, and Google returns exactly zero matches for his email address between 2014/01/01 and now.

Am I misunderstanding your comment?

No. My motivation is to have Basalt crunch on loading, but that does not preclude having another approach with the same algorithms.

The reactions here have been very interesting and I think everyone has supported the idea that crunched BASIC progams should not be distributed without the source. I would like to propose a couple of scenarios where that might not be best. The first is a commercial application, and how many of those written in C come with the source?. The second is a useful, complex application supported by an author who does not want modifications to be made, which might involve extra work. There are probably few of these, though.

Assuming that the source is available, crunching is a free boost to a program that should not be dismissed out of hand, I think. Admittedly, a user may well not see much difference with most programs on modern machines.

I started on a 16k Spectrum and I cannot shake the habit of looking for every saving of memory or gain in speed.

It might be worth mentioning that BB4W always crunches programs to produce the runtime code.

I suspect most modern RO machines are i/o bound rather than CPU bound and unless the crunching magically speeds up the access to the data files then it’s probably irrelevant.

Well put. I’m pretty sure that’s the case with all the apps I’ve written. The exceptions would be where a process grew more than linearly with data size, coupled with large data files – sorting huge files, for example. And then it would obviously be worth writing the heavily used core routines in Assembler.

I started on a 16k Spectrum and I cannot shake the habit of looking for every saving of memory or gain in speed.

8~) I can identify with that. I started on an 8k x 12bit PDP8…

Voilà, a nicely crunched program to distribute. ;-)

Your actual scenario would not work – you could not use your method to take advantage of the way BASIC crunches programs run with -quit, for example. However, there are alternatives to achieve the same. I do not want to imply that I would ever want to prevent this – it just happens to be so.

As an aside, a command to crunch in situ could also do other things to speed up a program, rather as BasaltInit does, and then it would be fruitless saving it out. ;-)

I believe bad-ass crunchers like StrongBS also did an amount of reworking of the program logic as well.

StrongBS does, but it looks more like a display of vituoso programming than making any real gains. The one thing that might help is concatening lines where possible.

The first is a commercial application,

Commercial applications are a different thing. People would want to protect their investment, especially given the very small market leads to software that may be seen as more expensive than elsewhere (it’s an economy of scale thing – a team of a dozen programmers looking to sell a million copies on Windows is really not the same thing as a single person selling copies with few zeros following the initial digit…).

For software released “for free”, there is less justification for scrambling the code, other than what ultimately transpires to be a variation of “I’m ashamed of my code” or the like. Speed, as has been pointed out plenty above, is likely these days to be more down to how good the machine is at retrieving the data rather than the brute force of the processor. This isn’t to say that there’s no benefit in crunching the code, but more that the small speed gain is offset by rendering the software as a big black box that can’t be fiddled with if things go wrong (either due to program errors or changes in the underlying machine).

and how many of those written in C come with the source?

By and large, exactly zero. And for a commercial application, it would be unreasonable to expect it to be supplied with source while it is still being actively marketed and supported.
However, when the author decides to drop a project or move on, that is when the source code should be made available (either to the public or to a new maintainer), because there are users who not only invested money in purchasing the software, but also the time in using it, the legacy of potentially loads of files that they have created with it. This, actually, goes a lot more for C (and the few assembler) programs than BASIC as such code is a lot more likely to be adversely affected by changes in the processor’s behaviour. A good example is how ARMv8 has removed SWP, which pretty instantly killed everything using a then-current GCCSDK library. Well, luckily as it was used in (mostly) open source software with an open source library, it could be fixed and updated versions made and all would resume. Well, what if that was an important application that people use, but is no longer being maintained? The entire project killed by a single instruction?

Active commercial programs should be expected to be closed source. It’s the author’s revenue stream. But when the author moves on, said programs should have their source released in some manner. Think how much richer the RISC OS scene would be had this been the norm? We wouldn’t need products like Aemulor as various people could work on their favourite pet program to bring it to the 32 bit world.

I started on a 16k Spectrum and I cannot shake the habit of looking for every saving of memory or gain in speed.

I used to think like that, but Android has broken me.
Around 100*MEGA*bytes for a friggin’ weather app? Are you kidding me? How the <long list of multilingual expletives>!!!

It’s good to be resource frugal rather than just treating memory as an endless dumping ground, but there are times when it just doesn’t make sense any more.

It’s like why I, the author of the assembler tutorials (back in the Dark Ages when SWP was still A Thing), advocate knowing some assembly (it’s dead easy to make API calls with a couple of lines of in-line assembly) but I would never ever advocate writing an entire application in assembler unless you’re doing it to learn, to show off, or just for the lulz. Both GCC and Norcroft are competent compilers that produce good code. Except for low level system stuff, speed sensitive code, inline API calls, or to use facilities not (yet?) available in the compiler (such as NEON or ARMv8 stuff), there’s little justification for going the assembler route these days … unless, of course, you still code in BASIC where there isn’t much of an alternative for fast code. :-)

you could not use your method to take advantage of the way BASIC crunches programs run with -quit, for example.

• Open TaskWindow
• Run Wimp program
• Wait for “WindowManager in use” error
• *BASIC then OLD then SAVE.
• Done.

;-)

it looks more like a display of vituoso programming than making any real gains

Tea → Keyboard.

Done. ;-)

That’s one alternative. Another is to extract the program from the application memory. ;-)

Having had several conversations over the years when people have said “App X is really unfriendly to install, because it comes with a whole bunch of source code that I don’t understand”, I’d suggest that the best option is to keep the two well and truly separate.

Most of the time – at least for BASIC stuff – it does not make sense to provide the source code as a separate download anyway, this just leads to problems like “why is that website no longer available” or “where did I download that” or “what is this source that is lying around in my download directory”. You just put the source into the app dir. Only interested parties will ever see it, all others can safely ignore it. Since we don’t need to worry anymore to transport stuff on 800KiB floppies…

this just leads to problems like “why is that website no longer available” or “where did I download that” or “what is this source that is lying around in my download directory”.

This.

You just put the source into the app dir. Only interested parties will ever see it, all others can safely ignore it.

This.

Put the original code in a separate folder alongside the application, it highlights its presence. Put it inside the application, it doesn’t.

Name it something like “Sourcecode”, it’ll be really obvious why that file is there. No confusion. Within the application itself, it is hidden in plain sight. There if necessary, yet pretty much invisible to non-geek users.

Put the original code in a separate folder alongside the application, it highlights its presence. Put it inside the application, it doesn’t.

!AntiSpam – the source is tucked away like that. The edit and re(new) compress is simple. The general user doesn’t know, or care, that it is there. People who want to modify find it easy, not least because of the quantity of commenting. Of course it was developed by one person¹ and further developed by another the latter would have been difficult with a cryptic ‘source’.

¹ With assistance/hindrance from various others.

So much stuff has been effectively lost due to crunching for ‘security’.

Talking of lost stuff. Does anyone have an un-crunched copy of !Usage? It would so much more use if we could fix the bug.

PS How do I use bq. ? I have never made it work!

You put a space after the dot!

PS Which you seem to have just found out!

Which you seem to have just found out!

Yes. At long last. So, I edited the post to remove the PS!

But thanks for the answer.

So, back to my original question ….

What did you have in mind – Just run !Usage 1.01 (05-Sep2012) by Ran Mokady !runimage through !BasFormat to uncrunch it.
!Usage also makes use of a Module as well.

See !Usage 1.04 (18-Sep-15) from RC15. Looks OK on the RPi3 and Titanium.

What did you have in mind

I wanted to try to fix the bug in v1.03 which causes it to die with “too many nested structures”.

Just run the !runimage through !BasFormat to uncrunch it.

Thanks. But, I have not yet found a copy of !BasFormat.

!Usage also makes use of a Module as well.

Yes. the module seems to work OK. I think it is the Basic !RunImage which dies.

See !Usage 1.04 (18-Sep-15) from RC15. Looks OK on the RPi3 and Titanium.

!Usage 1.03 Looks OK. But it eventually dies. I now have 1.04 running to see if that has the bug fixed.

The uncrunched sources for Usage (and its helper module) are available in CVS.

https://www.riscosopen.org/viewer/view/bsd/RiscOS/Sources/Utilities/Usage/

The VersionNum file (which is the main place to look for the change history) doesn’t mention any “too many nested structures” bug, so if you can find / fix the problem than that would be great.

The uncrunched sources are available in CVS.

uncrunched? Well, sort of.

I suspect that the only bits that are uncrunched had to be re-written because even the author could not understand the original bits of code when they needed changing.

Things like PROCa, FNb, j% etc are unintelligible.

Never mind. It was worth asking.

uncrunched? Well, sort of.

There’s CRUNCH and there’s crunch.

I have done a !BasFormat job on the compressed, crunched?, !RunImage so if or when the fault appears it should be easier to pin down, or not.

!Usage 1.04 dies in just the same way as 1.03 (“too many nested structures”), with a line number reference at the end of the file.

I will have another look for the cause of the fault, but it will likely not be easy with code mostly crunched up.