Pragmatic suggestions for the zero page restriction incompatibilities

I think cooperation between ROOL and Adrian Lees on making Aemulor function is the way forward. Either that or cooperation between Richard Keefe and Adrian Lees to make Impression have Aemulor baked in so that it works on current ROMS. The second way would provide a fee-based method. Or just pool resources to get the 32-bitting of Impression finished?

Had you noticed that the original Ovation was available in 32-bit form for free, or that the rather capable Ovation Pro has been 32-bit capable for a decade?
I don’t see any reason to fixate on throwing extra resource at a problem that already has a solution.

Aemulor provides a crutch functionality for programs that for some reason have a 26-bit limitation. The only connection with Impression is that there is a 26-bit limitation or five hidden away in the clever but probably rather obscure code

Also, unless I’m mistaken, there’s no reason why we can’t more the vectors high for safety, but leave a zero page there so that reads and writes don’t cause the program to instantly bomb out. A win all round.

Not really, because active developers need the programs to bomb out so that they can then fix the bugs. If you force people like me to go back to unprotected zero page (perhaps forcing us to build our own ROMs to keep the current state), you’ve just made life more difficult for us.

What’s wrong with retaining the current arrangement of high vectors, no zero page and ZeroPain to trap and log all accesses? You could make logging optional as Rick suggests, or simply ignore the logfile that’s generated – it stops after a while if you don’t wish to read it.

The fact that these companies make there own builds tells you that the ROOL distribution isn’t good enough – they are not willing to support it. It wouldn’t take any effort on ROOLs part to distribute a ‘Maximum compatibility’ version of their product.

I’m sure if those companies wish to pay for ROOL’s time to get things set up and keep them working, it would be an option.

Who on earth is saying that fixing bugs is a bad thing no-one in there right mind would, but banning programs that can work / may be needed is just an indulgence and quite frankly arrogance on the part of developers.

No-one is “banning” the software – stop spreading FUD. All the affected items that I use still work fine here with ZeroPain installed and active.

Intentional nonaligned reads and zero page access are not bugs

Could you enlighten us as to what “intentional … zero page accesses” software like Messenger Pro (to pick one example from this thread) is carrying out in normal use? Why is a User Mode application intentionally reading from Zero Page?

what is happening is that RISC OS has changed the API

I think you’ll find that ARM changed the behaviour of non-aligned loads and stores. No-one involved with RISC OS in any way had an option on that.

Could you enlighten us as to what “intentional … zero page accesses” software like Messenger Pro (to pick one example from this thread) is carrying out in normal use? Why is a User Mode application intentionally reading from Zero Page?

It’s SecureSockets rather than Messenger Pro, isn’t it? If one’s going to go for a commercial offender, maybe choose PlingStore.

Could you enlighten us as to what “intentional … zero page accesses” software like Messenger Pro (to pick one example from this thread) is carrying out in normal use? Why is a User Mode application intentionally reading from Zero Page?

That’s easy. If it uses the same construct as your ovation example

if (ptr != NULL && ptr[x] != NULL)

then it is not a bug. With a riscos API that doesn’t fault zero page that is perfectly valid code. I would love to know the percentage of zeropain reports that are actual bugs. Most of the changes will have been a port to a different API like changing code from 26bit to 32bit is a port to a different API.

I think you’ll find that ARM changed the behaviour of non-aligned loads and stores. No-one involved with RISC OS in any way had an option on that.

So what. It is a change in API for RISC OS. This is an OS. Its its job to virtualise hardware and obviously non aligned reads can be abstracted away otherwise they couldn’t be made to work. Had the virtualisation been impossible then I couldn’t agree with you more.

Poor old Ovationand other programs are getting panned as having buggy code when they do not.

No-one is “banning” the software – stop spreading FUD

Of course they are ROOL have an agenda which doesn’t involve maintaining backward compatibility so any programs it breaks is ok with them even though it is not necessary to break them at this point in time. If we are to avoid a fork we all have to accept this as a good thing and in doing so we banish perfectly good programs to the scrap heap.

I’m sure if those companies wish to pay for ROOL’s time to get things set up and keep them working, it would be an option.

Did ROOL pay for your time to port your software to the new API.

What ROOL are doing with ZeroPain/Alignment Exceptions/HV is breaking RISC OS. It is an exercise in determining which programs RO will be left with when RO is no longer able to virtualise cpu changes and as pointed out at the very start of this thread we’ve been doing this for 20 months and if a program has a problem now then it has either been fixed, the problem is known about but not fixed yet or is not going to be fixed. The experiment is over there’s nothing to be gained from continuing it. If you find – as a programmer – zeropage exceptions useful switch them on for yourself.

All the affected items that I use still work fine here with ZeroPain installed and active.

Absolutely – this cannot be too strongly emphasised! In the last 24 hours I have used Photodesk, MessengerPro, Netfetch, DPingScan, AMPlay, OvationPro, Thump, StrongEd, Paint, NetSurf 3.6, CloneDisc, SyncDiscs, CloudFS, LanMan98, TechWriter, DrawPrint and Otter on this high-vector Pi2 with ZeroPain enabled, all without any problems whatsoever (in fact Otter works better in a high-vector environment because the latter is more tolerant of Otter’s propensity relentlessly to build its dynamic area requirement during a day’s browsing). The only software that definitely won’t work are 26-bit apps requiring Aemulor, but if I want to run these (and I do from time to time) I can resort to ROS 4.02 under RPCEmu or a Pi1 running RC12. If that’s the price to be paid for ongoing OS development, I for one am happy to pay it.

that is perfectly valid code.

All bugs are perfectly valid code. They just don’t do what was anticipated…

but the example does exactly what was anticipated.

Could you enlighten us as to what “intentional … zero page accesses” software like Messenger Pro (to pick one example from this thread) is carrying out in normal use? Why is a User Mode application intentionally reading from Zero Page?

That’s easy. If it uses the same construct as your ovation example

if (ptr != NULL && ptr[x] != NULL)

then it is not a bug. With a riscos API that doesn’t fault zero page that is perfectly valid code.

Please give values of ptr and ptr[x] for which the above code will generate a zero page access on a high vector version of the OS.

My apologies I must remember to cut and past quotes. The construct was

if (*ptr != 0 && ptr != NULL)

Absolutely – this cannot be too strongly emphasised! In the last 24 hours I have used Photodesk, MessengerPro, Netfetch, DPingScan, AMPlay, OvationPro, Thump, StrongEd, Paint, NetSurf 3.6, CloneDisc, SyncDiscs, CloudFS, LanMan98, TechWriter, DrawPrint and Otter on this high-vector Pi2 with ZeroPain enabled, all without any problems whatsoever

There were only 3 applications with known problems and I added the ones mentioned in this discussion, which is still only 5 in total, has everyone reported the ones they know about?

still only 5 in total

I guess I should offer various mea culpas here since one of those apps mentioned belongs to me, viz. ID3TagEd. This was reported some while since and I fixed it. However, at the time I was going to make some other changes, and so didn’t release the fixed version immediately. Since then ID3TagEd has slipped below my horizon 8(

I had better do something about it.

This is an OS. Its its job to virtualise hardware and obviously non aligned reads can be abstracted away otherwise they couldn’t be made to work. Had the virtualisation been impossible then I couldn’t agree with you more.

Well, it was not possible to virtualise unaligned accesses. Or better: it was tried and it did not work in a sensible way.

You can either trap unaligned access exceptions (stops the software from “working”) or not trap them (unaligned memory access works differently from before, but nobody tells you). There is no option 3, no virtualisation.

By the way, I do not agree that the OS’ job is to provide a fully virtualised environment to protect buggy applications forever. There will always be changes in the OS that will break backwards compatibility, because a developer has not followed the API closely or has made assumptions about how things apparently work. We have emulators to cater for these things, or else you end up with an unmaintanable piece of crap that once was called an OS.

ROOL have an agenda which doesn’t involve maintaining backward compatibility so any programs it breaks is ok with them even though it is not necessary to break them at this point in time. If we are to avoid a fork we all have to accept this as a good thing and in doing so we banish perfectly good programs to the scrap heap.

Please re-read the original announcement:
https://www.riscosopen.org/news/articles/2015/07/05/important-software-compatibility-notice

The idea that ROOL have an (obviously secret) agenda that does not care for maintaining backward compatibility everywhere sensible is absurd. Of course there is a fine line between what is sensible and what is not, and it always needs to be discussed. Hopefully in a well-mannered way.

I know it’s not a secret agenda to drop backward compatibility with some programs and I know that the decision was not taken lightly and I don’t necessarily disagree with their decision for moving RO forward or when they did it. But while they have got an eye on the future the rest of us are in the now and it would be nice if all machines that were capable of running older software did so with all the other fixes/enhancements RO now has.

If you say that alignment exceptions can’t be fixed for all of the machines we have available now I will have to accept that and lose programs that use non aligned reads.

I really am not complaining about anything I’m just discussing what we could have now vs what we do have now. The changes don’t affect me personally in the slightest but in a forum full of programmers someone has to think about the users. If what we lose isn’t worth a jot then the whole discussion is academic.

What ROOL are doing with ZeroPain/Alignment Exceptions/HV is breaking RISC OS.

Ah, now there’s an example of conflating different changes:
Zeropain is a ROOL change (to pressure compliance with an Acorn API change made, in principle, a couple of decades ago) and adherence to the ZPP compliant version fixes a number of errors that have probably been triggering random failures for years. As Steve F said with some of his software he’s surprised it worked at all. The Ovation code actually working previously is probably best described as a happy accident.

Meanwhile the Alignment Exceptions issue is one where ARM have altered the processor in later cores and is thus one that no one in ROOL can do anything other than advise people to recode.

This is an OS. Its its job to virtualise hardware and obviously non aligned reads can be abstracted away otherwise they couldn’t be made to work.

While it would be nice to have a virtualisation “layer” available to the OS my recollection of conversations between Jon Abbott and Jeffrey Lee is that they seemed to agree that some changes to the OS, particularly around the ZPP element were required to make the virtualisation process neater/easier. I will leave to those two in combination, or singly, to explain the issues.

If I’m not mis-remembering then the proposed virtualisation would actually make use of various old (broken) programs possible once more. Maybe.

It is an exercise in determining which programs RO will be left with when RO is no longer able to virtualise cpu changes and as pointed out at the very start of this thread we’ve been doing this for 20 months and if a program has a problem now then it has either been fixed, the problem is known about but not fixed yet or is not going to be fixed.

There has a fair bit of noise on this subject, but when you check the zero page protection errors page and and do a swift count you find a grand total of 20 reported programs of which just5 (five) are still unresolved and all five of those have currently active authors. As Mike asked, has everyone reported the errors they know of? Reading further down this thread I see that Chris J has fixed ID3TagEd, so we appear to be down to four known faulty items.
Two from one source…

Does anyone recall the childrens story about the chicken hit on the head by an Acorn and then ran around claiming the sky was falling?

Edit At one point I was spending time checking what worked and what did not among the items I have in my software stash. For a good many months, running to double figures and some, I have been working so many hours at varying times that I’ve not had the time to do structured testing. Maybe when the current project finishes.

Random copy pastes with replies…

Not really, because active developers need the programs to bomb out so that they can then fix the bugs.

Important emphasis added.

you’ve just made life more difficult for us.

I trust you realise the irony here, that the exact same argument could be made in reverse. As has been pointed out several times previously, everybody who used Aemulor has been inconvenienced by this change. It just made life more difficult for them.

then it is not a bug. With a riscos API that doesn’t fault zero page that is perfectly valid code.

Interesting opinion. Now, I don’t know about you, but I would describe any memory access outside of the application’s memory space (unless specifically mandated by the API – data held in the RMA, use of dynamic areas, etc) is incorrect. A flaw. A bug.
It is completely irrelevant whether or not the operating system permits it. The next time you are running RISC OS 3.10 on an emulator, type !8=0 into BASIC. It’s perfectly valid code. The OS allows it. So it’s okay, right?

Poor old Ovation and other programs are getting panned as having buggy code when they do not.

I regret two things. I regret not remembering that the original bug was indirect pointer accesses, and I regret mentioning that it was Ovation. The specific line of code that I remember fixing was indeed in Ovation, but any other program with a similar construct would suffer the same outcome.
And yes, regardless of how you wish to classify it, a stray read from a null pointer while also checking if the pointer is null is a bug. Not a serious one. Certainly not one the application should be aborted over. But it, technically, is a bug.

Of course they are ROOL have an agenda which doesn’t involve maintaining backward compatibility

What a load of [euphemism for excrement].
Do you know how many hoops I had to jump through to fix some crappy code in CLib that was broken for decades? Do you know how much farting around I had to do to get an emulator running with RISC OS 3.10? This is all the more amusing given that my fix was involving multiple timezones where the current timezone wasn’t the first one in the list – something that never worked correctly in any Acorn release of RISC OS.
All for, what is in effect, a single line change to get CLib to use the correct SWI to read the current timezone, not just “the first one that the territory defines”.

Do you have any idea how much nicer RISC OS could be if backwards compatibility was not to be an issue? The Wimp can become UTF-8 by default. The data blocks can become 4K because, come on people, it’s 2017. You can’t even fit a full regular pathname into a Wimp block when the first 20 bytes are header words. And… and… and…

No. Backwards compatibility is always present. It is always going to be an issue. That is why my suggestion is to make ZeroPain a proper part of the OS. I’m not asking for high vector to go away, I’m just asking for programs not to be forcibly aborted as a result of stray page zero accesses.

What ROOL are doing with ZeroPain/Alignment Exceptions/HV is breaking RISC OS.

Alignment exceptions != ZeroPain/HV.

As for breaking RISC OS, ZeroPain does what is necessary to keep less well behaved programs working. It is only breaking things because this safety net is not an automatic part of the OS. It must be.
If a program has worked “as documented” for many years, it is the job of the OS to provide some form of mitigation if the OS decides to do things in a different manner that causes problems. While Steffen is right that the OS should not bend over backwards to support ancient stuff (haha, I bet the Wimp still supports starting tasks in the Arthur way!), it also shouldn’t implement changes that may prompt once-working programs to suddenly no longer work. Specifically in the manner in which they stop – total abort, data loss, program evaporated into the air.
Because of this, ZeroPain needs to be a part of the OS. And if people can’t handle that, then for god’s sake just map a page of memory into &0 and then forget about it.

If you say that alignment exceptions can’t be fixed for all of the machines we have available now I will have to accept that and lose programs that use non aligned reads.

Alignment exceptions can be fixed. Although the skills required are out of my abilities, I cannot imagine it would be difficult to sit on the exception and when it happens, read the instruction, read its address, and then execute some code that will fake the behaviour. Linux does this, look for something like CONFIG_EXCEPTION_TRAP.

The problem with this, however, is that it is quite different to the page zero issue in that continuing support for rotated loads (either in hardware on ARMv6 etc or by trapping the exception) means that nobody can write any software that uses proper unaligned loads.
Letting programs read from page zero is an issue for the program concerned that shouldn’t ordinarily affect other programs.
Letting a program use rotated loads means all programs are obliged to use rotated loads.

someone has to think about the users.

Won’t somebody please think about the children!

Sorry. Couldn’t resist. ;-)

But, yeah, that’s pretty much what I’ve been saying all along. I mean, I’m not really even remotely interested whether or not people in this forum think logging page zero accesses is a good idea for developers, because the very utterance of such an opinion means that they understand exactly what is going on and why. Seeing it from the point of view of a user that is not a developer, doesn’t want to get involved in that sort of stuff, and would kind of just like the machine to work… may find that some software that they use now blows up spectacularly. It does not have to be like this. Putting something at &0 (ZeroPain, scrap memory, kittens, whatever) means that for them, life will continue as before and apart from some very exceptional cases (such as Aemulor), they’ll probably never even notice a change.
Oh, wait, we probably ought to set the first few words to say oflaoflaofla. ;-)

If what we lose isn’t worth a jot then the whole discussion is academic.

To be frank, this whole discussion is an enormous tsunami in a tea cup.

working previously is probably best described as a happy accident.

Sort of. I said a while back that any specific bug of that nature that was a show stopper would have been noticed and dealt with, for obvious reasons. This is more an unwanted side effect.

has everyone reported the errors they know of?

Do you not think that, by asking here, we’re kind of preaching to the converted?

To be frank, this whole discussion is an enormous tsunami in a tea cup.

Chicken, sky falling. (Earlier)

Do you not think that, by asking here, we’re kind of preaching to the converted?

Do you not think that by having this discussion here we are reminding authors that they have broken bits that need fixing (Chris J recalled the fact he hadn’t got around to releasing a version he fixed some time back) and reminding everyone reading that it isn’t actually someone elses’s job to do the bug finding – it’s ours, all of us.
If there are no more programmes with ZPP or alignment bugs than have been reported then it isn’t a storm in a tea cup it’s actually a tiny ripple in a dolls thimble that is being portrayed as a world ending deluge.

If there are many more bugs being triggered, where are the reports?
Software authors tend not to work on fixing things they haven’t been told about and thus don’t know about.

My apologies I must remember to cut and past quotes. The construct was if (*ptr != 0 && ptr != NULL)

Which is a bug. C’s deliberate support for short circuit evaluation is in part to allow you to do the test the other way around:

if (ptr != NULL && *ptr != 0)

The whole point being that you verify that the pointer isn’t NULL before dereferencing it.

That is why my suggestion is to make ZeroPain a proper part of the OS. I’m not asking for high vector to go away, I’m just asking for programs not to be forcibly aborted as a result of stray page zero accesses.

Would be perfect.

Here, the only reason why I do not use HV ROM is too much logs, that could harm my SD card.
A SD card does not cost a lot. But the 86 GB of data present on it, yes.

Alignment exceptions can be fixed.

That would be just fantastic.

The module needed to trap it would be also a good start for some other ‘on the fly’ replacement of old instructions.
Or a FPA to VFP conversion? :)

Software authors tend not to work on fixing things they haven’t been told about and thus don’t know about.

They don’t know how to use their software, and can’t read a log file? Come on!
:)

The point is that users can help. But they don’t have too.
So give them a solution to help, but do not force them.
The only consequence here is that some people still use LV ROM.

If I’m forced to help fixing a software, where is the limit? Should I fix the code too? Write it?

I write and maintain my software.
But for the other, I’m a user.
Sometimes I fill bug report.

But I don’t care of ZPP logs for Messenger Pro, as long as it works.
And I would like not to care about AE things, since it’s the only way to use the two PNG converters made for RISC OS.

But, in the same time, I don’t want that these two choices mean that I can’t use HV ROM or ARMv7 Strict mode.

A better integration of ZPP for users is the solution. Some trapping of AE would be cool too. I don’t really understand why it’s so difficult to get these two changes.

Here, the only reason why I do not use HV ROM is too much logs, that could harm my SD card.

ZeroPain stops writing to the file when it gets to a certain size ¹, so that’s only an issue if you empty the log.

They don’t know how to use their software, and can’t read a log file?

Modern software isn’t linear: it can be used in many ways, with many combinations and orders of working.

In addition, for anything of any reasonable size, there will almost certainly be features that users have requested and the developer does not touch from day to day.

The point is that users can help. But they don’t have too. So give them a solution to help, but do not force them.

Testing is a way to give something back, especially if you’re not being asked to pay for the software, operating system or developers’ time…

The only consequence here is that some people still use LV ROM.

And that the HV one is harder for developers to get hold of, if they have to build their own.

¹ 1MB?

ZeroPain stops writing to the file when it gets to a certain size 1, so that’s only an issue if you empty the log.

Ah, OK, good to know :)