Programming things

Thanks Rick.
How funny this was ADA.
Our teachers were such ADA enthousiasts.
I would love to laugh at them now !
Thanks for the link, will send it to friends from these Uni years … For sure, they will laugh too !

I do have to point out that many newer vehicles actually do use a network with controllers for various parts now.

2002 Volvo V70 that I have does. The ECU seems very solid – although a temperature sensor failure giving an out-of-range value resulted in an undrivable vehicle, not a limp mode.

It’s meant to be an utter nightmare if a part fails because they tend to be paired with the ECU to deter theft.

Yup. Cost £150 to get a new controller for the electric windows because it had to be paired with the ECU!

Still – not too bad considering these are the only things to have gone wrong with the beast in 15 years and 149K miles.

Difficult enough in software. Harder in IT hardware. Bloody difficult in aerospace hardware!

Good summary :)

No. You never ignore. Never.

Only as an alternative to “just stop”.
When a car system crashes, it does not stop the engine.
It’s not really “ignore”, but more about maintaining a “status quo”… while waiting for new data.

Sometimes there is just one wrong data or calculation. It’s important to be able to ignore it, as the main aim of a critical system is to never stop. Even for an hardware failure.

When I studied ADA 21 years ago, one of my ADA teachers kept telling us that this kind of possible issues would have been detected by the ADA compiler.

Only if the contract is correct.
ADA is fantastic in KISS mode, and a nightmare in contract mode.

When a car system crashes, it does not stop the engine.

When a car system crashes, it should not stop the engine.

Well, okay, it didn’t stop the engine, but it did stop the engine producing any power. We were able to roll to a halt, fortunately managing to reach a fairly safe place. Declutching helped, but engine braking in top gear isn’t violent anyway. (My father’s Thames van skidded to a halt when a con rod broke and rammed the broken end into the cylinder wall, which really did stop the engine.)

It’s important to be able to ignore it, as the main aim of a critical system is to never stop.

Define “never stop”. What if the failed system is the one that controls location or direction? Would it be acceptable to run over a child in a self driving car because mud blocked the proximity detection from working correctly and… Never stop?

The ideal thing to do would be to abort safely if the fault is not something that is recoverable. Ironically, in the case of the rocket, that was a safe failure. Imagine if it veered out of control and ploughed into an urban area…?

Oops. There’s thick fog here today, been redirected because of an accident. Can just see a load of blue strobes. I’m guessing somebody tried to pass somebody and hit an oncoming car…? If humans cock it up, what hope a machine?

Define “never stop”.

I think it’s fair to say that “never stop” means the program should never stop, not that a vehicle should never stop. The program should continue to consider what best to do in the circumstances, which may well mean “stop the vehicle” – but probably not by shutting the engine down, unless you’ve lost steering control or control of speed. You might well want to reduce engine power to “limp mode” so you can proceed to a safe place to stop.

I think it’s fair to say that “never stop” means the program should never stop, not that a vehicle should never stop.

Correct. As hardware or data failure can and will happens, it’s important sometimes to wait for more data before really crashing (what I called “ignoring the error”). Of course, we now there is an error, and we raise an alert. But in the same time we choose to ‘ignore’ it to maintain the status quo.

All electronic systems in cars work like this. When you’ll connect a monitoring system, you’ll see a lot of abnormal data that is simply ignored. For example from the lambda sensor. But of course, the fuel injection will not stop working. It’ll just be in a very suboptimal state.

The program should continue to consider what best to do in the circumstances

Correct. It should never stop (the program).
A bug that translates in a system crash is not normal on critical systems.

How funny this was ADA. Our teachers were such ADA enthousiasts. I would love to laugh at them now !

If you have read the article Rick referenced to, and still think that it was somehow Ada’s (not ADA!) fault, read it again and try harder to understand the details.

In short, the problem was created by blindly reusing software and components from a previous, completely different rocket type.

Bottom line: Reusage is good. Reusage without engaging the brain is bad.

I would say that a major root cause of the loss of the rocket was poor testing. And thus poor project management. Like the space shuttle disaster, the real root cause was probably due to the way the project was being run from the top ?

After the incident, they were able to run a simulation of the rocket launch with simulated inputs to the computer. This simulation behaved exactly like the ‘real’ rocket using the same (or similar) process inputs.

Why wasn’t a full launch simulation done before the launch as part of a system software test, and not after ? Plus other ‘unit’ testing should have found such a basic error in the code. And they should never have been using mission-critical code with no exception handlers (apart from an exception causes the rocket to self-destruct in mid-air).

Ralph

@Stephen Huber : I do not need to try to understand ‘harder’, please spare me your insisting patronising me, with despise, once for all (Yes : I remember the Google group, and will for a very long time). ‘ROTFL’ : excellent choice.
Maybe you could try to understand me ‘harder’, and try ‘harder’ to speak politely to people ?
No shrink where you live to help you with your superiority complex ?

It is obvious no programming language will prevent from blatant misusage … It is why we (the students) were so appalled by our ADA enthousiasts teachers, because of their stubborn attitude, as though ADA was a ‘gem’ among all available programming languages.
In use, programming ADA is really painstaking, needing a constant and sustained effort, pretending its ‘overprotection’ on just everything will produce a more secure code.
In a way this is true, but at the expense of more terms (syntax) to use, and overall more ‘fatigue’ for the programmer, which, to all of us wannabe computer scientists, was the main cause of programming errors ;-)

Why wasn’t a full launch simulation done before the launch as part of a system software test, and not after?

Either or both of:

1. A project manager decided that such testing wasted time and would delay his/her “project delivery”¹
2. A bean counter who decided the the “cost benefit analysis” indicated that this was an expensive thing to do and there would be a “significant cost saving” of (2-3%) if this item was omitted.

¹ I have a colleague who says he wouldn’t trust most project managers to “deliver” pizza.

Why wasn’t a full launch simulation done before the launch as part of a system software test

For each mission, there are about 4 years of intensive tests and simulations.

All electronic systems in cars work like this. When you’ll connect a monitoring system, you’ll see a lot of abnormal data that is simply ignored. For example from the lambda sensor. But of course, the fuel injection will not stop working. It’ll just be in a very suboptimal state.

My previous car – Citroen Saxo. Possibly had a fault with the oxygen sensor. When the engine idled, it would slowly rev up, then down, then up, then down. It’d take maybe 15-20 seconds to go from around 1000rpm to 1500rpm, then back to 1000. We took it to a garage who hooked up fancy monitoring equipment. Verdict? Probably the oxygen sensor but we can’t tell for sure. €600 to swap it for a new one, but no guarantee it’ll change anything. I looked up the part online (cost: about a third, accessibility, dead easy it is the thing that looks like a spark plug on top of the exhaust manifold). So I said forget it, kept the car a little longer, took it for trade in on the C1 that I have now.

As it trims the engine to help keep emissions down, it wasn’t a particularly critical piece. Okay, we’d probably fail an MOT, but it’s not like the engine won’t work without it.

it’s important sometimes to wait for more data before really crashing (what I called “ignoring the error”).

Ignoring the error and crashing? Not quite the same thing. ;-)

And anyway, wait for what data? Somebody much smarter than me once said that doing something repeatedly expecting a different result was the very definition of madness. Now I do understand that there will be glitches in real world applications. Not so much sensor failure, more communications failures. A nearby lightning strike, driving under power lines, all sorts of things could mess with the data – though a well designed system will include some form of coding (hamming, CRC, whatever) to allow the receiver to know that the data was corrupted.
A failed sensor, on the other hand… is about as likely to spring back to life as a blown headlight is to illuminate.

Here is one to think about. You are an ECU. A fancy intelligent car. You know how fast the car is travelling, you know the engine RPM. These help you decide how best to mix the fuel for the way the user is driving… and maybe to also nag the user when they are in the wrong gear (hello Citroen C3, can you hear me?!?).
You’re doing 70. It’s 17.4C outside. Humidity at 73%, but it is not raining.

Now you are doing 0. All of a sudden, your wheel rotation sensor says the car’s speed is zero.

What action do you take?

Now you are doing 0. All of a sudden, your wheel rotation sensor says the car’s speed is zero.

Check the readings from the other wheel(s)

What action do you take?

I chose this specifically because there are two equally viable actions for two equally viable events.

Event #1: Sensor failure
In this case, it is no longer possible to determine car speed so the logical behaviour would be to lock down the last set of fuel adjustment parameters and flash the spanner logo like crazy. This will need fixed immediately, for without that sensor, the user may well not have any indication of how fast they are travelling.

Event #2: Head on collision
In this case, we cannot necessarily rely upon any other piece of hardware because the connections may have broken or been severed before communication got through (I’m thinking of the G-sensor hat triggers airbags). All we know is that we have gone from 70 to 0 in a hurry. The actions we should take are likely to be to disable the engine and the fuel pump. The car will suddenly lose power, but it can cruise to a stop (it isn’t like a broken cam belt or anything).

Remember, we are talking automotive devices here. It would be nice to say “okay, if one speed sensor has failed we can flag a warning and rely upon reading from the other one” but this is supposing that the car would be fitted with two. I doubt it. These are pieces of hardware selling for prices akin to ten grand or more, yet they still – in this day and age – cheap out on central locking and a frigging light bulb in the boot. I’m supposed to believe they’d fit redundant sensors? Yeah, right…

So. Your two choices. Which would you settle upon as being the best course of action?

There was a aircraft that had a speed sensor that froze up – the computer handed it over to the three pilot’s – which tried to work out what was wrong – while the plane stalled and went down like a stone.

Hmm… That’s kind of…

Machine 0 : 0 Human

Actually, it was a couple of hundred people hitting the middle of the Atlantic quite fast iirc.

What action do you take?

I wait a few milliseconds. Sensor can failed at time x, and not at time x+1. The same way some software works better with AE off . To be partly reliable is better than not to work at all… or not. Choices.

Most of time for mission critical tasks, you’ll opt for the status quo, rather than for a full and immediate stop (AKA stalled and went down). Simply because the critical point is the mission (continue flying). Of course that’s just the first step before something else. Stopping the engine, for a car, but probably not for a plane.

Gosh, I’m impressed with the calibre of the people posting to this thread. I hope the rocket and ECU companies are watching. They could hire people to solve all their problems.

The Airbus A330 (AF447) ‘handed-over’ to the two First Officers, who were on the flight deck. The captain was asleep having a programmed rest break when the ‘handover’ occurred.

The pilot flying (PF) continued to pull-back the ‘joystick’ until the plane stalled, dropped like a stone and eventually hit the South Atlantic. By the time the crew had awakened the sleeping Captain, there was not much time left to avert the crash. However, the captain did instruct the PF to push the stick forward (which was not
done).

Ralph

They could hire people to solve all their problems.

Be better to hire people to hack the hell out of the things, then present them with a list of what not to do…

I’ll give you an example of how ECU integration should never work: https://www.youtube.com/watch?v=MK0SrxBC1xs

I dont’t think it was resolved why the pilot flew into a thunderstorm instead of going around it?

Why the pilot ‘not in charge’ – kept pulling up on his joy stick?

(ie not touching his joy stick – when not flying)

Why the two joysticks didn’t have any feedback – bit like flying ‘Interdictor’ by two people with their own joysticks.

Gosh, I’m impressed with the calibre of the people posting to this thread. I hope the rocket and ECU companies are watching. They could hire people to solve all their problems.

They already have better people for this… But even a ‘near perfection team’ will not be 100% crash proof :)

Gosh, I’m impressed with the calibre of the people posting to this thread.

I’m not much impressed with the calibre of sarcasm on here. That there is criticism to be levelled at the rocket and ECU companies can scarcely be a matter of doubt. They do pretty well, but they ain’t perfect. Some of us on here are engineers, software or otherwise, and have worked in fairly high level jobs very comparable to those in rocket or ECU companies (in at least one case, actually in an ECU company). I for one am not levelling criticism at the engineers in those companies, but at the management – and the bean counters – and will continue to do so, sarcastic comments not withstanding.

It’s also easier to be wise after an event, like now.