Software verification, money and law

I am probably only revealing the depths of my ignorance in asking this question. Presumably software that is used in safety-critical situations or by branches of the state, must be verified and tested against documented specifications. I cannot believe that AI is up to this, and in any case some human has to carry the can in case of damaging errors. Do insurance companies employ expert software assessors (and have to pay them handsomely), before giving advice to guarantors? This hinterland at the triple juncture of business, law and software is something that I can only guess at. Has anybody on the forum any knowledge of what actually takes place?

Actual knowledge, no.

But looking at Tesla, and considering they’re calling their AI an “autopilot”, if anything goes wrong, it’s the meatsack who is culpable.

ISO 9001 is a general hand-wringing “we do things properly here” standard, which basically means writing down whatever awful thing you’re doing and then confirming that, yes, that’s what you’ve just done. It’s supposed to be “quality management” but it’s actually consistency management. [Yes we got it wrong, but we always get it wrong]

ISO 27001 is a much more thorough Information Security and rick management standard. [I meant risk, but Freud wins here]

But in both cases, what you claim the standard guarantees about your product, and how that is accredited and audited, is defined by the claimant, not by a standards body.

It may be better than nothing though.

Actually submitting source to escrow and having it analysed is rare and expensive. And won’t be happening in the USA because their legal culture is everything is fine until a lawsuit says otherwise. We have standards, they have lawyers.

But looking at Tesla, and considering they’re calling their AI an “autopilot”, if anything goes wrong, it’s the meatsack who is culpable.

I’m sure there was a court case with that finding

ISO 26262 is the automotive software standard.

Automotive being another one of my former industries, it fills me with horror that unverifiable statistical data models could be put in charge of a vehicle.

My Audi is not autonomous, but has several of the necessary features as driver aids. The Active cruise is pretty good, and useful. It very successfully keeps to the set speed, until it catches up to another vehicle, then keeps a safe distance behind, coming to a stop if necessary. However occasionally on corners it will lose sight of the vehicle in front and accelerate. Other times it starts to do a panic stop for something on the side of the road.
The Lane Assist however is dire. 50% of the time it disables itself as it can’t see the lane clearly enough. Sometimes old changes to lanes, e.g. for road works throw it out and it tries to lurch across the road. It has taught me to signal before changing lanes however, as otherwise it fights me.
The latest Satnav update doesn’t have updates to major roads that were in place 2 years ago (3 years in one case for the Netherlands). That needs to change.
If these technologies are going to drive vehicles, they need MASSIVE improvement. A car that can’t keep in lane, that doesn’t realise there’s something in front, and doesn’t know where the road goes? The mind boggles.

writing down whatever awful thing you’re doing and then confirming that, yes, that’s what you’ve just done

I used to work for an outfit that did QAs to 9001. Written the odd Quality Manual in my time.

Even talked with the guy in charge about whether it would be possible to fake up a business, create the most assinine process imaginable (customer complaints directly filled in the paper shredder, for example) and get it approved. He told me that if the process is clearly documented and scrupulously followed, it would be. Following the process is important. The process making sense isn’t.

and rick management standard

Rick management is easy. Just supply Tetley and don’t mention the B word (hint: 2016).
Garibaldi optional.

is defined by the claimant, not by a standards body.

Yup. It’s an interesting perversion of what most people might assume the word “quality” to mean.

that unverifiable statistical data models could be put in charge of a vehicle

I was allowed to drive, on real roads with others, with exactly zero experience. ;)

If you aren’t familiar with my little toy car… https://youtu.be/ae-UHFhIi0E

but has several of the necessary features as driver aids

Features that sound good but could go dangerously wrong? That’s not my definition of “necessary”.

The only three driving aids I’d like is power steering (no more fighting the wheel), ABS for improved braking, and there’s a thing where you can press a button to lock in the current speed and the car will keep to that. It doesn’t try to do anything else like detect the car in front, you are still responsible for braking when necessary. It just makes things easier if you’re on a long open road.
All the other “smarts” can remain in the box. It’s enough to have to second guess the other idiots on the road, I don’t want to have to deal with erratic behaviour from the car.

At my last permanent job, inspecting our backup procedure, the ISO 9001 inspector told a story of him just failing a company. Turns out their procedure stated that ‘tapes must be stored according to manufacturer’s recommendations’ in their fireproof safe. They were failed because they stood their DAT tapes up on their side (shortest edge at the base) so their labels could easily be read, but the manufacturer’s website said they should be stored in a different (I forget which) orientation.

As for car aids, when my MX-5 was in for service last year I got a brand-new CX-30 for the day. Bing! Something. Bing! Something else. The absolute worst was the lane keeping feature, which was distracting at best, and somewhat dangerous. Driving rural backroads, one often happily drifts over the centre line (when wide enough to have one). Car steering wheel vibrates furiously on encroaching, and was disturbed to find that trying to sweep across to use the other side, it actually actively tried to steer back. F-that.

Even my Subaru XV is tainted. Coming uphill around the sharp hairpin bend, 90% of the time it thinks ‘oh, high yaw rate, must need to muck about with the traction’ and flash warnings. Old Subarus just got on with it. MX-5 (LSD) just chucks the tail around and gets on with it.

Features that sound good but could go dangerously wrong? That’s not my definition of “necessary”.

I meant necessary to achieve autonomous vehicles. (Hopefuly not in my lifetime.)

ISO9001 compliance simply means you adhere to your own documented standards.
ISO27001 compliance is focused on current best practice around information security.

Way, way back, mission-critical systems used to be mathematically modelled prior to being coded. With the preference on outsourcing your risk to another entity these days, I doubt that’s still the case.

I vaguely remember mathematically modelling an air traffic control system in the late 80’s, that was quite interesting.

I knew someone in the late 80s who worked as an air traffic controller. He said they all had casters on their chairs so when a guy goes doolally they can just drag him away from the board PDQ. And someone DID have a breakdown at the board while he was there.

His patch included Mildenhall so he would sometimes see an SR71 on his scope – near stationary but for the whirling altitude changing by 200ft per second before it disappeared off the top of the radar.

Hope Boeing does better with the SR72 than the 737 Max.

Hope Boeing does better with the SR72 than the 737 Max.

Spying on Lockheed Martin to find out how to improve the X-51?

@ Gavin

Presumably software that is used in safety-critical situations or by branches of the state, must be verified and tested against documented specifications.

Yes software that has to be provided to governments, financial services, health care and more (and these days pretty much everywhere) has to be tested.

There are a number of Standards, some has been briefly mentioned here already, others are listed in this article and with a description of what they cover:

https://www.bcs.org/membership-and-registrations/member-communities/software-testing-specialist-group/the-tester/newsletter-archive/standards-for-software-testing/

It is important to note that there are also Standards that cover data processing and retention and, beside the available Standards, there are also regulations that can change from country to country.

In general, in terms of regulations, the major contributions come from EU and the US, but, obviously, other countries may implement similar or specific forms.

In the last few decades also Security standards and regulations have started to emerge and some famous one can be found on the NIST website for example, here you can find a recently proposed EU regulation on the matter of Cyber Security which also includes rules for software development:

https://commission.europa.eu/publications/proposal-cybersecurity-regulation_en

Please note: althought if some regulation and/or Standard may not directly refer to the software testing process as of the first link I shared, companies that work or provide software solutions to govs, health care (or even corporations these days due to supply chain security requirements) may be required to follow such regulations and standards as part of the daily work process, and as such it does fall in the cathegory you have requested.

There are also certifications, for example FIPS-140 that applyes to software and hardware solutions in an offer for certain governments if you wish to sell them your software solution, more details about this here:

https://csrc.nist.gov/publications/detail/fips/140/3/final

The above link also shows that, if software uses certain types of encryptions, then it must adhere to certain algorithms and they shall not be modified and such algorithms are certified as well.

For financial services there are also certifications like PCI-DSS and HIPPA and more.

Also, on top of all this, there are things like fault tollerance requirements, fail over requirements, and, in general, the more something becomes mission-critical in certain environments like health care or military infrastructure, the more strict certain requirements become.

Safety requirements can be part of a specific project and so may not be covered by a specific standard or regulation, however it would be required to be included as part of a deliverable and therefore prove it’s in place.

Hopefully this provides an understanding of the “playground” here.

For what concerns AI, things are a bit more complicated. AI algorithms may require some of the aforementioned certifications, standards, regulations and requirements like all other software provided to governments etc., but they also may have to adhere to privacy laws (which is a separate category of tests, data processing and retention rules and, again depends on the country). Again here the “may”, “can” etc. are because different countries may have different combinations of required stdandards, regulations and project’s requirements.

The correctness of a response is not always considered mission-critical, that depends on the way a solution is sold. Govs, big pharma, financial services have always invested (for different reasons) on tools that could provide some complex data analysis to help improving decisional processes, automation, costs reduction, military effectiveness and alike. That doesn’t means such tools are required to be absolutely right.

I understand that some people is trying to push the concept that these so called “AI” (let’s be careful with terms, there isn’t AI yet) tools will replace humans, but in reality it’s not that simple, here is an example:

Hypotesys = tools like CodePilot reaches capacity of writing “correct” software, where correct is within certain standards and regulations tollerance. This seems to be the biggest fear being shared over Youtube/Twitter etc.

In such an hypotesys, people believe that developers will be replaced by machines. The reality is not that simple, because having complete automated software solution design and development means no one in “company A” has any idea of how the software works, so the assumption for the first fear to become true would be: the machine has reached such a level of capability that would make the machine accountable for the software it generates.

Here is the problem, a computer is not accauntable for anything as it’s not a human being. So, someone may argue: ok then the CEO of “company A” is accauntable for the machine, that would be a plausible idea. But that CEO should be completely insane not to create a hirearchy of humans to share the accauntability with, which would lead to, well, have the engineers still at their place, just coding less and more controlling/supervising the process of software development.

There are tons of these examples, my reply is already extremely lenghty and I do not wish to write an essay.

Hope this helps somehow.

@Paolo

Yes, very much. Thanks for taking the trouble to make your answer. Over thirty years ago I was a (passive) host at a summer school on software development, where the guests were mostly from business or government, trying to sell the importance of specification. We were aghast to learn from one guest that at that time even software for flight-control systems were not properly verified.

Gavin is correct that over 30 years ago when the first fly by wire systems for commercial aircraft were being developed, software verification was in its infancy. You’ll be glad to know that it did mature very rapidly. However, software is only tested against the limited set of scenarios which can be envisioned in advance by the designers.

My favourite incident is an a plane which slid right off the end of an icy runway many years ago, with everything working completely to specification, but nothing slowing the plane down. On touching down on ice there was no friction to start the wheels turning, so the safety mechanism prevented the spoilers being deployed in flight. Because the spoilers didn’t deploy and push the aircraft down, there weight on wheels sensor didn’t register the plane was on the ground. Because the plane didn’t register it was on the ground the reverse thrust wouldn’t activate. That left the plane with no wheel braking, no aerodynamic braking, no thrust breaking, so off the end of the runway it went, but luckily everyone survived.

Two lessons from that, first; you can’t think of everything that could happen in advance, second; when something unexpected does happen ensure there are overrides which can be activated quickly enough for the pilots to do something about the situation.

ensure there are overrides which can be activated quickly enough for the pilots to do something about the situation

and of course make sure the pilots know about the overrides. ref: 737MAX.

I worked for a company with ISO 9001 accreditation. The rules said the examiner should look for one of each of system, subsystem, and detailed documentation. Since the company had one of each, it passed. I didn’t stay long.