Tuesday, October 6, 2015

Techy Tuesday - On VW, The EPA, and Benchmarks

By way of a short recap, on September 18, the EPA issued a Notice of Violation charging that Volkswagen had cheated on their emissions tests by running software in their Engine Control Units which responded differently when it "thought" it was being tested.  They allege that from model year 2009 through 2015, Turbocharged Direct Injection engines (apparently all models) were equipped with software that made the so-called "Clean Diesel" cars run in a different mode during the test profiles that regulators like the EPA use.  This resulted in cars on the road having Nitrogen Oxides (NOx) emissions up to 35 times higher than what the EPA measured.  Using their usual models, the EPA estimates that Volkswagen killed between 12 and 106 persons by diseases such as asthma and heart disease in the United States.

VW has been hit hard by this, with their stock price being hammered, the CEO leaving, and legal actions just getting started.  The head of their US operations has been called before congress this Thursday, and Germany's version of the EPA, the KBA, has demanded a plan for a fix by tomorrow, October 7th. 

I'm not in the auto industry, and don't own a VW, so I really don't have a dog in this fight.  A quick glance at the About Me box in the right column shows I'm a radio engineer, so my only familiarity with the whole question of Engine Control software and how to optimize diesel engines is from reading on the subject.  It's miles away from my expertise, but I wanted to touch on the subject of benchmark tests like this.  Benchmarks, standards, and making the numbers is part of engineering and a bigger part of marketing.

Consider the computer most of you are using to read this.  Computers have been subject to benchmarking how fast they operate for as long as they've been consumer items.  Electronic Design's editor Bill Wong writes:
... I was the first PC Labs Director for PC Magazine ... many decades ago. I helped put together the first benchmarks for PCs, printers, and networks. We distributed them via bulletin-board systems running on banks of modems as well as on 5.25-in floppy disks.
...
Eventually we had graphics benchmarks, which is where a lot of cheating occurred. In some instances, video drivers were specifically written to check if tests were being run and adjusted the way the driver performed often doing nothing. In one sense, this is a valid optimization since a set of operations that does no useful work such as changing what is on the screen could help improve the performance of the overall system. Of course, adding this check could lower it, too.
Surprised?  You shouldn't be.  People shopping for computers would read the benchmarks and tests searching to see which computer ran faster than the others, or the same speed but sold cheaper.  Having good published benchmarks could be the different between a successful and unsuccessful model.  That's big bucks to a place like Dell or another computer maker.  It's the same concept in ham radio; influential writers will point out the importance of some specification and soon all the hams looking for that sort of radio will be diligently reading specs to compare radios and see which radios are better at that particular specification.  In both cases, when consumers are shopping by one particular benchmark, they'll buy one that's an imperceptible amount better.  In this case the benchmarks were faked: when the engine was run in the test lab, doing programmed exercises on a dynamometer, it responded differently than it did with people driving.  Buying it based on its benchmarks would cheat the buyers. 

The bottom line is that there are huge incentives for companies to come up with ways to benchmark better and look better to buyers.  Software controlled engines are ripe for this sort of thing to happen.  It's especially likely to happen if the designers believe the test conditions are completely different from real world use and the test is just a barrier between them and selling their cars.  They think customers will be happier with cars that perform better in real driving than passing regulatory tests. 

But the regulators are the ones who can ruin your business and arrest you.


12 comments:

  1. I have worked in computers for 45 years. I began my career fixing them in the 60's and then programming them and towards the end of my career as project manager and database administrator. I could make the computer do whatever you wanted it to do. I worked with security systems including those used by and written by the CIA. What VW did was blatant dishonesty not fudging or kinda somewhat dishonest it was total computer fraud. But my point is that VW did it for no better reason than to keep their gas mileage high to improve their sales. What would a bureaucrat or the CIA or the Democrat party do? It is SOOOO easy to commit fraud with computers that I do not trust anything from computers. Someday we will vote online and even though voter fraud is common today it will be 100 times worse when computers are used.
    My favorite story of computer fraud was in the early years when banks began using computers a programmer got the brilliant idea to take the fraction of a cent interest paid on accounts that would have just been rounded off. He sent these fractions to his own account. But these tiny fractions of millions of accounts added up to a lot of money. So much money that an auditor caught it even though they didn't know where it came from and no money was missing.
    Make no mistake computer fraud(s) are rampant. You don't see it but it is there. The NSA is a small player in the bigger game of using computers to get what you want.

    ReplyDelete
  2. Theoretically - and it's entirely theoretical - testing should exactly emulate the Real World, whether it's cars, computers or condoms. Otherwise, testing is varying degrees of meaninglessness. Then again, what constitutes the Real World for me almost certainly does not match your Real World, so a well-controlled testing standard that can be reliably and accurately duplicated is a reasonable alternative despite its disconnect from actual Real World, but only in that comparisons may be made between test results.

    When testing varies from the Real World - which we see everywhere, "teaching to the test" has become endemic in our schools, as opposed to teaching what allows people to be successful in the Real World - it's entirely reasonable that competition for test results will become the norm.

    I would suspect that, first, Volkswagen isn't the only car manufacturer doing this, only that they're the first to get caught at it, and second, other manufacturers may be more subtle in how they approach the problem. I would be astounded to find that "designing to the test" isn't among the top ten or so considerations at every auto company; there's too much money at stake for it not to be that way.

    I wonder if there isn't middle ground; if testing better reflected the Real World, might VW's test results and Real World fuel mileage data been closer together? Is the proper question "how much fuel can society save" or "how clean should our air be"?

    ReplyDelete
  3. This one stinks. I am not a fan of VW because of their attitude towards their car owners and basic vehicle reliability (odd components develop quirks, are hard (physical access) to replace without exorbitant dealership costs, and they never fess up). Mostly very good cars, over-priced.
    What stinks in this one is that the US auto industry has been actively discouraging the import of diesel powered automobiles. If they were actually as bad as their US reputation, they would be dead overseas.
    I do not doubt that they were programming to pass test routines, there just ain't no way they were pioneering the practice.

    ReplyDelete
  4. I would hope that the team that wrote the software got a big bonus for figuring a way around the testing protocols. I see no problem with VW marketing a vehicle that meets the testing standards while offering better performance when not being tested.

    It's called innovation and thinking-outside-the-box ...

    ReplyDelete
  5. In the 1970s the GM products I bought had a compressor attached to the fan belt. It did nothing more than inject fresh air into the exhaust system. At the time pollution testing was done at the end of the exhaust pipe, and only measured the emissions there. Since the exhaust was diluted by the fresh air what was basically the same engine from the sixties passed the test. It did take power from the engine to run the compressor, resulting in lower gas mileage. So more gas was burned on each trip, and therefore more pollution was generated on each trip. Did GM "cheat" or did GM merely meet the requirements set forth by the Law?

    ReplyDelete
  6. The purpose of injecting air into the exhaust was to finish the burn of the fuel. Most gasoline engines use more fuel than there is sufficient oxygen to burn it all in the cylinder. This creates gases in the exhaust that are deemed more harmful than if they had been fully oxidized. Injecting air was/is a valid way to reduce pollution (at least as the pollution is defined by the EPA). It wasn't cheating and it was a legitimate solution at the time.

    ReplyDelete
  7. The VW fraud would probably never have eventuated if the problem - the setting of artificial "targets" for diesel engine testing - had not been created in error in the first place by a bloated bureaucracy with apparently little or no understanding of processes in statistical control (Shewhart, Deming).
    The fact that the targets were also set at evidently infeasible levels (QED) would have merely served to compound the problem.

    This looks very much like a textbook example of the sort of thing that W.E.Deming was on about when he published his 14-point philosophy, where point 11 was:
    11. [Eliminate targets with no basis in statistical veracity]
    a) Eliminate work standards (quotas) on the factory floor.
    Substitute leadership.
    b) Eliminate management by objective. Eliminate management by
    numbers, numerical goals. Substitute leadership.

    (from Chapter 2 of "Out of the Crisis", by W. Edwards Deming).

    However, from experience, I predict that, in common with a great many people, approx. 80% (Pareto Principle) of the people who might read this comment will fail to accept or understand the truth of point 11, primarily because it runs contrary to conventional wisdom, and they will be unlikely to have seen the proof of it in Deming's "Red Beads" teaching experiment.

    This could seem somewhat ironic, given that amongst his many awards and accolades, Prof. Deming was elected in 1983 to the National Academy of Engineering, and in 1986 to the Science and Technology Hall of Fame in Dayton and he was inducted into the Automotive Hall of Fame in 1991.
    Another case of "Pearls before swine" perhaps.

    ReplyDelete
  8. The problem is that a bureaucracy such as the EPA MUST continue to justify it's existence and to grow larger. They have long since passed the 80/20 rule with car emissions once they were able to eliminate or minimize the 80% of emissions that were easy and cheap to accomplish and are now after that last 20% which costs more than it's worth and probably cannot ever be eliminated. The simple fact is everything that burns/oxidizes creates "pollution" and while some fuels create more pollution they typically do so at dramatically lower costs than the "clean" fuels. It is likely that all of the effective pollution control that could be and should be achieved was accomplished by 1970 and everything that has happened since was simply justifying the high paying jobs at the EPA.

    ReplyDelete
  9. ^^ *Anonymous' comment)

    My prediction: "...approx. 80% (Pareto Principle) of the people who might read this comment will fail to accept or understand the truth of point 11..."

    Told you.

    ReplyDelete
  10. IMO VW did nothing wrong. They met these requirements:

    Pass the emission testing - DONE!

    Produce a car that gets really good mileage - DONE!

    Produce a car that performs well enough to be desirable to buyers - DONE!

    The fact that VW accomplished the above with outside the box thinking is admirable.

    itor

    ReplyDelete
  11. Itor - the more I learn about this situation, the more I agree with you. They figured out how make an adaptive ECU: when it was driven under certain unusual parameters, it changed mode to low NOx emissions, and when driven like a normal human would drive, it optimized performance and mileage.

    I can't say I'm 100% convinced of that, but call it 80%?

    ReplyDelete
  12. Same concept on climate monitoring instruments ... (and likely so many other things - financial markets?)

    "What answer do you want?"

    Q

    ReplyDelete