Why Boeing's Problems With 737 Max Aren't Just Boeing's Fault

Let me lead with my conclusion.  My opinion is that the problem is largely from Boeing’s way of dealing with the FAA’s requirements and that those laws are the root cause of this sort of problem.  This isn't surprising because in an industry as heavily regulated as aviation, the Fed.gov ends up as a partner in every decision.  Exactly who we mean by saying “Boeing” isn’t clear, either; that is, was it high level management, was it lower level management pushing optimistic (and wrong) information up the chain of command or just who?

A friend who knows I worked in the industry sent me this article on the Verge that says the reason they've been unable to fix the software for the aircraft (so far) is partly due to the ancient computers they're using.  This is what prompted this entire post.

Every 737 Max has two flight control computers. These take some of the workload off of pilots, whether that’s through full automation (such as autopilot) or through fine control adjustments during manual flight. These computers can literally fly the airplane — they have authority over major control surfaces and throttles — which means that any malfunction could turn catastrophic in a hurry. So it’s more important for manufacturers to choose hardware that’s proven to be safe, rather than run a fleet of airplanes on some cutting-edge tech with bugs that have yet to be worked out.

Boeing took that ethos to heart for the Max, sticking with the Collins Aerospace FCC-730 series, first built in 1996. Each computer features a pair of single-core, 16-bit processors that run independently of each other, which reduces computing power but also keeps a faulty processor from taking down the entire system.

While that first paragraph has a lot of truth in it, the preference for hardware that as been “proven to be safe” goes beyond preference to having strong economic incentives or disincentives.  These go so far beyond simple preference that the article is telling half the story.  The article makes passing reference to regulatory scrutiny for new hardware but doesn't truly capture the intricacies of why the FAA regulations are so expensive and so strangling.

The basic cause of the problems with the 737 Max is that it was refresh of an old aircraft; the very first of the 737 series, the 737-100, rolled out in 1965.  Aircraft do get modernized, though nowhere near as often as consumer items; the last refresh of the series, the 737 Next Gen, was over 20 years ago in 1997.  One reason for the 20 year difference is the FAA’s requirements for this sort of refresh or modernization; nothing is allowed to fly without re-certification, and re-certification is horrifically expensive.

Let me step back for a moment.  In the case of a totally new aircraft, like the 787 Dreamliner a few years before, the FAA requires that prototypes be tested to extremes and the reams of documentation be submitted and reviewed before the aircraft can be accepted for flight.  As part of the process, every system: computer, radio, hydraulics, mechanical, or whatever's on that aircraft must be certified on that aircraft.  It's a redundant test.  I’ll use the examples of the various radio systems on board because those were my specialty and I’m most familiar with the picture; those boxes are tested to conform with a group of industry standards and compliance verified with a document called a TSO (Technical Standard Order).  Then they're additionally tested to ensure they work on that aircraft.  Since the purpose of TSO testing is that every radio meets the same requirements, every TSOed radio should work on any aircraft.  That means, for example, that any manufacturer's HF radio system should be interchangeable with any other manufacturer's HF radio, if it has passed all the same TSO tests.  That’s not to say they’ll be identical; the TSO certifies compliance to what the industry considered the Minimum Operational Performance Standards and we always took pride in being better than required, but all radios that have a TSO meets those minimums.  

The FAA doesn't accept that logic, essentially insisting when everything is combined together the TSO doesn’t matter; what matters is how they play together.  This view is not that different than insisting it doesn’t matter that the boxes are assembled with interchangeable screws, what matters is that specific screws work in specific holes; it’s dismissing the idea of interchangeable parts, and ignoring the continuous improvement in manufacturing!  What that lack of accepting TSO does is lock an old aircraft, like the 737, into using whatever was certified on that aircraft for the life of that model.  If parts become obsolete and a certain system can't be bought any more, the manufacturer can't just supply a more recent model.  That requires a new round of certification tests for the new system and this applies to everything on the aircraft; from engines to electronics to doors.  Again, for example, if the first 737 was certified with a vacuum tube radio for air to ground communication, they would be required to use that vacuum tube radio as long as they manufactured a 737 or until they certified a more modern radio.

It’s difficult to understate just how big a portion this is of the costs to create a new product in aviation from the airplane itself down to every little portion of it.

That's what trapped the 737 Max.  The main reason for the new version was to improve fuel economy by putting new engines on the plane.  That was going to force some amount of retesting and certification; for example, the bigger (higher bypass) engines required longer landing gear; the longer landing gear interfered with the existing radar system on the 737, so that was replaced with a more modern radar.  That meant the bigger engines, the longer landing gear and the radar required certification tests.  You can see how this could cascade into lots of tests unless they did their best to rein in changes.  The more things that need certification, the more expenses climb and could drag out schedule for years.  The first commandment became, "Thou Shalt Change As Little As Possible":

The new engines, which were larger and heavier than the ones on the Next Generation, did indeed make the Max just as fuel-efficient as its rival. But they also disrupted the flow of air around the wings and control surfaces of the airplane in a very specific way. During high-angle climbs, this disruption would cause the control columns in the airplane to suddenly go slack, which might cause pilots to lose control of the aircraft during a dangerous maneuver.

Boeing could have fixed this aerodynamic anomaly with a hardware change: “adaptive surfaces” on the engine housing, resculpted wings, or even just adding a “stick pusher” to the controls that would push on the control column mechanically at just the right time. But hardware changes added time, cost, and regulatory scrutiny to the development process. Boeing’s management was clear: avoid changes, avoid regulators, stay on schedule — period.

The FAA's rules – these aren’t new rules, everyone knew them going in - put Boeing in a tight spot.  Boeing's management, and I don’t know exactly at what levels, compounded the situation by not recognizing that certifying a more powerful flight computer could help their software fixes.  When they got into trouble, the fixes continued to demand more performance out of software to avoid hardware re-certification, compounding the bad decision to not bite the bullet and certify a new Flight Control Computer.  It looks now like that would still be the smart thing to do, but it would fly in the face of the corporate promises so far as to when the 737 Max will be able to fly again. 

Although I worked in the industry for nearly 20 years at a contractor to Boeing, Airbus, and many smaller aircraft manufacturers, I'm basing this on my experience with other programs.  I worked on the 737 Max, but in a fairly minor role in moving the new radar system from its original use to the new plane.  I retired at the end of 2015 and have not been involved since. 

Rollout of the first (or one of) 737 Max aircraft dedicated to qualifications testing, 2015.  Also from The Verge, 2015.