Thursday, April 16, 2020

Why Boeing's Problems With 737 Max Aren't Just Boeing's Fault

Let me lead with my conclusion.  My opinion is that the problem is largely from Boeing’s way of dealing with the FAA’s requirements and that those laws are the root cause of this sort of problem.  This isn't surprising because in an industry as heavily regulated as aviation, the ends up as a partner in every decision.  Exactly who we mean by saying “Boeing” isn’t clear, either; that is, was it high level management, was it lower level management pushing optimistic (and wrong) information up the chain of command or just who?

A friend who knows I worked in the industry sent me this article on the Verge that says the reason they've been unable to fix the software for the aircraft (so far) is partly due to the ancient computers they're using.  This is what prompted this entire post.
Every 737 Max has two flight control computers. These take some of the workload off of pilots, whether that’s through full automation (such as autopilot) or through fine control adjustments during manual flight. These computers can literally fly the airplane — they have authority over major control surfaces and throttles — which means that any malfunction could turn catastrophic in a hurry. So it’s more important for manufacturers to choose hardware that’s proven to be safe, rather than run a fleet of airplanes on some cutting-edge tech with bugs that have yet to be worked out.

Boeing took that ethos to heart for the Max, sticking with the Collins Aerospace FCC-730 series, first built in 1996. Each computer features a pair of single-core, 16-bit processors that run independently of each other, which reduces computing power but also keeps a faulty processor from taking down the entire system.
While that first paragraph has a lot of truth in it, the preference for hardware that as been “proven to be safe” goes beyond preference to having strong economic incentives or disincentives.  These go so far beyond simple preference that the article is telling half the story.  The article makes passing reference to regulatory scrutiny for new hardware but doesn't truly capture the intricacies of why the FAA regulations are so expensive and so strangling.

The basic cause of the problems with the 737 Max is that it was refresh of an old aircraft; the very first of the 737 series, the 737-100, rolled out in 1965.  Aircraft do get modernized, though nowhere near as often as consumer items; the last refresh of the series, the 737 Next Gen, was over 20 years ago in 1997.  One reason for the 20 year difference is the FAA’s requirements for this sort of refresh or modernization; nothing is allowed to fly without re-certification, and re-certification is horrifically expensive.

Let me step back for a moment.  In the case of a totally new aircraft, like the 787 Dreamliner a few years before, the FAA requires that prototypes be tested to extremes and the reams of documentation be submitted and reviewed before the aircraft can be accepted for flight.  As part of the process, every system: computer, radio, hydraulics, mechanical, or whatever's on that aircraft must be certified on that aircraft.  It's a redundant test.  I’ll use the examples of the various radio systems on board because those were my specialty and I’m most familiar with the picture; those boxes are tested to conform with a group of industry standards and compliance verified with a document called a TSO (Technical Standard Order).  Then they're additionally tested to ensure they work on that aircraft.  Since the purpose of TSO testing is that every radio meets the same requirements, every TSOed radio should work on any aircraft.  That means, for example, that any manufacturer's HF radio system should be interchangeable with any other manufacturer's HF radio, if it has passed all the same TSO tests.  That’s not to say they’ll be identical; the TSO certifies compliance to what the industry considered the Minimum Operational Performance Standards and we always took pride in being better than required, but all radios that have a TSO meets those minimums.  

The FAA doesn't accept that logic, essentially insisting when everything is combined together the TSO doesn’t matter; what matters is how they play together.  This view is not that different than insisting it doesn’t matter that the boxes are assembled with interchangeable screws, what matters is that specific screws work in specific holes; it’s dismissing the idea of interchangeable parts, and ignoring the continuous improvement in manufacturing!  What that lack of accepting TSO does is lock an old aircraft, like the 737, into using whatever was certified on that aircraft for the life of that model.  If parts become obsolete and a certain system can't be bought any more, the manufacturer can't just supply a more recent model.  That requires a new round of certification tests for the new system and this applies to everything on the aircraft; from engines to electronics to doors.  Again, for example, if the first 737 was certified with a vacuum tube radio for air to ground communication, they would be required to use that vacuum tube radio as long as they manufactured a 737 or until they certified a more modern radio.

It’s difficult to understate just how big a portion this is of the costs to create a new product in aviation from the airplane itself down to every little portion of it.

That's what trapped the 737 Max.  The main reason for the new version was to improve fuel economy by putting new engines on the plane.  That was going to force some amount of retesting and certification; for example, the bigger (higher bypass) engines required longer landing gear; the longer landing gear interfered with the existing radar system on the 737, so that was replaced with a more modern radar.  That meant the bigger engines, the longer landing gear and the radar required certification tests.  You can see how this could cascade into lots of tests unless they did their best to rein in changes.  The more things that need certification, the more expenses climb and could drag out schedule for years.  The first commandment became, "Thou Shalt Change As Little As Possible":
The new engines, which were larger and heavier than the ones on the Next Generation, did indeed make the Max just as fuel-efficient as its rival. But they also disrupted the flow of air around the wings and control surfaces of the airplane in a very specific way. During high-angle climbs, this disruption would cause the control columns in the airplane to suddenly go slack, which might cause pilots to lose control of the aircraft during a dangerous maneuver.

Boeing could have fixed this aerodynamic anomaly with a hardware change: “adaptive surfaces” on the engine housing, resculpted wings, or even just adding a “stick pusher” to the controls that would push on the control column mechanically at just the right time. But hardware changes added time, cost, and regulatory scrutiny to the development process. Boeing’s management was clear: avoid changes, avoid regulators, stay on schedule — period.
The FAA's rules – these aren’t new rules, everyone knew them going in - put Boeing in a tight spot.  Boeing's management, and I don’t know exactly at what levels, compounded the situation by not recognizing that certifying a more powerful flight computer could help their software fixes.  When they got into trouble, the fixes continued to demand more performance out of software to avoid hardware re-certification, compounding the bad decision to not bite the bullet and certify a new Flight Control Computer.  It looks now like that would still be the smart thing to do, but it would fly in the face of the corporate promises so far as to when the 737 Max will be able to fly again. 

Although I worked in the industry for nearly 20 years at a contractor to Boeing, Airbus, and many smaller aircraft manufacturers, I'm basing this on my experience with other programs.  I worked on the 737 Max, but in a fairly minor role in moving the new radar system from its original use to the new plane.  I retired at the end of 2015 and have not been involved since. 

Rollout of the first (or one of) 737 Max aircraft dedicated to qualifications testing, 2015.  Also from The Verge, 2015.


  1. When Sea Launch went into Chapter 11, some of my Boeing friends were transferred to Boeing Flight Test at Edwards. They were working on the "new" 747 Freighter, and then on the 787.

    Ohhhh, goodness...they things they saw. We were a lot more pragmatic at Sea Launch, and would often implement a "Best Fix" solution immediately, and worry about getting the paperwork caught up at a later date, but that didn't fly with anybody my friends worked with post Sea Launch. Stacks of paperwork HAD to be completed, much of it redundant or superfluous, before any testing could be done. The Engineers would be working furiously, while the Techs, Ground Crew, and Flight Crew sat around drinking coffee.

    It's a far more costly business than people realize, and a good chunk of the cost is paperwork compliance.

    1. It's a far more costly business than people realize, and a good chunk of the cost is paperwork compliance.

      I used to tell new grads that it took us longer to certify the product, after development, than the life of a consumer product. A slight exaggeration, but definitely longer than the latest cellphone would be the featured generation. OTOH, our products were expected to be in service for 10 or 15 years.

    2. And those products will probably still be usable in 30~50 years even though the rest of the systems they were used with have moved on.

      I have some WWII gear that still functions 100%!

  2. It's my belief that the 737MAX will never fly again. At some point Boeing will have to cut their losses and scrap them.

  3. OR you could fix the FAA. That might be easier.

    1. That was really my point. In a regulated industry like Aviation, nobody does a thing without the regulator's approval. I'm sure they believe holding onto outdated regulations makes everyone safer, but if not uprooted, the FAA should be pruned back to 1/100th the current size.

      Just like the FDA doesn't keep your food safe, the FAA doesn't keep your flying safe.

  4. Collins Aerospace computers? That explains a lot.

  5. Fix the FAA...hehehehehe! Have you ever seen the material you are required to know in the A&P test in order to be a mechanic? Half of it is on radial engines. They were to upgrade the material when I was going to A&P school. I'm retired. The test is unchanged. There is a reason magnetos are still used on most light aircraft. It is proven technology and any change subjects you to the FAA and LAWYERS. Both are deadly to aviation.

  6. Yes the FAA is the problem in many ways. Safety is important BUT no one can insure 100% safety. The FAA is operating just like it did 60+ years ago but in a more draconian way. The FAA is caught up in the bureaucratic, deep state nonsense; they believe they have the power to dictate.

    An observation by my late father who died in Dec 2000. He was an FAA (individual) Designated Engineering Representative (DER). He had worked for the FAA from 1958 until 1967 as an engineer and gained his DER when laid off from a major defense aviation retrofit company. What he observed was that by the late 1990's the engineering ranks of the FAA were filling up with people whose only experience with aviation was flying on an airliner. He had young engineers dismissing his 30 plus years of experience and knowledge because they didn't know squat but were in authority. The management accept this. I have a feeling that is what is still going on. God save us!

  7. The same mentality pervades the Nuclear Power industry as regulated by the NRC. There are reasons why US companies can't compete in foreign markets for Nuclear Power Plants, but a big one is that countries without a regulatory system similar to the NRC will only buy US designs certified by the NRC. Ironically they will buy Russian or Chinese copies of US designs at half price, without a qualm.

  8. This sounds unfortunately like some of the NASA stories I've heard, in particular about computers on the Space Shuttle and why they were never upgraded from the original 8086-based units.

  9. Your post is what I mean by "I don't think you disbelieve central planning has been disproved and anyone who does it is a kook, like you disbelieve 100 MPG carburators that run on water" What if we use deionized water? Run the water past a magnet? Make the jet passage square? Use a homeopathic dose of octane increaser? You're just certain that central planning for flying vehicles must work. All those previous failures like Boeing weren't real socialism, so they don't count as disproofs. You think you're being critical of government but you aren't; you're just being critical of this one particular program at this one particular time. You aren't being critical of government in general.

    Telsa cars wouldn't exist without the electric car subsidy. Would SpaceX exist without the NASA subsidy? Maybe SpaceX now is in the position where Boeing used to be, their engineering is currently good but it's going to turn out like Boeing (and the joint strike fighter, and the Osprey tilt-rotor helicopter, and the Zumwalt ship, and the threatened cancellation of the A-10 and the B-52, both of which work). It doesn't matter how many printed pages of failed government technology development I list; government is a religious faith.