Thursday, February 6, 2020

Boeing's Starliner Contained a “Catastrophic” SW Bug

Boeing's mid-December launch of their crew capsule, the Starliner, intended to take US astronauts to the ISS seems to look worse the closer you look.  A few weeks ago, I posted a story about issues with thruster performance during the flight.  Today, Ars Technica reports that a potentially catastrophic software bug was found and corrected during the flight, a bug that could have led to loss of the spacecraft.
The software issue was identified during testing on the ground after Starliner's launch, said panel member Paul Hill, a former flight director and former director of mission operations at Johnson Space Center in Houston. The problem would have interfered with the service module's (SM) separation from the Starliner capsule.

“While this anomaly was corrected in flight, if it had gone uncorrected it would have led to erroneous thruster firing and uncontrolled motion during SM separation for deorbit, with the potential for catastrophic spacecraft failure,” Hill said during the meeting.
This story was revealed at a quarterly meeting of NASA's Aersopace Safety Advisory Panel meeting last Thursday, however Ars space reporter Eric Berger reveals he was told about this issue in January.
However, as part of reporting on a story about Starliner software and thruster issues three weeks ago, a source told Ars about this particular problem. According to the source, Boeing patched a software code error just two hours before the vehicle reentered Earth's atmosphere. Had the error not been caught, the source said, proper thrusters would not open during the reentry process, and the vehicle would have been lost.
Why wasn't this in the January Ars report?  Berger says that he questioned Boeing about the issue, but they downplayed the seriousness of the bug and he accepted their explanations.

NASA's Aersopace Safety Advisory Panel was not as accepting or forgiving.
But the public remarks by Hill on Thursday appear to underscore the seriousness of the issue, and the safety panel recommended several reviews of Boeing. "The panel has a larger concern with the rigor of Boeing's verification processes," Hill said. "As a result, the panel recommends that NASA pursue not just the root cause of these specific flight-software anomalies but also a Boeing assessment of and corrective actions for Boeing's flight-software integration and testing processes."

The safety panel also recommended that NASA conduct "an even broader" assessment of Boeing's Systems Engineering and Integration processes. Only after these assessments, Hill said, should NASA determine whether the Starliner spacecraft will conduct a second, uncrewed flight test into orbit before astronauts fly on board. (Boeing recently set aside $410 million to pay for that contingency).

Finally, before the meeting ended, the chair of the safety panel, Patricia Sanders, noted yet another ongoing evaluation of Boeing. "Given the potential for systemic issues at Boeing, I would also note that NASA has decided to proceed with an organizational safety assessment with Boeing as they previously conducted with SpaceX," she said.
Boeing, NASA, and U.S. Army personnel work around the Boeing CST-100 Starliner spacecraft shortly after it landed in December.  NASA/Bill Ingalls photo



Last Wednesday's launch of 60 Starlink Satellites for SpaceX proved a little rougher on the landing than we usually see.  Teslarati reports that the booster is probably reusable but shows an interesting picture: the same booster after its previous flight (left) and this one. 


The difference jumps out at you simply by looking at the height of the black-painted section of the booster.  Assuming those guys are around 6' tall, it looks to me that it's 2-3 feet lower.  Notice the landing legs are also at a lower angle.  Something bent? 
Taken from almost identical perspectives as the drone ship passed through the mouth of Port Canaveral, the difference in the booster’s height and stance are hard to miss, with B1051’s engine bells and the black ‘belt’ of its heat-shielded engine section clearly sitting several feet lower after Starlink V1 L3.

While subtle, the most important difference is near the tips of each visible landing leg’s telescoping boom, visible in the form of a final, smaller cylinder on the left (earlier) image. On the right, that cylinder has effectively disappeared. This is actually an intentional feature of Falcon 9’s landing leg design: known as a ‘crush core’, the tip of each leg boom holds a roughly 1m (3ft) long cylinder of aluminum honeycomb, optimized to lose structural integrity (crush) only after a specific amount of force is applied. In essence, those crush cores serve as dead-simple, single-use shock absorbers that can be reused as long as a given booster’s landing is gentle enough.
Whether that landing was “gentle enough” or not will require more thorough inspections than looking across a few hundred yards of water from Jetty Park in Port Canaveral.  Exactly why the booster landed hard enough to crush those crush cores hasn't been released - if it's known.  It could have been caused by the drone being at the bottom of a swell or an anomaly with the booster’s landing engine. 

We'll try to keep up with whether good old booster B1051 flies again or gets totaled out.


13 comments:

  1. Wow, move your headquarters and the evil headquarters curse hits one right after another (My Theory: When company is doing good, then builds a Taj Mahal of a headquarters, that move is an indication of the coming death of a company. May be related to not spending money on your employees and infrastructure.)

    I wouldn't be surprised that Boeing's lack of openness over what happened gets them derated. Heck, if I was NASA I'd require 2 cargo only trips to the ISS before I'd even look at man-rating it, and force Boeing to go over all their tests again.

    Yet... Still closer to space than Orion.

    And, again, SpaceX's openness is the complete opposite of Boeing or Lockheed. Which is why SpaceX is doing so much better than anyone else.

    ReplyDelete
    Replies
    1. The company I retired from went through that. Since I retired at the end of '15, they've merged once and now it looks like they'll be merging with another company this year.

      It looks like Boeing really needs the audits they just got assigned. Unless I'm mistaken, that "organizational safety assessment" they're going to put Boeing through is what SpaceX got hit with for Musk smoking weed with Joe Rogan. Which implies NASA thinks Boeing is so eff'ed up they must be doing drugs.

      Delete
  2. Thank you very much for the update. These are developments that I was unaware of/didn't follow as they were announced.

    ReplyDelete
    Replies
    1. Thanks, Larry. I'm thinking that since there's so much space activity close to home, this is something I can add some value about.

      Delete
  3. Different pad on the booster. White structural steel is missing, solid struts replaced by those cable/chain shackles.

    ReplyDelete
    Replies
    1. Both pics are of the booster on their drone barge over here, OCISLY (Of Course I Still Love You). When they land, none of that hardware is on the deck, it gets added to secure the booster - so they don't lose 40 or 50 million dollars worth of booster to bad seas or bad handling. My guess is they have alternate sets of the supports for contingency use.

      Delete
    2. Yup - wonder why they'd choose one over the other. I generally like lower centers of gravity . . . it makes 'em more stable and hard to push over, like The Mrs.

      Delete
  4. On a related topic, Moon of Alabama notes that there is a serious structural flaw in all 737 NG and related models. It consists of the use of defective fasteners. It showed up in the structural breakup of several 737's during rough landings. Boeing's management was informed of the fastener problem, but judged the likelihood of a catastrophic failure in flight was minimal. So far they have been right, all the breakups occurred on landing. But it is yet another indication of the Boeing management's disregard for safety.

    ReplyDelete
    Replies
    1. yes. It would appear that some project manager did such a great job "trimming fat" on the 737 Max program that he got promoted to do the same for the Starliner. Corporate group-think, gotta love it.
      Tom S.

      Delete
    2. Been out of town all day, so I missed this. Does Moon of Alabama mention if the fasteners are counterfeit or just what? To me, defective could mean just that: they were bought from a certified vendor and the vendor screwed up. Nowadays, it could also mean the parts were counterfeit. That's a big problem these days, and you'll never guess which People's Republic those tend to come out of. In either case, quality assurance should find those, but whenever statistical sampling is used, there's a chance it isn't found (and you can't test 100% of the fasteners to failure).

      Delete
  5. I always hack on the control software of my vehicles while they are in motion. If it compiles then I'm done, right?

    ReplyDelete
  6. Seems I remember Boeing priding themselves on their safety culture. And then bean counters....

    ReplyDelete