Boeing's Starliner Contained a “Catastrophic” SW Bug

Boeing's mid-December launch of their crew capsule, the Starliner, intended to take US astronauts to the ISS seems to look worse the closer you look.  A few weeks ago, I posted a story about issues with thruster performance during the flight.  Today, Ars Technica reports that a potentially catastrophic software bug was found and corrected during the flight, a bug that could have led to loss of the spacecraft.

The software issue was identified during testing on the ground after Starliner's launch, said panel member Paul Hill, a former flight director and former director of mission operations at Johnson Space Center in Houston. The problem would have interfered with the service module's (SM) separation from the Starliner capsule.

“While this anomaly was corrected in flight, if it had gone uncorrected it would have led to erroneous thruster firing and uncontrolled motion during SM separation for deorbit, with the potential for catastrophic spacecraft failure,” Hill said during the meeting.

This story was revealed at a quarterly meeting of NASA's Aersopace Safety Advisory Panel meeting last Thursday, however Ars space reporter Eric Berger reveals he was told about this issue in January.

However, as part of reporting on a story about Starliner software and thruster issues three weeks ago, a source told Ars about this particular problem. According to the source, Boeing patched a software code error just two hours before the vehicle reentered Earth's atmosphere. Had the error not been caught, the source said, proper thrusters would not open during the reentry process, and the vehicle would have been lost.

Why wasn't this in the January Ars report?  Berger says that he questioned Boeing about the issue, but they downplayed the seriousness of the bug and he accepted their explanations.

NASA's Aersopace Safety Advisory Panel was not as accepting or forgiving.

But the public remarks by Hill on Thursday appear to underscore the seriousness of the issue, and the safety panel recommended several reviews of Boeing. "The panel has a larger concern with the rigor of Boeing's verification processes," Hill said. "As a result, the panel recommends that NASA pursue not just the root cause of these specific flight-software anomalies but also a Boeing assessment of and corrective actions for Boeing's flight-software integration and testing processes."

The safety panel also recommended that NASA conduct "an even broader" assessment of Boeing's Systems Engineering and Integration processes. Only after these assessments, Hill said, should NASA determine whether the Starliner spacecraft will conduct a second, uncrewed flight test into orbit before astronauts fly on board. (Boeing recently set aside $410 million to pay for that contingency).

Finally, before the meeting ended, the chair of the safety panel, Patricia Sanders, noted yet another ongoing evaluation of Boeing. "Given the potential for systemic issues at Boeing, I would also note that NASA has decided to proceed with an organizational safety assessment with Boeing as they previously conducted with SpaceX," she said.

Boeing, NASA, and U.S. Army personnel work around the Boeing CST-100 Starliner spacecraft shortly after it landed in December.  NASA/Bill Ingalls photo

---

Last Wednesday's launch of 60 Starlink Satellites for SpaceX proved a little rougher on the landing than we usually see.  Teslarati reports that the booster is probably reusable but shows an interesting picture: the same booster after its previous flight (left) and this one. 

The difference jumps out at you simply by looking at the height of the black-painted section of the booster.  Assuming those guys are around 6' tall, it looks to me that it's 2-3 feet lower.  Notice the landing legs are also at a lower angle.  Something bent? 

Taken from almost identical perspectives as the drone ship passed through the mouth of Port Canaveral, the difference in the booster’s height and stance are hard to miss, with B1051’s engine bells and the black ‘belt’ of its heat-shielded engine section clearly sitting several feet lower after Starlink V1 L3.

While subtle, the most important difference is near the tips of each visible landing leg’s telescoping boom, visible in the form of a final, smaller cylinder on the left (earlier) image. On the right, that cylinder has effectively disappeared. This is actually an intentional feature of Falcon 9’s landing leg design: known as a ‘crush core’, the tip of each leg boom holds a roughly 1m (3ft) long cylinder of aluminum honeycomb, optimized to lose structural integrity (crush) only after a specific amount of force is applied. In essence, those crush cores serve as dead-simple, single-use shock absorbers that can be reused as long as a given booster’s landing is gentle enough.

Whether that landing was “gentle enough” or not will require more thorough inspections than looking across a few hundred yards of water from Jetty Park in Port Canaveral.  Exactly why the booster landed hard enough to crush those crush cores hasn't been released - if it's known.  It could have been caused by the drone being at the bottom of a swell or an anomaly with the booster’s landing engine. 

We'll try to keep up with whether good old booster B1051 flies again or gets totaled out.