Friday, December 20, 2019

Boeing Starliner Test Flight Failing Main Objective

As we reported on back on December 2nd, Boeing was preparing for an unmanned, Orbital Flight Test of its Starfire crew capsule this month.  The launch was before sunrise this morning, at 6:36 AM EST, so I got up at 6:15 to watch.  It wasn't until around 20 minutes into the mission that the problem surfaced.  At this time, it appears the main objective of the mission, to launch a man-rated capsule to ISS for autonomous rendezvous and docking is a failure.  Summary reporting from Ars Technica:
During a post-launch news conference, NASA Administrator Jim Bridenstine explained that the mission elapsed timing system had an error in it, with the net effect that the spacecraft thought it was performing an orbital insertion burn, when in fact it was not. The on-board computer then expended a significant amount of propellant to maintain a precise attitude, thinking it had reached orbit.

"Today, a lot of things went right," Bridenstine said. "But we did not get the orbital insertion burn we were hoping for."

When ground-based controllers realized the problem, they immediately sent a command to begin the orbital insertion burn, but due to a communications problem—which could have been a gap in coverage of NASA's Tracking and Data Relay Satellite System or some spacecraft error—those commands were not received right away by Starliner. So it continued to expend fuel to maintain a precise attitude.

By the time the commands got through, Starliner had expended too much fuel to make a safe rendezvous and docking with the International Space Station, the primary goal of this test flight.
This sounds like a silly mistake in software (and I realize it's always easier to see them in retrospect): the vehicle decided it was firing the orbital insertion burn based on one thing: the mission timer.  They could have used other sensors to cross check the mission timer's accuracy rather than just assume the timer is correct.  Starliner is safe and in a stable orbit, so some of the tests planned for the mission can be conducted, up to and including landing under parachutes in New Mexico on Sunday.  It seems that is likely to be an obstacle to Boeing getting this capsule approved for a manned flight to the ISS.
The spacecraft has operated nominally in a number of different ways so far, with good power systems, cooling, and more. However, the inability to perform a docking at the station raised the question of whether Boeing will need to perform a second uncrewed test flight before NASA allows its astronauts to fly on Starliner,.

“I think it’s too early to know," Bridenstine responded during the news conference. "We don’t know what the root cause is. I’m not saying yes and I’m not saying no."

NASA does not have a specific requirement that Starliner must demonstrate a rendezvous-and-docking before it clears Starliner for human flights, but certainly the agency's engineers would feel more comfortable with that task having been performed.

In any case, the primary issue appears to be one of software rather than hardware. Boeing may be able to convince NASA that had humans been on board the vehicle, they would have immediately recognized the problem and manually commanded the orbital insertion burn.

Whether there is an additional test flight or not, this problem almost certainly delays Boeing's crewed flight of Starliner beyond the first half of 2020. This would, for the time being at least, put SpaceX back into the lead for getting humans back into space from a US-based launch pad. A source said Friday that the company appears to be on track toward a springtime launch of astronauts on board the Crew Dragon vehicle.
Interestingly, from before this morning's launch, again sourcing Ars Technica:
After funding several development efforts in the early 2010s, NASA down-selected to Boeing and SpaceX in 2014 to finalize design and development of their commercial crew vehicles. All told, the agency has paid Boeing $4.8 billion and SpaceX $3.1 billion for Starliner and the Crew Dragon spacecraft, respectively.
I read within the last couple of days that SpaceX is planning their next major test of their Crew Dragon craft, the in-flight abort test, “no earlier than” January 11th.  SpaceX flew their equivalent of today's Boeing mission last March.

This morning's Boeing Starliner Atlas 5 mission, Trevor Mahlmann photo.  This appears to be across the Indian River lagoon, so probably from Titusville, looking southeast toward Cape Canaveral.


  1. I received several "Watch This..." emails from Boeing over the last week. I was going to watch it live, and saw what time it was scheduled to launch.

    0430? Sorry, I'll watch the replay!

    Not sure what to make of the IFA, but it's good they know what problem was. Root Cause Analysis and corrective action might take a bit....

    1. If I was in your location, I sure wouldn't get up at 0430 to watch it online! Getting up at just about 0630 is way better than 0430 but I only got up at all because going outside to watch it until the rumble gets here is way better than watching online.

      Not that I haven't gotten up in the middle of the night (MoTFN) to go outside and watch a launch. The winter is best for those, when the neighbors' trees have lost their leaves and I can see the vehicle lower in the sky.

  2. I am not at all surprised by this failure. I think what is behind it is endemic in our legacy government aerospace contractors. A former colleague of mine at the major aerospace firm that we worked at had a good analysis. The legacy companies have an ingrained attitude in their corporate culture from the days of cost plus contracting. This attitude is one of squeezing more money from the contracting agency by not meeting the quality needed to perform and therefore slipping schedule and demanding more budget. This sorta fits right into it. I am in a way sorry for Boeing as they are taking it in the shorts from more that one direction (i.e. the 737 Max problem) but I will say they brought this on themselves by not doing what it takes to get it right in the first place.

    1. Maybe, like the 737MAX issue, they outsourced the programming to an Indian low bidder...

    2. To BillB, I have to say the "cost plus" concept seems to be sunk into that world - I've worked in that world and seen the mindset myself, even when the job wasn't cost plus.

      When you consider that NASA has paid SpaceX 60% of what they've paid Boeing and is arguably ahead of them - closer to manned flight, I'd say SpaceX is the better deal for us taxpayers.

      Divemedic - the missing detail there is that Boeing gets told they can't sell into India (or China, or ...) unless a mandated percentage of jobs during design get outsourced into that country. It's not Indian coders that are responsible, it's Boeing's because they do software quality assurance and approve it before they accept it. Boeing told them what to write and verified it was written as they ordered. The software did everything it was supposed to do, the software was bad because they gave the software writers the wrong requirements.

      The fault is entirely in Boeing's lap.

    3. SiG what is really disheartening with these failures is that Boeing has gone down hill. Ten years ago when I took my Masters Degree in Systems Engineering, one of the adjunct professors held out Boeing as a great example with their 787 program. They were supposedly at a 6+ Sigma level on the operation and interfacing of the avionics that went with that aircraft. This was because of very good requirements definition and superior Interface Control Documents (or Descriptions according to some)(ICDs). I speak from experience of being caught in poor requirements definition and crappy ICDs.

    4. Understand. I worked on the 787, several radios on that aircraft have RF sections I designed, and I think the up front systems engineering work that was done on ICDs really saved time and made things better.

      Just a guess, but I'm thinking that the 787 got all the best graybeards in the company on the team while the 737Max got the kids. "It's just minor changes. Let the kids learn." It's a truism that the big, clean sheet of paper design jobs are best for the most experienced people and the "minor redesign" jobs get handed off to the newbies. Those jobs are good for the newbies, but the oversight and mentoring process fell apart.

      A comment on Ars pointed out that this very capsule was tested in a parachute landing and someone left out a pin that held one of the chutes in. It landed on two chutes instead of three. Boeing seems to have some QA problems - telling them what to inspect for and getting it right.

  3. 01 It's been said, at least they didn't kill anybody this time.

    10 Those 555s can be tricky.

    1. 10 Those 555s can be tricky.

      Thanks. That got an actual LOL.

  4. If they had made their engineers play Kerbal Space Program, this error would have been spotted before launch. I know, I have blown up a few ships through improper sequencing.

  5. I thought that the 555 had been withdrawn from sale a while back ....

    This makes me trust Boeing *less* - it's the same failure as in the 737 MAX. The software relies on a single input, and fails to check that input against easily available alternate inputs. As you said, at least they didn't kill anyone this time but jeez Louise this looks like an institutional problem in their software design.

    It's enough to make me reconsider flying Airbus.

    1. The 555 datasheet is still on pdf format and it has links to order them. It's one of the rare early parts that's well past voting age.

    2. Past voting age? I used them in college EE Lab, back in 1980 or so. :-/