Friday, April 24, 2020

Musk Posts Explanation of March Engine Failure

On Wednesday, before the successful SpaceX Starlink mission, Elon Musk replied to a question on Twitter about what caused the engine failure in the previous Starlink mission, the one that caused the loss of a booster on its fifth flight back on March 8th.

When another Twitter user asked what a "dead leg" is, a third user, David Urry, posted this diagram to explain it, after saying:
My best guess on what the heck Elon's talking about:
  1. Must wash out engine with alcohol between uses.
  2. Gravity must have defeated getting it all cleared out.
  3. Isopropyl would be ice in liquid O2, must be cause of pressure reading error day before. 

According to this week's Rocket Report on Ars Technica:
A source told Ars that the company has already replicated the problem during tests, and that fixing it will require changing some cleaning procedures. This should have no effect on the upcoming Crew Dragon launch.
If you haven't heard the results, Wednesday's Starlink mission was a very successful mission for SpaceX.  First off, all of the Starlink satellites were delivered to orbit as planned.  Next, the booster nailed its landing on the drone ship Of Course I Still Love You.  Third, both fairing halves were recovered; one caught in the net and the other retrieved from the ocean.  These were already re-used fairing halves, opening the possibility of a third flight for the fairings, which will be a first.  Finally, the mission put the Falcon 9 booster ahead of the Atlas V as the most experienced rocket in the US.
The Falcon 9 rocket has now launched 84 times. This surpasses the total flights by United Launch Alliance's Atlas V rocket.
The Atlas V rocket first launched on August 2, 2002—about three months after SpaceX was founded by Elon Musk and two other engineers, Tom Mueller and Chris Thompson. Since then, the Atlas V rocket has flown an average of a little fewer than five missions per year. All were rated as successes.

SpaceX first flew the Falcon 9 rocket on June 4, 2010, from Space Launch Complex 40 in Florida. It has caught up to the Atlas V rocket by flying an increasing cadence of missions from 2017 onward, averaging 17 flights a year over the last three full years. One of SpaceX's launches, CRS-7 in 2015, failed.
Perhaps more impressively, the Falcon 9 has assumed the mantle of most-experienced rocket while making multiple revisions to its design and incorporating first-stage reuse. In short, SpaceX increased the flight rate of its rocket even as it has aggressively sought to optimize its performance. The company that was once an upstart now stands among—if not above—the blue bloods of the US launch industry.
It's hard to find fault with that last statement from Eric Berger at ARS.

While talking about SpaceX, I should note that serial number 4 of the Starship prototypes was rolled to the test stand at Boca Chica, Texas today.  It's exactly 31 days since they started building SN4 and hours short of 21 days since SN3 failed during it's testing on the night of April 3rd. 

Drone's eye view of SN4 on the pad today.  I prefer this for the sense of scale you get looking down on the cars and trucks as opposed to looking up at the booster from the ground.  If I'm remembering correctly, these Starship prototypes have all been about 100 feet tall and more than 10 feet in diameter.  Photo credit to Elon Musk on Teslarati.


  1. Makes sense to use Iso to clean the system, and also makes sense that a small amount, not drained or dried out, would freeze and cause problems.

    I bet the fix to this issue is to run purified hot air through the system for a reasonably long time. Maybe even hook the engine into a rotating frame.

    Can you imagine Boeing being able to give a 2 month response time on any fault, let alone openly stating the issue rather than burying it deep in some report?

    Interesting news on SN4. Will be interesting to see what goes wrong this time.

  2. Great write-up, thanks!
    Every day of this march, Heinlein becomes more and more prescient: business is going to open space.

    Government, via NASA, from 1970-present, has been the problem, not the solution.

    We could've had our flying cars and jet packs, and been living in artificial G in orbit decades ago, dammit!

  3. Sensor dead leg. In instrumentation piping there is often multiple piping connections and maifold valves at the sensor input which may have been a static pressure, or differential pressure sensor which converts the physical process pressure to an electrical signal for use in the vehicle control system. The valves act to 1. isolate the sensor from the process, 2. open the sensor pipe to atmosphere to blow down the sensing line, and 3. to enable calibration of the sensor. The “dead leg” in this case could have been a capped off calibration line or separate blow down line. These lines are typically capped at the dead ends to prevent foreign objects from entering what would otherwise be an isolated open pipe end to atmosphere. The “dead” ends are these capped calibration or blow down lines. Dead meaning not containing process pressures.
    A blow down is when the sensing pipe is opened to atmosphere thru a blow down valve that blows any foreign materials that may have collected in the sensing line out to atmosphere. I use the terms “line” and “pipe” interchangeably.

    A Process and Instrumentation Diagram P&ID would detail such a sensors piping and valving

    We might reliably assume that the process sensor was not in an oxygen containing system as flourine containing cleaners are typically the standard in cleaning oxygen systems. For example HCFC-225

    However NASA does list isopropyl alcohol as a “test solvent “ for use in cleaning oxygen systems in the following manual. A test solvent is not a cleaning agent.


    My knowledge and experience is in industrial process controls not aerospace systems. However they share many processes and instrumentation techniques

    I think there is more to this mishap than Musk is revealing.

    1. Excellent write up, so thanks!

      My knowledge and experience is in industrial process controls not aerospace systems. However they share many processes and instrumentation techniques

      Oh, for sure. They had to start somewhere and you just have to know that the people who started knew the industrial processes and adapted some for the new environment.

  4. More than 10' in diameter? I should say so! They are 9 meters in diameter. You could fit a small McDonald's franchise inside one.

  5. HCFC-225

    My understanding is the ban on freons happened because the patent ran out for DuPont, and they needed to invent some new reason to ban free enterprise. My understanding is the ozone layer damage turned out to be natural and seasonal, and freon wasn't contributing.

    1. My understanding is the ban on freons happened because the patent ran out for DuPont, and they needed to invent some new reason to ban free enterprise.

      My take on it (from a documentary in the dim, distant past) was the patent ran out and DuPont needed a new refrigerant under patent to maintain their monopoly. They didn't start the ozone hole nonsense but happily pushed it and offered their new refrigerant that wasn't going to damage the ozone.

      It's like how one of the insulin formulations has always stayed under patent by constant, small tweaks to the formula. Keeps the price up and competitors away.

    2. I'm glad you agree the reasons for the ban on CFCs are all economically, scientifically, and morally bogus. Now imagine if an American was inspired to make and sell freon, and then defend himself from the depredations of the commies by shooting the enforcers and their leaders. I imagine most Americans would only approve of going so far as civil disobedience, to be a celebrity legal test case and attempt to get a different result from democratic majority rule. Most Americans are commies, and will cease work and go bankrupt when ordered.

  6. If anyone is interested in SN4's progress, keep a watch on the Lab Padre You Tube channel. They stream live video from Boca Chica.

    I think there is a road closure tonight on Texas State Highway 4 that runs out to Boca Chica Village/SpaceX. They may start pressure testing. Currently it appears that they are just working on SN4.

    1. Looking like they might do the cryo pressure test tonight. I'll keep this running in the background.

  7. This comment has been removed by a blog administrator.