Thursday, May 2, 2019

Small Details Emerge in SpaceX Anomaly

In the week since I first wrote about the SpaceX Crew Dragon "Anomaly" (aka - "cratering explosion on the test stand"), some small details have started to dribble out of the investigation.

Ars Technica reports that the vehicle exploded with less than one second left in the countdown to engine firing.
In the company's most expansive comments to date, [Hans] Koenigsmann said the "anomaly" occurred during a series of tests with the spacecraft, approximately one-half second before the firing of the SuperDraco thrusters. At that point, he said, "There was an anomaly and the vehicle was destroyed."
More importantly, to my way of thinking, they confirmed this was indeed the capsule that flew to the Space Station in early March.  That was reported, but not confirmed (that I know of) until this statement.
The lost spacecraft is the same one that successfully flew a demonstration mission to the International Space Station in March. (During that Demo-1 mission, the SuperDracos were not activated.) After the March flight, the Dragon spacecraft was being prepared for a launch abort test this summer.
Why is this important?  The capsule went through an entire flight sequence, which includes getting dunked in saltwater when the capsule splashes into the Atlantic (in this case).  Getting dunked raises the possibility of saltwater intrusion into portions of the controls, the wiring, or the engines themselves, which in turn raises the possibility of galvanic corrosion (dissimilar metals in a conductive electrolyte) or some other effect of saltwater.

There has been a lot of attention paid to the SuperDraco engines themselves as the source of failure.  That's reasonable when you think of their job as being to control a long-duration explosion and use it for an intended purpose, but it could well be leaping to a conclusion.
It is too early to determine a probable, or root, cause, but Koenigsmann expressed confidence in the SuperDraco thruster system. He noted that SpaceX has tested these powerful thrusters more than 600 times at its test facilities in McGregor, Texas. Moreover, they have performed well in hover tests as well as a launch pad abort test in 2015. "We have no reason to believe there’s an issue with the SuperDracos themselves," he said.
The Wikipedia article on the engines reports that SpaceX first developed the "Mark 1" engine over a decade ago, with the first engine being fired in June of 2012.  By the end of that year, the ground-test engines had been fired a total of 58 times for a total firing-time duration of 117 seconds

A second version of the engine was developed in 2013, this one manufactured with 3D printing rather than the traditional casting technique.  By July 2014, the 3D-printed engine combustion chamber had been fired over 80 times, for a total duration of more than 300 s, and it likewise completed a full qualification test.

This SpaceX montage of the 3D printed SuperDraco engines being test fired appears in the Ars Technica article.  It says each engine is tested 300 times.  There are four of these SuperDraco pairs (8 engines) on the Crew Dragon for the escape system.

Since this was a test firing on a dedicated test stand, their advantage in the failure analysis was that everything they could think to measure was measured.  The investigators are swimming in data, trying to understand.  The investigation is a joint SpaceX/NASA investigation, and Alabama senator Richard Shelby (a vocal backer of NASA's Space Launch System, and with it, Boeing) is voicing complaints/concern over that.
One of the sharpest critics of SpaceX, Alabama Senator Richard Shelby, said Wednesday that NASA should conduct an independent investigation of the accident. He appeared unhappy with NASA's decision to conduct a side-by-side investigation with SpaceX. "Can you be independent, and reach independent conclusions, if you're doing something jointly with somebody?" he asked during a Senate committee hearing. "That's not the norm, I think, and it's something we'll check out."
The goal for the Crew Dragon after this test was to be a live test of the abort system by launching a Falcon 9 with the Crew capsule in position and then firing the abort system, directly simulating a launch abort.  The tentative schedule was for that test to occur in the summer, with the first manned flight to the ISS possible by early October.  This failure has rocked that schedule, and without a definite root cause for the explosion issued there's no replacement schedule.  I think all these investigators know it's more important to get it right than just get it done.

A 2015 pad abort test similar to the one that failed 12 days ago.  The 8 SuperDraco engines can be seen lifting the capsule off the pad.


  1. Interesting. Since it happened so close to the planned firing, I wonder if it was a sequencing problem allowing the fuels to mix where they shouldn't.

  2. According to the (trusted) resource Wikipedia, the Super Draco uses a hypergolic mix. which yes, could have combusted in the chamber if released too early, but it should ignite and exit the nozzle. Mixing somewhere other than the combustion chamber would be Very Bad.
    But more interesting is the group complaining about using SpaceX services - a Senator with ties to NASA/SLS and by association -wait for it- Boeing.
    What? The same Boeing that doesn't have an issue self-certifying the flight control system for the 737 Max? Naw, couldn't be the same...
    Wandering Neurons

    1. My initial reaction to Senator Shelby was, "why would NASA be involved at all?" After all, SpaceX is a private company and they lost their own money here. The alternative argument, of course, is that they are under contract to NASA and it might be in the contract that NASA is in charge or has oversight if something like this happens.

      From my standpoint as an engineer I think I'd welcome their help. As an organization, they've investigated far more "anomalies" than I/we have. Whether or not there's really institutional knowledge at NASA about such things, or how relevant that is, I can't say, but it could be.

      Shelby's real motivation is apparently that he wants the SLS to "win" and the jobs to remain in his state. What we used to call pork barrel politics.

  3. Thanks for the update. As much as I don't like Elon, he is doing something amazing, putting a greater emphasis on non-government into space flight.

    Senator Shelby just can't let go of the Government having a monopoly on space flight. Boeing wouldn't be participating if it wasn't for Government contracts.

  4. I'm thinking that something cracked and allowed one of the fuels to leak into a large space, and then the other fuel line (or valve, or ...) cracked and mixed with it to produce the explosion.

    Remember that with hypergolic fuels, it takes two to tango. Given the very small probability that two separate structures would have failed at nearly the same time, I'm also guessing that whatever broke would have been the same source of failure on both sides of the system.

    I really hope they have enough sensors to find the root cause without too much ambiguity, or they may be forced to break quite a few of them to find it.

    1. FWIW, my guess is that there's a long, intricate checklist of things that need to be done before the test, and somebody screwed up something.

      I know that's not a specific answer, but I'm looking at He noted that SpaceX has tested these powerful thrusters more than 600 times... and thinking the hardware or low level software is probably not the problem. It's probably the proverbial "nut behind the steering wheel" as they say about cars - in this case, some tech working down a checklist got something wrong: skipped a step, did some steps backwards, something like that.

      The wildcard is something like saltwater getting into the engine while it was in the ocean and causing some sort of unseen damage. I'd expect that they tested the engines with a MIL standard saltwater test, but don't know that.

      This is manned spaceflight hardware and everyone knows the vehicle doesn't fly until the paperwork outweighs the launch vehicle.

    2. Tell me if this is reasonable: Two things do not have to fail, only one to fail to cause a leak which then comes into contact with the controlled burn. The result is an advancement of a flame front to areas where it is Very Bad for something to burn (explode).


    3. Rick - I think it's reasonable it could be a single point failure, like you say. I just don't know if that's consistent with the explosion being before the engine started.

      I don't want to speculate because I don't know enough about how the systems work for it to be meaningful.

    4. Si, thanks for responding. How corrosive or electrically conductive are the reagents used in the fuel?


    5. The fuel is Mono Methyl Hydrazine (MMH) and the oxidizer is Nitrogen Tetroxide (NTO). The combination explodes on contact (hypergolic fuel system).

      The first answer is they are extremely corrosive (and extremely toxic).

      Strangely, the MSDS listings on Wikipedia don't mention conductivity directly.

      My assumption is that if it's bad one of both of these two will do it.

  5. The Senator who is the critic, has zero real experience and has been a professional senator since 1987. His college education was as a lawyer.
    So great, we got a lawyer telling people how to build rockets. HAHHAHAHA