Friday, October 30, 2020

NASA's Crew-1 Mission on SpaceX Dragon Looks Rescheduled

The first operational crewed mission to the ISS on the SpaceX Crew Dragon had originally been scheduled for mid-October, than backed off until tomorrow, Halloween day.   Then there was a last seconds abort on another Falcon 9 mission, this one to carry a GPS-III satellite for the Space Force on October 2nd.  In an abundance of caution, SpaceX and NASA scrubbed tomorrow's launch, called Crew-1, while failure analysis took place.  

Any current or former technicians are going to love this story, courtesy of Teslarati.  They're calling it the nail polish delay. 

First some needed background.  The booster that auto-aborted two seconds before liftoff on October 2nd was a new Falcon 9 booster, which means it was built virtually side by side with Crew-1’s own new booster.  The rare last-second abort was quickly blamed on “unexpected pressure rise in the turbomachinery gas generator” by CEO Elon Musk.

Within 24 hours technicians had carefully gone over the engine and were unable to find any clues to the behavior.  But since the suspicious boosters were never-flown and not reused boosters, it led NASA and SpaceX to reschedule the Crew-1 for No Earlier Than (NET) 7:49 pm EST on Saturday, November 14th.  I'll note this adds to the argument that the cheaper, already-flown, boosters are actually safer because they've been tested more extensively. 
Now, during NASA’s October 28th Crew-1 briefing, SpaceX’s (Hans) Koenigsmann revealed that the company had ultimately decided to replace not one but two of Crew-1 booster B1061’s nine Merlin 1D engines. Thanks to Falcon 9’s namesake nine-engine booster design and SpaceX’s prolific rocket factory, that process was completed extraordinarily quickly, simply requiring the redirection of already qualified Merlin 1D engines from a fairly large pool. Based on Koenigsmann’s phrasing, SpaceX has already installed both replacement engines on the Crew-1 booster.
... in the course of the rapid and complex mechanical and electrical ballet preceding Falcon 9 first stage ignition, the rocket’s autonomous flight computer observed that two of the GPS III SV04 booster’s nine Merlin 1D engines appeared to be running ahead of schedule, so to speak. The computer immediately halted the ignition process to avoid what could have otherwise been a “hard” (i.e. stressful or damaging) start. SpaceX quickly began inspecting the rocket within 24 hours but was unable to detect anything physically or electrically wrong with Falcon 9’s Merlin 1D engines and engine section.

Out of an abundance of caution, SpaceX removed both misbehaving engines and shipped them to its McGregor, Texas development and test facilities where – somewhat miraculously – the same premature startup behavior was replicated on the test stand. After a great deal of increasingly granular inspections, SpaceX finally narrowed the likely cause down to a tiny plumbing line feeding one of the engine’s gas generator relief valves. In a seemingly random subset of relatively new Merlin 1D engines, SpaceX eventually discovered that a supplier-provided relief valve line was sometimes clogged by a protective lacquer Koenigsmann likened to “red nail polish.”

Used to selectively exclude parts of the engine tubing during a surface finishing process known as anodization, the lacquer was either unsuccessfully removed on a random selection of engine parts or was accidentally channeled into a blockage by over-enthusiastic cleaning. Ultimately, for whatever, reason that miniscule blockage was enough to cause affected Merlin 1D engines to consistently attempt to ignite a tiny fraction of a second early.

Crucially, when SpaceX discovered the possible cause and cleaned out the blocked plumbing, each previously affected Merlin 1D engine performed perfectly, all but directly confirming both the cause and the cure for Falcon 9’s October 2nd abort.
The only more "open and shut" way of proving a failure is put the bad part back in place and verify the problem happens again, but that may be impossible here.  The troubleshooting they've done seems to demonstrate that the lacquer used to coat the tubing was what caused the engine to misbehave.  Exactly how it could have sped up the process of starting is obscure to me, but since they say, "accidentally channeled into a blockage," maybe it's like putting your thumb over the garden hose to get it to spray harder. 

When I first read this, I thought of the lacquer like Glyptal, a compound often used to seal electronic circuits to keep adjustments from moving. On second thought, it seems more like masking tape that's masking off areas to keep them from getting anodized.  Nail polish is probably a better analogy.

A Merlin1D engine on the test stand at McGregor, Texas.  Probably a stock photo.  From Teslarati.

It's probably brash to say that Crew-1 is back on the schedule, but NASA has quarantined the crew for Crew-1, ordinarily a 14 day quarantine. The GPS-III satellite launch is currently set for Wednesday the 4th at 6:24 PM EST and they certainly won't reschedule Crew-1 if that doesn't go nominally.  That date is at least partially dependent on the NROL launch on Election Day at 5:58 PM EST because if they don't launch but delay 24 hours, they get priority over any other mission from the Cape. The NROL Delta IV scrubs have delayed SpaceX several times since June.


  1. You're correct on the comparison of Glyptal and the "varnish" they used on the tubing. The "varnish" that I've seen used to do that is translucent, more like a tinted clear, than a "solid" color. It's kinda like "Candy Apple Red". Glyptal is pretty thick, or at least the stuff I used was, although you could also get it in rattle cans. It was originally developed for insulating electric motor windings, and it's pretty tough stuff. The automotive guys use it to seal porosity on aluminum castings, and to aid oil drainback.

    1. Yeah, the real stuff is thick. I've used it for sealing adjustments after alignment where you only put one drop down. Glyptal is like Band-Aid where a trademarked name becomes the common name. Good products are like that.

    2. I first ran across it at Western Electric, where we used it as the "Factory Seal" on trimpots and such. Then again at McGraw-Edison, where I saw it used on big electric motors. Sometimes motors would get "leaky" due to dirt/grease/other on the stator windings, so they'd take the end bells off and the rotor out, and dip the stator in some kind of solvent bath, bake it out in a big oven, and then apply Glyptal. Put it back together with new bearings, and you have an economy class "Motor Rebuild".

      Called it a "Dip and Glyp"!

  2. Causation is hard to prove. The actual cause could always be the Bevis and Butthead of gray aliens parked on the back side of the moon messing with you.

  3. As a trouble shooting fool, I like this. For the Halloween special, WWND-What Would Nasa Do?

  4. Funny, I remember using Glyptal for locking in adjustable caps. It was the first thing I thought of, too. Took a few minutes for the neurons to fire. But Glyptal is not a lacquer, so probably not it.