NASA's Crew-1 Mission on SpaceX Dragon Looks Rescheduled

The first operational crewed mission to the ISS on the SpaceX Crew Dragon had originally been scheduled for mid-October, than backed off until tomorrow, Halloween day.   Then there was a last seconds abort on another Falcon 9 mission, this one to carry a GPS-III satellite for the Space Force on October 2nd.  In an abundance of caution, SpaceX and NASA scrubbed tomorrow's launch, called Crew-1, while failure analysis took place.  

Any current or former technicians are going to love this story, courtesy of Teslarati.  They're calling it the nail polish delay. 

First some needed background.  The booster that auto-aborted two seconds before liftoff on October 2nd was a new Falcon 9 booster, which means it was built virtually side by side with Crew-1’s own new booster.  The rare last-second abort was quickly blamed on “unexpected pressure rise in the turbomachinery gas generator” by CEO Elon Musk.

Within 24 hours technicians had carefully gone over the engine and were unable to find any clues to the behavior.  But since the suspicious boosters were never-flown and not reused boosters, it led NASA and SpaceX to reschedule the Crew-1 for No Earlier Than (NET) 7:49 pm EST on Saturday, November 14th.  I'll note this adds to the argument that the cheaper, already-flown, boosters are actually safer because they've been tested more extensively. 

Now, during NASA’s October 28th Crew-1 briefing, SpaceX’s (Hans) Koenigsmann revealed that the company had ultimately decided to replace not one but two of Crew-1 booster B1061’s nine Merlin 1D engines. Thanks to Falcon 9’s namesake nine-engine booster design and SpaceX’s prolific rocket factory, that process was completed extraordinarily quickly, simply requiring the redirection of already qualified Merlin 1D engines from a fairly large pool. Based on Koenigsmann’s phrasing, SpaceX has already installed both replacement engines on the Crew-1 booster.
...
... in the course of the rapid and complex mechanical and electrical ballet preceding Falcon 9 first stage ignition, the rocket’s autonomous flight computer observed that two of the GPS III SV04 booster’s nine Merlin 1D engines appeared to be running ahead of schedule, so to speak. The computer immediately halted the ignition process to avoid what could have otherwise been a “hard” (i.e. stressful or damaging) start. SpaceX quickly began inspecting the rocket within 24 hours but was unable to detect anything physically or electrically wrong with Falcon 9’s Merlin 1D engines and engine section.

Out of an abundance of caution, SpaceX removed both misbehaving engines and shipped them to its McGregor, Texas development and test facilities where – somewhat miraculously – the same premature startup behavior was replicated on the test stand. After a great deal of increasingly granular inspections, SpaceX finally narrowed the likely cause down to a tiny plumbing line feeding one of the engine’s gas generator relief valves. In a seemingly random subset of relatively new Merlin 1D engines, SpaceX eventually discovered that a supplier-provided relief valve line was sometimes clogged by a protective lacquer Koenigsmann likened to “red nail polish.”

Used to selectively exclude parts of the engine tubing during a surface finishing process known as anodization, the lacquer was either unsuccessfully removed on a random selection of engine parts or was accidentally channeled into a blockage by over-enthusiastic cleaning. Ultimately, for whatever, reason that miniscule blockage was enough to cause affected Merlin 1D engines to consistently attempt to ignite a tiny fraction of a second early.

Crucially, when SpaceX discovered the possible cause and cleaned out the blocked plumbing, each previously affected Merlin 1D engine performed perfectly, all but directly confirming both the cause and the cure for Falcon 9’s October 2nd abort.

The only more "open and shut" way of proving a failure is put the bad part back in place and verify the problem happens again, but that may be impossible here.  The troubleshooting they've done seems to demonstrate that the lacquer used to coat the tubing was what caused the engine to misbehave.  Exactly how it could have sped up the process of starting is obscure to me, but since they say, "accidentally channeled into a blockage," maybe it's like putting your thumb over the garden hose to get it to spray harder. 

When I first read this, I thought of the lacquer like Glyptal, a compound often used to seal electronic circuits to keep adjustments from moving. On second thought, it seems more like masking tape that's masking off areas to keep them from getting anodized.  Nail polish is probably a better analogy.

A Merlin1D engine on the test stand at McGregor, Texas.  Probably a stock photo.  From Teslarati.

It's probably brash to say that Crew-1 is back on the schedule, but NASA has quarantined the crew for Crew-1, ordinarily a 14 day quarantine. The GPS-III satellite launch is currently set for Wednesday the 4th at 6:24 PM EST and they certainly won't reschedule Crew-1 if that doesn't go nominally.  That date is at least partially dependent on the NROL launch on Election Day at 5:58 PM EST because if they don't launch but delay 24 hours, they get priority over any other mission from the Cape. The NROL Delta IV scrubs have delayed SpaceX several times since June.