Apparently NASA Forgot How to Work With Liquid Hydrogen

The persistent issues that have been arising with trying to fuel Artemis I for its maiden flight are drawing attention to an awkward conclusion.  Either the contractors for Artemis and whoever is running Mission Control forgot how to work with liquid hydrogen or everyone who knew how to do it has retired or otherwise left the business.  There's a simple root cause, though, embedded in the joke that the SLS is "the Shuttles' Leftover Shit."  The Shuttle program had trouble with liquid H₂ as well and since the SLS program is reusing the Space Shuttle Main Engines, they have to use Hydrogen and Oxygen just like the Shuttles did.

Over its lifetime, due to this complexity, the shuttle on average scrubbed nearly once every launch attempt. Some shuttle flights scrubbed as many as five times before finally lifting off. For launch controllers, it never really got a whole lot easier to manage the space shuttle's complex fueling process, and hydrogen was frequently a culprit.

There's a famous quote attributed to SF author Harlan Ellison that the two most common elements in the universe are hydrogen and stupidity.  Here we appear to be seeing both.  Hydrogen is the lightest and smallest element, with the most common form being simply one proton with one 1S electron orbiting it.  Molecular hydrogen is two of these atoms bonded together (it's called H₂ after all) with a cloud of both electrons enveloping both protons.  Because they're so tiny, hydrogen molecules can squeeze through the smallest of gaps. That's not a big deal at ambient temperatures and pressures, but in the super-cold and high pressure conditions of liquid hydrogen fuel, the molecules easily ooze through any available opening. 

The hardware that fuels any rocket is complex.  The pipes have to be very cold to keep too much liquid fuel from evaporating, and in the case of liquid H₂ that means less than -425 F. The pipes and joints shrinking as they chill and expanding as they warm add complexity, but the most complex part is that the vehicle must be refilled - "topped off" - continuously and the "quick disconnect" (QD) lines must disconnect within tenths of a second of ignition.  The temperature of the pipes compounds the difficulty.  One of the things you'll read about the Artemis issues over the weekend is that controllers would let the QD lines warm up and hope they'd naturally settle into a position that sealed better.  

Oh, yeah.  There's another problem.  Hydrogen is explosive; I mean that's why they use it as rocket fuel, right?  

NASA, therefore, has a tolerance for a small amount of hydrogen leakage. Anything above a 4 percent concentration of hydrogen in the purge area near the quick disconnect, however, is considered a flammability hazard. "We were seeing in excess of that by two or three times that," said Mike Sarafin, NASA's Artemis I mission manager, of Saturday's hydrogen leak. "It was pretty clear we weren’t going to be able to work our way through it. Every time we saw a leak, it pretty quickly exceeded our flammability limits."

And one story that makes my head hurt:

NASA officials are still assessing the cause of the leak, but they believe it may have been due to an errant valve being opened. This occurred during the process of chilling down the rocket prior to loading liquid hydrogen. Amid a sequence of about a dozen commands being sent to the rocket, a command was sent to a wrong valve to open. This was rectified within 3 or 4 seconds, Sarafin said. However, during this time, the hydrogen line that would develop a problematic quick-disconnect was briefly over-pressurized.

Here, we circle back to a reasonable question.  If hydrogen is so hard to work with and hard to handle compared to kerosene (RP-1) or methane, why use it?  Back to the first paragraph: when congress allocated funds to start the program they mandated the use of the Shuttle hardware.  At this point, it might add some perspective to consider much of the Shuttle hardware was designed around 45 years ago.  Asked about whether using it was good idea, NASA Administrator Bill Nelson, who was one of the architects of the SLS as a jobs program said, "We deferred to the experts."  That is, those with an interest in saying to do it that way.

Among the idea's opponents was Lori Garver, who served as NASA's deputy administrator at the time. She said the decision to use space shuttle components for the agency's next-generation rocket seemed like a terrible idea, given the challenges of working with hydrogen demonstrated over the previous three decades.

"They took finicky, expensive programs that couldn't fly very often, stacked them together differently, and said now, all of a sudden, it's going to be cheap and easy," she told Ars in August. "Yeah, we've flown them before, but they've proven to be problematic and challenging. This is one of the things that boggled my mind. What about it was going to change? I attribute it to this sort of group think, the contractors and the self-licking ice cream cone."

By now, I believe most folks know that the Artemis stack has to be rolled back to the VAB if for no other reason than to replace the batteries in the Flight Termination System (FTS).  Those are certified for 25 days which is expired now, and the system is apparently not accessible on the pad.  Today's update to the Artemis blog says:

After standing down on the Artemis I launch attempt Saturday, Sept. 3 due to a hydrogen leak, teams have decided to replace the seal on an interface, called the quick disconnect, between the liquid hydrogen fuel feed line on the mobile launcher and the Space Launch System (SLS) rocket while at the launch pad. 

This will require that an enclosure be built around the area they'll be working.  The enclosure will protect the hardware from the weather, but enable engineers to troubleshoot and test the system under the actual cryogenic conditions.  It's possible that while they'll be able to test under the most realistic conditions that some repairs or rework will need to be done in the VAB.

It's hard not to wonder if they're approaching some sort of limits on the amount of testing the vehicle can withstand.  Last Saturday's countdown - cancelled when the hydrogen tank was 11% filled - was the sixth time they've attempted to fuel this launch vehicle; it's tempting to call these tests a WDR (wet dress rehearsal) but I don't think they really accomplished enough to deserve that name.   

NASA's Space Launch System rocket at LC-39B last Thursday: September 1, 2022.  Trevor Mahlmann photo.

How much do you want to bet that the seventh time will be the charm?