Wednesday, January 31, 2018

NASA's (and Our) Worst Week in Spaceflight

We're almost at the end of the worst week of NASA history.  It's a peculiar coincidence that every accident that took the lives of the crew and destroyed the vehicle took place in the space of one calendar week, although those accidents are separated by decades.

January 27th, was the 51st anniversary of 1967's hellish demise of Apollo 1 and her crew, Gus Grissom, Roger Chaffee and Ed White, during a pad test, not a flight.  In that article, Ars Technica interviews key men associated with the mission and provides, for the first time I've seen, the audio of the test.  In the early days of the space program, one of the larger than life names we all came to recognize was Chris Kraft, the Capsule Communicator or CapCom who had directed all of the Mercury flights, many of the Gemini missions and was widely recognized for this masterful control.
Half a century later, the painful memories remain. “I was on console the day it burned,” he explained, sitting in his second-floor den, just a few miles from the control center that now bears his name at Johnson Space Center.

“I heard their screaming voices in the cockpit of the spacecraft,” Kraft recounted. “I heard them scream that they were on fire. I heard them scream get me out of here. And then there was dead silence on the pad. Within minutes we knew they were dead, and we were in deep, serious trouble. Nobody really said anything for 15 minutes, until they got the hatch open. We were sitting there, waiting for them to say what we knew they were going to say.”
....
There was plenty of blame to go around—for North American, for flight control in Houston, for technicians at Cape Canaveral, for Washington DC and its political pressure on the schedule and its increasingly bureaucratic approach to spaceflight. The reality is that the spacecraft was not flyable. It had too many faults. Had the Apollo 1 fire not occurred, it’s likely that additional problems would have delayed the launch.

“Unless the fire had happened, I think it’s very doubtful that we would have ever landed on the Moon,” Kraft said. “And I know damned well we wouldn’t have gotten there during the 1960s. There were just too many things wrong. Too many management problems, too many people problems, and too many hardware problems across the whole program.”
The ARS article is worth your time. 

The next day, January 28, is the anniversary of the Space Shuttle Challenger disaster.  Shuttle Challenger was destroyed on January 28, 1986, a mere 73 seconds into mission 51-L as a flaw in the starboard solid rocket booster allowed a secondary flame to burn through supports and cause the external tank to explode.  It was the kind of cold day that we haven't had here in some years.  It has been reported that it was between 20 and 26 around the area on the morning of the launch and ice had been reported on the launch tower as well as the external tank.  O-rings that were used to seal the segments of the stackable solid rocket boosters were too cold to seal.  Launch wasn't until nearly noon and it had warmed somewhat, but the shuttle had never been launched at temperatures below 40 before that mission.  Richard Feynman famously demonstrated that cold was likely the cause during the televised Rogers Commission meetings, dropping a section of O ring compressed by a C-clamp into his iced water to demonstrate that it had lost its resilience at that temperature.


There's plenty of evidence that the crew of Challenger survived the explosion.  The crew cabin was specifically designed to be used as an escape pod, but after most of the design work, NASA decided to drop the other requirements to save weight.  The recovered cabin had clear evidence of activity: oxygen bottles being turned on, switches that require a few steps to activate being flipped.  It's doubtful they survived the impact with the ocean and some believe they passed out due to hypoxia before that. 

Finally, at the end of this worst week, Shuttle Columbia, the oldest surviving shuttle flying as mission STS-107, broke up on re-entry 15 years ago tomorrow, February 1, 2003 scattering wreckage over the central southern tier of the country with most debris along the Texas/Louisiana line.  As details emerged about the flight, it turns out that Columbia and everyone on board had been sentenced to death at launch - they just didn't know it.  A chunk of foam had broken off the external tank during liftoff and hit the left wing's carbon composite leading edge, punching a hole in it.  There was no way a shuttle could reenter without exposing that wing to conditions that would destroy it.  They were either going to die on reentry or sit up there and run out of food, water and air.   During reentry, hot plasma worked its way into that hole, through the structure of the wing, burning through piece after piece, sensor after sensor, until the wing tore off the shuttle and tore the vehicle apart.  Local lore on this one is that the original foam recipe was changed due to environmental regulations, causing them to switch to a foam that didn't adhere to the tank or stand up to abuse as well. 

There's film from inside Columbia until the moment the vehicle is ripped apart by the aerodynamic forces.  I suspect the forces ripped apart their bodies just as fast.  

January 27 to February 1 is 6 days.  Not quite a full week.

On a personal note, I remember them all.  I was a kid living in Miami when Apollo 1 burned.  I was living here and watched Challenger live on satellite TV at work.  Instead of going outside to watch it as I always did, I watched it on NASA Select.  Mrs. Graybeard was working on the unmanned side on the Cape, next door to the facility that refurbished the SRB's between flights, and was outside watching the launch.  It took quite a while for the shock to ease up.  I saw those spreading contrails everywhere for a long time.  Columbia happened when it was feeling routine again.  Mom had fallen and was in the hospital; we were preparing to go down to South Florida to visit and I was watching the TV waiting to hear the double sonic booms shake the house as they always did.  

When this blog started, one of my regular reads was Dr. Sanity, a flight surgeon for NASA and board-certified psychiatrist.  She was Crew Surgeon for Challenger and has left posted a haunting recollection of the mission although her blog hasn't been updated since 2012.  It's still there.  I conclude with a quote from that post:
I remember the Challenger and her crew frequently and with love. They are a part of me now. All of them represent the best within the American spirit, and always will. Since that day in 1986, I have come to see NASA as one of the greatest impediments to the Dream of space exploration; but I have never given up the Dream itself. Nor have I forgotten any of the pioneers who have died in the service of that Dream. Some day we humans will leave this small planet and joyfully play in all the corners of the cosmos.

I eagerly look forward to it.
The failure reports and investigations of all three of these disasters center on the same things: the problems with NASA's way of doing things.  Tending to rely on "well, it worked last time" when dealing with dangerous situations, or leaning too much toward, "schedule is king"; gambling that someone else would be the one blamed for delaying a mission.  Spaceflight is inherently very risky, so some risk taking is inevitable, but NASA had taken stupid risks too often.  People playing Russian Roulette can say, "well, it worked last time", but that doesn't mean playing has reduced their odds of losing. 

"I have come to see NASA as one of the greatest impediments to the Dream of space exploration; but I have never given up the Dream itself."  We're in complete agreement.


29 comments:

  1. If you don't already know, look up the reason for making the solid boosters in sections............

    ReplyDelete
    Replies
    1. I forget that story.

      Got a link or care to tell it?

      Delete
    2. Long ago, I saw a claim that the boosters were made in segments so they could be transported from the manufacturer in a certain legislative district, in return for the cooperation of that rep/senator in moving the shuttle project forward. I have so far not been able to find a reference to this claim.

      Delete
  2. My dad knew all three of Apollo 1's crew. Closest I ever saw him hit someone was when some loudmouth idiot spouted the conspiracy theory that they lived, and it was all a coverup. And from what I remember, Grissom knew the capsule was a dog and he fought to make it better, only achieving that goal by his death. Sad days of cost-cutting and big egos not listening to those in the know. And they didn't ever learn.

    Challenger, my wife and I were watching the NASA feed on satellite. Seems like things fell and hit the water for half an hour or so. The down range feed stayed on for hours, and we watched as some boats and helos made it out there after the last of the debris finished hitting. That was horrible. And so much for a ship supposedly designed to launch from Vandenberg AFB (where below 40 degree temps are not uncommon.)

    A friend who worked on the boosters said removing the CFCs from the foam insulation is what doomed that flight. (He lays the blame directly on Al Gore and Bill Clinton for that.) It wasn't as if it wasn't a known problem. There was lots of footage and evidence from previous launches of chunks of foam and ice bouncing off the orbiter.

    Gee, imagine if we had kept with the Apollo expansion program? North America had already figured out how to reuse about 70% of the capsule, mainly replacing components damaged by explosive bolts and the heat shield.

    ReplyDelete
  3. Mr. Kraft is a photogenic spokesperson, but those with worse memories of Apollo 1 were on the pad just outside the capsule. They not only HEARD, but they also SAW. And were unable to do a damn thing to help because the fire raised the pressure inside the capsule to the point the hatch could not be opened.

    For Challenger, that lays directly on Thiokol management. The STS design criteria mandated that the system, including the boosters, be able to operate in the conditions that existed on that day. While there was significant discussion about the ability of the integrated system to successfully launch, NASA management specifically noted the design requirement to Thiokol management during the prelaunch review, and asked them if their hardware met that requirement. Thiokol management said "YES." Thiokol engineers not in the briefing room were trying to get to their management to make sure they knew of the previous issues with SRB joint seals, but were unable to do so. Do remember that this was before cell phones, and the landlines to the briefing room were fully occupied by engineers from NASA and all the companies associated with the program. I do not know if those Thiokol engineers were able to get to their management AFTER the launch decision, but management may have been unwilling to change their decision for fear of the company looking bad.

    Also note that there was supposed to be a test of an SRB at the stand in Utah under conditions to verify its ability to meet the cold weather design requirement. That test was cancelled because the government had more important things to do with the money. Of course, the SSME issues were also going on at that time and eating up a bunch of money. The agency figured the SSMEs were more likely to cause significant issues than the SRBs, since solid rocket motor issues were rather well known. And finally, note that the joint leakage would not have even caused loss of mission on its own. Thrust loss from that leak would have easily been made up for by a slightly longer SSME burn. Unfortunately, the plume coming out of the joint impinged directly on the external tank and caused the lower attachment strut to separate from the tank. This allowed the SRB nose to yaw towards the tank due to the thrust vector from the engine. When the SRC nose impacted the tank, it caused structural failure of the LOx tank (located at the front of the ET) which rapidly followed by structural failure of the entire tank, separating what was left of the tank from the Orbiter and the two boosters. Aero loads at this event were sufficient to break up the Orbiter, and in the videos you can see the pressurized crew cabin breaking out of the rest of the debris. If the joint leak had NOT impinged on the tank, but had instead been almost anywhere else along the circumference of that joint, the mission would have been able to proceed as planned. Of course, with that joint being near the aft attach point, the concentrated loads did make it more likely to fail in that direction.

    And in a final note, NASA at least to use the "second set of eyes" principle in that, besides the contractor engineer and his team doing the work, there was a NASA engineer with knowledge of the system operation approving their work under the "oversight" principle. In the interest of cost saving, NASA has gone to "insight", where the contractor can do as he wishes and NASA engineers have to hope they catch any errors before they cause disaster. And of course, SpaceX does not even use "insight"...

    Sorry about the length of this screed, but some things take a lot of words to describe properly. Hopefully I haven't made too many spelling or grammar errors herein.

    ReplyDelete
    Replies
    1. No problem at all. It's fantastic to get this kind of information. I hadn't thought that if the leak had occurred elsewhere on the circumference of the SRB it might have been a minor inconvenience.

      I do remember the Thiokol involvement and I seem to recall there was an engineer who briefly got a lot of press for talking about trying to tell NASA not to launch that day.

      I heard yesterday SpaceX is going for Tuesday to launch the Falcon Heavy, and someone on the radio is expecting something like 8 million people to come into the area to see it. I would think they don't use insight because they're the customer as well as provider. They sell launch services - they get a payload to orbit for a price. I'm sure if they lose a payload they end up paying in some way (or else their insurance isn't working). Will they do the same policy for manned flights for NASA? I guess that's up to NASA.


      Delete
    2. This comment has been removed by the author.

      Delete
    3. "Insight" is indeed a NASA thing. But "oversight" could be properly done within the company. The idea is that this is indeed "rocket science", and as such it is easy to get tunnel vision on part of what you're doing and miss some effect you're having on other parts of the system. The "second set of eyes" is a technically knowledgeable individual looking at what you're proposing to do and seeing if it makes sense. Not "Is this the way I'd do it?", since there are generally many different ways to solve an engineering problem, but instead "Is this technically acceptable?" The technical person doing such a review is more likely to catch any real issues than the engineer who developed the plan to do the work, as long as the reviewer wasn't intimately involved in developing the plan in the first place. Of course, you still have to worry about "incest", where the reviewer says "Johnny is a good engineer and he doesn't make mistakes, so I don't need to look at this too closely. And besides, he might end up reviewing my work next week, and I don't want him to think I'm picking on him if I find a problem in his plan because then he might get back at me later."

      Delete
    4. Are you saying they don't do internal design reviews? That goes beyond shocking to ... whatever you call beyond shocking. Even the smallest, startup, commercial product places I worked did that.

      For readers unfamiliar, every place I know of follows some sort of design review process. They typically use terms that I believe come from the military process; terms like preliminary design review, critical design review, as well as systems, test readiness and lots of other reviews. These are set up as milestones or gates to clear during the design process. A group of engineers that haven't worked on the program get together to effectively say, "did you think of X?" or more importantly "I did that once and we had trouble with Y". Independent oversight inside the company.

      I worked in a defense and space contractor for around 15 years, so I've been through the design review series on jobs for both manned and unmanned space programs, as well as Air Force, Army and others. While it's a good idea, the dominant lesson I got out of that is that nobody cares as much that my design should work as I do. They're not the folks that are going to be working at 10 at night or Saturday morning. The most useful part of the process is that it makes you sit back and question yourself. I've had glaring mistakes on schematics that weren't found by the reviewers.

      Perhaps an inquiry board or being dragged into congress to testify gets more out of the design reviewers, but the most important thing is hiring good, conscientious engineers. A good engineer bails out a bad process far more often than a good process bails out a bad engineer.

      Delete
    5. I worked at Boeing for 27 years (Defense and Space, and Commercial) as an R&D engineer, and I can tell you that the most important thing you need for a good, conscientious process is a culture that doesn't punish people for mistakes.

      As you say, no-one cares about the design more than the designer, but if that designer is afraid he will get demoted or fired over some aspect of his work, he may do his best to hide the problem area from other eyes.

      It's important that a team work together, with full expectation of criticism without reprisal.

      Delete
    6. This is not about design reviews. This is instead about reviewing operations. Procedures to prep a vehicle for flight are somewhat complex. And in the course of doing that prep, things frequently are different from what they were the last time they prepped a vehicle. Turning an engineer and techs loose without documenting what they're going to do and without making them follow that procedure or get an appropriate review for changes to same create "entertainment". I saw that frequently during the 30+ years I worked at KSC. Especially with first time hardware coming from elsewhere. The designers and manufacturers were supposed to develop procedures to operate their equipment, and test that on said equipment at their site before shipping to KSC. I don't think I ever saw one of their procedures that could actually be run as written. And when you launch something and it goes "BOOM!!!", it's not good to try to rely on the memory of the techs and engineers who put the toy together. But then again: "We're engineers. We don't need no steenkin' procedures..."

      Delete
    7. Gotcha. That's something that's covered by the review process in the contractor world, too. Procedures for the equivalent sorts of tests are written, reviewed and approved. The test can't be modified while it's running without several engineers getting together and approving it; typically the chief engineer running the particular test (say the chief of the EMI or environmental lab), test engineer, design engineer, etc. Document, document, document. It's OK if it's in red pen after hours as long as the master copy is kept up to date and under configuration control.

      Delete
    8. For the last several years I was there, that was NOT the standard for Shuttle processing. The contractor engineers most likely did coordinate with each other when drafting the procedure, but when it was time to run, they had carte blanche to rearrange steps without any review or documentation, other than the time stamp that the techs or QC were supposed to put by each step as it was finished. And if they wanted to add or delete any of the steps in the procedure, they could do that on their own without any further approval, although they were supposed to write down the changes in the procedure. It's possible that has changed since I retired at the end of June 2009, but not very likely. In fact, with Bolden being the NASA Administrator since then, my bet is that it has actually gotten worse...

      Delete
  4. I visited the Kennedy Space Center this week and was somewhat disappointed. Very little to actually see. Sure there were mockups of spaceships and a bus trip to the launch pad. There were movies too but I felt it was all back in the 60's and remember this or that with almost nothing new or even since the mid-70's. Our guide/bus driver seemed knowledgeble but I will never know because English is his second language and I understood maybe 10% of what he said. The long drive-by of the launch pad was a waste of half an hour. Very disappointing.

    ReplyDelete
    Replies
    1. I'm afraid that's the way it has been for quite a while. It definitely had the air of "our best days were long in the past".

      I haven't been there in a long time, and got that feeling back then.

      Sad, really.

      Delete
    2. You obviously didn't get to go through any of the employee appreciation tours. When Shuttle was active, there were hazardous operations going on continuously in most of the processing facilities, and the general public was not allowed in those facilities for safety reasons. KSC did try to hold an Open House roughly once a year when possible, shutting down hazardous ops and safing systems for one day so employees and their families and a limited number of their friends escorted by them (nothing larger than a 9 passenger vehicle) could show where they worked and what they did.

      Today I believe that SpaceX is processing in one of the VAB high bays so that is off limits for proprietary reasons. I don't know if they have any loaded booster segments in there for the new SLS system that is supposed to fly next year, but that would be a reason to deny entry as well. Unintentional ignition of an SRB or segment is a Bad Day. And much of the other Shuttle facilities should be empty and waiting further use, so there wouldn't be much to see in them either. All this flows from the decision to terminate Shuttle and not fund a replacement at levels that could get it developed in a reasonable timeframe, which is why I retired when I did.

      Oh, and Lockheed Martin is assembling the Orion capsule in the O&C building, so that is off limits for proprietary reasons as well.

      Delete
    3. I got to go to a few of those when my wife worked on the KSC, fairly near the VAB. I was on the Cape for the return to flight open house after Challenger, which I recall as July 4, 1987. Another time I went inside the OPF, stood under Discovery, saw lots of cool stuff. But she retired in ‘97, so that was long ago. Plus, the comment was about the Visitors’ Center’s tours so that’s what I was responding to.

      Delete
    4. As a child, I remember vividly visiting KFC! They were assembling Apollo 12 in the Assembly Building, so of course we didn't get to go in. Wow, was that a great time to be a kid. Now...I worry that my grand-kids are going to ask me, "Pa-paw, did we REALLY go to themoon." Space-X and company have given me new hope though.

      Delete
  5. Here's a link to the NASA NOAA N-Prime Mishap Investigation Board. This is the multi-million dollar satellite that was dumped onto the floor because of complacent attitudes and lack of discipline in following proper procedures. Interesting reading on organizational dysfunction.
    https://www.nasa.gov/pdf/65776main_noaa_np_mishap.pdf

    ReplyDelete
    Replies
    1. Oh, the famous one where they dropped the quarter-billion dollar satellite on the floor. That was a regular meme for a while.

      Gravity wins.

      Delete
  6. One more for ya. Since you're a radio guy you'll like this. This was one of the first missions with which I was involved; I had the opportunity to build and test the data processing/memory board for one of the instruments on IMAGE.
    https://skyriddles.wordpress.com/2018/01/21/nasas-long-dead-satellite-is-alive/

    ReplyDelete
    Replies
    1. At the risk of disappointing you, I saw the story when it showed up last week (I think). It's an interesting development to have it wake itself back up and come to life again. By the link you sent, it apparently came back to life in October, 2016. Very cool website, by the way.

      That could be an interesting way to spend time, though. Looking for satellite signals. Another hobby! With exotic (= expen$ive) equipment! What more do I need?

      The last satellite I worked on was an earth observing satellite for JPL. It suddenly went dead a couple of years into the mission, but their failure analysis implied something killed the power and it died fairly quickly. I recall talk about collision with space junk but not why they thought that was possible.

      Delete
  7. My bad. Actual link is:
    https://skyriddles.wordpress.com/2018/01/21/nasas-long-dead-satellite-is-alive/

    ReplyDelete
  8. Argggh! nasas-long-dead-image-satellite-is-alive

    My apologies...

    ReplyDelete
  9. Oh, dear - bad week indeed. I've worked Boeing Space & Defense, Lockheed Space Systems, and NASA over the course of a long career ... and some things never change. Good or bad. While I miss working with all or any of them, I suspect those days are gone and the hope lies with Space-X clones if not Space-X itself. Attitudes are well-ingrained in the cultures. NASA seems to have followed Pournelle's Iron Law.

    I still have the NASA blurb from when Challenger blew ... and I heard the same stories of a change in recipe for the tiles - which I tend to believe. I once had a couple of pieces of the old tile. It looked and felt like styrofoam ... but I could hold a 3/8 thick chunk in my hand, blow a welding torch on it, and not feel the heat.

    I well remember Feynman's demo with the O-ring - about the time the axe was going to fall on some poor engineer at Thiokol when the blame belonged on whoever authorized a launch "out of spec". Icicles indeed. The spec as I recall was "above 10C". I'm sure it had nothing to do with the SOTU speech scheduled that evening.

    But my point in commenting. Here's a book you might be interested in reading: "Truth, Lies, and O-Rings - Inside the Space Shuttle Challenger Disaster" by Allan J McDonald with James R. Hansen. (Univ of FL Press) ISBN 978-0-8130-4193-3

    I got my copy at a lecture by the author given at one of NASA's facilities - which was sort of a surprise given the topic and the lecture host.

    Q

    ReplyDelete
    Replies
    1. I am unable to find the spec that the SRBs were originally required to meet, but I believe that was supposed to be "F", not "C". The requirement was to be able to launch from Vandenberg for USAF missions. 10C is 50F. They would rarely be able to put one up from Vandenberg if that was the requirement.

      Delete
    2. Yes, I was flabbergasted when the seal failed under those temps. Lived at Vandenberg from '67 to '70 and just above 32 degree temps were not uncommon in winter-time.

      Just one test. One stinking test of a solid booster under freezing conditions would have found the issue, but, noooo, costs too much.

      And it wasn't the tiles on the Orbiter, it was the foam coating over the main tank that was changed during the Clinton/Gore era due to CFC reductions. Wow, rather save absolutely nothing since at that time the ozone layer was already repairing itself than save human lives and weight (the new tank foam required more foam to achieve the same insulation results.)

      Delete
  10. Mark Matis:

    Yes, I heard 50F ... I translated it to C to make the SI folks happy. They shouldn't have launched from Florida at that temp either. I find that icicles are a good indicator of "not warm".

    It does get fairly chilly at the Thiokol site in Utah though. But if the test spec said 50F - or 10C - I can't see a contractor going the extra mile (spending the extra money - or inviting undesired extra "oversight") to go beyond test requirements.

    Q

    ReplyDelete
    Replies
    1. 50F is not accurate. The design requirements called for being able to launch from Vandenberg to meet USAF requirements. I do not have access to those criteria, but they were definitely LOWER than 10C. As Thiokol admitted when they were asked in the meeting.

      Delete