Arianespace, the European space consortium, had a launch at 8:52 PM Monday night (EST) of one of their Vega rockets that failed to achieve orbit. It was the second failure of a Vega in two years, after a spotless six years of service. The vehicle was carrying earth observing satellites for both France and Spain; both satellites were lost.
One of the satellites that was lost, while being readied for launch. Arianespace photo.
Yesterday, Elon Musk announced in his Twitter feed what the failure analysis showed as the root cause of the problem with SN8 that led to dripping molten metal and the other things I talked about last Friday.
It originally seemed to me that this is the opposite kind of problem that Ariane had with their Vega. It's not a human error like swapping two cables, it's more a design error of picking the wrong materials for the launch pad (the martyte) or not anticipating what can go wrong. On second thought, I don't think it's that simple. It could be the martyte wasn't the right choice of material, but perhaps it would have been fine if it was mixed or applied differently.
Teslarati author Eric Ralph is optimistic that we may see testing resume before the end of the year. From the information I've seen, including the chatter by the full-time (or nearly so) observers on the Lab Padre cameras, the vehicle has already been fixed, with the one damaged engine replaced already. If any design changes are known to be coming, especially to the vehicle itself, it would seem to be best to incorporate those before any more attempts.
The Vega is designed for relatively small satellites, typically handling total weights in the area of about 1,000 kilograms (2200 lbs), though it can lift heavier items into lower orbits or take lighter ones higher. The trip to space is powered by a stack of three solid rocket stages; once in space, a reignitable liquid-fueled rocket can perform multiple burns that take payloads to specific orbits.Although the solid rocket stages operated normally, when the liquid-fueled stage ignited to push the satellites into their desired polar orbits, something went wrong. According to Roland Lagier, chief technical officer of Arianespace, the problem started the moment the stage ignited.
However, “straightaway after ignition” of the upper stage, he said, the vehicle started to tumble out of control. “This loss of control was permanent, inducing significant tumbling behavior, and then the trajectory started to deviate rapidly from the nominal one, leading to the loss of the mission.”Ouch. That kind of human error is always exasperating and especially when it takes out a mission. The entire infrastructure around launches is aimed at catching operator error like this. Everything: quality inspections, the workers performing the connection signing off on every operation, stamping the work to verify it agrees with the requirements, everything. Picture two cables marked 1 and 2, or something, that are supposed to go to connectors marked the same way. Instead of seeing 1 connected to 1 and 2 to 2, it was 1 to 2 and 2 to 1. And it got by.
Analysis of the telemetry from the mission, along with data from the production of the vehicle, led them to conclude that cables to two thrust vector control actuators were inverted. Commands intended to go to one actuator went instead to the other, triggering the loss of control.
One of the satellites that was lost, while being readied for launch. Arianespace photo.
Yesterday, Elon Musk announced in his Twitter feed what the failure analysis showed as the root cause of the problem with SN8 that led to dripping molten metal and the other things I talked about last Friday.
About 2 secs after starting engines, martyte covering concrete below shattered, sending blades of hardened rock into engine bay. One rock blade severed avionics cable, causing bad shutdown of Raptor.As I understand it, martyte is a ceramic filled epoxy intended to help fireproof and protect the area under the test launch pad. Moments after this tweet, he added that avionics cables would be moved to steel pipe shields and they would be adding water-cooled steel pipes to the test pad. He also stated that an overview of the Starship development program will be delayed to account for some “notable” design changes. While we're all looking forward to seeing Starship and SN8 flying and proceeding in development, this all seems like things that must be done. After all, the Raptors are producing up to 200 metric tons (~450,000 lbf) of thrust each and an exhaust stream traveling some 3.3 kilometers per second (2 mi/s, Mach ~10). Don't leave anything loose under the vehicle that you ever expect to see again.
It originally seemed to me that this is the opposite kind of problem that Ariane had with their Vega. It's not a human error like swapping two cables, it's more a design error of picking the wrong materials for the launch pad (the martyte) or not anticipating what can go wrong. On second thought, I don't think it's that simple. It could be the martyte wasn't the right choice of material, but perhaps it would have been fine if it was mixed or applied differently.
Teslarati author Eric Ralph is optimistic that we may see testing resume before the end of the year. From the information I've seen, including the chatter by the full-time (or nearly so) observers on the Lab Padre cameras, the vehicle has already been fixed, with the one damaged engine replaced already. If any design changes are known to be coming, especially to the vehicle itself, it would seem to be best to incorporate those before any more attempts.
Prudent design would seem to call for incompatible connectorsin that area, and probably more. When I started at the laser mine, I was slightly stunned to find that none of the ribbon cable IDC connectors were mechanically keyed. And lo, it was not uncommon to find a connector off by 1 pin in sideways or lengthwise. It became an early troubleshooting stop.
ReplyDeleteThere has been a lot of questioning about SpaceX not having any water baffling or flame diverters set up on previous launches and tests.
ReplyDeleteBethcha that the launch stands are reworked with metal flame diverters forthwith.
And maybe some sort of water baffling for the Heavy stand they're building.
We used to joke that we weren't ready to launch until the stack of documents was as tall as the rocket.
ReplyDelete*EVERYTHING* was photographed during the close-out procedures, and that was added to the stack that already had hundreds of photos leading up to the close-out procedure.
I've lost track of the hours spent repairing/replacing the ablative coating on our camera enclosures that were at the pad. It's nasty stuff, and required a waiver from some state agency so we could use it.
Our "Hot Gas Deflector" hung down under the pad about 50'or so, and had four very large nozzle structures for water deluge.
The second (?) Saturn V launch had an issue with cabling. One engine had issues and the computer shut it down, but that command went to a different engine, resulting in two engines out. Oops. Among some other problems. They changed the cable length so that couldn't happen again.
ReplyDeleteRocket science: it's not rocket science.
I agree with Ritchie. It is criminally negligent design not to connectorize so that it is literally impossible to put it together wrong. Amphenol and most others have keying schemes that allow a huge number of different plug/socket combinations, so there's no real excuse for this. It's really basic.
ReplyDeleteThe problem with the Martyte might be because SpaceX isn't NASA. Their rules might not be as strict and that allows some freedom to try stuff that hasn't been done before. Will it work? I dunno, nobodies ever tried it. It should work, let's try it.
ReplyDeleteYes Murphy isn't just a theory, it's the law. Marking cables doesn't always work, after all you are relying on someone to make sure cable 1 is actually marked cable 1. Having gotten (well) paid to do root cause of failure analysis for various nuclear power plants, I have come to the conclusion that human error is the cause of almost all failures. You can engineer systems to be as robust as time, money and physical limitations allow, however, nothing is perfect. That's why we test, it's better to find the oopsies in a controlled environment rather than when lives are on the line. Even then "Stuff" happens. What is amazing is that considering how complex, how dangerous and how hostile the environment of space flight is, we have such a remarkable safety records we do. We owe it to ourselves and those who went before us to learn from our mistakes, and to do our best not to make the same mistakes twice.
ReplyDeleteYes Murphy isn't just a theory, it's the law. Marking cables doesn't always work, after all you are relying on someone to make sure cable 1 is actually marked cable 1. Having gotten (well) paid to do root cause of failure analysis for various nuclear power plants, I have come to the conclusion that human error is the cause of almost all failures. You can engineer systems to be as robust as time, money and physical limitations allow, however, nothing is perfect. That's why we test, it's better to find the oopsies in a controlled environment rather than when lives are on the line. Even then "Stuff" happens. What is amazing is that considering how complex, how dangerous and how hostile the environment of space flight is, we have such a remarkable safety records we do. We owe it to ourselves and those who went before us to learn from our mistakes, and to do our best not to make the same mistakes twice.
ReplyDeleteMurphy's Law is as much a law of nature as Newton's.
ReplyDeleteSo the cables were connected wrong. 1. Shouldn't be able to happen (Poka-yoke). 2. Should have been caught in QA inspection. 3. Should have been caught in test. I mean, nobody noticed that the wrong nozzle gimbled? I BET the test moves both/all the nozzles at the same time and the same direction to save time.
ReplyDeleteALMOST as bad as the Lockheed satellite they dropped because some took the retaining bolts out of the fixture.