Special Pages

Wednesday, April 14, 2021

Sometimes I Come Up Empty Handed

Maybe that's empty headed?

I've been trying to determine why the Mars Ingenuity Helicopter hasn't flown.  For those who haven't followed, on April 9th, Ingenuity was commanded to spin up its rotors as a test before being allowed to fly.  The test didn't pass.  To quote from the JPL Ingenuity Page:

During a high-speed spin test of the rotors on Friday, the command sequence controlling the test ended early due to a “watchdog” timer expiration. This occurred as it was trying to transition the flight computer from ‘Pre-Flight’ to ‘Flight’ mode. The helicopter is safe and healthy and communicated its full telemetry set to Earth.

The watchdog timer oversees the command sequence and alerts the system to any potential issues. It helps the system stay safe by not proceeding if an issue is observed and worked as planned.

The previously announced date it would fly was April 11th, with data from JPL available by the 12th.  I went looking for updates on the 12th and found reports the flight was put off until NET the 14th.  (As always, NET = No Earlier Than).  That's today as I write this. 

There isn't a test flight date I can find on the JPL, just this interesting statement:

The Ingenuity team has identified a software solution for the command sequence issue identified on Sol 49 (April 9) during a planned high-speed spin-up test of the helicopter’s rotors. Over the weekend, the team considered and tested multiple potential solutions to this issue, concluding that minor modification and reinstallation of Ingenuity’s flight control software is the most robust path forward. This software update will modify the process by which the two flight controllers boot up, allowing the hardware and software to safely transition to the flight state. Modifications to the flight software are being independently reviewed and validated today and tomorrow in testbeds at JPL.

They then list the steps in making sure that software is safe to upload to Mars and say:

Once we have passed these milestones, we will prepare Ingenuity for its first flight, which will take several sols, or Mars days. Our best estimate of a targeted flight date is fluid right now, but we are working toward achieving these milestones and will set a flight date next week. We are confident in the team’s ability to work through this challenge and prepare for Ingenuity’s historic first controlled powered flight on another planet.

It's safe to say there won't a flight on the 14th or 15th, and very likely not before the 21st.  

When you consider the years of development of this system, you'd think the software would have been verified to not have hangups like this.  It brings to mind a software error that didn't show up until well into the Space Shuttle program.  During the countdown, ground based computers were to hand over control to the Shuttle's internal computers at T-31 seconds.  Long after I would have expected handshaking problems between the computers to have been verified impossible, a handshaking problem aborted a countdown.  

This may be the same sort of problem.  Despite Software Quality Assurance and testing, modules were aligning in time to work properly, not by design but by pure chance.  


  1. I thought they ran that thing before it was packed up for shipment.
    "Naw, just launch it, it'll be fine."

  2. I wonder if it was all one-off proprietary chips and software. There are a lot of artifacts in the Chinese code. I'm not suggesting sabotage, but am saying that I've run into that before - and sometimes that manifests in odd ways downstream.

    1. I have no inside information, but based on other programs I worked on, it's one-off proprietary software with off the shelf chips.

      The gray area is programmable chips, Field Programmable Gate Arrays or FPGAs, are many times more common than custom chips, Application Specific Integrated Circuits or ASICs, because they're a tiny fraction of the price. FPGAs are off the shelf chips that your proprietary software turns into custom chips.

  3. Ha! Happened to me in high school programming. Had to deal a deck of cards.

    Went great, output was perfect. Got 100%.

    Then I ran it again. Multiple 4 of diamonds. Oops.

  4. Honestly, when I saw it was delayed, I just sighed. I sort of wish they had downplayed the helicopter until after they had it up and taking footage. Underpromise, overdeliver.