Later this month, NASA is expected to launch its latest Mars rover, Perseverance, on a first-of-its-kind mission to the Red Planet. Its job is to collect and store geological samples so they can eventually be returned to Earth. Perseverance will spend its days poking the Jezero Crater, an ancient Martian river delta, and the samples it collects may contain the first evidence of extraterrestrial life. But first it has to find them. For that, it needs some damn good computers—at least by Martian standards.
How a chaotic skunkworks race in the desert launched what's poised to be a runaway global industry.
Perseverance is significantly more autonomous than any of NASA’s previous four rovers and is designed to be what Philip Twu, a robotics system engineer at NASA’s Jet Propulsion Laboratory, calls a “self-driving car on Mars.” Like the ones on Earth, Perseverance will navigate using an array of sensors feeding data to machine vision algorithms. But whereas terrestrial autonomous vehicles are packed with the best computers money can buy, the main computer on Perseverance is about as fast as a high-end PC … from 1997. The only way Perseverance’s poky brain is able to handle all this autonomous driving is because NASA gave it a second computer that acts like a robotic driver.
On previous rovers, the navigation software had to share limited computing resources with all the other systems. So to get from one point to another, the rover would take a picture to get a sense of its surroundings, drive a little, and then stop for a few minutes to figure out its next move. But since Perseverance can offload many of its visual navigation processes to a dedicated computer, it won’t have to take this stop-and-go approach to Martian exploration. Instead, its main computer can figure out how to get Perseverance where it’s supposed to go, and its machine vision computer can make sure it doesn’t hit any rocks on the way. “We’re moving closer and closer to being able to continuously drive and think,” Twu says.
Autonomy is critical for Perseverance’s mission. The distance between Earth and Mars is so large that it can take a radio signal traveling at the speed of light up to 22 minutes to make a one-way trip. The long delay makes it impossible to control a rover in real time, and waiting nearly an hour for a command to make a round trip between Mars and the Earth isn’t practical either. Perseverance has a packed schedule—it needs to drop off a small helicopter for flight tests, then collect dozens of rock samples and find a place on the surface to store them. (A later mission will bring the cache back to Earth so it can be studied for signs of life.) If the rover has any hope of accomplishing all of this in the year allotted for its primary mission, it has to be able to make a lot of navigation decisions by itself.
Terrestrial autonomous vehicles typically use lasers to determine where an object is and how far away it might be, but these Lidar systems are bulky, energy hungry, and prone to mechanical failure. Instead, Perseverance will use stereo vision and visual odometry to figure out where it is on the Red Planet. Stereo vision combines two images from a “left camera” and a “right camera” to create a 3D picture of the rover’s surroundings, while visual odometry software analyzes images separated in time to estimate how far the rover has moved.
“We were concerned about the mechanical reliability of Lidar for a space mission,” says Larry Matthies, a senior research scientist and supervisor of the computer vision group at NASA’s Jet Propulsion Laboratory. “We started using stereo vision for 3D perception at JPL decades ago when Lidars were far less mature, and it’s worked out pretty well.”
Matthies has helped build the visual navigation systems for every rover that’s ever gone to Mars. Aside from Sojourner, NASA’s first rover on the Red Planet, all of its mobile explorers have used a combination of stereo vision and visual odometry to get around. But what makes Perseverance special is that it has dedicated hardware and a suite of fancy new algorithms for machine vision.
Perseverance’s new digital glasses will allow it to autonomously navigate its surroundings several times faster than its predecessors, which means the rover has more time to focus on its main scientific objectives. Still, it will take Perseverance a full day to drive the same distance a sloth can cover in an hour. But compared to NASA’s previous Mars rovers, Perseverance is a hot rod. “The longest drive that any Martian rover has ever done in a day is 219 meters,” says Twu. “We’re able to drive around 200 meters per day, so on average Perseverance will be hitting or exceeding the current track record for Mars rovers.”
It’s not Perseverance’s fault that it thinks slowly; blame the radiation. Mars doesn’t have a magnetic field or a thick atmosphere to shield it from the charged particles streaming from the sun, and these particles can wreak havoc on a computer. They can cause transistors to turn on and off when they’re not supposed to, and if enough of these errors accumulate, they can cause a computer to crash. This could cause a loss of valuable data—or the failure of the entire mission—so engineers at NASA do everything they can to prevent crashes from happening in the first place.
There are a lot of techniques to make a computer immune to radiation. For example, it’s possible to add additional transistors that are harder to turn on and off, which makes them less likely to be flipped by a wayward ion. Minal Sawant, the space systems architect at Xilinx, a California technology company that designed and built the machine vision chip for Perseverance, says that the chip is radiation hardened by design. Based on qualification tests conducted by the company, the chip shouldn’t experience more than two bit flip errors—in which an ion causes a bit of information stored in memory to change from one to zero or vice versa—per year.
But, generally speaking, protecting a processor from radiation requires compromising its performance. This partly has to do with the design of the processor, and partly with the fact that it simply takes a long time to test a component’s immunity to radiation. By the time a component is qualified, the performance of state of the art processors has surged ahead. NASA engineers don’t want to use old technology; but they do want to use technology they know will work. The type of Xilinx chip Perseverance uses has flown on several previous space missions and has nearly a decade of performance data to back it up.
“The US space industry is traditionally very risk averse, and there’s a logic to that,” says Sawant. “One small error can cause a whole mission to go south, so they want to use a component that’s already been to space rather than try new technology. Reliability is key.”
Xilinx’s machine vision computer will be running brand-new vision algorithms developed by Twu, Matthies, and their colleagues at NASA. Unlike self-driving cars on Earth, Perseverance doesn’t have the luxury of a bank of powerful computers in its trunk for image processing. Energy and processing power are precious resources on the Red Planet, which means the algorithms that Perseverance uses to navigate must be as lean and efficient as possible, without compromising their accuracy.
“The algorithm can always make a mistake, even if the hardware is perfect,” says Matthies. “In computer vision, there are outliers that cause the algorithm to make mistakes. And so we have to overwhelm that possibility.” Outliers might include a situation in which the rover can’t see an object, or mistakes it for something else. One solution to this problem is to feed the rover’s navigation system data from other sensors so it’s not just relying on sight to get around. For example, gyroscopes and accelerometers help the rover understand the slope and roughness of the surface.
The other solution is to expose the rover’s algorithms to as many scenarios as possible before it launches so there aren’t any surprises when it gets to Mars. At NASA’s Jet Propulsion Laboratory in Pasadena, there’s a large outdoor field strewn with boulders and red dirt that simulates a Martian landscape. This is the Mars Yard, and for the past few years it's served as a proving ground for the algorithms that will guide Perseverance. Twu and his colleagues have regularly taken a replica of the rover out to the Mars Yard and deliberately constructed scenarios that they thought would confuse the rover. For example, if the rover drove itself into a dead end, could it backtrack and try a new route?
“The more complicated the system is, the more types of decisions it could make,” says Twu. “Making sure you’ve covered every possible scenario that the rover might run into has been very challenging. But it’s by doing a lot of really hands-on tests like this that we find quirks in the algorithm.”
But there are only so many different ways to arrange boulders in a giant sandbox. Most tests of Perseverance’s navigation algorithms were tested in virtual simulations, where the rover team threw every conceivable scenario at the rover’s software to get an idea of how it would perform in those situations. This was still mostly shuffling (virtual) rocks around, but there wasn’t really a limit to the types of landscapes and scenarios that could be modeled. Twu says this extensive testing of the visual algorithms combined with all the sensor data pulled in by the rover will allow Perseverance to navigate much more difficult terrain than any of the other Mars rovers.
But even the most perfect simulations pale in comparison to the real thing. The rover will undergo its most high stakes test yet when it touches down on the Red Planet next February. If all goes well, the path it plots may lead us to evidence of life beyond Earth.
Updated 7-23-20, 5:15 pm ET: Xilinx built the chip for the Perseverance vision compute element. The computer was built at NASA's Jet Propulsion Laboratory.