“Usually we don’t do this during rush hour,” says Ben Shukman. He’s driving a Lincoln MKZ sedan, trying to exit a gas station driveway and cross four lanes of traffic so he can make a left at the light 20 yards ahead. It’s 5 pm in Palo Alto, and Silicon Valley commuters are crawling home, leaving few gaps between the cars. Finally, the car in the closest lane stops, leaving a space for him. The car in the next lane over does too. Shukman slides in and makes the left.
“Good job, Ben,” says Shai Magzimof, giving a wave of thanks to those gracious humans. He’s sitting in the driver’s seat, while, in a garage miles away, Shukman controls the Lincoln from the kind of setup you’d find in the bedroom of a too-serious fan of racing video games. And he’s showing off the type of remote-control capability that every major player in the nascent world of robotic driving will end up relying on (at least for now) in some form or other.
Picking Up the Remote
Magzimof is the cofounder and CEO of Phantom Auto, a startup betting that this sort of long distance driving will be crucial to anyone looking to deploy (mostly) autonomous vehicles. “We’re not aiming to replace AVs,” he says. He wants to provide the ability to have a human briefly take control of the car whenever the robot encounters what engineering types call “edge cases.” It could be a construction zone, or a cop using hand signals to direct traffic, or the gaping maw of a sinkhole. Anything weird and complex enough to upset a computer system that's happiest when everyone follows the rules.
Phantom’s system is not for emergencies. If a crash seems imminent, it’s up to the car, not the human, to keep everyone safe. This is a backup system for moments when the car isn’t sure what to do, so it comes to a safe stop on its own and calls for help.
To make this work, Magzimof says he found a way to use cellular networks to maintain a strong connection between the human and the car, without the latency that has long bedeviled other teleoperation setups. Phantom Auto plans to establish call centers where a few humans will keep watch over a fleet of someone else’s robocars. If one gets in trouble, a human can use the car’s cameras and microphone to grok what’s going on. Then they can use a steering wheel and pedal combo there in the call center to do whatever needs doing. Once the world around the car is back to normal, the autonomous driving system resumes control. Magzimof says he has been in talks with various autonomy providers and has lined up a few (unnamed) customers. The last thing anybody wants is a robot car that gets stuck, stranding its passenger, blocking traffic, enraging others, after all.
For an industry based around the idea of ripping control away from the oh-so-fallible human, a system that plugs one back in seems counterintuitive. But as would-be robo-car makers race to hit the streets, they’re grappling with all the practicalities that stand between developing technology and deploying it. One thing they’ve realized is that no matter how good their system, no matter how constrained its operating conditions, they can’t yet reproduce that most remarkable human ability—to see a complex scene and decide what to do, especially in a world built for and populated by other carbon-based lifeforms.
We know what you're thinking: When the car gets confused, why not just have the human inside grab the wheel and figure it out? Well, you've just betrayed your lack of imagination. Self-driving cars won't just be regular cars with piles of sensors. They'll take all sorts of shapes and move all sorts of things. Relying on a person inside the vehicle means relying on a method of control (say, a steering wheel) that will soon be outdated. If you want to make an autonomous vehicle that can haul packages, or people who can't drive, your backup system can't be onboard.
Phantom, then, is far from alone in its efforts. Starsky Robotics is making a system that lets trucks drive themselves in simple highway conditions, but has a human driver take control from afar when it’s time to rumble down tricky city streets. Other self-driving car companies take different approaches, but they almost all keep a human in the driver’s seat in some fashion or other—even if that seat is miles and miles away.
Drawing the Line
At CES last year, Nissan became the first major developer of self-driving tech to announce a plan for keeping a human in the loop. Nissan calls it Seamless Autonomous Mobility, and it’s basically a help center much like the one Phantom imagines. But when cars send out a distress signal, the humans won’t take over the driving. They’ll use the car’s sensors to look around, determine the best course of action, and issue fresh instructions to the computer on wheels, like “Cross the double yellow line, follow the line of cones, then cross back over after 15 yards.” They draw a path on a digital map using a computer mouse, and let the car execute it, remaining in autonomous mode all the while. If a kid chasing an augmented reality Charizard runs into the human-issued path, the car won’t move.
Even better, every time the car does get stuck, its developers can feed the problem and the human-derived solution into their machine learning system, and teach the robot to do better the next time around. This approach avoids the latency problem (Magzimof's solution is proprietary, he says), and works even when cell connections are weak.
No other major company has publicized its cars’ occasional reliance on humans, but most of them are quietly planning for it in some form. Waymo’s cars, which are already roaming Arizona with no one inside, will be able to ask humans in a remote call center for help. Example: “That cop seems to be saying I should go the wrong direction down the one-way street, can you confirm that’s cool?”
General Motors, which plans to launch a Chevy Bolt without a steering wheel or pedals next year, is building a similar system, which it calls “expert mode,” to give those vehicles extra information to help it navigate tricky situations. So is Uber. Spokesperson Sarah Abboud calls it “a way for us to prepare for the unique edge cases that might come up on the road that don't necessarily require someone to retrieve the vehicle to resolve the issue.”
“We definitely want humans supervising,” says Tory Smith, technical program manager for Drive.ai, a startup working with Lyft. Zoox, a stealthy, well-funded startup based in San Francisco, is working on a setup that looks a lot like Nissan’s. So is Nutonomy, the MIT spinoff that automotive supplier Delphi bought last year.
Ford confirmed this sort of system is part of its development effort too. A spokesperson declined to offer specifics, but noted Ford explored the idea of remotely controlling vehicles back in 2015.
Speaking Human
A person in a call center can offer more than a timely translation of a traffic cop’s hand gestures. With any novel technology, especially one that people trust with their lives, the ability for users to ask a human a question could be crucial. All the developers listed above, including Phantom, plan to use remote humans not just as backup “drivers,” but as counselors, ready to provide friendly explanations.
Another reason so many companies are developing these capabilities: The law might well demand they do so. California’s DMV has proposed rules that would allow human-free cars onto public roads, but would require that their operators maintain a way to communicate with the vehicle from afar. Florida’s self-driving regulations require the vehicle’s operator—who doesn’t have to be inside the thing—retain the ability to control, or at least stop, the car.
A federal bill winding its way through the legislature would supersede such state regulations, and might end up demanding this kind of feature, too. Senator Richard Blumenthal of Connecticut, among the most nervous of Nellies when it comes to unleashed robocars, would like to see the human stick around. “It is critical that this bill holds automakers accountable for necessary safeguards—including robust fallback mechanisms such as safety drivers or teleoperators,” he said through a spokesperson.
The remote human, however necessary now, will be less and less needed over time, and as these systems get better, a single human overseer will be able to watch over more cars. (One example: Waymo’s cars used to ask for confirmation before turning right at red lights. Now, they’re confident enough to do it on their own.) The public will get used to autonomous cars, just like they’re now comfortable riding elevators that once had onboard operators. But even elevators get stuck sometimes—and they still have buttons that connect whoever’s inside to another human, somewhere far away, ready to lend a hand.