In early August, the enterprise security firm Armis got a confusing call from a hospital that uses the company's security monitoring platform. One of its infusion pumps contained a type of networking vulnerability that the researchers had discovered in a few weeks prior. But that vulnerability had been found in an operating system called VxWorks—which the infusion pump didn't run.
Hospital representatives wondered if it was just a false positive. But as Armis researchers investigated, they started to see troubling signs of a connection between VxWorks and the infusion pump's operating system. What they ultimately discovered has disturbing implications for the security of countless critical systems—patient monitors, routers, security cameras, and more—across dozens of manufacturers.
Today Armis, the Department of Homeland Security, the Food and Drug Administration, and a broad swath of so-called real-time operating system and device companies disclosed that Urgent/11, a suite of network protocol bugs, exist in far more platforms than originally believed. The RTO systems are used in the always-on devices common to the industrial control or health care industries. And while they're distinct platforms, many of them incorporate the same decades-old networking code that leaves them vulnerable to denial of service attacks or even full takeovers. There are at least seven affected operating systems that run in countless IoT devices across the industry.
"It's a mess and it illustrates the problem of unmanaged embedded devices," says Ben Seri, vice president of research at Armis. "The amount of code changes that have happened in these 15 years are enormous, but the vulnerabilities are the only thing that has remained the same. That's the challenge."
>
The bugs endured for so long because they all trace back to the same popular early-aughts implementation of network protocols that make up the "TCP/IP stack," allowing devices to connect to networks like the internet. The Swedish software firm Interpeak created a version of this stack called IPnet that it licensed to an array of customers, including numerous real-time operating system developers. Then in 2006, Wind River, the developer of VxWorks, acquired Interpeak and absorbed IPnet. Once Wind River acquired Interpeak and dissolved the company there was no more support for IPnet licenses, so whatever bugs were already there lived on, unbeknownst to Wind River or Interpeak's old customers.
That's why the infusion pump, made by the medical device manufacturer Becton Dickinson Alaris, had Urgent/11 bugs despite not running VxWorks. Instead, it uses a real-time platform called Operating System Embedded by the Swedish IT company ENEA—which also incorporates IPnet. In its original July Urgent/11 security advisory, Wind River noted the possibility that other operating systems and devices might be vulnerable as well, because of IPnet's distribution prior to 2006.
"As a strong proponent for responsible disclosure practices, Wind River believes it is critically important with matters like Urgent/11 that the extent of industry impact is determined and disclosed as soon as possible," Arlen Baker, Wind River chief security architect, told WIRED in a statement.
The timing of the hospital alert was auspicious; the researchers were already in Las Vegas for a hacking conference. To validate their theory, the Armis researchers met with BD Alaris representatives at Defcon's Biohacking Village, a setting where hackers, manufacturers, and regulators work together to solve industry security issues. There, they tested a potentially vulnerable infusion pump at the Defcon Biohacking Village, and confirmed the presence of some of the IPnet vulnerabilities.
"Our approach was to encourage trust and collaboration in the Biohacking Village," says Beau Woods, a cybersafety innovation fellow at the Atlantic Council and an organizer of the Medical Device Village. "That created the conditions for device manufacturers to make their equipment available, for researchers to test for vulnerabilities safely, and for FDA to help with disclosure to preserve patient safety and public trust."
A BD Alaris spokesperson told WIRED that the vulnerabilities could not be exploited en masse on Alaris PC Units; any attempted attacks would need to target each device individually. And even if a hacker successfully exploited the bugs, she still wouldn't be able to interrupt an in-progress infusion. She could only create a situation where medical professionals would need to reboot the device before they could start a new infusion or change the parameters of a treatment. An attack on the device would also trigger flashing red dome lights and a loud alarm, along with the message "Communications error" on the screen. Urgent/11 exploits are different in different devices—some attacks are limited, as is the case for the infusion pump, but some would be easier for a hacker to carry out and potentially more damaging.
"The FDA actively engages with device makers to ensure that they are aware of cybersecurity vulnerabilities, such as those like URGENT/11, which can be found in new devices as well as in legacy systems," an FDA spokesperson told WIRED in a statement. "The software that contains these vulnerabilities may be incorporated into other software applications, equipment, and systems which may be used in a variety of medical and industrial devices that are still in use today. Awareness is a key because without it, industry cannot begin their risk assessment and mitigation activities."
>
BD Alaris isn't directly issuing a patch for vulnerable devices, but is publishing a list of mitigation techniques in its product security bulletin, including a specific firewall rule to block any remote attempts to exploit the IPnet bugs. It's a response that illustrates a crucial part of the challenge in dealing with vulnerabilities like those in IPnet. As software components that are open source or can be licensed get adapted for various software products over time they undergo a sort of divergent evolution. This lack of standardization then makes it almost impossible to develop a one-size-fits-all security patch if those modules turn out to contain vulnerabilities. In the case of Urgent/11, even the updates Wind River released for VxWorks at the end of July don't work as a template for patching other systems.
And add to all of that the broader challenge of securing IoT devices: Even if patches existed for every device that has IPnet lurking inside, the owners often lack the resources or time time apply them.
"The thing is once you identify what is vulnerable, how do you actually update these devices?" Armis's Seri says. "Often the update mechanism is almost nonexistent or it's such an analog process it's almost like it's with a screwdriver. It's not something that can be done at scale. So I don't know if it will ever be accomplished to update all of these machines."
In its legislative proposals, the FDA has advocated that manufacturers adopt a "software bill of materials" that outlines which stacks, libraries, and open source components are in devices so it's easier to track vulnerabilities like Urgent/11 across all sorts of devices when they inevitably crop up.
In addition to the BD Alaris PC Unit infusion pump, the researchers found patient monitors, cameras, printers, routers, Wi-Fi mesh access points, and a Panasonic door bell camera that are all vulnerable to Urgent/11 bugs. Along with VxWorks, the researchers identified six other platforms that they believe have at least some vulnerable versions—ENEA's OSE, INTEGRITY by Green Hills, ITRON, Microsoft's ThreadX, Mentor's Nucleus RTOS, and zebOS.
It will take time to determine exactly what is exposed, and how, though. For example, Microsoft didn't develop ThreadX itself, instead absorbing it through an April acquisition of the real-time IoT company Express Logic. A Microsoft spokesperson told WIRED in a statement that, "We’ve investigated these reports and confirmed that these vulnerabilities do not impact any ThreadX release." This doesn't preclude the possibility, though, that there are vulnerable devices out there running versions of ThreadX alongside an IPnet license.
Similarly, Kevin McCombe, senior director of embedded platform solutions engineering at Mentor (which is owned by the industrial control giant Siemens) told WIRED in a statement that, "Mentor Nucleus RTOS and all its related products do not include the Interpeak IPnet Stack. We are currently in contact with ICS-CERT to clarify the situation regarding Mentor Nucleus RTOS."
The Department of Homeland Security's Cybersecurity and Infrastructure Security Agency released both a regular security advisory and a medical advisory about the IPnet findings on Tuesday. "CISA is aware of a public report detailing vulnerabilities found in the Interpeak IPnet TCP/IP stack," the agency wrote. "These vulnerabilities have expanded beyond the affected VxWorks systems and affect additional real-time operating systems. CISA recommends users take defensive measures to minimize the risk of exploitation."
Armis is releasing an open source tool that anyone can use to detect potentially vulnerable devices on their network and determine possible defense strategies. And the potential exposure from Urgent/11 will likely take years to decline—after all, these bugs have already made it through nearly two decades, an eternity in tech time. The even bigger concern, though, is all the other dormant bugs in ancient, ubiquitous code that are just waiting to be found. Or already have been.
Updated October 1, 2019 2:00 pm ET to emphasize Microsoft's findings that no version of ThreadX before or after its acquisition directly incorporated IPnet.
Updated October 4, 2019 11:00am ET to include comment from Mentor.