By Philippe Laulheret of McAfee
Avaya is the second largest VOIP solution provider (source) with an install base covering 90 percent of the Fortune 100 companies (source), with products targeting a wide spectrum of customers, from small business and midmarket, to large corporations. As part of the ongoing McAfee Advanced Threat Research effort into researching critical vulnerabilities in widely deployed software and hardware, we decided to have a look at the Avaya 9600 series IP Deskphone. We were able to find the presence of a Remote Code Execution (RCE) vulnerability in a piece of open source software that Avaya likely copied and modified 10 years ago, and then failed to apply subsequent security patches to. The bug affecting the open source software was reported in 2009, yet its presence in the phone’s firmware remained unnoticed until now. Only the H.323 software stack is affected (as opposed to the SIP stack that can also be used with these phones), and the Avaya Security Advisory (ASA) can be found here ASA-2019-128.
The video below demonstrates how an attacker can leverage this bug to take over the normal operation of the phone, exfiltrate audio from its speaker phone, and potentially “bug” the phone. The current attack is conducted with the phone directly connected to an attacker’s laptop but would also work via a connection to the same network as a vulnerable phone. The firmware image Avaya published on June 25th resolves the issue and can be found here.
What are Researchers Looking for?
When studying the security of embedded and IoT devices, researchers generally have a couple of goals in mind to help kickstart their research. In most cases, two of the main targets are recovering the files on the system so as to study how the device functions, and then finding a way to interact directly with the system in a privileged fashion (beyond what a normal user should be able to do). The two can be intertwined, for instance getting a privileged access to the system can enable a researcher to recover the files stored on it, while recovering the files first can show how to enable a privileged access.
In this case, recovering the files was straightforward, but gaining a privileged access required a little more patience.
Recovering the Files From the Phone
When we say recovering the files from the phone, we mean looking for the operating system and the various pieces of software running on it. User files, e.g. contacts, settings and call logs, are usually not of interest to a security researcher and will not be covered here. To recover the files, the easiest approach is to look for firmware updates for the device. If we are lucky, they will be freely available and not encrypted. In most cases, an encrypted firmware does not increase the security of the system but rather raises the barrier of entry for security researchers and attackers alike. In this case, we are in luck, Avaya’s website serves firmware updates for its various phone product lines and anyone can download them. The download contains multiple tar files (a type of archive file format). We can then run a tool called binwalk on the extracted files. Binwalk is a big dictionary of patterns that represents known file formats; given an unknown firmware file, it will look for any known pattern and, upon finding potential matches, will attempt to process them accordingly. For instance, if it finds what looks like a .zip file inside the firmware, it will try to unzip it. Running this tool is always a good first step when facing an unknown firmware file as, in most cases, it will identify useful items for you.
When processing the phone’s firmware, extracting the files and running binwalk on them gave us the program the phone runs at startup (the bootloader), the Linux kernel used by the phone, and a JFFS filesystem that contains all the phone’s binaries and configuration files. This is a great start, as from there we can start understanding the inner workings of the device and look for bugs. At this stage however, we are limited to performing a static analysis: we can look at the files and peek at the assembly instructions of various binaries, but we cannot execute them. To make life easier, there are usually two options. The first one is to emulate the whole phone, or at least some region of interest, while the other is to get a privileged access to the system, to inspect what is running on it as well as run debugging tools. Best results come when you mix and match all these options appropriately. For the sake of simplicity, we will only cover the latter, but both were used in various ways to help us in our research.
Getting the privileged access
In most cases, when talking about gaining a privileged access to an IoT/embedded device, security researchers are on the lookout for an administrative interface called a root shell, that lets them execute any code they want with the highest level of privilege. Sometimes, one is readily available for maintenance purposes; other times more effort is required to gain access to it, assuming one is present in the first place. This is when hardware hacking comes into play; security researchers love to rip open devices and void warranties, looking for potential debug ports, gatekeepers of the sought-after privileged access.
Close up of the phone’s circuit board. UART ports in Red and the EEPROM in blue
In the picture above, we can see two debug ports labeled UART0 and UART1. This type of test point, where the copper is directly exposed, is commonly used during the manufacturing process to program the device or verify everything is working properly. UART stands for Universal asynchronous receiver-transmitter and is meant for two-way communication. This is the most likely place where we can find the administrative access we are looking for. By buying a $15 cable that converts UART to USB and soldering wires onto the test pads, we can see debug information being printed on screen when the phone boots up, but soon the flow of debug information dries up. This is a curious behavior—why stop the debug messages?—so we need to investigate more. By using a disassembler to convert raw bytes into computer instructions, we can peek into the code of the bootloader recovered earlier and find out that during the boot process the phone fetches settings from an external memory to decide whether the full set of debug features should be enabled on the serial console. The external memory is called an EEPROM and is easily identifiable on the board, first by its shape and then by the label printed on it. Labels on electronic components are used to identify them and to retrieve their associated datasheet, the technical documentation describing how to use the chip from an electrical engineering standpoint. Soldering wires directly to the chip under a microscope, and connecting it to a programmer (a $30 gizmo called a buspirate), allows us to change the configuration stored on it and enable the debug capabilities of the phone.
EEPROM ready to be re-programmed
Rebooting the phones gives us much more debug information, and eventually we are greeted with the root shell we were after.
Confirmation we have a root shell. Unrelated debug messages are being printed while we are invoking the “whoami” command
Alternative roads
The approach described above is fairly lengthy and is only interesting to security researchers in a similar situation. A more generic technique would be to directly modify the filesystem by altering the flash storage (a NAND Flash on the back of the circuit board) as we did for previous research, and then automatically start an SSH server or a remote shell. Another common technique is to tamper with the NAND flash while the filesystem is loading in memory to get the bootloader in an exception state that will then allow the researcher to modify the boot arguments of the Linux kernel. Otherwise, to get remote shell access, using an older firmware with known RCE vulnerabilities is probably the easiest method to consider; it can be a good starting point for security researchers and is not threatening to regular users as they should already have the most up-to-date software. All things considered, these methods are not a risk to end-users and are more of a stepping stone for security researchers to conduct their research.
In search of vulnerabilities
After gaining access to a root shell and the ability to reverse engineer the files on the phone, we are faced with the open-ended task to look for potentially vulnerable software. As the phone runs Linux, the usual command line utilities people use for administering Linux systems are readily available to us. It is natural to look at the list of processes running, find the ones having network connection and so forth. While poking around, it becomes clear that one of the utilities, dhclient, is of great interest. It is already running on the system and handles network configuration (the so-called DHCP requests to configure the phone’s IP address). If we invoke it in the command line, the following is printed:
Showing a detailed help screen describing its expected arguments is normal behavior, but a 2004-2007 copyright is a big red flag. A quick search confirms that the 4.0.0 version is more than 10 years old and, even worse, an exploit targeting it is publicly available. Studying the exploit code and how the bug was patched—dhclient code is open source, so finding the differences between two successive version is straightforward—helps us to narrow down which part of the code could be vulnerable. By once again using a disassembler, we confirm the phone’s version of dhclient is indeed vulnerable to the bug reported in 2009. Converting the original exploit to make it work on the phone requires a day or two of work, while building the proof of concept demonstrated in the above video is a matter of mere hours. Indeed, all the tools to stream audio from the phone to a separate machine are already present on the system, which greatly reduces the effort to create this demo. We stopped the exploitation here, but we can assume that at this point, building a weaponized version able to threaten private networks is more of a software engineering task and a skilled attacker might only need a few weeks, if not days, to put one together.
Remediation
Upon finding the flaw, we immediately notified Avaya with detailed instructions on how to reproduce the bug and suggested fixes. They were able to fix, test and release a patched firmware image in approximately two months. At the time of publication, the fix will have been out for more than 30 days, leaving IT administrators ample time to deploy the new image. In a large enterprise setting, it is pretty common to first have a testing phase where a new image is being deployed to selected devices to insure no conflict arises from the deployment. This explains why the timeline from the patch release to deployment to the whole fleet may take longer than what is typical in consumer grade software.
Conclusion
IoT and embedded devices tend to blend into our environment, in some cases not warranting a second thought about the security and privacy risks they pose. In this case, with a minimal hardware investment and free software, we were able to uncover a critical bug that remained out-of-sight for more than a decade. Avaya was prompt to fix the problem and the threat this bug poses is now mitigated, but it is important to realize this is not an isolated case and many devices across multiple industries still run legacy code more than a decade old. From a system administration perspective, it is important to consider all these networked devices as tiny black-box computers running unmanaged code which should be isolated and monitored accordingly. The McAfee Network Security Platform (NSP) detects this attack as “DHCP: Subnet Mask Option Length Overflow” (signature ID: 0x42601100), ensuring our customers remain protected. Finally, for the technology enthusiasts reading this, the barrier of entry to hardware hacking has never been this low, with plenty of online resources and cheap hardware to get started. Looking for this type of vulnerability is a great entry point to information security and will help make the embedded world a safer place.
