Nozomi Networks Labs is committed to conducting cyber security research that makes industrial organizations more secure. We do this in multiple ways, including malware research, vulnerability disclosures, curating threat information and more. You can find a summary of our work here.
One way we help is by creating or improving tools that facilitate research or check for threats. For example, we recently enhanced Radamsa, an open source fuzzing tool for testing software. Our new code makes it faster and easier to test devices that communicate over industrial networks, such as PLCs and RTUs, for security vulnerabilities.
Read on to find out why this is important and how we developed our code contributions.
Radamsa Speeds Up Testing and Enhances Software Security
Radamsa is a well-known test case generator, i.e. fuzzing tool, that improves the robustness and security of software. It was created by Aki Helin at the University of Oulu, in Finland.
Radamsa is a general-purpose tool for identifying security issues without a lot of human input in terms of understanding the target. The concept is that the tool performs a certain number of mutations on sample input to test how a program behaves with mutated data.
It is quite useful and effective in finding various kinds of vulnerabilities, as its authors outline on the Radamsa website.
Nozomi Networks has a long history with Radamsa — we’ve been using it since our company was founded. We use it in our automated testing platform to make Nozomi Networks products more robust and identify bugs early in the software development life cycle.
We also use Radamsa as part of our scripting / tooling set to (semi) automatically find vulnerabilities in PLCs, RTUs, etc.
About Radamsa
- Radamsa is a general-purpose test case generator which others can easily contribute to, enabling more secure software development.
- It has been developed at Oulu University by the Secure Programming Group (OUSPG).
- The tool attempts to generate a wide variety of malformed input documents for programs to expose errors easily and early in the development life cycle. The goal is to identify vulnerabilities before they are found and possibly exploited by threat actors using one of the many similar ad-hoc tools.
- Radamsa has been engineered to make state-of-the-art automatic black-box1 robustness testing more accessible to vendors and independent software developers. It was designed to be a command line tool, used by simply pointing to the data set that needs to be tested.
Adapted from the University of Oulu, Radamsa
Radamsa Project Accepts Our PCAPNG Contribution
The challenge with a general-purpose test case generator is that it needs more cycles to generate interesting cases than white-box testing3. Radamsa just doesn’t focus on key areas, that is, eliminating all the variations that don’t utilize a piece of software with suspected issues.
To this extent, the Radamsa project has recently accepted our pull request that allows the tool to behave with more focus when mutating packet capture (PCAPNG2) files.
The Nozomi Networks Labs contribution allows Radamsa to mutate PCAPNG files focusing only on the packets themselves. It eliminates the bytes and data structures used by the PCAPNG format itself to store the additional things needed to manage the recording of packets into a file.
Technical Summary of Our Radamsa Contribution
To understand how we developed our Radamsa PCAPNG code, let’s take a step back and consider the characteristics of Radamsa. It is written in Owl Lisp, a functional dialect of the Scheme programming language. It does not support any form of plugins or extensions.
This peculiarity led us to directly modify it with care, to avoid altering the original program behavior unless the user specifically enables our changes. To explain how we did that, it is first necessary to understand how the program works.
Radamsa operates by conditionally applying mutations to the stream of byte blocks produced by specific internal components called generators. For each input block, Radamsa determines whether to mutate it or not. It then outputs either the mutation result, or the original block unchanged.
Mutations happen with a certain probability that is dynamically updated along the whole run. They can be performed in various ways, depending on how the patterns options have been configured (e.g. apply the same mutation multiple times, or not).
By default, multiple generators are used at the same time, effectively mixing different inputs. In our case, we were interested in a single generator scenario, namely, only that in which has a PCAPNG file as an input. By using a single input generator, we can control what will be mutated and, therefore, what the output will look like.
The default generator for input files is called file and it simply splits its input into random-sized blocks. This way of processing inputs, however, destroys the structure of PCAPNG files. A PCAPNG file
is a sequence of blocks, each of which has its own structure. Each block contains either metadata (e.g. capture information) or actual traffic data (a packet). Among the several kinds of blocks that exist, the Enhanced Packet Block is the one we focused on, as it is the block generally used for storing packets.
Our first step consisted of forcing Radamsa to apply mutations on a PCAPNG-block basis and not to random-sized blocks, as the file generator does. This was needed to ensure that Radamsa’s output was still a sequence of blocks that more or less resembled a PCAPNG file.
We thus created a new generator called pcapng that precisely produces a stream of PCAPNG blocks out of a PCAPNG file.
At that point we had two other problems to solve:
- Blocks containing a packet have a structure and metadata that we want to preserve, and that we might have to adapt, due to the changes introduced by mutations.
- We don’t want to mutate blocks that don’t contain a packet (e.g. network interface blocks).
To address 1. we applied an instrumentation to all mutations so that only the actual packet contained in a block gets mutated. Specifically, the instrumentation:
- Extracts the packet from a block
- Applies the underlying mutation
- Uses the result to build a new block based on the original block metadata plus the outcome of the underlying mutation
For instance, when the underlying mutation adds a new byte, the instrumentation makes sure that the packet size is properly incremented by one and that the alignment requirement is still satisfied.
To address 2. we defined a block predicate that allowed us to discern between interesting blocks (those that we wanted to mutate) and uninteresting blocks (those that we wanted to preserve). The first solution we tried amounted to applying the NOP mutation, which does nothing, whenever our instrumentation function encountered uninteresting blocks.
This, however, turned out to be a bad strategy, since it broke the checksum mechanism that Radamsa uses to ensure that the output files it produces are all unique. Therefore, we had to move to a more invasive, but nonetheless necessary approach. It consisted of strengthening the condition employed by Radamsa to determine whether to apply mutations on a given block, or not.
With those changes in place, activated only when the PCAPNG generator is used, we were finally able to achieve our initial goals.
If you want to try our contribution out, run the current development version of Radamsa as follows:
radamsa --generators pcapng --count 10 --output mutation_%02n.pcapng input.pcapng
To summarize, Nozomi Networks Labs contribution to Radamsa consists primarily of three components:
- A new generator that parses PCAPNG files
- A mutation instrumentation that ensures we only mutate packet data and produce valid PCAPNG blocks
- A predicate that prevents Radamsa from mutating PCAPNG blocks that do not contain traffic
Nozomi Networks Labs Enhances Radamsa for Safer ICS Software
With our Radamsa contribution accepted, researchers and engineers worldwide have a better tool to test the robustness of network-enabled devices.
In particular, our enhancement tests the robustness of protocol stacks. This is an area where PLC, RTU and, in general, OT-device software, needs more focus. Higher quality is required to help prevent zero-day vulnerabilities and other security issues.
We hope that many individuals and companies will find our change to Radamsa valuable, and add their own contributions to the project. Together, we can create better ways to ensure that ICS software is secure.
