What is Fuzz Testing?

Andrei Serban
March 12, 2021 ~ 8 min read
Info

Fuzz testing detects the hard-to-find bugs that developers & other security tools like SAST (static analysis) and SCA (software composition analysis) typically miss.

Instead of testing with a small, pre-defined set of test cases that are handwritten by developers like unit tests or SAST, fuzz testing constantly tests your code with newly generated inputs, using the feedback and results it gathers to generate additional test cases, in an effort to exercise all of your application’s code paths. This feedback loop is run thousands of times per second, making it an effective way to test your application very thoroughly.

An animation of what the fuzz testing feedback loop looks like
The fuzzer is sending inputs into the application and using the coverage generated by that input to guide the next set of inputs until it is able to hit a branch of code that causes a bug.

Basic Definition

At its core, fuzz testing is an advanced type of random testing. It can be helpful to imagine fuzzing as a million monkeys typing on a million keyboards, all of which are hooked up to your application. These inputs are random, often completely invalid and unexpected by your program.

Fuzz testing is so powerful because developers tend to only test the happy paths in their code (ie, the inputs that the user should be sending), leaving the malicious inputs untested. You can see how once you unleash all of those “monkeys” onto your application, you’ll end up uncovering all the edge cases your application doesn’t handle with grace. More often than not, your users end up playing the role of the “monkeys”, causing all sorts of unexpected and unhandled behavior in your live instance. Fuzzing allows you to discover these bugs before they ever hit production.

You can think about it in terms of what a developer knows and doesn’t know.

  • Known knowns: bugs that developers expect and write tests for: typically covered by manually written checks like unit and integration tests
  • Known unknowns: bugs that developers know exist, but aren’t quite sure where or how they might manifest: usually found by SAST and SCA tools, which act like spell checkers looking for common errors
  • Unknown unknowns: bugs that developers don’t expect, and are therefore also unsure where they may present themselves. Truly new bugs and vulnerabilities like Heartbleed, Urgent/11 and the tens of thousands of Chrome vulnerabilities, which are otherwise discovered by users in production.
known unknowns table

Types of Bugs

The types of bugs that fuzz testing can find vary between languages, but ultimately, a fuzzer is used to invalidate assumptions about your code. If you can define a property that must hold true for any given input (also called an invariant), then the fuzzer will look for inputs that break your invariant. Some invariants are true for every application. For example, your code should never produce:

  • Crashes, Timeouts and Hangs
  • Memory Corruption errors like overflows and leaks
  • Race Conditions
  • Excessive Resource Consumption
  • Unhandled exceptions
  • Undefined Behavior

These are bugs that fuzz testing can easily discover in any code: your application should never crash, corrupt memory, or time out. It also just so happens that these types of bugs also make up over 50% of all CVEs (the largest database of publicly disclosed vulnerabilities - 149,000 of them).

Developers can write their own invariants to discover bugs specific to their application by defining properties that must always be true. If the fuzzer detects that a property is not true for some input, it knows it has found a logic bug. Applications with complex business logic typically have a lot of underlying bugs that can only be found with fuzz testing.

What Should I Fuzz?

Hopefully by now, you’re convinced that fuzz testing is an effective way to find bugs in your code and are starting to think about its applications. In our opinion, if you’re writing complex, mission-critical software, fuzz testing is the most effective way to mitigate unexpected bugs. Any part of your software that consumes data across a trust boundary is a perfect candidate for fuzzing.

In practice, trust boundaries can come in the form of:

  • Publicly exposed APIs
  • Embedded software (potentially consuming data from radios and sensors)
  • Cryptography & encryption code
  • External interfaces in Databases and Cryptocurrency nodes
  • Medical, automotive and other IoT applications
  • Industrial Control Systems
  • Parsers & marshallers for text, video and audio protocols

While this isn’t an exhaustive list, it gives you a sense of the type of software that should be fuzzed. If you don’t control the source of the data your software processes, that software should be thoroughly fuzz tested.

On the other hand, if you’ve written a parser or framework that is meant to work with a custom or less well-known protocol, then it absolutely makes sense to fuzz that protocol implementation. This is very common in software that uses industry-specific protocols with custom framework implementations such as DICOM or BACNET.

Coverage-Guided Fuzz Testing

Coverage-guided fuzz testing (sometimes referred to as mutational fuzzing) is one of the most important breakthroughs in the fuzzing space, which has made it the de-facto fuzzing technique over the last 5 years. Taking our monkey example from above, it’s a way to make the “monkeys” much smarter - allowing them to cover more lines of code in a fraction of the time.

The idea behind coverage-guided fuzzing is that the fuzzing engine monitors the code coverage output of your application as it processes an input, which then guides the generation of the next set of inputs. While the fuzzer may generate inputs “randomly” at first, it saves the inputs that trigger new coverage in your application to the corpus (a set of interesting inputs), allowing it to drive deeper into your code with each set of new inputs.

When you run this feedback loop thousands of times per second, the fuzzing engine very quickly picks up on structure and patterns within your program, essentially learning how to generate inputs tailored specifically to your application.

A great example of this is Michal Zalewski’s blog post on pulling JPEGs out of thin air (spoiler alert: in just 6 hours, the fuzzer taught itself to generate valid JPEG images from scratch).

Coverage-guided fuzz testing is the most generally applicable fuzzing technique for developers - if you can write a unit test, you can write a fuzz test. It can also fit seamlessly into continuous integration workflows, making it a great choice for many dev teams.

Protocol Fuzz Testing

Protocol fuzzing (occasionally called generational fuzzing) acts as a low-friction alternative to coverage-guided fuzzing. Instead of knowing nothing about the application it’s fuzzing and learning as it goes along, it starts off with an in-depth understanding of the protocol the code expects, and sends inputs that mostly conform to the specification, with some exceptions - it’s important to test the robustness of the parser as well as the underlying business logic.

This can be very effective against obscure and complex protocols that often have proprietary implementations or where the protocol is impossible to sufficiently test manually. Protocol fuzzing can craft inputs specifically for the protocol your application expects, enabling the fuzzer to not only test the parser layer, but the core business logic of your application as well.

The other benefit of protocol fuzzing is the ability to fuzz large pre-compiled applications and complex systems. Protocol fuzzing typically only needs a thin agent to be deployed alongside the software under test in order to monitor for crashes, system errors, path traversal and unexpected behavior.

We recommend protocol fuzzing as a great solution when testing complex, obscure or custom protocols, as well as a good alternative to coverage-guided fuzzing for when you have to fuzz black-box systems that are hard to instrument or modify.

How does Fuzzbuzz help?

Fuzzing sounds like a magic bullet, and is incredibly powerful when used properly, but in reality it’s difficult to get right - especially when it comes to integrating into your CI/CD pipeline and convincing your developers to adopt it as part of their workflow.

Current fuzzing tools (open source or otherwise), aren’t very developer friendly, often forcing users to learn completely new testing paradigms, work with low-level structures they don’t understand, and significantly modify their application to get any results at all. Add in the need for custom infrastructure code to keep the fuzzer testing applications 24/7, and the set up time becomes too prohibitive for most teams to even consider.

Fuzzbuzz completely abstracts the complexities of fuzzing, so you don’t have to be a security or infrastructure expert to test your code. Fuzzbuzz requires no code to get started (although we do support all of the open source fuzzers), integrates directly into existing developer workflows, and ensures all bugs are reproducible and actionable, so developers don’t waste their time fiddling with infrastructure or chasing false positives.

Fuzzbuzz uses automation and intelligence to make fuzz testing as developer friendly as possible, but you don’t have to take our word for it - sign up for a demo with the founders by clicking “Get Early Access” in the top right (or shoot me an email [email protected]). We’re onboarding new users every week!