Fuzzing FreeImage project with Sydr and AFLplusplus
"Mistakes are also important to me. I don’t cross them out of my life, or memory. And I never blame others for them."
― Geralt of Rivia.
From this article you will learn how to use hybrid fuzzing tool sydr-fuzz, that combines the power of Sydr - dynamic symbolic execution tool and AFLplusplus. Sydr-fuzz also supports another fuzzing engine libFuzzer, here is the previous guide that shows how to apply sydr-fuzz with libFuzzer engine. In this guide we will focus on preparing fuzz targets for AFLplusplus and Sydr, hybrid fuzzing, code coverage collection. Also, we will use our crash triage tool Casr, and apply Sydr to check security predicates for finding interesting bugs using symbolic execution techniques.
Like last time, it is a hard task to find a single open source project to show all declared above features at once. I succeeded to find FreeImage project that suits us perfectly. Let's dive into our fuzzing journey with Sydr and AFLplusplus!
"If I have to choose between one evil or another, I'd rather not choose at all"
― Geralt of Rivia.
In the guide for xlnt project I covered some aspects of fuzz target creation and how to add new project to oss-sydr-fuzz. Here I want to remind the basic concepts and explain the difference between fuzz targets for libFuzzer and AFLplusplus.
So, before we could start fuzzing, we have to do the following steps:
- Create a fuzz target for AFLplusplus in persistent mode. If you aren't familiar with AFLplusplus, you may look at this tutorial.
- Create a fuzz target for Sydr and code coverage. For this purpose, you just need to add main function that reads file and passes its contents to LLVMFuzzerTestOneInput.
- Build three executable binaries for AFLplusplus, Sydr, and coverage.
- Prepare corpus.
FreeImage is already added to oss-sydr-fuzz. So, you can just clone this repository and build docker container following the instructions. We use afl-clang-fast/afl-clang-fast++ compiler to build fuzz target for AFLplusplus (-fsanitize=fuzzer for persistent mode). This is all the difference. Ok, let's build docker container for FreeImage and start fuzzing!
"Sometimes there’s monsters, sometimes there’s money. Rarely both. That’s the life."
― Geralt of Rivia.
Before starting sydr-fuzz for fuzzing we have to write simple config in toml format. Here below is a configuration file load_from_memory_tiff-afl++.toml for FreeImage:
exit-on-time = 3600
[sydr]
args = "-l debug"
target = "/load_from_memory_tiff_sydr @@"
jobs = 4
[aflplusplus]
target = "/load_from_memory_tiff_afl"
args = "-t 60000 -i /corpus_tiff"
jobs = 16
[cov]
target = "/load_from_memory_tiff_cov @@"Let's have a brief look at this config file:
exit-on-time - is an optional parameter that takes time in seconds. If during this time (1 hour in our case) the coverage does not increase, fuzzing is automatically terminated.
[sydr] table may contain the following parameters:
args is an arguments string for Sydr. Options for log files and input files are set automatically.
target is a command line for target program to run. Instead input file name use @@.
jobs is a number of Sydr's to run. Default is 1.
[aflplusplus] table contains arguments for AFLplusplus.
[cov] table contains target run string for code coverage binary.
To sum up, we will start fuzzing with 16 AFLplusplus instances and 4 Sydr jobs. Fuzzing process will stop if coverage will not increase for 1 hour. The most important thing is that for running parallel fuzzing with AFLplusplus sydr-fuzz uses smart algorithm to start fuzzing instances with different additional arguments. This algorithm is based on recomendations given by AFLplusplus developers. Well, let's build docker:
$ sudo docker build -t oss-sydr-fuzz-freeimage .
And run:
$ sudo docker run -v /etc/localtime:/etc/localtime:ro --privileged --network host --rm -it -v $PWD:/fuzz oss-sydr-fuzz-freeimage /bin/bash
Change directory to /fuzz:
# cd /fuzz
Run hybrid fuzzing:
# sydr-fuzz -c load_from_memory_tiff-afl++.toml run
Finally, we started sydr-fuzz. Firstly, sydr-fuzz copies initial corpus directory to it's project corpus directory and minimizes it with afl-cmin. After cmin step fuzzing is started. Let's wait until fuzzing is finished.
According to log we found 394 crashes. Fuzzing was stopped by exit-on-time (Testing aborted programmatically). Sydr was executed 613 times. Before we start collecting coverage and checking security predicates, let's minimize resulting corpus:
$ sydr-fuzz -c load_from_memory_tiff-afl++.toml cmin
Good, we have minimized corpus (2526 files).
Now we can collect code coverage and check security predicates.
"Jaskier: Actually, I've always wanted to know, do witchers ever retire?
Geralt: Yeah, when they slow and get killed."
― Geralt of Rivia.
To collect coverage in lcov format we run this command:
# sydr-fuzz -c load_from_memory_tiff-afl++.toml cov-export -- -format=lcov > load.lcov
After getting .lcov file let's get html report:
# genhtml -o load-html load.lcov
Great, now we can see which parts of code sydr-fuzz has reached. It's time to check security predicates!
"Toss a coin to your witcher
O' valley of plenty
O' valley of plenty
O'
Toss a coin to your witcher
O' valley of plenty"
— Jaskier.
The idea behind security predicates shortly described in xlnt guide. Let's check security predicates. Well, resulting corpus is still not small. I suggest to check security predicates on subset of inputs (for example 500 inputs) using 32 Sydr jobs.
$ sydr-fuzz -c load_from_memory_tiff-afl++.toml security --jobs 32 --runs 500
Let's wait for a while...
After security predicates checking we have 398 crash inputs (4 new inputs) and 831 verified inputs by sanitizers: intoverflow/bounds/zerodiv/null: 811/20/0/0. Detailed analysis of verified integer overflow that was found during security predicates check you can find in this issue.
Okay, we still have 398 crashes. Let's run Casr for crash triage.
"Beware of an old man in a profession where men usually die young."
― Geralt of Rivia.
Let's start crash triage using this command:
$ sydr-fuzz -c load_from_memory_tiff-afl++.toml casr
Firstly, crash reports (.casrep) for each crash are created. We use AFLplusplus binary compiled with sanitizers for that. The main component of report is a stack trace. Our crash triaging is based on stack trace comparison. Next step is a deduplication of equal stack traces (we consider that such crashes are the same). After deduplication phase crash clustering begins. Then crash inputs are copied next to reports and some information is updated. At the last step for each clustered crash input our caesar tool is used to get crash report on binary without sanitizers. It might be useful during crash analysis to see how it works on executable without sanitizers.
NOTE: casr is available on github. caesar is replaced by casr-gdb.
Deduplication reduces 398 crashes to 9 crashes, that were put into 6 buckets, cool! Let's look at some crash report. For that I use casr-cli tool.
$ casr-cli load_from_memory_tiff-afl++-out/casr/cl2/crash-17492804a7e9699fbbab8e2b92a691b56dde98e6.casrep
A segmentation fault occurred on memcpy call. From caesar report I've learned that src_tag value is equal to NULL that is the crash reason. It's look like a real bug, nice catch!
In this article we learned how to use sydr-fuzz with AFLplusplus and apply well known features like: code coverage collection, security predicates check and crash triage. We found some interesting crashes on FreeImage project. To sum up, combining symbolic executors with different fuzzing engines are very promising to find novel bugs. I hope this small guide was interesting and usefull for you:)
Andrey Fedotov