A Generative and Mutational Approach for Synthesizing Bug-exposing Test Cases to Guide Compiler Fuzzing
COMFUZZ is a compiler fuzzing framework that combines generative and mutation techniques. Unlike prior work, COMFUZZ devotes to generating bug-exposing test cases by using historical test programs and performs focused testing by leveraging carefully designed bug-guided mutators.
We provide a code image and a database image to run "out of box". The required environment information can be found in REQUIREMENT.pdf, and the installation insreuctions are introduced in INSTALL.md.
We provide two ways to execute COMFUZZ: one is the Quick Run; the other is to Run Step-by-Step with the following scripts:
- step1_generator.py: the script that generates test programs according to historical test programs.
- step2_init.py: the script that builds the initial seed pool.
- step3_harness.py: the script that performs the differential testing on target compilers.
- step4_mutation.py: the script that mutates the interesting test cases for focused and intensive testing.
- step5_filter.py: the script that filters faulty and analyzed test cases.
We created a Jupyter Notebook that contains a concise demonstration and instructions for reproducing COMFUZZ during the AE period. Reviewers can access it to replicate the entire COMFUZZ process and observe the corresponding results. The instructions for accessing the notebook are provided within the Jupyter file.
Use the following commands to test JS compilers:
cd COMFUZZ_js/workline
python main.py --enrich_limit_num=10 --loop_times=5 --clean_project
The parameter enrich_limit_num controls the number of tese cases in the seed pool (we set it to 10 for demonstration)， loop_times controls the iterative number during testing, and clean_project means cleaning the data in the database. We also provide many configurable parameters for customized execution. You can use python3 main.py --help to see what these parameters mean.
Use the following commands to test JVM compilers:
cd COMFUZZ_Java/workline
python main.py --clean_database=True --max_iterator=1
The parameter --clean_database controls cleaning the data in the database, and --max_iterator is used to specify the number of iteration testing.
You can also use the following step-by-step instructions to run COMFUZZ. Note that the step-by-step instructions are the same for testing JS and JVM compilers.
This step will generate JS functions or Java methods and save them into Table_Function.
python step1_generator.py
Notably, when running in COMFUZZ_Java, the arguments file_num should meet the constraint: file_num % 10 == 0.
This step will assemble JS functions or Java methods from Table_Function, and save them into Table_Testcase.
python step2_init.py
This step will apply differential testing on the selected test cases, (1)origin test cases generated by the trained model, (2)mutated test cases marked as interesting, or (3)mutated test cases marked as non-interesting. For JS, output results are stored in Table_Result and Table_Suspicious_Result; for Java, results of running javac are stored in Table_javac_Result and Table_javac_Suspicious_Result, and results of running java are stored in Table_Result and Table_Suspicious_Result.
python step3_harness.py
This step will mutate test cases by two kinds of mutation methods: general-purpose mutation and bug-guided mutation. For those interesting test cases, COMFUZZ mutates them by bug-guided mutators and others by general-purpose mutators. Mutated test cases will be saved in Table_Testcase, and the column Mutation_method record the selected mutator id.
python step4_mutation.py
This step will filter suspicious test cases to avoid re-analysis of faulty and analyzed test cases.
python step5_filter.py
Released under the terms of the Apache-2.0 License. See LICENSE for details.