Cling Application

This repository contains the replication package for the application of Cling (automated class integration testing tool) and EvoSuite (automated unit testing tool) on 140 pairs of classes from five different open-source projects.

This document contains the following instructions:

• Cling Application
• Docker image
• Run case filtering
• Run test generation tools
  • Prepare input CSV file
  • Run test generation script
  • Output
• RQ1: CBC Coverage
  • Collect CBC Coverage
  • Data Analysis
• RQ2: Mutation Score and line coverage
  • Run PIT
  • Collect Line coverages
  • Collect Mutation Scores
  • Collect the mutants Killed By Cling
  • Data Analysis
• RQ3: Captured Faults
  • Collect Captured Stack Traces
  • Collect Interesting Stack Traces
  • Manual Analysis

Docker image

Docker image of this replication package is available in DockerHub.

Also, you can build and image by running build-image bash file:

. scripts/docker/build-image.sh

For running the container, you can use the run-container bash file:

. scripts/docker/run-container.sh

Run case filtering

Run subjects-filtering bash file:

. scripts/main/subjects-filtering.sh

docker

docker exec -it cling-application-container bash scripts/main/subjects-filtering.sh

The filtered cases will be saved in data/filtered-class-pairs.csv.

Also, the sampled class pairs from the filtered ones are available in subject_generator/selected-class-pairs.csv.

Run test generation tools

Prepare input CSV file

The test generation script works with a CSV file as the input. The csv file is subjects.csv, which is located in subject_generator directory. Each row in this CSV file contains a caller_class and calee_class (one of the selected class pairs). The rows in this CSV file is categorized by their value for tool column:

• cling executes class integration testing for caller_class and callee_class for 5 minutes.
• evosuite-caller5 executes EvoSuite (unit testing) on caller_class for 5 minutes.
• evosuite-callee5 executes EvoSuite (unit testing) on callee_class for 5 minutes.
• randoop-caller5 executes Randoop (unit testing) on caller_class for 5 minutes.
• randoop-callee5 executes Randoop (unit testing) on callee_class for 5 minutes.

To generate this CSV file, you need to execute generate.py python file:

python subject_generator/generate.py 

docker

docker exec -it cling-application-container python subject_generator/generate.py 

To change the number of execution times, you need to change this python file (line 9). Also, if you want to filter the class pairs, you need to filter them from subject_generator/selected-class-pairs.csv and rerun generate.py.

Run test generation script

Run test-generation.sh bash file with one input parameter, which is number of parallel processes:

. scripts/main/test-generation.sh <number of parallel processes>

docker

docker exec -it cling-application-container bash scripts/main/test-generation.sh <number of parallel processes>

Output

Test cases generated by Cling and EvoSuite will be stored in the results directory. Also, the execution logs of these two tools will be saved in the logs directory.

We do note that the test suites and execution logs, which are used in our study, are currently available in results and results directory, respectively.

RQ1: CBC Coverage

Collect CBC Coverage

Run the following bash file:

. scripts/main/rq1-cbc-coverage.sh <number of parallel processes>

docker

docker exec -it cling-application-container bash scripts/main/rq1-cbc-coverage.sh <number of parallel processes>

The CBC Coverages of Cling and EvoSuite applied to the caller classes will be saved in data/rq1/cbc-coverage.csv. The current cbc-coverage.csv is the CSV file that we have used in our study.

Moreover, the execution results of the CBC coverage tool applied to EvoSuite generated tests are stored in logs/cbc/.

Data Analysis

Run the following R Script:

Rscript data_analysis/r-scripts/cbc-coverage.r 

The output is two boxplots indicating the CBC coverage of EvoSuite and Cling. These boxplots are saved as data_analysis/figures/cbc-total.pdf and data_analysis/figures/cbc-per-project.pdf.

RQ2: Mutation Score and line coverage

Run PIT

To calculate the mutation score of the generated test cases by Cling and EvoSuite, we need to execute PIT. to run PIT on all of the test cases, generated in this study, run the following bash file:

. scripts/main/rq2-mutation-score.sh <number of parallel processes>

docker

docker exec -it cling-application-container bash scripts/main/rq2-mutation-score.sh <number of parallel processes>

The output of the PIT tool will be saved in the data/rq2/pit/ directory. Also, the execution logs of PIT runs will be stored in the logs/pit/ directory.

This script will also detect and ignore the flaky tests detected afted 5 times of execution. To calculate the totla number of flaky test cases, run the following Python script:

python scripts/python/test/parse_ignored_tests.py

docker

docker exec -it cling-application-container python scripts/python/test/parse_ignored_tests.py

Collect Line coverages

To collect all of the line coverages achieved by each generated test case, run the following python script:

python scripts/python/line-coverage/collect-line-coverages.py

docker

docker exec -it cling-application-container python scripts/python/line-coverage/collect-line-coverages.py

Collect Mutation Scores

To collect all of the mutation scores reported in PIT outputs, run the following python script:

python scripts/python/mutation/collect-mutation-score.py

docker

docker exec -it cling-application-container python scripts/python/mutation/collect-mutation-score.py

The output of this script will be saved as a CSV file: data/rq2/mutation-scores.csv.

Collect the mutants Killed By Cling

In this step, we want to collect the mutants which are killed by Cling. Also, we want to detect mutants killed by Cling but not by TE, mutants killed by Cling but not by TR, and mutants killed by Cling but not by any test suite generated by EvoSuite.

To collect all of this information, run the following bash script:

. scripts/main/rq2-killed-mutants.sh 

docker

docker exec -it cling-application-container bash scripts/main/rq2-killed-mutants.sh 

This script creates multiple CSV files in data/rq2. Each CSV file indicates the mutants which are killed by cling but not with the other tools and congifurations.

Data Analysis

After generating all of these CSV files, you can run the following R scripts to create tables and figures reported in the paper. For line coverage:

Rscript data_analysis/r-scripts/line-coverage.r 

For mutation scores:

Rscript data_analysis/r-scripts/rq2-mutation-scores.r 

For mutation operators:

Rscript data_analysis/r-scripts/rq2-mutation-operators.R 

All of the tables and figures will be saved in data_analysis/tables and data_analysis/figures, respectively.

RQ3: Captured Faults

Collect Captured Stack Traces

By running the following bash script, the stack traces captured by all of the generated test cases will be stored id data/rq3/stack-traces

. scripts/main/rq3-collect-captured-exceptions.sh 

docker

docker exec -it cling-application-container bash scripts/main/rq3-collect-captured-exceptions.sh 

Currently, the captured stack traces in our study are available in data/rq3/stack-traces.

Collect Interesting Stack Traces

After collecting the stack traces, you can collect the interesting stack traces, which has been thrown only in the test cases generated by Cling, by executing the following python script:

python scripts/python/captured-exceptions/analyze-stack-traces.py 

docker

docker exec -it cling-application-container python scripts/python/captured-exceptions/analyze-stack-traces.py  

The interesting stack traces will be saved in data/rq3/stack-traces/interesting. The current content of this directory is the interesting stack traces that we have found during our study.

Manual Analysis

We have analyzed all of the interesting stack traces to find the failure and non-failure stack traces. The achieved results in data_analysis/manual-analysis. Moreover, the full explanations related to stack traces stemming from failure are available in data_analysis/manual-analysis/failure-explanation.md.