mETRICS - rEproducible sofTware peRformance analysIs in perfeCt Simplicity

Authors

• Thibault Falque - Exakis Nelite
• Romain Wallon - CRIL, Univ Artois & CNRS
• Hugues Wattez - Laboratoire d'Informatique de l'X (LIX), École Polytechnique

About Metrics

Metrics is an open-source Python library developed at CRIL, designed to facilitate the conduction of experiments and their analysis.

The main objective of Metrics is to provide a complete toolchain from the execution of software programs to the analysis of their performance. In particular, the development of Metrics started with the observation that, in the SAT community, the process of experimenting solver remains mostly the same: everybody collects almost the same statistics about the solver execution. However, there are probably as many scripts as researchers in the domain for retrieving experimental data and drawing figures. There is thus clearly a need for a tool that unifies and makes easier the analysis of solver experiments.

The ambition of Metrics is thus to simplify the retrieval of experimental data from many different kinds of inputs (including the solver's output), and provide a nice interface for drawing commonly used plots, computing statistics about the execution of the solver, and effortlessly organizing them. In the end, the main purpose of Metrics is to favor the sharing and reproducibility of experimental results and their analysis.

Installation

To execute Metrics on your computer, you first need to install Python (at least version 3.8).

You may install Metrics using pip, as the metrics library is available on PyPI.

pip install crillab-metrics

Note that, depending on your Python installation, you may need to use pip3 to install it, or to execute pip as a module, as follows.

python3 -m pip install crillab-metrics

To improve the reproducibility of the experiments, we highly recommend to use a virtual environment for each analysis you create with Metrics, and thus to install the metrics library in this virtual environment rather than with a system-wide installation.

Using Metrics

You may find more information on how to use Metrics in the documentation we provide for the package.

Citing Metrics

If you are using Metrics in your papers, we kindly ask you to either refer to this repository or to one of the following papers:

• Metrics : Mission Expérimentations. Thibault Falque, Romain Wallon and Hugues Wattez. 16es Journées Francophones de Programmation par Contraintes (JFPC'21), 2021.
• Metrics: Towards a Unified Library for Experimenting Solvers. Thibault Falque, Romain Wallon and Hugues Wattez. 11th International Workshop on Pragmatics of SAT (POS'20), 2020.