Skip to content

Commit

Permalink
Fix benchmark instructions: differentiate tasty-bench & gauge options…
Browse files Browse the repository at this point in the history
…. [skip ci]
  • Loading branch information
wismill committed Sep 20, 2022
1 parent 37d7d03 commit c34cf20
Showing 1 changed file with 55 additions and 2 deletions.
57 changes: 55 additions & 2 deletions benchmark/README.md
Original file line number Diff line number Diff line change
@@ -1,13 +1,55 @@
# Benchmark library choice

You may choose between two libraries to perform the benchmarks:

- `tasty-bench`: default
- `gauge`: alternative, requires cabal flag `use-gauge`.

# Comparing benchmarks for regression

The following commands are tested with `cabal` version 3.0.

## `tasty-bench`

Run the benchmarks for the baseline code i.e. without the changes:

```
# Remove any old benchmark results file first
$ rm results.csv
$ cabal run bench -- --csv=old.csv
```

It will collect the benchmark results in `old.csv` file.

If you want more accurate benchmark results you can add `--stdev 1`
(default: 5).

You may generate a SVG chart by adding `--svg chart.svg`.

Modify the code and then run:

```
$ cabal run bench -- --csv=new.csv --baseline=old.csv
```

It will collect the new benchmark results in `new.csv` file.

To generate a benchmark comparison in CSV between old and new changes from the
benchmark results:

```
# Source: see “Comparison against baseline” in tasty-bench documentation.
awk 'BEGIN{FS=",";OFS=",";print "Name,Old,New,Ratio"}FNR==1{next}FNR==NR{a[$1]=$2;next}{print $1,a[$1],$2,$2/a[$1];gs+=log($2/a[$1]);gc++}END{print "Geometric mean,,",exp(gs/gc)}' old.csv new.csv
```

## `gauge`

Run the benchmarks for the baseline code i.e. without the changes:

```
# Remove any old benchmark results file first
$ rm results.csv
$ cabal run bench -- --csvraw=results.csv --quick
$ cabal run bench --flag use-gauge -- --csvraw=results.csv --quick
```

It will collect the benchmark results in `results.csv` file.
Expand Down Expand Up @@ -51,11 +93,22 @@ $ export LIBRARY_PATH=/usr/lib/:/opt/local/lib
$ cabal bench --extra-lib-dirs=/usr/local/opt/icu4c/lib --extra-include-dirs=/usr/local/opt/icu4c/include
```

## `tasty-bench`

Remove any old `results.csv` and run benchmarks with `has-icu` flag enabled:

```
$ rm results.csv
$ cabal run bench --flag has-icu -- --csv=results.csv
```

## `gauge`

Remove any old `results.csv` and run benchmarks with `has-icu` flag enabled:

```
$ rm results.csv
$ cabal run bench --flag has-icu -- --csvraw=results.csv --quick
$ cabal run bench -f use-gauge -f has-icu -- --csvraw=results.csv --quick
```

The following command will now show the comparison between `text-icu` and
Expand Down

0 comments on commit c34cf20

Please sign in to comment.