Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add cpa and snr benchmarks #5

Merged
merged 5 commits into from
Mar 29, 2024
Merged

Conversation

TrAyZeN
Copy link
Member

@TrAyZeN TrAyZeN commented Mar 26, 2024

No description provided.

@kingofpayne
Copy link
Contributor

When testing (on my computer), criterion prints many warnings during the benchmark:
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 22.7s, or reduce sample count to 20.
Can you configure it to increase that time limit ? (I would suggest 60 seconds)

Also when running it twice, many benches show regression or improvement above noise margin; this is probably because not enough samples are taken due to the time limit.

@TrAyZeN
Copy link
Member Author

TrAyZeN commented Mar 27, 2024

The measurement time if too low is automatically increased1. Thus, in this case criterion will record 100 samples taking more than 5s. criterion performs measurements by sample, but a sample can contain multiple iterations of the function2. From my understanding what changes if the measurement time is increased is the number of iterations per sample. In the case where the measurement time is too low, criterion with record 100 samples with 1 iterations. Increasing the measurement time would allow criterion to perform more iterations per samples. Increasing the measurement time seems indeed to yield more robust results.

To reduce benchmark variance, it is advised to disable CPU frequency scaling and to set task affinity3.

Footnotes

  1. https://docs.rs/criterion/0.5.1/criterion/struct.Criterion.html#method.measurement_time

  2. https://bheisler.github.io/criterion.rs/book/analysis.html#measurement

  3. https://google.github.io/benchmark/reducing_variance.html

@TrAyZeN TrAyZeN force-pushed the benchmark branch 2 times, most recently from 72022b3 to b60a141 Compare March 28, 2024 14:58
@TrAyZeN TrAyZeN marked this pull request as ready for review March 29, 2024 10:20
@kingofpayne kingofpayne merged commit 7a4a0d4 into Ledger-Donjon:master Mar 29, 2024
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants