Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chain microbenchmarks: run the chains once before benching to avoid measuring init time #1668

Merged
merged 2 commits into from
Jun 18, 2024

Conversation

rukai
Copy link
Member

@rukai rukai commented Jun 16, 2024

While assisting with #1649 I noticed that our chain benches are slightly inaccurate in that they run the against a fresh transform chain every time.
In a more realistic scenario the chain is reused across requests and the transforms can cache values internally after processing requests that can be reused on the following request.

Due to the way criterion works we cant reuse the chain from the previous iteration of the bench.
So the best way to solve this is to send an unmeasured request batch through the chain during setup.
This ensures that each iteration of the bench will operate on a chain which has had exactly one request batch pass through it already.

For the query_counter bench it will be useful to measure in both ways, so I've split it into query_counter_fresh and query_counter_pre_used. Which use a fresh chain and a pre used chain respectively.

For all other existing benches I converted them to use the pre used chain since that is most realistic.

@rukai rukai force-pushed the chain_benches_prerun branch from c119a68 to 2aff88f Compare June 16, 2024 23:36
Copy link

codspeed-hq bot commented Jun 16, 2024

CodSpeed Performance Report

Merging #1668 will not alter performance

Comparing rukai:chain_benches_prerun (2f5f934) with main (911e8a0)

Summary

✅ 37 untouched benchmarks

🆕 2 new benchmarks
⁉️ 1 dropped benchmarks

⚠️ Please fix the performance issues or acknowledge them on CodSpeed.

Benchmarks breakdown

Benchmark main rukai:chain_benches_prerun Change
⁉️ query_counter 13.8 µs N/A N/A
🆕 query_counter_fresh N/A 14.1 µs N/A
🆕 query_counter_pre_used N/A 14 µs N/A

@rukai rukai force-pushed the chain_benches_prerun branch from 2aff88f to 2281316 Compare June 17, 2024 02:17
@rukai rukai marked this pull request as ready for review June 17, 2024 02:55
@rukai rukai enabled auto-merge (squash) June 18, 2024 00:44
@rukai rukai merged commit b1fabdd into shotover:main Jun 18, 2024
40 of 41 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants