chain microbenchmarks: run the chains once before benching to avoid measuring init time #1668
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
While assisting with #1649 I noticed that our chain benches are slightly inaccurate in that they run the against a fresh transform chain every time.
In a more realistic scenario the chain is reused across requests and the transforms can cache values internally after processing requests that can be reused on the following request.
Due to the way criterion works we cant reuse the chain from the previous iteration of the bench.
So the best way to solve this is to send an unmeasured request batch through the chain during setup.
This ensures that each iteration of the bench will operate on a chain which has had exactly one request batch pass through it already.
For the query_counter bench it will be useful to measure in both ways, so I've split it into
query_counter_fresh
andquery_counter_pre_used
. Which use a fresh chain and a pre used chain respectively.For all other existing benches I converted them to use the pre used chain since that is most realistic.