Add feature flags for each protocol in shotover #1441
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This improves compile times of windsock without compromising on performance like #1438 ended up doing.
Previously
cargo windsock
took 2m 45s to compile after a change to our windsock benches.Now it takes:
cargo windsock-redis
- 1 min 32scargo windsock-kafka
- 1 min 21scargo windsock-cassandra
- 2 min 23scargo windsock
- 2m 45sWe would be getting similar improvements to our integration tests but we rarely need to run them in release so I didnt bother adding cargo aliases for them or measuring them.
Additionally, this PR improves compile times of shotover when developing custom transforms that only need to support a single protocol by allowing developers to disable unneeded dependencies within shotover, this will be quite helpful for our internal kafka project where we are also struggling with build times and have no need for any protocol other than kafka.
As a result of compiling less, the binary size with just kafka enabled is now 13.9MB, while the standard build with everything enabled is 19MB.
Additionally, additionally, this will be quite helpful for investigating the cause of build time issues in the future as we can easily eliminate certain protocols from the build.
Updates:
Reduced cfgs
With the help of #1442 #1443 #1444 and #1445 I have reduced the number of cfgs from 283 down to 181
Improvements for custom transform devs
Testing on our internal kafka-specific project:
As a result with this PR we will cut 4 minutes off the time to run CI as we have no caching setup for our CI on that project.
We will also cut 2 mins 30s from our iterative benchmark workflow which is super important to the project as its very performance sensitive.