Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add python-kafka integration test #1782

Merged
merged 1 commit into from
Oct 22, 2024
Merged

Conversation

rukai
Copy link
Member

@rukai rukai commented Oct 22, 2024

This PR introduces kafka integration tests using the kafka-python-ng driver, following a similar approach as #1690

This includes:

  • a uv python project which runs runs a simple smoke test against a kafka cluster.
  • a rust helper ensure_uv_is_installed to install uv to shotover's target directory
  • a rust helper run_python_smoke_test that calls ensure_uv_is_installed and then runs the uv project, passing in required configuration.
  • an integration test which calls run_python_smoke_test

uv was chosen as it drastically simplifies python package management and is easy to install.
Compare the current implementation to https://github.com/shotover/shotover-proxy/pull/1267/files where we had to manually wrangle venvs in order to create an isolated and reproducible python environment. uv handles all of that for us and is a lot faster.

Time to run

  • First run of the test completes in 14s due to installing uv
  • On reruns the test completes in 6s

Future work

This sets us up to write an integration test for the python driver that uses sasl scram which is known to be broken in shotover.
However I've left that out of this initial PR to keep scope down.

Copy link

codspeed-hq bot commented Oct 22, 2024

CodSpeed Performance Report

Merging #1782 will not alter performance

Comparing rukai:kafka_python_test (51f91ad) with main (61b5a98)

Summary

✅ 38 untouched benchmarks

@rukai rukai force-pushed the kafka_python_test branch 3 times, most recently from 7cdf9c5 to 1c177b5 Compare October 22, 2024 02:23
@rukai rukai force-pushed the kafka_python_test branch from 1c177b5 to 51f91ad Compare October 22, 2024 02:25
@rukai rukai marked this pull request as ready for review October 22, 2024 03:06
@conorbros conorbros merged commit 9930709 into shotover:main Oct 22, 2024
41 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants