Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support kafka java driver 3.8 #1819

Merged
merged 1 commit into from
Nov 18, 2024

Conversation

rukai
Copy link
Member

@rukai rukai commented Nov 17, 2024

In order to reproduce the issue, this PR bumps the java driver used in integration tests from 3.7 to 3.8.
j4rs does not appear to support multiple versions of dependencies, so I think the best way to get coverage across different protocol versions is to use the java driver (well maintained and kept up to date with the latest kafka protocol) as the upper bound and the cpp driver as the lower bound (basic maintenance and uses far older versions of the kafka protocol)

The cause of the failures on 3.8 is that the driver is now sending out DescribeCluster requests which shotover does not yet support.
So this PR implements support for the DescribeCluster message type.
The implementation is very similar to the Metadata request. In fact the functionality provided by DescribeCluster is just a subset of that in Metadata. (The remaining functionality was split off into DescribeTopicPartitions)
Since they are so similar I was able to share some of the logic between both DescribeCluster and Metadata by adding a rewrite_controller_id method to handle the controller_id field rewriting.

Copy link

codspeed-hq bot commented Nov 18, 2024

CodSpeed Performance Report

Merging #1819 will not alter performance

Comparing rukai:support_kafka_java_driver_3.8 (18da0d4) with main (e50d528)

Summary

✅ 38 untouched benchmarks

@rukai rukai force-pushed the support_kafka_java_driver_3.8 branch from 72e6b4d to 3c6e542 Compare November 18, 2024 00:34
@rukai rukai force-pushed the support_kafka_java_driver_3.8 branch from 3c6e542 to 18da0d4 Compare November 18, 2024 02:03
@rukai rukai marked this pull request as ready for review November 18, 2024 02:03
@rukai rukai merged commit 8045ecd into shotover:main Nov 18, 2024
41 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants