ApiVersions rewrite unit test

[rukai]

Background on ApiVersions

The ApiVersions response is used by the broker to describe which message types and which versions of those message types are supported by the broker.
This allows clients to make use of new functionalities where possible but fallback to more basic function if the new message versions are not supported by the broker.

The problem

Consider the following scenario:

• Broker supports message type Foo V2
• Client supports message type Foo only up to V1
• Shotover can parse Foo V1 and Foo V2 via kafka-protocol crate.
  • However, shotover's rewrite or routing logic for Foo is only compatible with Foo v1. Foo V2 has a new field that needs to be taken into account but since the client currently only supports Foo V1 we have no way to observe this incompatibility in an integration test.

A new version of the client is then released that makes use of Foo v2.
A user begins using this new client before shotover has a chance to discover this issue.
Kafka/shotover breaks in strange ways and everyone is sad.

Thankfully kafka gives us the tools to prevent this from happening, via the ApiVersions message type.

Solution

We can rewrite the ApiVersions response to squeeze the version values into something that shotover is compatible with.
This means:

• we take the max version supported by the broker and lower it to what shotover is compatible with. (But never increase it, we need to remain compatible with the broker)
• we take the min version supported by the broker and increase it to what shotover is compatible with. (But never decrease it, we need to remain compatible with the broker)

Note that I'm not aware of kafka itself or shotover ever setting a min value other than 0.
But we should be prepared for kafka one day increasing a minimum value and handle it assuming they might.

This solution is already merged in another PR

Another problem

However this introduces another problem.
Its now very easy for shotover to fall behind on supported messages.
Keeping track of the 81 (and growing) message types and all of their versions is a non-trival task.
The consequences of this are shotover missing new features and efficiencies gained from the new message versions, that it would have been able to support with no changes if it werent for the ApiVersions response being rewritten.

Solving "Another problem"

To make this problem more tractable I have introduced a unit test that asserts the diff between the message types supported by shotover and the message types supported by the kafka-protocol crate.
kafka-protocol crate is generated from the JSON message type definitions in the upstream kafka repository.

So assuming that:

• kafka-protocol is regenerated on recent JSONs every now and then
• and we keep shotover up to date with the new releases of kafka-protocol

We should receive notifications of new message types and versions in the form of a failing unit test when updating kafka-protocol crate.

This PR implements this solution

PR Status

The unittest relies on some changes landing upstream in kafka-protocol crate.
So I will land the implementation separately and leave this as a draft for now.

See:

• ApiKey additions tychedelia/kafka-protocol-rs#98
• VersionRange additions tychedelia/kafka-protocol-rs#99

kafka-protocol 0.14 is released so this PR can go ahead now

[codspeed-hq]

CodSpeed Performance Report

Merging #1832 will not alter performance

_{Comparing rukai:api_versions_rewrite (f48889a) with main (d754774)}

Summary

✅ 38 untouched benchmarks