Skip to content

Commit

Permalink
better understanding
Browse files Browse the repository at this point in the history
  • Loading branch information
rukai committed Nov 18, 2024
1 parent 5ac944d commit 19ff474
Showing 1 changed file with 5 additions and 15 deletions.
20 changes: 5 additions & 15 deletions shotover/src/transforms/kafka/sink_cluster/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -3085,21 +3085,11 @@ The connection to the client has been closed."

// List of keys that shotover doesnt support and so should be removed from supported keys list
let disable_keys = [
// This message type has very little documentation available but it seems that due to the way the pagination
// functionality is designed, there is no straightforward way to implement this message type in shotover.
// I reccomend we avoid implementing this message type for as long as possible.
//
// The problem is that the cursor spans across topics, but each partition in the request needs to be split up and routed to the owner of that partition.
// So shotover is going to need to split the cursor across different brokers which is ill-defined.
//
// A possible solution could be something like:
// 1. split and route request to brokers by location of partition and remove the cursor from the request.
// 2. Store the result in a cache.
// 3. Apply our own cursor logic upon the data in the cache.
// 4. On follow up requests make use the existing cache to respond with the next section of the paginated data.
//
// But this will be quite difficult to implement and makes pagination incredibly inefficient which is the opposite of the intention of pagination.
// So for now, lets not.
// This message type has very little documentation available and kafka responds to it with an error code 35 UNSUPPORTED_VERSION
// So its not clear at all how to implement this and its not even possible to test it.
// Instead lets just ask the client to not send it at all.
// We can consider supporting it when kafka itself starts to support it but we will need to be very
// careful to correctly implement the pagination/cursor logic.
ApiKey::DescribeTopicPartitionsKey as i16,
// This message type is part of the new consumer group API, we should implement support for it in the future.
// I've disabled it for now to keep the scope down for kafka 3.9 support.
Expand Down

0 comments on commit 19ff474

Please sign in to comment.