[Bug]: Python KafkaIO read transform is inefficient when using the commit_offsets_in_finalize option

[chamikaramj]

What happened?

Seems like Python Kafka IO read transform requires significant more resources when using"commit_offsets_in_finalize" option.

'commit_offsets_in_finalize' adds an extra Reshuffle and a callback to Kafka to commit messages.

For streaming jobs, seems like this results in backlog of the Kafka read work item increasing by a large amount time to time and Dataflow jobs are unable to downscale due to this backlog.

Transform can operate efficiently when not using the "commit_offsets_in_finalize" option.

Issue Priority

Priority: 2 (default / most bugs should be filed as P2)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[chamikaramj]

Java pipelines may exhibit similar behaviors but I haven't tested that.

@johnjcasey is this something we can track as a part of the ongoing effort to improve the Kafka connector ?

[johnjcasey]

For sure. The reshuffle is essentially mandatory, but it is a side path, so it shouldn't hold up the main flow of the data

[Abacn]

related Java issue (commitOffsetsInFinalize): #20689

[scwhittle]

#31682 removes the reshuffle when commit offsets in finalized is enabled