[Bug]: KafkaIO does not make use of Kafka Consumer Groups

[srfrnk]

What happened?

I'm using KafkaIO to consume events from a Kafka topic.
I've added "group.id" to the consumer properties.
When running the pipeline I can see this value sent to Kafka in the consumer properties.
The consumers created by KafkaIO fail to join the consumer group.
Looking into the code I can see that no where is the consumer "subscribing" to the topic which is how KafkaConsumer should join a consumer group. It seems the code attempts to circumvent the partition assignment mechanism provided by Kafka to use it's own.
By doing that it prevents the user from using consumer groups.
Is that by intention? Is there any reason why the decision to avoid using consumer groups has been taken?
I would love to see any documentation about that if possible please.

Issue Priority

Priority: 2 (default / most bugs should be filed as P2)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[aromanenko-dev]

Thanks for reporting but it looks more as a question for [email protected] or [email protected]
Would you mind to start a discussion on this topic there?

[srfrnk]

Thanks @aromanenko-dev.
I wasn't aware of these.
Happy to do so.
Can you send me a link to any documentation about how to create such a discussion please?

[aromanenko-dev]

@srfrnk Sure!
https://beam.apache.org/community/contact-us/

[johnjcasey]

Closing as this has moved to the user lists

[kschat]

For those who stumbled upon this thread (like me), you can find the continued conversation here.

[gabrywu]

It is a pity that beam doesn't use consumer group, without which, it will be inconvenient for us

[srfrnk]

@gabrywu if you'd like you can clone my original attempt to rewrite that part to use consumer groups.
It's partially limited and needs some cleaning up but as a POC it was working.
You can find it here: https://github.com/srfrnk/demo/blob/cg-reader/beam-kafka2file/src/main/java/kafka/Read.java