Seek to beginning or reset offset to 0 to reconsume a Kafka topic

[scspijker]

When we make a boo boo while consuming topics, sometimes we want to re-consume the whole topic.
SmallRye Reactive Messaging does not support this (at least on Kafka?).

Our current approach is either:
Implement a rebalance listener and maintain the offset and partitions ourselves (undesirable for such a small request)
Change the consumer group and delete the old one (undesirable as we are on a managed cluster and need to ask another team to clean up consumer groups, also this makes for ugly consumer group ID’s).

Spring Kafka recently implemented seekToBeginning: https://docs.spring.io/spring-kafka/docs/2.6.10/api/org/springframework/kafka/listener/ConsumerSeekAware.ConsumerSeekCallback.html#seekToBeginning(java.lang.String,int)

We would love for SmallRye’s Reactive Messaging to support this as well, since this is the preferred consumer for Quarkus.

[ozangunalp]

A couple of questions:

• We are talking about topic subscription, not manual partition assignment right?
• Do you want to seek to particular offset at runtime, while consuming records or on the first rebalance? Calling seekTo* on the consumer without a rebalance is a bit tricky, it is doable but you need to accept to loose some messages.
• And why would you need to change the consumer group, because an offset is potentially already committed, then you also need to clean the committed offset when using seek?

[scspijker]

Thanks for your reply :)

• Yes
• At runtime, the use case is a restore of our data storage, so we detect that our data store has been restored and we want to restart consumption at the offset the data store is (the offset is also stores in the data store).
• We do not need to change the consumer group, this is just our current approach to completely re-consume a topic.

So the use-case:

• Have a consumer topic subscription
• Save the offset in your datastore
• Restore a backup of the datastore to an earlier version
• Detect non-matching offset of the datastore
• Seek to the offset that the datastore has to restart consumption at that point and re-fill datastores.