How do you guys will get & update CDC offset for stoppable Flink jobs?

[cloorc]

In my scenario, we might pause or stop our flink jobs due to fixed maintainence. But we don't want to lose the offset of flink job sources.

For example, we are using flink to incrementally synchronize data from DB2 instance to MySQL cluster. We specified the startup offset(supposed A) the first time we submitted our flink job. However, 3 months later we need to restart the job but we do not want to figure out the right offset(might B, and the latest offset most probably C). How should I make it unattended?

Could we just give the CDC connector a startup offset and specify a distributed storage(such as redis) to save the right offset for future job recover? If so, will this conflict with checkpoints saved automatically?

[cloorc]

After digging into org.apache.flink.cdc.debezium.internal.FlinkOffsetBackingStore#OFFSET_STATE_VALUE, it is easy to figure out that Db2Source.Builder#build() will build an CheckpointedFunction that will save offset automatically.

I guess this is the answer, checkpoint should always be first-class citizen.