#30903: Do not reemit data from impulse #30905
Conversation
je-ik
force-pushed
the
30903-do-not-reemit-data-from-impulse
branch
2 times, most recently
from
d6bdeea to
8ab8b75
Compare
|
Assigning reviewers. If you would like to opt out of this review, comment R: @Abacn for label build. Available commands:
The PR bot will only process comments in the main thread (not review comments). |
|
R: @JozoVilcek |
|
Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control |
je-ik
force-pushed
the
30903-do-not-reemit-data-from-impulse
branch
from
8ab8b75 to
af7089b
Compare
je-ik
force-pushed
the
30903-do-not-reemit-data-from-impulse
branch
2 times, most recently
from
86d0009 to
a41dcb0
Compare
| @@ -61,8 +55,6 @@ | |||
| */ | |||
| public class FlinkBoundedSourceReader<T> extends FlinkSourceReaderBase<T, WindowedValue<T>> { | |||
| private static final Logger LOG = LoggerFactory.getLogger(FlinkBoundedSourceReader.class); | |||
| private static final VarLongCoder LONG_CODER = VarLongCoder.of(); | |||
| private final Map<Integer, Long> consumedFromSplit = new HashMap<>(); | |||
There was a problem hiding this comment.
Can you please elaborate how does removing tracking of consumption from state helps to fix the bug, to not reemit elements from impulse source?
There was a problem hiding this comment.
This is related, because this was introduced in an attempt to fix exactly this issue of avoiding re-emission of elements from Impulse. Unfortunately, this is not working as expected and - more seriously - can cause data-loss for sources that are not 'stable' - e.g. NoSQL databases, where the data in a split can change between restarts, which might cause skipping an element that was always present (but changed its ordered position).
je-ik
force-pushed
the
30903-do-not-reemit-data-from-impulse
branch
from
a41dcb0 to
cfe7be6
Compare
Closes #30903
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, commentfixes #<ISSUE NUMBER>instead.CHANGES.mdwith noteworthy changes.See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)
See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.