pulsar connector produces message asynchronously #22026
Conversation
|
Can one of the admins verify this patch? |
4 similar comments
|
Can one of the admins verify this patch? |
|
Can one of the admins verify this patch? |
|
Can one of the admins verify this patch? |
|
Can one of the admins verify this patch? |
| @@ -45,12 +54,48 @@ public void setup() throws PulsarClientException { | |||
|
|
|||
| @ProcessElement | |||
| public void processElement(@Element byte[] messageToSend) throws Exception { | |||
| producer.send(messageToSend); | |||
| producer.sendAsync(messageToSend) | |||
There was a problem hiding this comment.
Is there any concern of data loss here? Typically Beam is supposed to handle any threading or async itself. I'm thinking of the case where async is delayed, and Beam thinks a given element is completed, but it hasn't actually been sent to pulsar.
There was a problem hiding this comment.
I agree with the concerns here, I've seen very troublesome async writers in Beam that were prone to loosing data. I suppose in this case it depends on the behavior of producer.flush() in finishBundle(). Does it block until all pending messages are send? If so, I'd suggest to add some comments to clarify.
|
Wondering, is there a lack of test coverage if you could change the producer from sync to async without having to change tests? |
|
This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the [email protected] list. Thank you for your contributions. |
|
This pull request has been closed due to lack of activity. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time. |
|
added this to [Parent issue] Support for Apache Pulsar #31078 |
Please add a meaningful description for your change here
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
R: @username).addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, commentfixes #<ISSUE NUMBER>instead.CHANGES.mdwith noteworthy changes.See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)
See CI.md for more information about GitHub Actions CI.