Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Aren't writes acknowledged too early due to the batching mechanism? #24

Open
arianvp opened this issue Nov 4, 2024 · 0 comments
Open

Comments

@arianvp
Copy link

arianvp commented Nov 4, 2024

We seem to call send_feedback imemdiately after self.writer.put_message

https://github.com/surbas/pg2kinesis/blob/master/pg2kinesis/__main__.py#L103

However put_message aggregates records into batches. This means that we might lose access to LSNs that have not been written to Kinesis yet if we crash before writing or during writing the batch.

I think we should only send feedback after the entire batch has been written?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant