Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add num of tries while pushing to kafka #86

Open
vesely-david opened this issue Aug 3, 2021 · 0 comments
Open

Add num of tries while pushing to kafka #86

vesely-david opened this issue Aug 3, 2021 · 0 comments

Comments

@vesely-david
Copy link
Collaborator

As of now, we are pushing to Kafka via an infinite loop, which might cause problems in the future.
We should consider defining "number of tries" after which exception will be raised.

py2k/py2k/producer.py

Lines 24 to 39 in dfe8d5e

def produce(self, record):
while True:
try:
self._producer.produce(
topic=self._topic,
key=record.key_to_avro_dict(),
value=record.value_to_avro_dict(),
on_delivery=self._delivery_report
)
self._producer.poll(0)
break
except BufferError as e:
print(
f'Failed to send on attempt {record}. '
f'Error received {str(e)}')
self._producer.poll(1)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant