Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Plugin to provide better error logging #172

Open
Jymit opened this issue Jan 11, 2018 · 0 comments
Open

Plugin to provide better error logging #172

Jymit opened this issue Jan 11, 2018 · 0 comments

Comments

@Jymit
Copy link

Jymit commented Jan 11, 2018

Logstash v5.6.4
Kafka version : 0.10.0.1

Ingestion coming from a variety of log sources like beats and syslog and sending to Kafka.

The INFO level log lines do not give a lot of detail into what is causing the failure.

2018-01-09T09:31:46,097][INFO ][logstash.outputs.kafka ] Sending batch to Kafka failed. Will retry after a delay. {:batch_size=>1, :failures=>1, :sleep=>0.01} 
[2018-01-09T09:31:46,113][INFO ][logstash.outputs.kafka ] Sending batch to Kafka failed. Will retry after a delay. {:batch_size=>1, :failures=>1, :sleep=>0.01} 
[2018-01-09T09:31:46,129][INFO ][logstash.outputs.kafka ] Sending batch to Kafka failed. Will retry after a delay. {:batch_size=>1, :failures=>1, :sleep=>0.01}

Once DEBUG logging is enabled then we see the particular:

[2018-01-09T15:58:26,995][DEBUG][logstash.outputs.kafka  ] KafkaProducer.send() failed: org.apache.kafka.common.errors.RecordTooLargeException: The message is
10486822 bytes when serialized which is larger than the maximum request size you have configured with the max.request.size configuration. {:exception=>java.util
.concurrent.ExecutionException: org.apache.kafka.common.errors.RecordTooLargeException: The message is 10486822 bytes when serialized which is larger than the maximum request size you have configured with the max.request.size configuration.}
[2018-01-09T15:58:26,997][INFO ][logstash.outputs.kafka  ] Sending batch to Kafka failed. Will retry after a delay. {:batch_size=>1, :failures=>1, :sleep=>0.01

Is it possible to have better error logging with this plugin?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant