Additional information from Kafka output logging

[hartfordfive]

Logstash version

6.1.1 (with X-Pack)

OS

CentOS 7.3

Description

I sometimes encounter issues with some of my pipelines where the kafka output is unable to publish messages to Kafka. For example, when the failures start occurring, messages such as the following start to flood the log file:

[2018-07-16T08:48:35,632][INFO ][logstash.outputs.kafka   ] Sending batch to Kafka failed. Will retry after a delay. {:batch_size=>2, :failures=>2, :sleep=>0.01}

My issue right now is that there is no indication of which logstash kafka output is failing in this case. Including the pipeline ID and kafka output ID (as well as any other potentially useful details) in the error message would definitely help in the debugging process. Would it be able to add these to the standard log message of the kafka output?

[jsvd]

please reopen this issue in the kafka output plugin repository: https://github.com/logstash-plugins/logstash-output-kafka

[hartfordfive]

Done.
logstash-plugins/logstash-output-kafka#201