Split this plugin into one-per-kafka-version

[jordansissel]

It has been a trouble to try and support all the versions of kafka that people use. This is difficult because Kafka clients are incompatible with various kafka server versions.

The result is that we have 4 branches that seem to be maintained to each serve exactly one version of Kafka.

This hurts users because they have to perfectly match the version of our kafka plugin, using the version matrix in our documentation, with their kafka server. Further, for every single kafka release, we are asked by users nearly immediately for support for the new-and-incompatible release of kafka.

Given the incompatibilities between clients and servers and versions, it makes sense to me to treat each version of Kafka as a new protocol/product, and for any other thing (say, stomp and redis) we have independent plugins.

I propose the following:

• For every kafka version which introduces an incompatibility, create a new plugin named by that version of kafka. For example, "logstash-output-kafka-0.9" or "logstash-output-kafka-0.10.0"

My hope is this solves the following:

• We no longer need to maintain the kafka version matrix
• Users don't have to be burdened by that matrix
• It becomes clear what plugin works with what version of kafka.

If agreed, the next steps are as follows:

• Create a new plugin per supported-version of kafka: 0.8, 0.9, 0.10.0, 0.10.1, and maybe 0.10.2 (see Kafka 0.10.2.0 #127).
• Deprecate this plugin
• Do the same for the kafka input plugin.

[allenmchan]

I am not sure if this matters but as of 0.10.2, kafka supports better compatibility between clients and server. (https://cwiki.apache.org/confluence/display/KAFKA/KIP-97%3A+Improved+Kafka+Client+RPC+Compatibility+Policy)

With this change, kafka clients version newer than kafka server can communicate with newer features disabled. (forward compatibility)
We always had the ability for older kafka clients version to communicate to newer kafka servers. (backward compatibility)

So if client -> server communication is solved, it is all about kafka features. I think it is up to the user to figure out the features since they have to figure it out on the kafka side too. (ie which kafka server/client supports SASL)

With that said, I have questions about your proposal

1. What version of kafka plugin would be shipped with every new logstash release? Latest kafka client version available? Previous stable?
2. Can multiple versions of kafka plugins be installed together (ie one logstash instance with both "logstash-output-kafka-0.9.0" and "logstash-output-kafka-0.10.0" installed) and the config just call the client that is to be used?

[jordansissel]

I am aware of the compatibility policy that Kafka announced recently but this will take time to actually earn my trust. As for what version to include, we don't have to pick -- with my proposal, we could ship all supported versions because each Kafka version gets a dedicated plugin. I am open to also delaying or deferring this proposal. I am most interested in making Kafka integration a good user experience (and hopefully a nice maintainer experience) and today we don't have this -- I am open to ideas :)

…

On Fri, Mar 31, 2017 at 4:07 PM Allen Chan ***@***.***> wrote: I am not sure if this matters but as of 0.10.2, kafka supports better compatibility between clients and server. ( https://cwiki.apache.org/confluence/display/KAFKA/KIP-97%3A+Improved+Kafka+Client+RPC+Compatibility+Policy ) With this change, kafka clients version newer than kafka server can communicate with newer features disabled. (forward compatibility) We always had the ability for older kafka clients version to communicate to newer kafka servers. (backward compatibility) So if client -> server communication is solved, it is all about kafka features. I think it is up to the user to figure out the features since they have to figure it out on the kafka side too. (ie which kafka server/client supports SASL) With that said, I have questions about your proposal 1. What version of kafka plugin would be shipped with every new logstash release? Latest kafka client version available? Previous stable? 2. Can multiple versions of kafka plugins be installed together (ie one logstash instance with both "logstash-output-kafka-0.9.0" and "logstash-output-kafka-0.10.0" installed) and the config just call the client that is to be used? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#130 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAIC6iYcaCV0SfQTnxfAI7Tc5Bdteb88ks5rrYc-gaJpZM4MvK-P> .