Kafka compatible client API

[AshleySchaeffer]

Hello!

Firstly, Iggy looks great. Well played!

I've (sadly) spent quite a bit of time looking at message queues and streaming platforms like this and what I can't get away from is the Kafka client API. Because of it's widespread adoption there are countless projects and tools that are compatible with it.

I've not done too much of a deep dive into the code base yet, but it would be good to understand what the appetite for a Kafka compatible client API would be? Based purely on the features list, it implies that Iggy uses a similar internal architecture so I question whether it'd require much work beyond the API itself.

Besides tooling, iggy would benefit from battle tested client libraries in numerous languages, while also being entirely familiar with developers. If a Kafka client API worked perfectly, it could even be a drop in replacement.

[spetz]

Hey,

Once again, thank you for the kind words and the interesting idea. While we already had a discussion last week on our Discord, I'd also like to sum up a few things in this topic.

Kafka is an extremely popular, mature and reliable tool, thus it could be really helpful and useful for all the developers out there already using Kafka, to provide a client compatible API on the Iggy side.

I think that it wouldn't be too difficult (to be verified of course) to implement the most important methods being sending and polling the messages, as some of the concepts, patterns as well as the underlying data storage are quite similar. However, the Kafka API is much bigger than these two, and it's hard to say at this point if it would be possible to provide the seamless integration with all the available features - certainly, we wouldn't like the developers to experience some undefined behavior or weird errors being returned on our side, since we couldn't provide a 100% compatible SDK.

I'd like this to remain an open topic and encourage anyone willing to help with this subject (research, ideas, PoC etc.) - I would be more than happy to support you with this task :)