integrate with latest kafka_protocol

[zmstone]

brod is still using {kafka_protocol, "2.3.6"}
kafka_protocol 3.0 introduced non-backward compatible changes due to changes made in Kafka 2.0 schemas.
see all the API and data structure change details here:
https://github.com/kafka4beam/kafka_protocol/pull/76/files#diff-3bd14d078188074c410028847113ceae68865d0ad5b844a27183ef87fbe2fcc3

Worth noting that this part of the changelog.md is generated from a script which compares old and new schema

Ideally we should be able to upgrade to kafka_protocol 4.0 without breaking brod APIs compatibility

[tpitale]

What would you say is the minimal change to brod to support the latest kafka_protocol, just updating the fields on the data structure?

[zmstone]

Yes.
Also try to use the latest API versions (at least for the most critical APIs: fetch and produce)

[tpitale]

Great, thanks! I will push up a draft PR soon (tomorrow).

[tpitale]

There's one or two changes in the tests that have got me stuck. I'm still working on getting them all to pass. Could I push up the draft PR and get some feedback from you?

[zmstone]

sure.

[tpitale]

I pushed up a PR #459 But having some challenges with snappyer compress.