Add feature to request consumer group information

[Ledostuff]

Add feature to request consumer group information(start, end offsets, lags, consumer-instance-id, consumer-instance-ip)
Added rest paths:

1. /groups - returns consumer groups list

[{"groupId":"perf-consumer-46894","coordinator":{"host":"{some_host_name}","port":9496}}]

2. /groups/{groupId}/topics - returns topics for wich group is subscribed

[{"name":"1"}]

3. /groups/{groupId}/topics/{topic} - returns consumer group information(start, end offsets, lags, consumer-instance-id, consumer-instance-ip...) restricted by spesified topic

{"topicPartitionList":[{"consumerId":"consumer-1-88792db6-99a2-4064-aad2-38be12b32e88","consumerIp":"/{some_ip}","topicName":"1","partitionId":0,"currentOffset":15338734,"lag":113812,"endOffset":15452546},{"consumerId":"consumer-1-88792db6-99a2-4064-aad2-38be12b32e88","consumerIp":"/{some_ip}","topicName":"1","partitionId":1,"currentOffset":15753823,"lag":136160,"endOffset":15889983},{"consumerId":"consumer-1-88792db6-99a2-4064-aad2-38be12b32e88","consumerIp":"/{some_ip}","topicName":"1","partitionId":2,"currentOffset":15649419,"lag":133052,"endOffset":15782471}],"topicPartitionCount":3,"coordinator":{"host":"{coordinator_host_name}","port":9496}}

4. /groups/{groupId}/partitions - returns consumer group information(start, end offsets, lags, consumer-instance-id, consumer-instance-ip...) for all topics for wich group is subscribed

{"topicPartitionList":[{"consumerId":"consumer-1-88792db6-99a2-4064-aad2-38be12b32e88","consumerIp":"/{some_ip}","topicName":"1","partitionId":0,"currentOffset":15338734,"lag":113812,"endOffset":15452546},{"consumerId":"consumer-1-88792db6-99a2-4064-aad2-38be12b32e88","consumerIp":"/{some_ip}","topicName":"1","partitionId":1,"currentOffset":15753823,"lag":136160,"endOffset":15889983},{"consumerId":"consumer-1-88792db6-99a2-4064-aad2-38be12b32e88","consumerIp":"/{some_ip}","topicName":"1","partitionId":2,"currentOffset":15649419,"lag":133052,"endOffset":15782471}],"topicPartitionCount":3,"coordinator":{"host":"{coordinator_host_name}","port":9496}}

Add unit and integration tests

[ghost]

It looks like @Ledostuff hasn't signed our Contributor License Agreement, yet.

The purpose of a CLA is to ensure that the guardian of a project's outputs has the necessary ownership or grants of rights over all contributions to allow them to distribute under the chosen licence.
Wikipedia

You can read and sign our full Contributor License Agreement here.

Once you've signed reply with [clabot:check] to prove it.

Appreciation of efforts,

clabot

[Ledostuff]

[clabot:check]

[ghost]

@confluentinc It looks like @Ledostuff just signed our Contributor License Agreement. 👍

Always at your service,

clabot

[gAmUssA]

@Ledostuff PR build is failing. Could you check?

[cmccabe]

The build failed:

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.6.1:testCompile (default-testCompile) on project kafka-rest: Compilation failure
[ERROR] /home/jenkins/workspace/entinc-pr_kafka-rest_PR-435-42EK4ZKFX7WUDVKYQNLWCJBUA3J5IBRM4CNHID3FE32PTTTFMBRQ/kafka-rest/src/test/java/io/confluent/kafkarest/TestUtils.java:[263,34] cannot find symbol
[ERROR]   symbol:   method createConsumerProperties(java.lang.String,java.lang.String,java.lang.String,int)
[ERROR]   location: class kafka.utils.TestUtils
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <goals> -rf :kafka-rest

[cmccabe]

This PR shouldn't need to depend on ZooKeeper. We should be able to use the public AdminClient.

What's the security model here? How do we decide who can and cannot access admin APIs?

[Ledostuff]

@gAmUssA Thank you for review. I`ll fix build and push changes asap.

[Ledostuff]

@cmccabe Thank you for review.
I`ll fix build and push changes asap.
Zookeeper needs to get broker information. I mean security protocol in EndPoint class.
Do you mean I should move method from AdminClientWrapper to another place?
Could you explain in detail your sentence about security model?

[cmccabe]

Zookeeper needs to get broker information. I mean security protocol in EndPoint class.

Why is this information needed for REST proxy? If it is needed, then we should at least think about providing it through the AdminClient.

Could you explain in detail your sentence about security model?

This PR seems to give rest-proxy users a lot of powerful abilities to access administrative APIs. I'm not sure that rest-proxy is the right place to manage the Kafka cluster.

If we did want to do this we would need to figure out how to give certain users the ability to use the admin APIs and block everyone else. For example, users should not be able to list groups unless they have DESCRIBE permission on the group, or DESCRIBE permission on the Cluster resource.

To be honest, I'm not sure this is the right direction to go-- at least not without a lot more discussion.

[Ledostuff]

@cmccabe Thank you for the answer.

This PR seems to give rest-proxy users a lot of powerful abilities to access administrative APIs. I'm not sure that rest-proxy is the right place to manage the Kafka cluster.

We are only show information about groups who consumes messges from clusters topics. We don`t try to change settings or do any other work with cluster.

If we did want to do this we would need to figure out how to give certain users the ability to use the admin APIs and block everyone else. For example, users should not be able to list groups unless they have DESCRIBE permission on the group, or DESCRIBE permission on the Cluster resource.

I don't see any problem about that. Because we allow users to consume messages through the rest-proxy. But why do we can't show information about consumer groups? Even if we allow users DESCRIBE permission on groups or DESCRIBE permission on the Cluster resource .
I'm open for discussion this PR)

Why is this information needed for REST proxy? If it is needed, then we should at least think about providing it through the AdminClient.

I didn't found similar method in the API of AdminClient. If you show me alternative method to get information about brokers in cluster with security protocol, I'll repair code in this PR gladly.)

[stanislavkozlovski]

Hey @Ledostuff, thanks for the PR!

I haven't had time to give this a proper look yet but I wanted to ask - do we need ZooKeeper to access the cluster information only ( {"brokerId":0,"endPointList":[{"protocol":"SSL","host":"127.0.0.1","port":9092}]}) ?
In regards to that, I feel it would be best if we do not re-introduce a ZooKeeper dependancy in the proxy.

The consumer information should be fetchable through the adminClient only, correct?
To me, it makes sense to expose consumer information. I think endpoints 2., 3., 4. and 5. will be useful.

There is a question of whether that information should be limited to consumers managed by this proxy instance or consumers in the Kafka Cluster.
As @cmccabe said, permissions are a valid concern:

For example, users should not be able to list groups unless they have DESCRIBE permission on the group, or DESCRIBE permission on the Cluster resource.
As far as I know, in the current model we allow a consumer to be created and join any arbitrary group present in the Kafka cluster. That means exposing consumer group information for the whole cluster would be consistent with the security model already present.

[rootex-]

Hi @stanislavkozlovski,

"in the current model we allow a consumer to be created and join any arbitrary group present"

No, we don't. This is only valid for the case where authentication and acl's are off OR we elevate kafka-rest principal's resource permissions like READ/DESCRIBE on cluster/group/topic, depending on the operation type we want. We are free to choose which ACLs to give to kafka-rest's principal, should consumer groups be exposed or not. That is, users are free to choose their own security model.

users should not be able to list groups unless they have DESCRIBE permission on the group, or DESCRIBE permission on the Cluster resource.

That's right, unless we provide DESCRIBE permission on the Cluster resource.

Viewing consumer group offsets or listing consumer groups is just a feature that can be enabled or disabled, according to a security model adapted by the user.

[stanislavkozlovski]

Hi @rootex-,

That's true. I agree with exposing consumer group information.

[analytik]

Not to be annoying, but... is there still interest in this? I'd welcome having a list of consumer groups available via REST proxy.

[rootex-]

Not to be annoying, but... is there still interest in this? I'd welcome having a list of consumer groups available via REST proxy.

Yes, there is still a huge interest in this feature.
So much useful work done and no one have ever reviewed this stuff.
We are also sick and tired of endless patching Kafka Rest in Sberbank.

Could we merge this pull request?

[Ledostuff]

Hello there! Please review this PR. Thanks)

[Ledostuff]

Hi @stanislavkozlovski,
Please, review my changes. Sorry for the delay)

[Ledostuff]

Hello, @stanislavkozlovski :)
Review my changes, please.
Check my suggestions about your last comment.

[Ledostuff]

Hellooo. One month has already passed since last comment. PLease, review this pull-request.

[cla-assistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}