ureplicator - JMX metrics - branch "kafka-0.8.2.1"

[mpkd]

Hello,

We were able to setup ureplicator for our kafka 0.8 --> confluent kafka 5.4 using branch kafka-0.8.2.1.

We are trying to setup the JMX metrics and facing some challenges.

1. controller JMX metrics is not getting. Also, metricsPrefix parameter is not accepting in the startup command.
2. worker JMX metrics is getting, but not all metrics mentioned in the jmxtrans.xml(jmxtrans.xml is not there in kafka-0.8.2.1 branch, we have taken it from master branch - https://github.com/uber/uReplicator/blob/master/config/jmxtrans.xml). Majority of the metrics are not getting in graphite.

• type=WorkerInstance metrics are not getting

Controller startup command

ureplicator_home="/opt/uReplicator"
exec "$JAVACMD" $JAVA_OPTS -Dapp_name=uReplicator-Controller \
  -Dlog4j.configuration=file:${ureplicator_home}/config/tools-log4j.properties -Xms4g -Xmx4g -Xmn1g -XX:NewSize=1g -XX:MaxNewSize=1g -XX:+AlwaysPreTouch -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+DisableExplicitGC -XX:+PrintCommandLineFlags -XX:CMSInitiatingOccupancyFraction=80 -XX:SurvivorRatio=2 -XX:+PrintTenuringDistribution -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime -XX:+PrintGCTimeStamps -Xloggc:/var/log/ureplicator/controller/gc-ureplicator-controller.log \
  -classpath "$CLASSPATH" \
  -Dapp.name="start-controller" \
  -Dapp.pid="$$" \
  -Dapp.repo="$REPO" \
  -Dapp.home="$BASEDIR" \
  -Dbasedir="$BASEDIR" \
  com.uber.stream.kafka.mirrormaker.starter.MirrorMakerStarter \
  startMirrorMakerController \
  -helixClusterName uReplicator \
  -mode customized \
  -zookeeper zk1.example.com:2181, zk2.example.com:2181, zk3.example.com:2181 \
  -port 9000 \
  -instanceId ureplicator-1-001 \
  -graphiteHost graphite-example.com \
  -graphitePort 2003 \
  "$@"

Worker startup command

ureplicator_home="/opt/uReplicator"
exec "$JAVACMD" $JAVA_OPTS -Dapp_name=uReplicator-Worker \
  -Dlog4j.configuration=file:${ureplicator_home}/config/tools-log4j.properties -Xms6g -Xmx6g -Xmn1g -XX:NewSize=1g -XX:MaxNewSize=1g -XX:+AlwaysPreTouch -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+DisableExplicitGC -XX:+PrintCommandLineFlags -XX:CMSInitiatingOccupancyFraction=80 -XX:SurvivorRatio=2 -XX:+PrintTenuringDistribution -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime -XX:+PrintGCTimeStamps -Xloggc:/var/log/ureplicator/worker/gc-ureplicator-worker.log -javaagent:${ureplicator_home}/bin/libs/jmxtrans-agent-1.2.6.jar=${ureplicator_home}/config/jmxtrans.xml \
  -classpath "$CLASSPATH" \
  -Dapp.name="start-worker" \
  -Dapp.pid="$$" \
  -Dapp.repo="$REPO" \
  -Dapp.home="$BASEDIR" \
  -Dbasedir="$BASEDIR" \
  com.uber.stream.kafka.mirrormaker.starter.MirrorMakerStarter \
  startMirrorMakerWorker kafka.mirrormaker.MirrorMakerWorker --consumer.config $ureplicator_home/config/consumer.properties --producer.config $ureplicator_home/config/producer.properties --helix.config $ureplicator_home/config/helix.properties --topic.mappings $ureplicator_home/config/topicmapping.properties "$@"

[mpkd]

Looks like the metrics in this kafka-0.8.2.1 branch is different than master.
https://github.com/uber/uReplicator/blob/kafka-0.8.2.1/uReplicator-Worker/src/main/scala/kafka/mirrormaker/CompactConsumerFetcherManager.scala

• Is it TotalLag between kafka source and destination for mirroring - https://github.com/uber/uReplicator/blob/master/uReplicator-Worker/src/main/scala/kafka/mirrormaker/CompactConsumerFetcherManager.scala#L94 ?
• Is it possible to get these changes in master to kafka-0.8.2.1 branch ?

[mpkd]

What we are trying to get is the

• total Lag in the mirroring process
• total latency between source and destination kafka cluster(ignore network latency)

[yangy0000]

@mpkd Yes, the total lag is the lag between the current replicated offsets and high water mark in source broker.
We can definitely try to backport this into 0.8.2 branch.

Also, Have you tried the ureplicator-worker-3.0 ? Is it workong on 0.8.2

[mpkd]

@yangy0000 Thank you for the update. It will be really awesome if we can get the 0.8.2 branch get changes in Master branch especially for JMX, currently we are getting some metrics, but not all the metrics mentioned in the jmxtrans

Currently we couldn't see ureplicator-worker-3.0 in 0.8.2 branch. We are just using 0.8.2 branch as it is, looks like this branch is missing lots of updates from master branch. Any plan to make 0.8.2 branch updated with master branch changes?

[yangy0000]

pr to backport total lag.

Please elaborate on the requirement

• total latency between source and destination kafka cluster (ignore network latency)

we can backport the critical bug fix to 0.8.2 branch, but it's very difficult to update with every master branch changes because of 0.8.2 branch uses 0.8.2 Kafka clients which is a very old version, some Kafka API that been used on master branch might not be available on 0.8.2 Kafka client