Is it possible to replicate messages with headers from source kafka to destination?

[bombardun]

I have a problem with test environment, when i tried to replicate topic to destination kafka, all messages was replicated without headers.
Controller & worker runs in dockers, maybe i set incorrectly consumer.properties?
This is my config:

# Zookeeper connection string
# comma separated host:port pairs, each corresponding to a zk
# server. e.g. "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002"
zookeeper.connect=zoo1:2181,zoo2:2181,zoo3:2181
bootstrap.servers=kafka1:9092,kafka2.9092,kafka3:9092
# timeout in ms for connecting to zookeeper
zookeeper.connection.timeout.ms=30000
zookeeper.session.timeout.ms=30000

#consumer group id
group.id=urepl_mirror

consumer.id=urepl_mirror_01
socket.receive.buffer.bytes=1048576
fetch.message.max.bytes=8388608
queued.max.message.chunks=5
key.deserializer=org.apache.kafka.common.serialization.ByteArrayDeserializer
value.deserializer=org.apache.kafka.common.serialization.ByteArrayDeserializer
#consumer timeout
#consumer.timeout.ms=5000

auto.offset.reset=smallest

[yangy0000]

Please try to use uReplicator-Worker-3.0 which supports Kafka header

[bombardun]

Is it possible to run it in docker-compose environment with non-federation mode?

[yangy0000]

yes, it's possible to run on docker-compose. the readme is a bit outdated. I'm going to submit a PR with docker-compose example.

[bombardun]

Maybe you write in comments, how to build docker image with worker-3.0 support? because if i build image as usual, how you describe in README,i can't see start shell scripts for worker-3.0 in ./uReplicator-Distribution/target/uReplicator-Distribution-pkg/bin directory

[yangy0000]

To run worker-3.0 on docker, the start shell scripts need to update.

1: you will need to update docker-entrypoint.sh with

**
case "$1" in
manager)
exec java -Dlog4j.configuration=file:config/tools-log4j.properties -server -cp uReplicator-Manager/target/uReplicator-Manager-2.0.1-SNAPSHOT-jar-with-dependencies.jar com.uber.stream.kafka.mirrormaker.manager.ManagerStarter ${@:2}
;;
controller)
exec java -Dlog4j.configuration=file:config/tools-log4j.properties -server -cp uReplicator-Controller/target/uReplicator-Controller-2.0.1-SNAPSHOT-jar-with-dependencies.jar com.uber.stream.kafka.mirrormaker.controller.ControllerStarter ${@:2}
;;
worker)
exec java -Dlog4j.configuration=file:config/tools-log4j.properties -server -cp uReplicator-Worker-3.0/target/uReplicator-Worker-3.0-2.0.1-SNAPSHOT-jar-with-dependencies.jar com.uber.stream.ureplicator.worker.WorkerStarter ${@:2}
;;
*)
exec $@
;;
esac
**

2: update Dockerfile
FROM maven:3.5-jdk-8

RUN apt-get update &&
apt-get install -y netcat

ARG MAVEN_OPTS="-Xmx1024M -Xss128M -XX:MetaspaceSize=512M -XX:MaxMetaspaceSize=1024M -XX:+CMSClassUnloadingEnabled"
COPY . /usr/src/app
WORKDIR /usr/src/app

RUN mvn clean package -DskipTests
RUN chmod +x /usr/src/app/bin/pkg/*.sh

ENTRYPOINT [ "./docker-entrypoint.sh" ]
CMD [ "controller" ]

After the above steps, you shall able to run worker-3.0 docker with the command
worker \
-consumer_config consumer.properties
-producer_config producer.properties
-helix_config helix.properties
-topic_mappings topicmapping.properties

[yangy0000]

worker-3.0 is the preferred option now, I'm going to update the start shell and Dockerfile soon.