Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

support confluent connectors for apache install #16

Open
christopherkeller opened this issue Aug 5, 2019 · 1 comment
Open

support confluent connectors for apache install #16

christopherkeller opened this issue Aug 5, 2019 · 1 comment
Labels
enhancement New feature or request

Comments

@christopherkeller
Copy link
Contributor

christopherkeller commented Aug 5, 2019

confluent supports ES sinks with a library package. we should install this for apache mode by default.

https://www.confluent.io/hub/confluentinc/kafka-connect-elasticsearch

@christopherkeller christopherkeller added the enhancement New feature or request label Aug 5, 2019
@christopherkeller christopherkeller self-assigned this Sep 17, 2019
@christopherkeller
Copy link
Contributor Author

Connector uploaded into S3 bucket, but need IAM work on each EC2 instance to make it seamless. listed are the manual steps after uploading into S3.

  • sudo yum install -y unzip
  • sudo unzip /tmp/confluentinc-kafka-connect-elasticsearch-5.3.1.zip -d /usr/share/java
  • sudo chown -R kafka:kafka /usr/share/java/confluentinc-kafka-connect-elasticsearch-5.3.1
  • sudo systemctl restart apache-kafka-connect (this seems unecessary as the connectors picked up the connectors instantly)

verify connector is running

  • curl http://`ifconfig eth1 | grep -e 'inet\s' | awk '{ print $2 }'`:8083/connector-plugins

On ansible host provision connectors as usual

AWS_PROFILE=slower ec2_region=us-east-2 ./deploy connect $platform_root/ucsd/inventory/hostsfile-us-east-2-demo.none demo aws us-east-2 development centos $platform_root/ucsd $key_path

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant