Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

can not output to kafka when logstash runs in docker #153

Open
cgyim opened this issue Sep 1, 2017 · 1 comment
Open

can not output to kafka when logstash runs in docker #153

cgyim opened this issue Sep 1, 2017 · 1 comment

Comments

@cgyim
Copy link

cgyim commented Sep 1, 2017

Hi,

  • Version: 5.5.1
  • Operating System: centos ,Linux es-logger-kibana 3.10.0-327.18.2.el7.x86_64 #1 SMP Thu May 12 11:03:55 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
  • Config File (if you have sensitive info, please remove it):
    logstash config:
    input {
    file{
    path => "/var/log/dpkg.log"
    start_position => "beginning"
    sincedb_path => "/dev/null"
    tags => ["test"]
    # codec => plain {
    # charset => "GB2312"
    # }
    }
    }
    output {
    kafka {
    # message_key => "hangqing"
    topic_id => "you2"
    bootstrap_servers => "10.88.115.20:9092,10.88.115.20:9093,10.88.115.20:9094"
    codec => plain {
    format => "%{message}"
    }
    }
    # stdout { codec => rubydebug}
    }
    docker-compose config:
    logstash:
    image: logstash:5.5.1
    net: host
    volumes:
    - /opt/docker-compose/logstash/logstash.conf:/etc/logstash/conf.d/logstash.conf
    - /opt/docker-compose/logstash/jvm.options:/etc/logstash/jvm.options
    user: root
    command: logstash --debug -f /etc/logstash/conf.d/logstash.conf
  • Steps to Reproduce:
  1. docker-compose up logstash
  2. (another terminal in kafka home)bin/kafka-console-consumer.sh --bootstrap-server 0.0.0.0:9092 --topic you2 --zookeeper 0.0.0.0:2181 --from-beginning .
  3. no output , but the topic was created. It's was wierd.

I am sure that the kafka works fine because when logstash runs directly in os as a service, everything works fine , i can see the output typing the same command as a kafka consumer.
iptables config:
Chain INPUT (policy ACCEPT)
target prot opt source destination
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpts:3000:9999
ACCEPT 112 -- 0.0.0.0/0 0.0.0.0/0
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED
ACCEPT icmp -- 0.0.0.0/0 0.0.0.0/0
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:22

Chain FORWARD (policy ACCEPT)
target prot opt source destination
DOCKER-ISOLATION all -- 0.0.0.0/0 0.0.0.0/0
DOCKER all -- 0.0.0.0/0 0.0.0.0/0
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0

Chain OUTPUT (policy ACCEPT)
target prot opt source destination
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpts:3000:65535
ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 udp dpts:3000:65535

Chain DOCKER (1 references)
target prot opt source destination
ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 udp dpts:3000:65535
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpts:3000:65535

Chain DOCKER-ISOLATION (1 references)
target prot opt source destination
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0

Could some1 give me a hand :( ?

@cgyim
Copy link
Author

cgyim commented Sep 1, 2017

logstash-kafka.txt

logstash debug output log file also provided

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant