Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Kafka Prometheus metrics export #128

Merged
merged 14 commits into from
Feb 3, 2018
Merged

Add Kafka Prometheus metrics export #128

merged 14 commits into from
Feb 3, 2018

Commits on Jan 19, 2018

  1. Current config from the metrics-improve-scrape-times branch, 

    see #49
    @solsson
    solsson committed Jan 19, 2018
    Configuration menu
    Copy the full SHA
    b275895 View commit details
    Browse the repository at this point in the history

  2. The metrics part of #49

    @solsson
    solsson committed Jan 19, 2018
    Configuration menu
    Copy the full SHA
    d82b419 View commit details
    Browse the repository at this point in the history

  3. Upgrades jmx exporter to 0.2.0

    @solsson
    solsson committed Jan 19, 2018
    Configuration menu
    Copy the full SHA
    162902c View commit details
    Browse the repository at this point in the history

  4. Adds liveness probe from the metrics-improve-scrape-times branch

    @solsson
    solsson committed Jan 19, 2018
    Configuration menu
    Copy the full SHA
    74a5177 View commit details
    Browse the repository at this point in the history

  5. This is an optional feature, but belongs to the broker pods, 

    so let's evaluate kubectl patch to add it
    @solsson
    solsson committed Jan 19, 2018
    Configuration menu
    Copy the full SHA
    da113d2 View commit details
    Browse the repository at this point in the history

  6. Evaluates JVM memory limit awareness 

    Interesting input for #112, for use with broker and zk pods
in addition to KAFKA_HEAP_OPTS.
    @solsson
    solsson committed Jan 19, 2018
    Configuration menu
    Copy the full SHA
    eaf9ebd View commit details
    Browse the repository at this point in the history

  7. On start the metrics pod gesses that it has ~45% 

    of the memory limit as "Max. Heap Size (Estimated)"

Reducing limits as experiment for #112.
    @solsson
    solsson committed Jan 19, 2018
    Configuration menu
    Copy the full SHA
    66255af View commit details
    Browse the repository at this point in the history

  8. With kubectl patch, this folder kan be scoped as Prometheus support

    @solsson
    solsson committed Jan 19, 2018
    Configuration menu
    Copy the full SHA
    e05b790 View commit details
    Browse the repository at this point in the history

  9. With 120M you get OOMKilled even on brokers with modest metrics volumes 

    This reverts commit 66255af.
    @solsson
    solsson committed Jan 19, 2018
    Configuration menu
    Copy the full SHA
    f1e6e96 View commit details
    Browse the repository at this point in the history

  10. Removes liveness probes, to focus on memory limits, 

    and rely on metric staleness alerts instead for exporter liveness.

This reverts commit 74a5177.
    @solsson
    solsson committed Jan 19, 2018
    Configuration menu
    Copy the full SHA
    0d78e08 View commit details
    Browse the repository at this point in the history

Commits on Jan 29, 2018

  1. It was the liveness probe that killed the metrics container, 

    and we might not need liveness if we have alerts for stale metrics.

This reverts commit f1e6e96.
    @solsson
    solsson committed Jan 29, 2018
    Configuration menu
    Copy the full SHA
    cfe434c View commit details
    Browse the repository at this point in the history

Commits on Feb 3, 2018

  1. ”In a production Kafka cluster, an offline partition 

    may be impacting the producer clients, losing messages or causing back-pressure in the application.
This is most often a “site down” type of problem and will need to be addressed immediately.”

Excerpt from: Neha Narkhede, Gwen Shapira, and Todd Palino. ”Kafka: The Definitive Guide”.

We now export kafka_controller_kafkacontroller_value{name="OfflinePartitionsCount",} and friends.
See #140 for why.
    @solsson
    solsson committed Feb 3, 2018
    Configuration menu
    Copy the full SHA
    152bb19 View commit details
    Browse the repository at this point in the history

  2. At first i suspected that order matters, but this also works

    @solsson
    solsson committed Feb 3, 2018
    Configuration menu
    Copy the full SHA
    f022138 View commit details
    Browse the repository at this point in the history

  3. There's only values in this metric type

    @solsson
    solsson committed Feb 3, 2018
    Configuration menu
    Copy the full SHA
    a564ed2 View commit details
    Browse the repository at this point in the history