Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Initial Stab at prometheus exporter #13

Open
wants to merge 4 commits into
base: master
Choose a base branch
from
Open

Conversation

djnym
Copy link
Member

@djnym djnym commented Jul 1, 2019

If configured this will start an inets httpd server which can be scraped by prometheus.
In the process of working on this many functions were elevated from the mondemand_statdb module up to the mondemand module so that full CRUD operations could be done using just the mondemand module. While I've tested locally, this has not been added to any components or tested with large amounts of metrics.
A few things still remain which is why this is WIP

  1. any metrics sent via mondemand:send_stats/3 will currently not be available for export. I'm thinking of caching the previous result and exporting those.
  2. flushing to a valid lwes channel is still required. This just needs a bit of unwinding and rearranging to get going
    To try this out, I've been doing the following
% ./mondemand-dev

then paste in

mondemand:create_counter(foo, baz, [{another, thing}], "A Baz counter for Foo").
mondemand:create_counter(foo, bob, "A BOB counter for Foo").
mondemand:create_gcounter(foo, rate, "A rate gcounter for Foo").
mondemand:create_gauge(foo, blah, "A blah gauge for Foo").
mondemand:create_sample_set(foo, bar,[{other,thing}], "A Bar counter for Foo", 1000, all).
spawn(fun() ->
        [ begin
            [ begin
                S = rand:uniform(100),
                mondemand:add_sample(foo, bar, [{other,thing}], S)
              end || _ <- lists:seq(1,100+rand:uniform(100)) ],
            [ begin
                C = rand:uniform(100),
                mondemand:increment(foo,baz,[{another,thing}],C),
                mondemand:increment(foo,bob,rand:uniform(5)),
                mondemand:gincrement(foo,rate,rand:uniform(10))
              end || _ <- lists:seq(1,1+rand:uniform(10)) ],
            mondemand:set(foo,blah,rand:uniform(1000)),
            io:format("Added stats, sleeping...~n",[]),
            timer:sleep(60000)
          end || _ <- lists:seq(1,1440)
        ]
       end).

If you have prometheus locally you can add the following to your prometheus.yml

  - job_name: 'mondemand'

    scrape_interval: 60s
    metrics_path: '/md/metrics'

    static_configs:
    - targets: ['localhost:8082']
      labels:
        group: 'production'

The output of the above is

# TYPE foo_bar summary
# HELP foo_bar A Bar counter for Foo
foo_bar{other="thing",quantile="0.5"} 51 1562016900000
foo_bar{other="thing",quantile="0.75"} 77 1562016900000
foo_bar{other="thing",quantile="0.90"} 88 1562016900000
foo_bar{other="thing",quantile="0.95"} 95 1562016900000
foo_bar{other="thing",quantile="0.98"} 98 1562016900000
foo_bar{other="thing",quantile="0.99"} 99 1562016900000
foo_bar{other="thing",quantile="1.0"} 99 1562016900000
foo_bar_sum{other="thing"} 9904 1562016900000
foo_bar_count{other="thing"} 193 1562016900000
# TYPE foo_rate gauge
# HELP foo_rate A rate gcounter for Foo
foo_rate 0
# TYPE foo_blah gauge
# HELP foo_blah A blah gauge for Foo
foo_blah 561
# TYPE mondemand_erlang_flush_total_millis_total counter
# HELP mondemand_erlang_flush_total_millis_total The total amount of time in milliseconds taken to flush
mondemand_erlang_flush_total_millis_total 5
# TYPE mondemand_erlang_flush_count_total counter
# HELP mondemand_erlang_flush_count_total The total number of times events have been flushed
mondemand_erlang_flush_count_total 1
# TYPE foo_baz_total counter
# HELP foo_baz_total A Baz counter for Foo
foo_baz_total{another="thing"} 336
# TYPE foo_bob_total counter
# HELP foo_bob_total A BOB counter for Foo
foo_bob_total 36

@djnym djnym changed the title [WIP] Initial Stab at prometheus exporter Initial Stab at prometheus exporter Jul 2, 2019
@djnym
Copy link
Member Author

djnym commented Jul 2, 2019

Both issues have been resolved, this might still have some roughness, but should be complete enough to try out.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant