Skip to content

Commit

Permalink
ENUM to support e.g. Light Switches (#27)
Browse files Browse the repository at this point in the history
ENUM is a not so common metric, that is made to suport boolen states.
Typical use case are predefined states like on/off or low/medium/high in string form.
Regarding ENUM see https://github.com/prometheus/client_python/blob/9a24236695c9ad47f9dc537a922a6d1333d8d093/prometheus_client/metrics.py#L640-L698 and https://github.com/OpenObservability/OpenMetrics/blob/main/specification/OpenMetrics.md#stateset

Co-authored-by: poggenpower <[email protected]>
  • Loading branch information
poggenpower and poggenpower authored Sep 15, 2021
1 parent 86d79ec commit 1afdb5a
Show file tree
Hide file tree
Showing 8 changed files with 213 additions and 19 deletions.
34 changes: 32 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,30 @@ Configurable general purpose Prometheus exporter for MQTT.

Subscribes to one or more MQTT topics, and lets you configure prometheus metrics based on pattern matching.

[![Test with pytest](https://github.com/fhemberger/mqtt_exporter/actions/workflows/test.yml/badge.svg)](https://github.com/fhemberger/mqtt_exporter/actions/workflows/test.yml)[![CodeQL](https://github.com/fhemberger/mqtt_exporter/actions/workflows/codeql-analysis.yml/badge.svg)](https://github.com/fhemberger/mqtt_exporter/actions/workflows/codeql-analysis.yml)
[![Test with pytest](https://github.com/fhemberger/mqtt_exporter/actions/workflows/test.yml/badge.svg)](https://github.com/fhemberger/mqtt_exporter/actions/workflows/test.yml)[![CodeQL](https://github.com/fhemberger/mqtt_exporter/actions/workflows/codeql-analysis.yml/badge.svg)](https://github.com/fhemberger/mqtt_exporter/actions/workflows/codeql-analysis.yml)![Docker Pulls](https://img.shields.io/docker/pulls/fhemberger/mqtt_exporter)


## Features

- Supported Metrics:
- standard metrics
- Gauge, Counter, Histogram, Summary
- additional
- **Counter (Absolute):**
- Same as Counter, but working with absolute numbers received from MQTT. Which is far more common, than sending the diff in each publish.
- e.g. a network counter or a rain sensor
- **Enum:**
- is a metric type not so common, details can be found in the [OpenMetrics docs](https://github.com/OpenObservability/OpenMetrics/blob/main/specification/OpenMetrics.md#stateset) and [Python client code](https://github.com/prometheus/client_python/blob/9a24236695c9ad47f9dc537a922a6d1333d8d093/prometheus_client/metrics.py#L640-L698).
- Allows to track as state by a know set of strings describing the state, e.g. `on/off` or `high/medium/low`
- Common sources would be a light switch oder a door lock.
- Comprehensive rewriting for topic, value/payload and labels
- similar to prometheus label rewrites
- regex allows almost every conversion
- e.g. to
- remove units or other strings from payload
- convert topic hierarchy into labels
- normalize labels
- check example configs `./exampleconf` and the configs in `./test/test_data/`


## Usage
Expand Down Expand Up @@ -43,9 +66,16 @@ docker run -d \
- PyYAML
- yamlreader

Functional tests are written in `pytest` (see [tests/readme.md](tests/readme.md)), code formatting uses [`autopep8`](https://pypi.org/project/autopep8/) with default settings. If you submit a PR to this repo, please make sure it follows its formatting guidelines.

## Contribution

* Contribution is welcome. Fork and then PR.
* Discussions in Issues.
* Functional tests are written in `pytest` (see [tests/readme.md](tests/readme.md))
* Code formatting uses [`autopep8`](https://pypi.org/project/autopep8/) with default settings. If you submit a PR to this repo, please make sure it follows its formatting guidelines.


## TODO

- Add persistence of metrics on restart
- forget/age out metrics receiving no updates anymore
23 changes: 23 additions & 0 deletions exampleconf/histogram.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
# histogram metric. with Buckets <= 0.5, 5, 10, +inf
- name: 'network_ping_ms'
help: 'ping response in ms'
type: 'histogram'
topic: 'network/+/+/ping'
parameters:
buckets:
- 0.5
- 5
- 10
label_configs:
- source_labels: ['__msg_topic__']
target_label: '__topic__'
- source_labels: ["__msg_topic__"]
regex: "network/([^/]+).*"
target_label: "network"
replacement: '\1'
action: "replace"
- source_labels: ["__msg_topic__"]
regex: "network/[^/]+/([^/]+).*"
target_label: "server"
replacement: '\1'
action: "replace"
9 changes: 8 additions & 1 deletion exampleconf/metric_example.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,14 @@ metrics:
- name: 'mqtt_example' # Required(unique, if multiple, only last entry is kept)
help: 'MQTT example gauge' # Required
type: 'gauge' # Required ('gauge', 'counter', 'summary' or 'histogram')
#buckets: '.1, 1.0, 10.0, inf' # Optional (Passed as 'buckets' argument to Histogram)
#parameters: # Optional parameters for certain metrics
# buckets: # Optional (Passed as 'buckets' argument to Histogram)
# - .1
# - 1.0
# - 10.0
# states: # Optional (Passes as 'states' arguments to Enum)
# - on
# - off
topic: 'example/topic/+' # Required

# Inspired by 'https://prometheus.io/docs/operating/configuration/#<relabel_config>'
Expand Down
23 changes: 23 additions & 0 deletions exampleconf/switchstate.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
# metric for a switch with the state on and off.
# states are case sensitive and must match exactly
# use label_config to rewrite other values, see below.
metrics:
- name: "fhem_light_state"
help: "Light state on/off"
type: "enum"
topic: "fhem/+/+/light"
parameters:
states:
- 'on'
- 'off'
label_configs:
- source_labels: ['__value__'] # replace uppercase ON and 0 with on
regex: "(ON|0)"
target_label: '__value__'
replacement: 'on'
action: "replace"
- source_labels: ['__value__'] # replace uppercase OFF und 1 with off
regex: "(OFF|1)"
target_label: '__value__'
replacement: 'off'
action: "replace"
56 changes: 41 additions & 15 deletions mqtt_exporter.py
Original file line number Diff line number Diff line change
@@ -1,30 +1,31 @@
#!/usr/bin/env python

from copy import copy, deepcopy
from copy import deepcopy
import json
import prometheus_client as prometheus
from collections import defaultdict
import logging
import argparse
import paho.mqtt.client as mqtt
import yaml
import os
import re
import operator
import time
import signal
import sys
import paho.mqtt.client as mqtt
import yaml
import prometheus_client as prometheus
from yamlreader import yaml_load
import utils.prometheus_additions
import version

VERSION = version.__version__
SUFFIXES_PER_TYPE = {
"gauge": [''], # add at least an empty suffix
"gauge": [],
"counter": ['total'],
"counter_absolute": ['total'],
"summary": ['sum', 'count'],
"histogram": ['sum', 'count', 'bucket'],
"enum": [],
}


Expand Down Expand Up @@ -187,7 +188,7 @@ def _log_setup(logging_config):
raise TypeError(f'Invalid log level: {log_level}')

if logfile != '':
logging.info('Logging redirected to: ' + logfile)
logging.info(f"Logging redirected to: {logfile}")
# Need to replace the current handler on the root logger:
file_handler = logging.FileHandler(logfile, 'a')
formatter = logging.Formatter('%(asctime)s %(levelname)s: %(message)s')
Expand Down Expand Up @@ -280,12 +281,11 @@ def _update_metrics(metrics, msg):
if not _apply_label_config(labels, metric['label_configs']):
continue

# try to convert to float, but leave as is if conversion not possible
try:
labels['__value__'] = float(labels['__value__'].replace(',', '.'))
except ValueError:
logging.exception(
f"__value__ must be a number, was: {labels['__value__']}, Metric: {metric['name']} on __msg_topic_: {labels['__msg_topic__']}")
continue
logging.debug(f"Conversion of {labels['__value__']} to float not possible, continue with value as is.")

logging.debug('_update_metrics all labels:')
logging.debug(labels)
Expand Down Expand Up @@ -352,7 +352,9 @@ def _export_to_prometheus(name, metric, labels):
'counter': CounterWrapper,
'counter_absolute': CounterAbsoluteWrapper,
'summary': SummaryWrapper,
'histogram': HistogramWrapper}
'histogram': HistogramWrapper,
'enum': EnumWrapper,
}
valid_types = metric_wrappers.keys()
if metric['type'] not in valid_types:
logging.error(
Expand All @@ -378,8 +380,11 @@ def _export_to_prometheus(name, metric, labels):
metric['prometheus_metric']['parent'] = prometheus_metric
else:
prometheus_metric = metric['prometheus_metric']['parent']

prometheus_metric.update(label_values, value)
try:
prometheus_metric.update(label_values, value)
except ValueError as ve:
logging.error(f"Value {value} is not compatible with metric {metric['name']} of type {metric['type']}")
logging.exception('ve:')

logging.debug(
f"_export_to_prometheus metric ({metric['type']}): {name}{labels} updated with value: {value}")
Expand Down Expand Up @@ -421,12 +426,10 @@ class GaugeWrapper():
"""
Wrapper to provide generic interface to Gauge metric
"""

def __init__(self, name, help_text, label_names, *args, **kwargs) -> None:
self.metric = prometheus.Gauge(
name, help_text, list(label_names)
)

def update(self, label_values, value):
child = self.metric.labels(*label_values)
child.set(value)
Expand All @@ -448,6 +451,10 @@ def update(self, label_values, value):
child.inc(value)
return child

class HistogramWrapper():
"""
Wrapper to provide generic interface to Summary metric
"""

class CounterAbsoluteWrapper():
"""
Expand Down Expand Up @@ -489,7 +496,10 @@ class HistogramWrapper():
def __init__(self, name, help_text, label_names, *args, **kwargs) -> None:
params = {}
if kwargs.get('buckets'):
params['buckets'] = kwargs['buckets'].split(',')
if isinstance(kwargs['buckets'], str):
params['buckets'] = kwargs['buckets'].split(',')
else:
params['buckets'] = kwargs['buckets']

self.metric = prometheus.Histogram(
name, help_text, list(label_names), **params
Expand All @@ -501,6 +511,22 @@ def update(self, label_values, value):
return child


class EnumWrapper():
def __init__(self, name, help_text, label_names, *args, **kwargs) -> None:
params = {}
if kwargs.get('states'):
params['states'] = kwargs['states']

self.metric = prometheus.Enum(
name, help_text, list(label_names), **params
)

def update(self, label_values, value):
child = self.metric.labels(*label_values)
child.state(value)
return child


def add_static_metric(timestamp):
g = prometheus.Gauge('mqtt_exporter_timestamp', 'Startup time of exporter in millis since EPOC (static)',
['exporter_version'])
Expand Down
58 changes: 58 additions & 0 deletions tests/test_data/test_enum/conf.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@

# Logging
logging:
# logfile: 'conf/mqttexperter.log' # Optional default '' (stdout)
level: 'debug' # Optional default 'info'

timescale: 0

# Metric definitions
metrics:
- name: "fhem_light_state"
help: "Light state on/off"
type: "enum"
topic: "fhem/+/+/light"
parameters:
states:
- 'on'
- 'off'
label_configs:
- source_labels: ['__value__']
regex: "(ON|0)"
target_label: '__value__'
replacement: 'on'
action: "replace"
- source_labels: ['__value__']
regex: "(OFF|1)"
target_label: '__value__'
replacement: 'off'
action: "replace"
- source_labels: ['__msg_topic__']
target_label: '__topic__'
- source_labels: ["__msg_topic__"]
regex: "fhem/([^/]+).*"
target_label: "location"
replacement: '\1'
action: "replace"
- name: 'network_ping_ms'
help: 'ping response in ms'
type: 'histogram'
topic: 'network/+/+/ping'
parameters:
buckets:
- 0.5
- 5
- 10
label_configs:
- source_labels: ['__msg_topic__']
target_label: '__topic__'
- source_labels: ["__msg_topic__"]
regex: "network/([^/]+).*"
target_label: "network"
replacement: '\1'
action: "replace"
- source_labels: ["__msg_topic__"]
regex: "network/[^/]+/([^/]+).*"
target_label: "server"
replacement: '\1'
action: "replace"
23 changes: 23 additions & 0 deletions tests/test_data/test_enum/mqtt_msg.csv
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
in_topic;in_payload;out_name;out_labels;out_value;delay;assert
fhem/room01/desk/light01;on;fhem_light_state;{"location":"Garten","topic": "fhem/room01/desk/light01"};0;4;True
fhem/room01/desk/light01;on;fhem_light_state;{"location":"Garten","topic": "fhem/room01/desk/light01"};0;3;True
fhem/room01/desk/light01;off;fhem_light_state;{"location":"Garten","topic": "fhem/room01/desk/light01"};1;5;True
fhem/room01/desk/light01;on;fhem_light_state;{"location":"Garten","topic": "fhem/room01/desk/light01"};0;4;True
fhem/room01/desk/light01;off;fhem_light_state;{"location":"Garten","topic": "fhem/room01/desk/light01"};1;3;True
fhem/room01/desk/light01;ON;fhem_light_state;{"location":"Garten","topic": "fhem/room01/desk/light01"};0;5;True
fhem/room01/desk/light01;OFF;fhem_light_state;{"location":"Garten","topic": "fhem/room01/desk/light01"};1;4;True
fhem/room01/desk/light01;off;fhem_light_state;{"location":"Garten","topic": "fhem/room01/desk/light01"};1;3;True
fhem/room01/desk/light01;1;fhem_light_state;{"location":"Garten","topic": "fhem/room01/desk/light01"};1;5;True
fhem/room01/desk/light01;0;fhem_light_state;{"location":"Garten","topic": "fhem/room01/desk/light01"};0;4;True
fhem/room01/desk/light01;on;fhem_light_state;{"location":"Garten","topic": "fhem/room01/desk/light01"};0;3;True
fhem/room01/desk/light01;off;fhem_light_state;{"location":"Garten","topic": "fhem/room01/desk/light01"};1;5;True
network/vlan11/srv01.local/ping;2;network_ping_ms;{"network": "vlan11","topic": "network/vlan11/srv01.local/ping", "server": "srv01.local", "le": "5.0"};{"_count": 1, "_sum": 2, "_bucket": 1};2;True
network/vlan11/srv01.local/ping;4;network_ping_ms;{"network": "vlan11","topic": "network/vlan11/srv01.local/ping", "server": "srv01.local", "le": "5.0"};{"_count": 2, "_sum": 6, "_bucket": 2};6;True
network/vlan11/srv01.local/ping;7;network_ping_ms;{"network": "vlan11","topic": "network/vlan11/srv01.local/ping", "server": "srv01.local", "le": "10.0"};{"_count": 3, "_sum": 13, "_bucket": 3};1;True
network/vlan11/srv01.local/ping;0.4;network_ping_ms;{"network": "vlan11","topic": "network/vlan11/srv01.local/ping", "server": "srv01.local", "le": "0.5"};{"_count": 4, "_sum": 13.4, "_bucket": 1};4;True
network/vlan11/srv01.local/ping;20;network_ping_ms;{"network": "vlan11","topic": "network/vlan11/srv01.local/ping", "server": "srv01.local", "le": "+Inf"};{"_count": 5, "_sum": 33.4, "_bucket": 5};5;True
network/vlan11/srv01.local/ping;11.1;network_ping_ms;{"network": "vlan11","topic": "network/vlan11/srv01.local/ping", "server": "srv01.local", "le": "+Inf"};{"_count": 6, "_sum": 44.5, "_bucket": 6};2;True
network/vlan11/srv01.local/ping;5;network_ping_ms;{"network": "vlan11","topic": "network/vlan11/srv01.local/ping", "server": "srv01.local", "le": "5.0"};{"_count": 7, "_sum": 49.5, "_bucket": 4};4;True
network/vlan11/srv01.local/ping;6;network_ping_ms;{"network": "vlan11","topic": "network/vlan11/srv01.local/ping", "server": "srv01.local", "le": "10.0"};{"_count": 8, "_sum": 55.5, "_bucket": 6};1;True
network/vlan11/srv01.local/ping;0.05;network_ping_ms;{"network": "vlan11","topic": "network/vlan11/srv01.local/ping", "server": "srv01.local", "le": "0.5"};{"_count": 9, "_sum": 55.55, "_bucket": 2};4;True
network/vlan11/srv01.local/ping;30;network_ping_ms;{"network": "vlan11","topic": "network/vlan11/srv01.local/ping", "server": "srv01.local", "le": "+Inf"};{"_count": 10, "_sum": 85.55, "_bucket": 10};5;True
6 changes: 5 additions & 1 deletion tests/test_mqtt_exporter.py
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,11 @@ def _get_mqtt_data(file_name):
# covert payloud to bytes, as in a MQTT Message
row[MqttCVS.in_payload] = row[MqttCVS.in_payload].encode('UTF-8')
# parse labels, to a python object.
row[MqttCVS.out_labels] = json.loads(row.get(MqttCVS.out_labels, '{}'))
try:
row[MqttCVS.out_labels] = json.loads(row.get(MqttCVS.out_labels, '{}'))
except json.decoder.JSONDecodeError as jde:
logging.error(f"json.decoder.JSONDecodeError while decoding {row.get(MqttCVS.out_labels, '{}')}")
raise jde
# Value could be a JSON, a float or anthing else.
try:
row[MqttCVS.out_value] = float(row.get(MqttCVS.out_value))
Expand Down

0 comments on commit 1afdb5a

Please sign in to comment.