From 23fb28630d8f82f1254f836a355408d605685ccb Mon Sep 17 00:00:00 2001 From: Alex Garel Date: Mon, 11 Dec 2023 15:48:01 +0100 Subject: [PATCH] =?UTF-8?q?docs:=C2=A0some=20hints=20about=20prometheus?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/observability.md | 40 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 40 insertions(+) diff --git a/docs/observability.md b/docs/observability.md index 35bc99a6..3b7deb42 100644 --- a/docs/observability.md +++ b/docs/observability.md @@ -33,3 +33,43 @@ The observability stack used in the OFF stack is comprised of the following appl The observability stack diagram is as follows: ![Observability stack](./img/obs_stack.png) + +## Prometheus + +Some interesting pages of prometheus include: +* https://prometheus.openfoodfacts.org/alerts to show currently triggered alerts +* https://prometheus.openfoodfacts.org/targets where you can check the status of the targets (services observed by prometheus) +* https://prometheus.openfoodfacts.org/rules for alerting rules + +### multi-target exporter pattern + +We use the [multi-target exporter pattern](https://prometheus.io/docs/guides/multi-target-exporter/#querying-multi-target-exporters-with-prometheus) on some configurations. + +This pattern helps writting a lot of targets that follow same rules in the same way. + +Some points that are not easy to understand: +* targets will be in `__address__` at the begining of the process +* the last rule overwrite `__address__` to put some static value, but it's ok as previous rules where already processed +* simple params like `instance`, `app` are parameters for the job configuration, not for the target url +* to add params to target url we use `__param_xxxx` (for exemple `__param_target` to add a `target` parameter to url) +* `instance` is very important, as each job must have their separate instance name. + +You can look at https://prometheus.openfoodfacts.org/targets to see if your targets are processed correctly. +A mouse over in the "labels" column shows you parameter before processing. + + +## Blackbox exporter + +Is a service that can be used by Prometheus to probe for websites. + +Prometheus will call the service as if it was a metric exporter with the appropriate target (that you set through `target` on `__param_target` in the configuration (if you use [multi-target exporter pattern](https://prometheus.io/docs/guides/multi-target-exporter/#querying-multi-target-exporters-with-prometheus)) + +### Testing it on monitoring container + +Blackbox is on port 9105 so you can test it, for example using: +```bash +# http probe +curl "http://localhost:9115/probe?module=http_probe&target=https%3A%2F%2Fsearch.openfoodfacts.org%2F" +# icmp probe +curl "http://localhost:9115/probe?module=icmp&app=ovh1&target=ov1.openfoodfacts.org" +```