Add Monitoring to test local performance improvements

[Jeffrey-Vervoort-KNMI]

When running just local you can access the monitoring.

• E.g. in Grafana you can view statistics about the gRPC methods
• E.g. in Grafana you can view statistics about the most expensive and most called database queries.

[github-actions]

API Unit Test Coverage Report

File	Stmts	Miss	Cover	Missing
\_\_init\_\_.py	0	0	100%
datastore_pb2.py	61	48	21%	34–81
datastore_pb2_grpc.py	54	27	50%	15–16, 19, 65–80, 121–123, 128–130, 135–137, 142–144, 148–173, 219, 246, 273, 300
export_metrics.py	10	0	100%
grpc_getter.py	20	11	45%	15–19, 23–26, 30–32, 36–38
locustfile.py	15	15	0%	1–31
main.py	43	7	84%	45, 50, 60, 70–71, 81–82
metadata_endpoints.py	65	31	52%	45–54, 58, 85, 100–219, 223
response_classes.py	5	0	100%
utilities.py	174	46	74%	20, 38, 45, 67–70, 78–89, 94–101, 121, 125, 127, 155, 161, 179, 193–194, 198, 214–218, 222–228, 232–234, 264, 268, 290, 295
custom_geo_json
edr_feature_collection.py	6	0	100%
formatters
\_\_init\_\_.py	11	0	100%
covjson.py	59	1	98%	91
geojson.py	21	2	90%	27, 52
openapi
custom_dimension_examples.py	4	0	100%
edr_query_parameter_descriptions.py	11	0	100%
openapi_examples.py	13	0	100%
routers
\_\_init\_\_.py	0	0	100%
edr.py	101	4	96%	348–349, 438–439
feature.py	47	19	60%	99–132, 148–153, 159–181
TOTAL	720	211	71%

API Unit Test Coverage Summary

Tests	Skipped	Failures	Errors	Time
30	0 💤	0 ❌	0 🔥	1.824s ⏱️

[jo-asplin-met-no]

The gRPC Server dashboard in Grafana has the same three panel groups (or whatever you call them) for each of three jobs. The panel groups are gRPC General, gRPC Methods, and Go, while the jobs are prometheus, prometheus-postgres-exporter, and store. Does this mean that prometheus and prometheus-postgres-exporter are also written in Go and use gRPC for communication?

[Jeffrey-Vervoort-KNMI]

The gRPC Server dashboard in Grafana has the same three panel groups (or whatever you call them) for each of three jobs. The panel groups are gRPC General, gRPC Methods, and Go, while the jobs are prometheus, prometheus-postgres-exporter, and store.

I think the Grafana term is variables. At this moment, except for the interval, you always want to keep them at their default value. The other values are not that interesting. For example, database names is a dynamic list retrieved from Prometheus. It consists of all the databases in the Postgres container, namely our database and the default databases. I did not mind it, but for clarity, it could be better to only have data as value. What do you think?

Does this mean that prometheus and prometheus-postgres-exporter are also written in Go and use gRPC for communication?

They are written in Go. I don't think they use gRPC. I made the dashboard this way as I thought for our most common use case store everything is in one place. If we want to view the Go performance of "prometheus" or the "exporter" we can. But as I said I expect we seldom look at it, so I think there is something to say to change the variable to only store. What do you think?

[jo-asplin-met-no]

To avoid confusion I would personally have preferred to move the two monitoring servers (prometheus and exporter) out in a separate dashboard (typically called Monitoring Infrastructure or something like that) and no longer associate them with gRPC metrics (since that's n/a as you say). Or even delete them altogheter, but that's not important I think. It's interesting to have them there.

[jo-asplin-met-no]

Looks good!