L3 231/add production monitoring #80
Conversation
clusters/l3-sqnc/nginx/values.yaml
Outdated
| enabled: true | ||
| ingressClassResource: | ||
| name: nginx | ||
| default: false |
There was a problem hiding this comment.
we can probably set this one to be the default.
There was a problem hiding this comment.
Somewhere in here there should be entries related to the ingress and ingressClass we are using. As well as how we expose both prometheus and grafana dashboards.
| persistence: | ||
| enabled: true |
There was a problem hiding this comment.
We should be setting the strorageclass and check that the default size of the PVC is sufficient for our needs.
| prometheusSpec: | ||
| retention: 24h |
There was a problem hiding this comment.
The retention is way too long for production. @mattdean-digicatapult thoughts on how long we want to keep stuff?
| prometheus: | ||
| serviceMonitor: | ||
| selfMonitor: true | ||
| namespace: monitoring | ||
| prometheusSpec: | ||
| retention: 24h | ||
| resources: | ||
| requests: | ||
| cpu: 200m | ||
| memory: 200Mi |
There was a problem hiding this comment.
We need to set a persistence option here, as well as potentially have these pods autoscaling. Those default resources look a bit minuscule for a production cluster so we should beef this up somewhat.
There was a problem hiding this comment.
Running horizontal scaling on prometheus is a mess. It can be configured by using thanos in this chart, but that is a lot of overhead for our needs ATM.
Instead I've set it up as HA, with vertical scaling on the prometheus instances.
|
All the resource requirements for the monitoring stack are best guesses. We'd probably have to fine-tune this according to actual usage. |
msterle
requested review from
mattdean-digicatapult,
jonmattgray and
n3op2
as code owners
PR 2/2 For L3-231
This PR adds the monitoring stack defined for the kind cluster to the production cluster. It also adds the service monitors for the other components.