Skip to content
This repository has been archived by the owner on Dec 5, 2023. It is now read-only.

Alert when shard goes down #12

Open
zumo64 opened this issue Nov 17, 2023 · 0 comments
Open

Alert when shard goes down #12

zumo64 opened this issue Nov 17, 2023 · 0 comments

Comments

@zumo64
Copy link

zumo64 commented Nov 17, 2023

We have a client that experienced a shard loss without any standard redis provided Prometheus alerts firing.

I tested the alert below that requires shards ids (master shards) - it works but requires hard coded values
Maybe that can be improved to be made generic

- alert: Shard went down 
    expr: absent(redis_up{job="redis-enterprise", status="active", redis="42"}) == 1 or absent(redis_up{job="redis-enterprise", status="active", redis="39" }) == 1
    for: 20s
    labels:
      severity: critical
    annotations:
      description: "Shard is down  - Cluster: {{ $labels.cluster }} Shard: {{$labels.redis}} Node: {{$labels.node}} Database:{{$labels.bdb}}"

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant