Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Alerts suddenly start getting inhibited #2773

Open
bbczeuz opened this issue Jun 21, 2023 · 0 comments
Open

Alerts suddenly start getting inhibited #2773

bbczeuz opened this issue Jun 21, 2023 · 0 comments

Comments

@bbczeuz
Copy link
Contributor

bbczeuz commented Jun 21, 2023

This evening, the alert node suddenly started inhibiting alerts even there is no .inhibit() property defined. Events from influxdb continued to be received by kapacitor.

Tick Script Template:

var measurement string

var where_filter = lambda: TRUE

var groups = [*]

var window = 10s

var deadman_interval = 900s

var topic string

var info = lambda: TRUE

var warn = lambda: FALSE

var crit = lambda: FALSE

var data_raw = stream
    |from()
        .measurement(measurement)
        .where(where_filter)
        .groupBy(groups)

data_raw
    |alert()
        .category('threshold')
        .info(info)
        .warn(warn)
        .crit(crit)
        .topic(topic)
[...]

Variables:

Vars:
Name                          Type      Value                                   
crit                          lambda    "used_percent" > 90.0                   
measurement                   string    disk                                    
topic                         string    telegraf_disk                           
warn                          lambda    "used_percent" > 75.0                   

Graph:

digraph disk {
graph [throughput="20.01 points/s"];

stream0 [avg_exec_time_ns="0s" errors="0" working_cardinality="0" ];
stream0 -> from1 [processed="111222"];

from1 [avg_exec_time_ns="0s" errors="0" working_cardinality="0" ];
from1 -> alert2 [processed="111222"];

alert2 [alerts_inhibited="18148" alerts_triggered="93074" avg_exec_time_ns="0s" crits_triggered="0" errors="0" infos_triggered="92811" oks_triggered="0" warns_triggered="263" working_cardinality="352" ];

Version:
kapacitor-1.6.5-1.x86_64 on Centos 7, Influxdb is 1.8.10

Data is gathered from telegraf and sent to influxdb; kapacitor subscribed to the DB. Any pointers?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant