Skip to content

Commit

Permalink
Drop some latency buckets (#54)
Browse files Browse the repository at this point in the history
The majority of metrics sent by the agent are latency buckets.
`linkerd-proxy` exports 25-bucket histograms:
```
1, 2, 3, 4, 5,
10, 20, 30, 40, 50,
100, 200, 300, 400, 500,
1000, 2000, 3000, 4000, 5000,
10000, 20000, 30000, 40000, 50000,
```

Decrease the number of time series the agent sends by decreasing latency
buckets from 25 to 10:
```
1, 5,
10, 50,
100, 500,
1000, 5000,
10000, 50000,
```

Also, modify the `buoyant-cloud-agent` scrape job to only opt-in
specific metrics needed for debugging.

Signed-off-by: Andrew Seigner <[email protected]>
  • Loading branch information
siggy authored Mar 1, 2022
1 parent 378e454 commit 02bd1a8
Showing 1 changed file with 17 additions and 0 deletions.
17 changes: 17 additions & 0 deletions charts/linkerd-buoyant/templates/metrics-agent.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,11 @@ data:
- action: labelmap
regex: __meta_kubernetes_pod_label_(.+)
metric_relabel_configs:
- source_labels: [__name__]
regex: ^go_memstats_alloc_bytes|go_memstats_alloc_bytes_total|go_memstats_heap_inuse_bytes|go_memstats_stack_inuse_bytes|process_resident_memory_bytes|process_virtual_memory_bytes|up$
action: keep
# scrape_configs copied from `linkerd install`
- job_name: 'kubernetes-nodes-cadvisor'
Expand Down Expand Up @@ -198,6 +203,12 @@ data:
regex: 'response_latency_ms_bucket;outbound'
action: drop
# drop some latency buckets
- source_labels:
- le
regex: ^2.*|3.*|4.*$
action: drop
# drop high-cardinality outbound tcp open connections
- source_labels:
- __name__
Expand Down Expand Up @@ -315,6 +326,12 @@ data:
- source_labels: [__name__]
regex: ^gateway_alive|gateway_probe_latency_ms_bucket$
action: keep
# drop some latency buckets
- source_labels:
- le
regex: ^2.*|3.*|4.*$
action: drop
---
#
# Metrics Agent
Expand Down

0 comments on commit 02bd1a8

Please sign in to comment.