Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add shotover metrics profiler #1378

Merged
merged 5 commits into from
Dec 4, 2023

Conversation

rukai
Copy link
Member

@rukai rukai commented Nov 29, 2023

Add a new windsock profiler (--profilers shotover_metrics) that fetches metrics from shotovers prometheus web interface.

When I implemented sar based system metrics, I set it up to extract specific hardcoded metrics from the full pool of metrics collected by sar. This made sense there as:

  • sar collects a lot of metrics and only some of them are relevant
  • sar's naming was quite poor and we can give better names for each item
  • we can define an appropriate Goal value for each metric.

However for prometheus metrics I opted to include all metrics as long as we know how to handle that type of metric.
My reasoning is:

  • Metrics are very dyanamic, new metrics are added or removed just by adding or removing transforms, so we need to dyanamically adapt to that.
  • All metrics are relevant - no need to filter some out
  • The names are fine as is - no need to rename them

Unfortunately this does come at the cost of not being able to define goals for some types of metrics.

Running with --cloud doesnt work yet due to EC2 security groups, this will be complicated to resolve, so for now lets land as is because I dont need it to work in the cloud yet.

Running:

cargo windsock --profilers shotover_metrics "name=kafka shotover=message-parsed topology=cluster1|single"
cargo windsock --results-by-tags ""

we get:
image

@rukai rukai force-pushed the add_shotover_metrics_profiler branch from fd5d20b to b1e6d3a Compare November 30, 2023 02:52
@shotover shotover deleted a comment from github-actions bot Nov 30, 2023
@rukai rukai force-pushed the add_shotover_metrics_profiler branch 3 times, most recently from 83e6fb2 to 658d8fd Compare November 30, 2023 20:48
@rukai rukai force-pushed the add_shotover_metrics_profiler branch from 658d8fd to 77e53f5 Compare November 30, 2023 22:22
@rukai rukai force-pushed the add_shotover_metrics_profiler branch from 307aad9 to 7a0f2a7 Compare December 1, 2023 02:55
@rukai rukai force-pushed the add_shotover_metrics_profiler branch from 7a0f2a7 to d0d5840 Compare December 1, 2023 03:03
@rukai rukai marked this pull request as ready for review December 1, 2023 03:03
@rukai rukai requested a review from conorbros December 1, 2023 03:53
@rukai rukai merged commit 4f89245 into shotover:main Dec 4, 2023
40 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants