Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable MetricsExporter and PrometheusRules on each StorageCluster namespaces #2293

Conversation

aruniiird
Copy link
Contributor

@aruniiird aruniiird commented Dec 1, 2023

In a multi storagecluster scenario, both metrics-exporter and PrometheusRules should be enabled on all the namespaces where the StorageClusters are installed.

Follow up PR: #2313

@openshift-ci openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Dec 1, 2023
Copy link
Contributor

openshift-ci bot commented Dec 1, 2023

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@aruniiird aruniiird force-pushed the multicluster-changes-to-metrics-exporter-n-prometheus-rules branch 3 times, most recently from 9c3163c to ad51dca Compare December 4, 2023 19:53
@agarwal-mudit
Copy link
Member

@aruniiird is this still WIP?

@aruniiird aruniiird force-pushed the multicluster-changes-to-metrics-exporter-n-prometheus-rules branch 2 times, most recently from cfcb1d7 to 58a9d9b Compare December 8, 2023 10:58
@aruniiird aruniiird marked this pull request as ready for review December 8, 2023 11:00
@openshift-ci openshift-ci bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Dec 8, 2023
@aruniiird aruniiird changed the title Enable MetricsExporter and PrometheusRules only on main namespace Enable MetricsExporter and PrometheusRules on each StorageCluster namespaces Dec 8, 2023
controllers/storagecluster/exporter.go Outdated Show resolved Hide resolved
controllers/storagecluster/exporter.go Outdated Show resolved Hide resolved
controllers/storagecluster/exporter.go Outdated Show resolved Hide resolved
controllers/storagecluster/exporter.go Show resolved Hide resolved
controllers/storagecluster/exporter.go Outdated Show resolved Hide resolved
controllers/storagecluster/exporter.go Outdated Show resolved Hide resolved
controllers/storagecluster/prometheus.go Outdated Show resolved Hide resolved
@aruniiird aruniiird force-pushed the multicluster-changes-to-metrics-exporter-n-prometheus-rules branch from 58a9d9b to 21f4ac4 Compare December 8, 2023 12:08
Copy link
Contributor

@umangachapagain umangachapagain left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Commit heading needs to be updated. It's still referring to old implementation.

@aruniiird aruniiird force-pushed the multicluster-changes-to-metrics-exporter-n-prometheus-rules branch 3 times, most recently from f8aa8b9 to 42a14e7 Compare December 8, 2023 12:29
@aruniiird aruniiird force-pushed the multicluster-changes-to-metrics-exporter-n-prometheus-rules branch 4 times, most recently from b7b9f4b to cdf0cf4 Compare December 10, 2023 17:45
controllers/storagecluster/exporter.go Show resolved Hide resolved
controllers/storagecluster/exporter.go Outdated Show resolved Hide resolved
controllers/storagecluster/exporter.go Show resolved Hide resolved
controllers/storagecluster/exporter.go Outdated Show resolved Hide resolved
controllers/storagecluster/exporter.go Outdated Show resolved Hide resolved
controllers/storagecluster/exporter.go Show resolved Hide resolved
controllers/storagecluster/exporter.go Outdated Show resolved Hide resolved
controllers/storagecluster/exporter.go Show resolved Hide resolved
controllers/storagecluster/exporter.go Show resolved Hide resolved
controllers/storagecluster/exporter.go Show resolved Hide resolved
Copy link
Contributor

@umangachapagain umangachapagain left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generated alert changes should also be committed with mixin changes.

Copy link
Contributor

@umangachapagain umangachapagain left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since we are creating ServiceAccount and RBACs from ocs-operator, we must provide those access to ocs-operator serviceaccount as well.

Copy link
Member

@jmolmo jmolmo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM after addressing suggestions.
@aruniiird , can you comment briefly how have you tested the changes?

@aruniiird aruniiird force-pushed the multicluster-changes-to-metrics-exporter-n-prometheus-rules branch from 5b5a249 to bbb9fba Compare December 11, 2023 13:11
@aruniiird aruniiird force-pushed the multicluster-changes-to-metrics-exporter-n-prometheus-rules branch from ee2c993 to 22c80ad Compare December 11, 2023 15:09
@aruniiird aruniiird force-pushed the multicluster-changes-to-metrics-exporter-n-prometheus-rules branch 7 times, most recently from 1e4734b to 7af36ad Compare December 12, 2023 05:39
@aruniiird
Copy link
Contributor Author

All the changes are done (except for the functional tests).
Verified the fix on @iamniting provided cluster...

image

@jmolmo , fix is now tested.
@umangachapagain , please take a look

metrics/internal/collectors/cluster-advance-feature-use.go Outdated Show resolved Hide resolved
metrics/main.go Outdated Show resolved Hide resolved
Copy link
Contributor

@umangachapagain umangachapagain left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please update the commit messages as per our Contribution Guide.
Commit details should explain what changed and why we changed it.

Also, would be nice to see test results for newly updated alerts.

@aruniiird aruniiird force-pushed the multicluster-changes-to-metrics-exporter-n-prometheus-rules branch 4 times, most recently from c778247 to 86428e0 Compare December 12, 2023 08:01
@aruniiird
Copy link
Contributor Author

Please update the commit messages as per our Contribution Guide. Commit details should explain what changed and why we changed it.

Also, would be nice to see test results for newly updated alerts.

Updated the commit messages

@aruniiird aruniiird force-pushed the multicluster-changes-to-metrics-exporter-n-prometheus-rules branch from 86428e0 to e6fd970 Compare December 12, 2023 10:42
@openshift-merge-robot openshift-merge-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Dec 12, 2023
In a multi storagecluster scenario, we have to create the following
resources on each of the storagecluster's namespace
a. metrics exporter deployment and needed cluster roles
b. exporter service and servicemonitor
c. prometheus rules

Signed-off-by: Arun Kumar Mohan <[email protected]>
In multicluster scenario, metric exporter should get resources only
for the namespace which it is deployed to.
Changes are backward compatible and should work on all scenarioes.

Signed-off-by: Arun Kumar Mohan <[email protected]>
In a multicluster mode, each alert should have clear indication
about which cluster and namespace from which the it is being fired.

Signed-off-by: Arun Kumar Mohan <[email protected]>
@aruniiird aruniiird force-pushed the multicluster-changes-to-metrics-exporter-n-prometheus-rules branch from e6fd970 to 9ae46b2 Compare December 12, 2023 12:54
@openshift-merge-robot openshift-merge-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Dec 12, 2023
@openshift-ci openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Dec 12, 2023
Copy link
Contributor

openshift-ci bot commented Dec 12, 2023

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: aruniiird, umangachapagain

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Dec 12, 2023
@openshift-merge-bot openshift-merge-bot bot merged commit 000607b into red-hat-storage:main Dec 12, 2023
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. lgtm Indicates that a PR is ready to be merged.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants