Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Take over agent configs #10

Merged
merged 14 commits into from
Apr 30, 2024
Merged

Take over agent configs #10

merged 14 commits into from
Apr 30, 2024

Conversation

QuentinBisson
Copy link
Contributor

What this PR does / why we need it

This PR takes over the remote write configuration from prometheus-meta-operator if mimir is enabled

Checklist

  • Update changelog in CHANGELOG.md.

@QuentinBisson QuentinBisson self-assigned this Apr 9, 2024
@QuentinBisson QuentinBisson requested a review from a team as a code owner April 9, 2024 11:46
@QuentinBisson
Copy link
Contributor Author

Configmap testing:

PMO CM:

k get cm -n org-giantswarm golem-remote-write-config -oyaml
apiVersion: v1
data:
  values: |
    global: {}
    prometheus-agent:
      externalLabels:
        cluster_id: golem
        cluster_type: management_cluster
        customer: giantswarm
        installation: golem
        organization: giantswarm
        pipeline: testing
        provider: capa
        region: eu-west-2
        service_priority: highest
      image:
        tag: v2.50.1
      shards: 1
      version: v2.50.1
kind: ConfigMap
metadata:
  creationTimestamp: "2023-05-09T13:34:16Z"
  finalizers:
  - observability.giantswarm.io/monitoring
  labels:
    app-operator.giantswarm.io/watching: "true"
  name: golem-remote-write-config
  namespace: org-giantswarm
  resourceVersion: "1006002975"
  uid: 16a0ff42-0365-4511-a443-cccf876a9416

Then edited by observability-operator:

k get cm -n org-giantswarm golem-remote-write-config -oyaml
apiVersion: v1
data:
  values: |
    prometheus-agent:
      externalLabels:
        cluster_id: golem
        cluster_type: management_cluster
        customer: giantswarm
        installation: golem
        organization: giantswarm
        pipeline: testing
        provider: capa
        region: eu-west-2
        service_priority: highest
      image:
        tag: v2.51.1
      shards: 1
      version: v2.51.1
kind: ConfigMap
metadata:
  creationTimestamp: "2023-05-09T13:34:16Z"
  finalizers:
  - observability.giantswarm.io/monitoring
  labels:
    app-operator.giantswarm.io/watching: "true"
  name: golem-remote-write-config
  namespace: org-giantswarm
  resourceVersion: "1006007639"
  uid: 16a0ff42-0365-4511-a443-cccf876a9416

@QuentinBisson QuentinBisson force-pushed the take-over-agent-configs branch from 28f9976 to 111ec8f Compare April 9, 2024 12:02
// QueryTSDBHeadSeries performs an instant query against Mimir.
func QueryTSDBHeadSeries(ctx context.Context, clusterName string) (float64, error) {
config := api.Config{
Address: "http://mimir-gateway.mimir.svc/prometheus",
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This now uses Mimir

api := v1.NewAPI(c)

queryContext, cancel := context.WithTimeout(ctx, 2*time.Minute)
query := fmt.Sprintf("sum(max_over_time(prometheus_agent_active_series{cluster_id=\"%s\"}[6h]))", clusterName)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Uses the active series in agents instead of prometheus tsdb head series so we can get rid of the MC proms

@QuentinBisson QuentinBisson force-pushed the take-over-agent-configs branch from 111ec8f to 769a6f5 Compare April 9, 2024 12:17
@@ -24,11 +24,16 @@ spec:
image: "{{ .Values.image.registry }}/{{ .Values.image.name }}:{{ default .Chart.Version .Values.image.tag }}"
args:
- --leader-elect
- --management-cluster-base-domain={{ $.Values.managementCluster.baseDomain }}
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently configured to send to be .installationName.baseDomain but

@QuentinBisson QuentinBisson force-pushed the take-over-agent-configs branch from dded51c to 3530c3f Compare April 10, 2024 08:49
@marieroque marieroque self-requested a review April 10, 2024 10:32
pkg/monitoring/prometheusagent/service.go Outdated Show resolved Hide resolved
pkg/monitoring/prometheusagent/service.go Show resolved Hide resolved
pkg/monitoring/prometheusagent/service.go Show resolved Hide resolved
pkg/monitoring/prometheusagent/service.go Outdated Show resolved Hide resolved
pkg/monitoring/prometheusagent/service.go Outdated Show resolved Hide resolved
pkg/monitoring/prometheusagent/service.go Show resolved Hide resolved
pkg/monitoring/prometheusagent/service.go Show resolved Hide resolved
pkg/monitoring/prometheusagent/config.go Outdated Show resolved Hide resolved
pkg/monitoring/prometheusagent/config.go Outdated Show resolved Hide resolved
pkg/monitoring/prometheusagent/secret.go Show resolved Hide resolved
@QuentinBisson QuentinBisson force-pushed the take-over-agent-configs branch from 177d3f7 to bc1f838 Compare April 10, 2024 12:21
Signed-off-by: QuentinBisson <[email protected]>
@QuentinBisson QuentinBisson force-pushed the take-over-agent-configs branch from bc1f838 to 2d49d29 Compare April 10, 2024 12:25
@QuentinBisson QuentinBisson enabled auto-merge (squash) April 30, 2024 09:19
@QuentinBisson QuentinBisson disabled auto-merge April 30, 2024 09:21
@QuentinBisson QuentinBisson merged commit fc5802b into main Apr 30, 2024
7 checks passed
@QuentinBisson QuentinBisson deleted the take-over-agent-configs branch April 30, 2024 09:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants