Promtail deployment POC

[TheoBrigitte]

Towards: #311

We need to evaluate how to deploy Promtail into clusters.

TODOs:

• Add Promtail into observability-bundle, with empty configuration and disabled by default
• Check how to enable Promtail via the observability-bundle config
• Check how to update Promtail config

[marieroque]

PR to add promtail-app to observability-bundle and disabled it by default:
giantswarm/observability-bundle#43

[marieroque]

Follow instructions in the table below to enable promtail-app:

Type	Who	Where	How
Vintage	All MC	config	clusterID: {{ .codename }} organization: giantswarm apps: promtail-app: enabled: true
Vintage	Specific MC	config	1. create a directory observability-bundle 2. add configmap-values.yaml.patch apps: promtail-app: enabled: true
CAPA	All MC	default-apps-aws	Add to userConfig section: observabilityBundle: configMap: values: | apps: promtail-app: enabled: true
CAPA	Specific MC	management-clusters-fleet	Add to data.values section: userConfig: observabilityBundle: configMap: values: | apps: promtail-app: enabled: true
CAPG	All MC	TODO
CAPG	Specific MC	TODO
CAPZ	All MC	TODO
CAPZ	Specific MC	TODO
CAPVCD	All MC	TODO
CAPVCD	Specific MC	TODO

[marieroque]

PR to enable/disable promtail-app on Vintage clusters: https://github.com/giantswarm/config/pull/1614
PR to enable/disable promtail-app on CAPA clusters: giantswarm/default-apps-aws#171

[TheoBrigitte]

I see you are actually merging a lot of things here. I would like to remind and point out that the scope of this issue is to prototype our idea. Let's first make sure we can achieve what we want :

• deploy promtail disabled
• enable it
• update its config

Once we know this is possible we should gather and reflect on how we can implement this in a dynamic way, probably via a logging-operator.

[marieroque]

=> We reduce the scope to Vintage AWS only

[primeroz]

CAPZ | All MC | TODO

FYI i deployed, by hand, promtail on glippy MC with the help of @hervenicol

Ideally if i can help i would like to get into CAPZ MC properly so when i rebuild the MC i don't loose it. Let me know if / how i can help

[marieroque]

To deploy/undeploy promtail on a Vintage WC, we need to:

1. Create a configmap for the WC observability-bundle

apiVersion: v1
data:
  values: |
    apps:
      promtail-app:
        enabled: true
kind: ConfigMap
metadata:
  name: t3yw2-observability-bundle-user-values
  namespace: t3yw2

2. Reference this configmap in the App CR

....
spec:
  catalog: default-test
  userConfig:
    configMap:
      name: t3yw2-observability-bundle-user-values
      namespace: t3yw2
    secret:
      name: ""
      namespace: ""

Then promtail-app is created on WC-namespace on MC and promtail is deployed on the WC.

To disable (undeploy) it, you need to edit the configmap and set enabled: false

[QuentinBisson]

@marieroque if the procedure is tested, then I think we can close this

[marieroque]

Ok enabling/disabling promtail on WC works.

Now, I'd like to write the process to

• update promtail config on WC
• enable/disable promtail on MC
• update promtail config on MC

[marieroque]

• enable/disable promtail on MC
  Edit the observability-bundle-konfigure configmap:

apiVersion: v1
data:
  configmap-values.yaml: |
    clusterID: gauss
    organization: giantswarm
    apps:
      promtail-app:
        enabled: true
kind: ConfigMap

• update promtail config on MC/WC:
  PR which describe the process to customize promtail secret and configmap:
  Adding default configmap and secret to promtail-app observability-bundle#49

[marieroque]

How do we proceed with this ticket:

We keep the PR to base our logging operator development on it ?
or
We write the process somewhere else ?

[QuentinBisson]

I would probably merge this one as long as everyone is okay with the cm and secret name giantswarm/observability-bundle#49

and then we can proceed with the logging-controller.

I would ideally write everything down in the intranet for now yes

[marieroque]

Intranet documentation which describe the process:
https://docs.google.com/document/d/16YjS86jAXqIqEU5K-AQfYNrhcHVS-CQmq0kqWw6gtfU/edit?usp=share_link

[TheoBrigitte]

I think the term watch is not exact here. Logging Operator will not watch observability bundle but rather Cluster CR I believe. Because Cluster CR is the source of truth for which Cluster we are ingesting logs for and this is also where we will define if logging should be enabled or not. We can elaborate on this when we get there.

Nice job! 🚀

[marieroque]

@TheoBrigitte I updated the schema, if you want to take a look...