Automatically add workload cluster kubeconfig to Argo

[teemow]

Some customers use ArgoCD on the management cluster. To be able to deploy on all workload clusters with ArgoCD the kubeconfig needs to be added to ArgoCD.

See upstream issue: argoproj/argo-cd#4651

[teemow]

@puja108 @QuentinBisson this is the issue I've created yesterday. Afaik there is another one. Where is it?

[puja108]

Honeybadger has the bigger context right now and will try to quickly enable a workaround for the customer, but involving Rainbow to move the context of access to WCs from MC operators and MC isolation/security to rainbow

[gianfranco-l]

team Honeybadger will document the context and process of RBAC management and then will kick off a pairing session with Rainbow to enable this use case

[puja108]

Upstream Issue with pointer to a Kyverno-based solution: argoproj/argo-cd#4651

[puja108]

I also cannot find the other issue anymore, maybe Quentin knows. Anyway, we'll focus on solving the customer problem now and taking up the use case mentioned by 2 customers.

[teemow]

@gianfranco-l the generic issues that needs to be handed over to rainbow are these:

• Access to clusters in other organizations #1666
• Cluster-admin on management cluster for customers #1711

This one here is about a solution for the customers. And the result of the discussion was that honey badger will solve this.

[gianfranco-l]

thank you very much for clarifying :)

[uvegla]

Update: added a kyverno policy to all clusters - except KVM ones at the moment, because they are special to other ones in this regard. The policy creates the Argo secret in the same namespace where the original kube config secret is.

The secrets follow the naming scheme of: <CLUSTER_NAME>-kubeconfig-argo and contain the following fields under .data:

config: <SEE_EXAMPLE_BELOW>
name: <CLUSTER_NAME>
server: <API_URL>

An example what is base64 encoded into the config field:

{
  "tlsClientConfig": {
    "insecure": false,
    "caData": "...",
    "certData": "...",
    "keyData": "..."
  }
}

Will look into KVM, but other cluster seem fine at the moment.

[teemow]

Thanks @uvegla ! There is no need to do this on KVM.

I'm not sure if we really should roll this out to all MCs. Copying cluster credentials where it isn't needed increases risk. This is only necessary for MCs where argo is installed. I'd say the kyverno policy belongs into the helm chart of argo.

[uvegla]

@teemow Fair point, we deemed it fine to add to all cluster because it is the same namespace as the kubeconfig resides in anyway. I feel it can be confusing in certain scenarios to find a secret called *-argo if you don't use argo. The argo chart actually makes sense, what do you think @mproffitt?

[uvegla]

@teemow @mproffitt One important note tho: the policy differs for legacy and CAPx clusters which may prove challenging to decide from a Helm chart.

[mproffitt]

The helm approach has a couple of drawbacks that I see.

• As you point out, differentiating between legacy and CAPx clusters
• Detecting when a customer decides to install argo and automatically installing the policy when applicable.

That said, I also agree with the points made by @teemow about risk.

kyverno itself can potentially come to the rescue here and by using an additional generate, it should be possible to have the policy on standby in all MCs but only have it in existence if it detects the existence of CRDs owned by ArgoCD. See https://kyverno.io/docs/writing-policies/generate/

WDYT? Is this something that would potentially satisfy all scenarios?

[teemow]

One customer is using a service account on the workload cluster. This seems to be a much better solution instead reusing the same kubeconfig to connect to the clusters. I'll add this information to the more general issue: #1666

We can deploy the kyverno policy on the manamagent cluster for the other customer that wants to use Argo imo. And then discuss the general solution in the other issue.

[uvegla]

Deployed it to the select customers management cluster. Closing this in favour of #1666