Cluster-admin on management cluster for customers

[teemow]

We approaching a point where it makes sense to think about how we can enable cluster-admin for customers on the management cluster. Obviously this should be considered as an option for an advanced platform team. And we need to think about rail guards and responsibilities. But this would allow our customers to build their own platform solutions without handcuffs as the management cluster is a central place for the platform teams to offer services to their development teams.

Pre-requirement: We need to sort out the scope of our secrets on the management:

• https://github.com/giantswarm/giantswarm/issues/22385
• https://github.com/giantswarm/giantswarm/issues/20106

Example: #948 (comment)

[kuosandys]

@weatherhog would like to take care of following up

[teemow]

@weatherhog we talked about this story in the honey badger planning session and @gianfranco-l will hand this over to you.

[gianfranco-l]

the plan is for honeybadger to write down some process docs first and then hold a knowledge sharing session with Rainbow (specifically about the knowledge sharing, there was an informal chat between the two teams TFs already)

[mproffitt]

There are a few key questions coming up over enabling cluster-admin for customers that need to be addressed as part of this issue.

• How can we ensure that changes made by a customer do not have a detrimental affect on the operation of the MC
• How do we handle support of the MC when a customer has made unfettered changes to its general operation?
• how do we protect gs owned resources

A question came in to me recently on how a customer can deploy core resources / submit PRs to enable functionality they require for a solution running on their MC and this provoked a discussion within @giantswarm/team-honeybadger with a fairly simple answer.

The question is, what is the problem we're trying to solve with granting customers cluster-admin on a management cluster. IMHO for the most part, they require the ability to create resources there but largely don't care how those resources are created. They also want a managed solution which in my view, means they want us to help with any changes they are going to make to the MC.

Whether the customer is using GitOps or not, their management clusters are, for the most part, provisioned using a GitOps driven approach.

This presents an alternative solution for us which I believe is worth exploring as it's relatively clean, doesn't require giving cluster-admin and allows us to maintain a better support-model for customers who desire this functionality.

This potential solution would be to enable kustomize remote-build inside the respective MC gitops configuration which would call out to a location within the customer shared repository. This location could contain resources the customer needs to enable with cluster-admin permissions.

Being in the shared-repository, this then enables any GS engineer to view changes a customer makes. A customer would raise a PR against this which we have the opportunity to sign off on before it goes live, rejecting any which would breach our capability of managing/maintaining the MC.

This does not necessarily negate granting cluster-admin in the future, but could be used as a stepping stone to building confidence both within GS, and with our customers that we are able to effectively operate an MC that a customer is actively adapting to suit their purpose.

WDYT?

[teemow]

For me this is less a technical question but a mindset question. Customers have full access to workload clusters too. Should they use this? No. Can they use this. Yes. Workload clusters are for their development teams.

The management cluster does "the platform management" and the platform team of the customer is working with the management cluster. But they are often limited and have to find workarounds.

In general we've always seen us as enablers and not as a managed service. We did'nt want to stand in the way of our customers. This is why we've given full access to the workload clusters although customers can break them. The questions @mproffitt wrote down apply to workload clusters too. Still we gave them admin access.

This is why I'd like us to review the shared responsibility on management clusters. Why are we so afraid of platform teams breaking the management cluster? Why aren't we afraid that customers break workload clusters. And how often will they do this? Imo the freedom for the customer (gs not standing in the way) and us learning about the customers use cases (to adapt our own offering) outweights the risks.

The proposed gitops solution is imo falling too short. Customers need access to debug their services. They need to be able to restart their services. They should learn and see how our services are running etc.

[teemow]

Tracked here: https://github.com/giantswarm/giantswarm/issues/30449