OIDC Login does not work

[maaft]

Environmental Info:
K3s Version:

k3s version v1.29.9+k3s1 (e92d3b3b)
go version go1.22.6

Node(s) CPU architecture, OS, and Version:

Linux staging-control-plane-fsn1-irn 6.11.5-1-default #1 SMP PREEMPT_DYNAMIC Wed Oct 23 04:27:11 UTC 2024 (b4e3aa9) aarch64 aarch64 aarch64 GNU/Linux

Cluster Configuration:
3 servers, 7 agents

Describe the bug:
I've started the k3s servers with these kubelet args:

"kube-api-server_arg": |-
  - "oidc-issuer-url=https://sts.windows.net/4ee07d4d-79a8-4a9e-a038-e7c2e314b532/"
  - "oidc-client-id=59237398-07f6-4b9f-925b-8cb0428c4f26"
  - "oidc-groups-claim=groups"
  - "oidc-groups-prefix=oidc:"
  - "oidc-username-claim=upn"
  - "oidc-username-prefix=oidc:"

I get a new token with
kubectl oidc-login get-token --oidc-issuer-url=https://sts.windows.net/4ee07d4d-79a8-4a9e-a038-e7c2e314b532/ --oidc-client-id=59237398-07f6-4b9f-925b-8cb0428c4f26 --oidc-client-secret=<secret> --force-refresh

I created this cluster-role-binding:

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: cluster-adminbinding-group
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
  - apiGroup: rbac.authorization.k8s.io
    kind: Group
    name: "oidc:927d1cc3-fded-4a12-9d7d-f19b84903cd1"

When trying to authenticate against the k8s API with the fetched token I just get:

curl -H "Authorization: Bearer $TOKEN" \
     -X GET https://example.com:6443/api --insecure

{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "Unauthorized",
  "reason": "Unauthorized",
  "code": 401
} 

When looking at the k3s server logs with journalctl -fu k3s, I'll just get a generic:

authentication.go:73] "Unable to authenticate the request" err="invalid bearer token"

Here is the decoded token in JSON Format:

{
  "aud": "59237398-07f6-4b9f-925b-8cb0428c4f26",
  "iss": "https://sts.windows.net/4ee07d4d-79a8-4a9e-a038-e7c2e314b532/",
  "iat": 1730727909,
  "nbf": 1730727909,
  "exp": 1730731809,
  "amr": [
    "pwd",
    "mfa"
  ],
  "groups": [
    "927d1cc3-fded-4a12-9d7d-f19b84903cd1"
  ],
  "nonce": "wo_8QJ246MxNbT8-uOvb4W7P5NZEOZ_XmO5WLJbxznE",
  "oid": "77cac413-7ed2-40de-843b-206e5abb4965",
  "rh": "1.AS8ATX3gTqh5nkqgOOfC4xS1M5hzI1n2B59LkluMsEKMTyewANYvAA.",
  "sub": "nezZWhZkKpe2mbbrieekcJ0xWAUDp6VyZB6t1K-ID9M",
  "tid": "4ee07d4d-79a8-4a9e-a038-e7c2e314b532",
  "upn": "<redacted>",
  "uti": "vjivzFHEO0u5AfjRoP5AAA",
  "ver": "1.0"
}

Steps To Reproduce:

• Installed K3s: through k3s-terraform-hetzner project (self-managed k3s on hetzner cloud)
• follow this guide on how to create azure app registrations and how to use oidc-login
• create token,ClusterRoleBindings and try to auth

Expected behavior:
As I configured everything to be matching, I'd expect to be able to auth myself.

Otherwise, I'd expect that the server logs give me a better error message, i.e. WHAT is wrong with my token.

Actual behavior:
Instead, wasn't able to login and got only generic error message.

Additional context / logs:
Also tried to start k3s with --debug but this didn't change the verbosity of the failing authentication.
Disclaimer: I changed the UUIDs for privacy reasons.

[brandond]

"kube-api-server_arg": |-
  - "oidc-issuer-url=https://sts.windows.net/4ee07d4d-79a8-4a9e-a038-e7c2e314b532/"
  - "oidc-client-id=59237398-07f6-4b9f-925b-8cb0428c4f26"
  - "oidc-groups-claim=groups"
  - "oidc-groups-prefix=oidc:"
  - "oidc-username-claim=upn"
  - "oidc-username-prefix=oidc:"

Does it even start like this? I have no idea what you are doing here. It needs to be an actual YAML list and not a string. Also the key is kube-apiserver-arg not kube-api-server_arg:

"kube-apiserver-arg":
  - "oidc-issuer-url=https://sts.windows.net/4ee07d4d-79a8-4a9e-a038-e7c2e314b532/"
  - "oidc-client-id=59237398-07f6-4b9f-925b-8cb0428c4f26"
  - "oidc-groups-claim=groups"
  - "oidc-groups-prefix=oidc:"
  - "oidc-username-claim=upn"
  - "oidc-username-prefix=oidc:"

As I configured everything to be matching, I'd expect to be able to auth myself.
Otherwise, I'd expect that the server logs give me a better error message, i.e. WHAT is wrong with my token.

These are notes to take to the upstream project. K3s is a Kubernetes distro; we do not maintain the apiserver or other components ourselves. if you have complaints about how the apiserver behaves, those should be directed to https://github.com/kubernetes/kubernetes/

[maaft]

Omg, I'm so sorry. I'm not sure where and how in my setup the "kube-apiserver-arg" key became so messed up.

It works now. Closed.