Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

k3s service failed to start #11366

Closed
NoSpread opened this issue Nov 25, 2024 · 1 comment
Closed

k3s service failed to start #11366

NoSpread opened this issue Nov 25, 2024 · 1 comment

Comments

@NoSpread
Copy link

Environmental Info:
K3s Version:

k3s version v1.30.6+k3s1 (1829eaae)
go version go1.22.8

Node(s) CPU architecture, OS, and Version:

Linux main 6.11.0-9-generic #9-Ubuntu SMP PREEMPT_DYNAMIC Mon Oct 14 13:19:59 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Linux gpu 6.11.0-9-generic #9-Ubuntu SMP PREEMPT_DYNAMIC Mon Oct 14 13:19:59 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Linux pi 6.11.0-1004-raspi #4-Ubuntu SMP PREEMPT_DYNAMIC Fri Sep 27 22:25:34 UTC 2024 aarch64 aarch64 aarch64 GNU/Linux

Cluster Configuration:
1 Server, 2 Agents

Describe the bug:
After a reboot, the server node won't start.

2024-11-25T16:29:42.897362+00:00 main systemd[1]: k3s.service: Scheduled restart job, restart counter is at 201.
2024-11-25T16:29:42.899480+00:00 main systemd[1]: Starting k3s.service - Lightweight Kubernetes...
2024-11-25T16:29:43.066326+00:00 main k3s[6509]: time="2024-11-25T16:29:43Z" level=info msg="Starting k3s v1.30.6+k3s1 (1829eaae)"
2024-11-25T16:29:43.067185+00:00 main k3s[6509]: time="2024-11-25T16:29:43Z" level=info msg="Configuring sqlite3 database connection pooling: maxIdleConns=2, maxOpenConns=0, connMaxLifetime=0s"
2024-11-25T16:29:43.067270+00:00 main k3s[6509]: time="2024-11-25T16:29:43Z" level=info msg="Configuring database table schema and indexes, this may take a moment..."
2024-11-25T16:29:43.067454+00:00 main k3s[6509]: time="2024-11-25T16:29:43Z" level=info msg="Database tables and indexes are up to date"
2024-11-25T16:29:43.071738+00:00 main k3s[6509]: time="2024-11-25T16:29:43Z" level=info msg="Kine available at unix://kine.sock"
2024-11-25T16:29:43.076193+00:00 main k3s[6509]: time="2024-11-25T16:29:43Z" level=info msg="Saving cluster bootstrap data to datastore"
2024-11-25T16:29:43.076949+00:00 main k3s[6509]: time="2024-11-25T16:29:43Z" level=fatal msg="starting kubernetes: preparing server: failed to normalize server token; must be in format K10<CA-HASH>::<USERNAME>:<PASSWORD> or <PASSWORD>"

Steps To Reproduce:
I am new to Kubernetes and k3s, so I tried a lot.
I have reinstalled k3s multiple times today, did many upgrades and configuring.
The last thing I was doing was installing longhorn and applying some cert-manager configuration.
Then I installed the longhorn drivers and added a default certificate.
After a reboot, the server node wouldn't start.

The /var/lib/rancher/k3s/server/node-token was empty.
Since I joined all the other servers today, I still have the original contents.
Like in #5345 I tried to remove the file, but k3s does not want to start. Also, adding the contents that I used to join the other nodes didn't work either.

Expected behavior:

  • k3s should start normal.

Actual behavior:

  • k3s service exits with a fatal error

Additional context / logs:

  • Scrolling through the logs, it seems like I installed k3s and k3s-agent on the same machine. Stopping k3s-agent and trying to start k3s did not work. Is there something to be overwritten, or can I still salvage this install? Thanks for your help.
@NoSpread
Copy link
Author

I misread some parts in the issue, it's not node_token, it's server/token....

@github-project-automation github-project-automation bot moved this from New to Done Issue in K3s Development Nov 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Done Issue
Development

No branches or pull requests

1 participant