Plugins Backend is unhealthy - Error 54113

[ianwilliams1]

Problem with this page

Jenkins Plugins pagesource file

Expected behavior

Expect to see plugins on plugins page

Actual behavior

Click on "Browse", nothing comes up.
Type in parameter to search for and search, nothing comes up.
Click on a plugin in the New /Trending / Top section and you get:

Error 503 Backend is unhealthy
Backend is unhealthy

Error 54113
Details: cache-yyc1430032-YYC 1696617802 4248943131

Varnish cache server

Initially got the same error by clicking on a plugin link from with my own instance plugin page.

Possible solution

After Googling the error, figured out to clear the jenkins.io specific cookies and data, which clears the error.
However, I would not expect everyone to know to take that action based on the site main page simply appearing not to work or the error message provided. It suggests it's a server issue, not a local cache issue.

[halkeye]

@dduportal @lemeurherve are we using varnish anywhere for plugin site?

Either way its a blip indicating some sort of outage, not something regular.

[dduportal]

Thanks for raising the issue! The problem is now gone: we had issues when updating the internal of the service (kubernetes label and selectors) as part of #3719.

I'll update this issue with more details but it looks fixed now. Can you confirm ?

[dduportal]

FTR:

• Hotfix in the helm chart: hotfix(plugin-site) separate svcs helm-charts#844
• Deployment: hotfix(publick8s) bump plugin-site to 0.3.2 kubernetes-management#4505

[dduportal]

@dduportal @lemeurherve are we using varnish anywhere for plugin site?

Varnish is what Fastly uses as the site is cached on Fastly. But as I broke the backend service plugin.origin.jenkins.io with jenkins-infra/helm-charts#843 unexpectedly, then Fastly started to decache and forwarding backend errors after ~1h

[dduportal]

For info and better tracking, I'm moving this issue from jenkins-infra/plugin-site to the Jenkins infra issue tracker in jenkins-infra/helpdesk

[dduportal]

Closing this issue as the incident was closed and did not repeat itself.