Retire pending workers

[BitTheByte]

Based on discussion at #817

[BitTheByte]

Everything looks good at least for now. I deployed the change to our production cluster and will provide updates if needed. Hopefully, everything works well 😄

[BitTheByte]

Surprisingly I had the most stable run ever. One note to mention is if a pod is restarting which means it's deployment in an unready state a small possibility might happen:

1. The operator gets the unready state
2. pod restarts fast enough and starts executing work
3. The operator kills the pod mid-run

So I was thinking of a way to execute the logic only on deployment that have pods in a pending state however I can't find a way to do that using kr8s

[BitTheByte]

Done! now the operator takes actions based on pending pods rather than the deployments

[jacobtomlinson]

Thanks for taking the time here. A few thoughts.

[jacobtomlinson]

I think you might be confusing the Pod phases with the status that kubectl displays. The possible values for phase are Pending, Running, Succeeded, Failed and Unknown.

https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.28/#podstatus-v1-core

[BitTheByte]

Yup, you're correct I've been debugging this using k9s

[BitTheByte]

Sorry for the long delay, unfortunately, I don't have much time to work on this. meanwhile, I'll close this PR to allow someone else to finish this

[BitTheByte]

@jacobtomlinson I believe this is ready to merge

[jacobtomlinson]

The CI appears to be hanging as a result of these changes. This needs looking into before we can merge.

[BitTheByte]

It is something related to building some go code. I tried to check what is wrong but didn't figure it out. it appears to be something related to the CI itself as I see most of the PRs are falling too.

[BitTheByte]

@jacobtomlinson If possible can you suggest any solution to solve this issue?

[jacobtomlinson]

Thanks for being patient here. I've nudged the CI back into a happy state and pulled main into this PR. So hopefully now any failures will only be related to this PR.

[jacobtomlinson]

Ok the tests are still failing here so they are certainly hanging due to the changes in this PR.

[BitTheByte]

Seems like they're timeout issues not something failing

[jacobtomlinson]

Sure but I expect the CI is timing out because the tests are hanging. The tests on main do not hang, which suggests the changes here cause the tests to hang.