Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Change error when draining a node with Pod Disruption Budget #797

Closed
OttaviaB opened this issue Nov 25, 2024 · 0 comments · Fixed by #798
Closed

Change error when draining a node with Pod Disruption Budget #797

OttaviaB opened this issue Nov 25, 2024 · 0 comments · Fixed by #798

Comments

@OttaviaB
Copy link
Contributor

SUMMARY

When k8s_drain tries to evict a pod which cannot be evicted due to a pod disruption budget an error "Too Many Requests" is returned.
Changing this message to something more meaningful would make it easier to work with retries/until loops in an Ansible playbook to drain the node.

ISSUE TYPE
  • Feature Idea
COMPONENT NAME

kubernetes.core.k8s_drain

ADDITIONAL INFORMATION

Current behaviour:

TASK [Drain node] *************************************************************************
Montag 25 November 2024  09:37:09 +0100 (0:00:00.015)       0:00:00.015 ******* 
fatal: [host -> localhost]: FAILED! => {"changed": false, "msg": "Failed to delete pod kube-public/draintest-6b84677b99-c6s5w due to: Too Many Requests"}

Proposed behaviour:

TASK [Drain node] *************************************************************************
Montag 25 November 2024  09:20:28 +0100 (0:00:00.014)       0:00:00.014 ******* 
fatal: [host -> localhost]: FAILED! => {"changed": false, "msg": "Failed to delete pod kube-public/draintest-6b84677b99-9jf7m due to: Cannot evict pod as it would violate the pod's disruption budget."}
---
- hosts: "{{ target }}"
  serial: 1
  gather_facts: false
  tasks:
    - name: Drain node
        kubernetes.core.k8s_drain:
        kubeconfig: "{{ kubeconfig_path }}"
        name: "{{ inventory_hostname }}"
        delete_options:
            ignore_daemonsets: true
            delete_emptydir_data: true
            wait_timeout: 100
            disable_eviction: false
            wait_sleep: 1
        delegate_to: localhost

See also:

#711

kubernetes/kubernetes#106286

softwarefactory-project-zuul bot pushed a commit that referenced this issue Dec 11, 2024
…798)

SUMMARY
Closes #797 .
The error message "Too Many Requests" is confusing and is changed to a more meaningful message:
TASK [Drain node] *************************************************************************
Montag 25 November 2024  09:20:28 +0100 (0:00:00.014)       0:00:00.014 ******* 
fatal: [host -> localhost]: FAILED! => {"changed": false, "msg": "Failed to delete pod kube-public/draintest-6b84677b99-9jf7m due to: Cannot evict pod as it would violate the pod's disruption budget."}


The new task output would allow to deal with a pod disruption budget with the retries/until logic in a more controlled way:
---
- hosts: "{{ target }}"
  serial: 1
  gather_facts: false
  tasks:
    - name: Drain node
      kubernetes.core.k8s_drain:
        kubeconfig: "{{ kubeconfig_path }}"
        name: "{{ inventory_hostname }}"
        delete_options:
          ignore_daemonsets: true
          delete_emptydir_data: true
          wait_timeout: 100
          disable_eviction: false
          wait_sleep: 1
      delegate_to: localhost
      retries: 10
      delay: 5
      until: drain_result is success or 'disruption budget' not in drain_result.msg
      register: drain_result

ISSUE TYPE


Feature Pull Request

COMPONENT NAME
k8s_drain

Reviewed-by: Mike Graves <[email protected]>
patchback bot pushed a commit that referenced this issue Dec 11, 2024
…798)

SUMMARY
Closes #797 .
The error message "Too Many Requests" is confusing and is changed to a more meaningful message:
TASK [Drain node] *************************************************************************
Montag 25 November 2024  09:20:28 +0100 (0:00:00.014)       0:00:00.014 *******
fatal: [host -> localhost]: FAILED! => {"changed": false, "msg": "Failed to delete pod kube-public/draintest-6b84677b99-9jf7m due to: Cannot evict pod as it would violate the pod's disruption budget."}

The new task output would allow to deal with a pod disruption budget with the retries/until logic in a more controlled way:
---
- hosts: "{{ target }}"
  serial: 1
  gather_facts: false
  tasks:
    - name: Drain node
      kubernetes.core.k8s_drain:
        kubeconfig: "{{ kubeconfig_path }}"
        name: "{{ inventory_hostname }}"
        delete_options:
          ignore_daemonsets: true
          delete_emptydir_data: true
          wait_timeout: 100
          disable_eviction: false
          wait_sleep: 1
      delegate_to: localhost
      retries: 10
      delay: 5
      until: drain_result is success or 'disruption budget' not in drain_result.msg
      register: drain_result

ISSUE TYPE

Feature Pull Request

COMPONENT NAME
k8s_drain

Reviewed-by: Mike Graves <[email protected]>
(cherry picked from commit 52f2cb5)
patchback bot pushed a commit that referenced this issue Dec 11, 2024
…798)

SUMMARY
Closes #797 .
The error message "Too Many Requests" is confusing and is changed to a more meaningful message:
TASK [Drain node] *************************************************************************
Montag 25 November 2024  09:20:28 +0100 (0:00:00.014)       0:00:00.014 *******
fatal: [host -> localhost]: FAILED! => {"changed": false, "msg": "Failed to delete pod kube-public/draintest-6b84677b99-9jf7m due to: Cannot evict pod as it would violate the pod's disruption budget."}

The new task output would allow to deal with a pod disruption budget with the retries/until logic in a more controlled way:
---
- hosts: "{{ target }}"
  serial: 1
  gather_facts: false
  tasks:
    - name: Drain node
      kubernetes.core.k8s_drain:
        kubeconfig: "{{ kubeconfig_path }}"
        name: "{{ inventory_hostname }}"
        delete_options:
          ignore_daemonsets: true
          delete_emptydir_data: true
          wait_timeout: 100
          disable_eviction: false
          wait_sleep: 1
      delegate_to: localhost
      retries: 10
      delay: 5
      until: drain_result is success or 'disruption budget' not in drain_result.msg
      register: drain_result

ISSUE TYPE

Feature Pull Request

COMPONENT NAME
k8s_drain

Reviewed-by: Mike Graves <[email protected]>
(cherry picked from commit 52f2cb5)
softwarefactory-project-zuul bot pushed a commit that referenced this issue Dec 11, 2024
…798) (#815)

This is a backport of PR #798 as merged into main (52f2cb5).
SUMMARY
Closes #797 .
The error message "Too Many Requests" is confusing and is changed to a more meaningful message:
TASK [Drain node] *************************************************************************
Montag 25 November 2024  09:20:28 +0100 (0:00:00.014)       0:00:00.014 ******* 
fatal: [host -> localhost]: FAILED! => {"changed": false, "msg": "Failed to delete pod kube-public/draintest-6b84677b99-9jf7m due to: Cannot evict pod as it would violate the pod's disruption budget."}


The new task output would allow to deal with a pod disruption budget with the retries/until logic in a more controlled way:
---
- hosts: "{{ target }}"
  serial: 1
  gather_facts: false
  tasks:
    - name: Drain node
      kubernetes.core.k8s_drain:
        kubeconfig: "{{ kubeconfig_path }}"
        name: "{{ inventory_hostname }}"
        delete_options:
          ignore_daemonsets: true
          delete_emptydir_data: true
          wait_timeout: 100
          disable_eviction: false
          wait_sleep: 1
      delegate_to: localhost
      retries: 10
      delay: 5
      until: drain_result is success or 'disruption budget' not in drain_result.msg
      register: drain_result

ISSUE TYPE


Feature Pull Request

COMPONENT NAME
k8s_drain
softwarefactory-project-zuul bot pushed a commit that referenced this issue Dec 11, 2024
…798) (#816)

This is a backport of PR #798 as merged into main (52f2cb5).
SUMMARY
Closes #797 .
The error message "Too Many Requests" is confusing and is changed to a more meaningful message:
TASK [Drain node] *************************************************************************
Montag 25 November 2024  09:20:28 +0100 (0:00:00.014)       0:00:00.014 ******* 
fatal: [host -> localhost]: FAILED! => {"changed": false, "msg": "Failed to delete pod kube-public/draintest-6b84677b99-9jf7m due to: Cannot evict pod as it would violate the pod's disruption budget."}


The new task output would allow to deal with a pod disruption budget with the retries/until logic in a more controlled way:
---
- hosts: "{{ target }}"
  serial: 1
  gather_facts: false
  tasks:
    - name: Drain node
      kubernetes.core.k8s_drain:
        kubeconfig: "{{ kubeconfig_path }}"
        name: "{{ inventory_hostname }}"
        delete_options:
          ignore_daemonsets: true
          delete_emptydir_data: true
          wait_timeout: 100
          disable_eviction: false
          wait_sleep: 1
      delegate_to: localhost
      retries: 10
      delay: 5
      until: drain_result is success or 'disruption budget' not in drain_result.msg
      register: drain_result

ISSUE TYPE


Feature Pull Request

COMPONENT NAME
k8s_drain
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant