KubernetesPodOperator with GCS FUSE sidecar sometimes stays running indefinitely #39625
Open
2 tasks done
Labels
area:providers
kind:bug
This is a clearly a bug
provider:cncf-kubernetes
Kubernetes provider related issues
Apache Airflow Provider(s)
cncf-kubernetes
Versions of Apache Airflow Providers
No response
Apache Airflow version
airflow-2.7.3
Operating System
linux
Deployment
Google Cloud Composer
Deployment details
No response
What happened
I created a dag with a KubernetesPodOperator which uses the annotations to mount GCS buckets using the Google Cloud Storage FUSE Container Storage Interface (CSI) Plugin.
Due to what I believe to be another bug, GoogleCloudPlatform/gcs-fuse-csi-driver#257, the GCS FUSE sidecar would sometimes stay running.
From log messages, I determined the operator was stuck in an infinite loop here:
airflow/airflow/providers/cncf/kubernetes/utils/pod_manager.py
Line 623 in 029cbae
This appears to be a somewhat known issue, as the above function seems to have a special case for the istio sidecar. The same treatment should hold for all sidecars.
What you think should happen instead
It should not be possible for a "rogue" sidecar container to cause the KubernetesPodOperator to end up in an infinite loop. This behavior ends up hogging resources on a cluster and eventually clogs up the whole cluster with zombie pods.
One possible fix would be an optional timeout for how long to wait for the pod to do its work.
Another possible fix would be to generalize the treatment for the istio sidecar for all types of sidecars.
How to reproduce
The container runs a simple python script that writes 100 files to /out
Anything else
I can work around the issue with:
Are you willing to submit PR?
Code of Conduct
The text was updated successfully, but these errors were encountered: