Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

(Solved) failure when minikube enable ingress #16828

Closed
ongiant opened this issue Jul 6, 2023 · 15 comments
Closed

(Solved) failure when minikube enable ingress #16828

ongiant opened this issue Jul 6, 2023 · 15 comments
Labels
addon/ingress kind/support Categorizes issue or PR as a support question. lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed.

Comments

@ongiant
Copy link

ongiant commented Jul 6, 2023

What Happened?

when I try to enable ingress addon, but it fails.
Command for reproducing the issue:

minikube start --profile polar

console output:

console output details

😄  [polar] minikube v1.30.1 on Arch "23.0.0"                                                                                                                
✨  Automatically selected the docker driver                                                                                                                 
📌  Using Docker driver with root privileges                                                                                                                 
❗  Local proxy ignored: not passing HTTP_PROXY=socks5://localhost:7891 to docker env.                                                                       
❗  Local proxy ignored: not passing HTTPS_PROXY=socks5://localhost:7891 to docker env.                                                                      
❗  Local proxy ignored: not passing HTTP_PROXY=socks5://localhost:7891 to docker env.                                                                       
❗  Local proxy ignored: not passing HTTPS_PROXY=socks5://localhost:7891 to docker env.                                                                      
👍  Starting control plane node polar in cluster polar                                                                                                       
🚜  Pulling base image ...                                                                                                                                   
🔥  Creating docker container (CPUs=2, Memory=3900MB) ...                                                                                                    
❗  Local proxy ignored: not passing HTTP_PROXY=socks5://localhost:7891 to docker env.                                                                       
❗  Local proxy ignored: not passing HTTPS_PROXY=socks5://localhost:7891 to docker env.                                                                      
❗  Local proxy ignored: not passing HTTP_PROXY=socks5://localhost:7891 to docker env.                                                                       
❗  Local proxy ignored: not passing HTTPS_PROXY=socks5://localhost:7891 to docker env.                                                                      
🌐  Found network options:                                                                                                                                   
    ▪ HTTP_PROXY=socks5://localhost:7891                                                                                                                     
    ▪ HTTPS_PROXY=socks5://localhost:7891                                                                                                                    
    ▪ NO_PROXY=localhost,127.0.0.1,192.168.1.1,::1,*.local,10.96.0.0/12,192.168.59.0/24,192.168.49.0/24,192.168.39.0/24                                      
    ▪ http_proxy=socks5://localhost:7891                                                                                                                     
    ▪ https_proxy=socks5://localhost:7891                                                                                                                    
    ▪ no_proxy=localhost,127.0.0.1,192.168.1.1,::1,*.local,10.96.0.0/12,192.168.59.0/24,192.168.49.0/24,192.168.39.0/24                                      
❗  This container is having trouble accessing https://registry.k8s.io                                                                                       
💡  To pull new external images, you may need to configure a proxy: https://minikube.sigs.k8s.io/docs/reference/networking/proxy/                            
🐳  Preparing Kubernetes v1.26.3 on Docker 23.0.2 ...                                                                                                        
    ▪ env NO_PROXY=localhost,127.0.0.1,192.168.1.1,::1,*.local,10.96.0.0/12,192.168.59.0/24,192.168.49.0/24,192.168.39.0/24                                  
🔗  Configuring bridge CNI (Container Networking Interface) ...                                                                                              
🔎  Verifying Kubernetes components...                                                                                                                       
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5                                                                                                 
🌟  Enabled addons: default-storageclass, storage-provisioner                                                                                                
🏄  Done! kubectl is now configured to use "polar" cluster and "default" namespace by default

I notice that 'This container is having trouble accessing https://registry.k8s.io ' in above output logs, but I have set proxy for minikube, and referring to this official documentation (I adopted the regular install scheme), I configured a proxy for Docker daemon on my local machine, and it is working properly, so I can't understand this hint.

Then I input minikube addons enable ingress --profile polar, the output is:

💡  ingress is an addon maintained by Kubernetes. For any concerns contact minikube on GitHub.                                                               
You can view the list of minikube maintainers at: https://github.com/kubernetes/minikube/blob/master/OWNERS                                                  
    ▪ Using image registry.k8s.io/ingress-nginx/controller:v1.7.0                                                                                            
    ▪ Using image registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20230312-helm-chart-4.5.2-28-g66a760794                                                
    ▪ Using image registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20230312-helm-chart-4.5.2-28-g66a760794                                                
🔎  Verifying ingress addon...                                                                                                                               
                                                                                                                                                             
❌  Exiting due to MK_ADDON_ENABLE: enable failed: run callbacks: running callbacks: [waiting for app.kubernetes.io/name=ingress-nginx pods: timed out waitin
g for the condition]                                                                                                                                         
                                                                                                                                                             
╭───────────────────────────────────────────────────────────────────────────────────────────╮                                                                
│                                                                                           │                                                                
│    😿  If the above advice does not help, please let us know:                             │                                                                
│    👉  https://github.com/kubernetes/minikube/issues/new/choose                           │                                                                
│                                                                                           │                                                                
│    Please run `minikube logs --file=logs.txt` and attach logs.txt to the GitHub issue.    │                                                                
│    Please also attach the following file to the GitHub issue:                             │                                                                
│    - /tmp/minikube_addons_ae4fcb4856619e6ac42e679469959984454bfd00_0.log                  │                                                                
│                                                                                           │                                                                
╰───────────────────────────────────────────────────────────────────────────────────────────╯   

Then I try to debug, there are some commands and outputs:
1:

kubectl get pods -A

kubectl get pods -A

NAMESPACE NAME READY STATUS RESTARTS AGE
ingress-nginx ingress-nginx-admission-create-6twp2 0/1 ImagePullBackOff 0 19m
ingress-nginx ingress-nginx-admission-patch-6578w 0/1 ImagePullBackOff 0 19m
ingress-nginx ingress-nginx-controller-6cc5ccb977-26nj7 0/1 ContainerCreating 0 19m
kube-system coredns-787d4945fb-jkqp5 1/1 Running 2 (29m ago) 91m
kube-system etcd-polar 1/1 Running 1 (30m ago) 91m
kube-system kube-apiserver-polar 1/1 Running 1 (30m ago) 91m
kube-system kube-controller-manager-polar 1/1 Running 1 (30m ago) 91m
kube-system kube-proxy-77rcn 1/1 Running 1 (59m ago) 91m
kube-system kube-scheduler-polar 1/1 Running 1 (30m ago) 91m
kube-system storage-provisioner 1/1 Running 3 (29m ago) 91m

2:

kubectl describe pod -n ingress-nginx ingress-nginx-controller-6cc5ccb977-26nj7

kubectl describe pod -n ingress-nginx ingress-nginx-controller-6cc5ccb977-26nj7

Name: ingress-nginx-controller-6cc5ccb977-26nj7
Namespace: ingress-nginx
Priority: 0
Service Account: ingress-nginx
Node: polar/192.168.49.2
Start Time: Thu, 06 Jul 2023 17:31:40 +0800
Labels: app.kubernetes.io/component=controller
app.kubernetes.io/instance=ingress-nginx
app.kubernetes.io/name=ingress-nginx
gcp-auth-skip-secret=true
pod-template-hash=6cc5ccb977
Annotations:
Status: Pending
IP:
IPs:
Controlled By: ReplicaSet/ingress-nginx-controller-6cc5ccb977
Containers:
controller:
Container ID:
Image: registry.k8s.io/ingress-nginx/controller:v1.7.0@sha256:7612338342a1e7b8090bef78f2a04fffcadd548ccaabe8a47bf7758ff549a5f7
Image ID:
Ports: 80/TCP, 443/TCP, 8443/TCP
Host Ports: 80/TCP, 443/TCP, 0/TCP
Args:
/nginx-ingress-controller
--election-id=ingress-nginx-leader
--controller-class=k8s.io/ingress-nginx
--watch-ingress-without-class=true
--configmap=$(POD_NAMESPACE)/ingress-nginx-controller
--tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
--udp-services-configmap=$(POD_NAMESPACE)/udp-services
--validating-webhook=:8443
--validating-webhook-certificate=/usr/local/certificates/cert
--validating-webhook-key=/usr/local/certificates/key
State: Waiting
Reason: ContainerCreating
Ready: False
Restart Count: 0
Requests:
cpu: 100m
memory: 90Mi
Liveness: http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=5
Readiness: http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3
Environment:
POD_NAME: ingress-nginx-controller-6cc5ccb977-26nj7 (v1:metadata.name)
POD_NAMESPACE: ingress-nginx (v1:metadata.namespace)
LD_PRELOAD: /usr/local/lib/libmimalloc.so
Mounts:
/usr/local/certificates/ from webhook-cert (ro)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-cncqn (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
webhook-cert:
Type: Secret (a volume populated by a Secret)
SecretName: ingress-nginx-admission
Optional: false
kube-api-access-cncqn:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional:
DownwardAPI: true
QoS Class: Burstable
Node-Selectors: kubernetes.io/os=linux
minikube.k8s.io/primary=true
Tolerations: node-role.kubernetes.io/master:NoSchedule
node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message


Normal Scheduled 20m default-scheduler Successfully assigned ingress-nginx/ingress-nginx-controller-6cc5ccb977-26nj7 to polar
Warning FailedMount 4m35s (x7 over 18m) kubelet Unable to attach or mount volumes: unmounted volumes=[webhook-cert], unattached volumes=[webh
ook-cert kube-api-access-cncqn]: timed out waiting for the condition
Warning FailedMount 54s kubelet Unable to attach or mount volumes: unmounted volumes=[webhook-cert], unattached volumes=[kube
-api-access-cncqn webhook-cert]: timed out waiting for the condition
Warning FailedMount 23s (x17 over 20m) kubelet MountVolume.SetUp failed for volume "webhook-cert" : secret "ingress-nginx-admission" not fou
nd

3: my docker version on my local machine:

docker version

docker version

Client:
Version: 24.0.0
API version: 1.43
Go version: go1.20.4
Git commit: 98fdcd769b
Built: Thu May 18 09:55:36 2023
OS/Arch: linux/amd64
Context: default

Server:
Engine:
Version: 24.0.0
API version: 1.43 (minimum version 1.12)
Go version: go1.20.4
Git commit: 1331b8c39a
Built: Thu May 18 09:55:36 2023
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: v1.7.1
GitCommit: 1677a17964311325ed1c31e2c0a3589ce6d5c30d.m
runc:
Version: 1.1.7
GitCommit:
docker-init:
Version: 0.19.0
GitCommit: de40ad0

4: In the end, I noticed this sentence in the official documentation of minikube: It is recommended to run this tutorial on a cluster with at least two nodes that are not acting as control plane hosts. ,** What does 'two node' mean here?**

After using minikube node list -p polar command,
The results indicate that there is just a single node:

polar   192.168.49.2

Attach the log file

minikube logs --file=logs.txt --profile polar :
logs.txt

Operating System

Manjaro Linux

Driver

Docker


update 1:2023-07-10 20:40

I guess that this problem only happens on the ingress-nginx addon, because I can enable other addons maintained by Kubernetes . Especially, metrics-server addon's DEFAULT REGISTRY is same as ingress, both are registry.k8s.io . see :

➜  ~ minikube addons enable default-storageclass -p polar                                                      
💡  default-storageclass is an addon maintained by Kubernetes. For any concerns contact minikube on GitHub.
You can view the list of minikube maintainers at: https://github.com/kubernetes/minikube/blob/master/OWNERS
🌟  The 'default-storageclass' addon is enabled
➜  ~ 
➜  ~ 
➜  ~ 
➜  ~ minikube addons enable metrics-server -p polar                            
💡  metrics-server is an addon maintained by Kubernetes. For any concerns contact minikube on GitHub.
You can view the list of minikube maintainers at: https://github.com/kubernetes/minikube/blob/master/OWNERS
    ▪ Using image registry.k8s.io/metrics-server/metrics-server:v0.6.3
🌟  The 'metrics-server' addon is enabled
@ongiant ongiant changed the title failure when use minikube addons enable ingress command failure when minikube enable ingress Jul 7, 2023
@ongiant
Copy link
Author

ongiant commented Jul 10, 2023

Have you seen this? https://stackoverflow.com/questions/71976117/mountvolume-setup-failed-for-volume-webhook-cert-secret-ingress-nginx-admis

I don't think this is the root cause because when I use command kubectl describe pod on the ingress-nginx-admission-create-6twp2 and ingress-nginx-admission-patch-6578w , the output shows ErrImagePull just like this:

Normal   Pulling    32s (x3 over 75s)  kubelet            Pulling image "registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20230312-helm-chart-4.5.2-28-g
66a760794@sha256:01d181618f270f2a96c04006f33b2699ad3ccb02da48d0f89b22abce084b292f"                                                                           
  Warning  Failed     32s (x3 over 74s)  kubelet            Failed to pull image "registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20230312-helm-chart-4.5
.2-28-g66a760794@sha256:01d181618f270f2a96c04006f33b2699ad3ccb02da48d0f89b22abce084b292f": rpc error: code = Unknown desc = Error response from daemon: Get "
https://registry.k8s.io/v2/": proxyconnect tcp: dial tcp 0.0.0.0:7891: connect: connection refused                                                           
  Warning  Failed     32s (x3 over 74s)  kubelet            Error: ErrImagePull                   
  Normal   BackOff    5s (x4 over 74s)   kubelet            Back-off pulling image "registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20230312-helm-chart-4
.5.2-28-g66a760794@sha256:01d181618f270f2a96c04006f33b2699ad3ccb02da48d0f89b22abce084b292f"                                                                  
  Warning  Failed     5s (x4 over 74s)   kubelet            Error: ImagePullBackOff

@spowelljr
Copy link
Member

I believe it's your proxy flags that's causing This container is having trouble accessing https://registry.k8s.io. Could you try unsetting HTTP_PROXY, HTTPS_PROXY & NO_PROXY and then minikube delete and then start minikube again and see if there's any difference.

@ongiant
Copy link
Author

ongiant commented Jul 11, 2023

I believe it's your proxy flags that's causing This container is having trouble accessing https://registry.k8s.io. Could you try unsetting HTTP_PROXY, HTTPS_PROXY & NO_PROXY and then minikube delete and then start minikube again and see if there's any difference.

Your method worked :

minikube start details

➜ ~ minikube start -p polar --nodes 2
😄 [polar] minikube v1.30.1 on Arch "23.0.0"
✨ Automatically selected the docker driver
📌 Using Docker driver with root privileges
👍 Starting control plane node polar in cluster polar
🚜 Pulling base image ...
💾 Downloading Kubernetes v1.26.3 preload ...
> preloaded-images-k8s-v18-v1...: 397.02 MiB / 397.02 MiB 100.00% 6.34 Mi
🔥 Creating docker container (CPUs=2, Memory=2200MB) ...
🐳 Preparing Kubernetes v1.26.3 on Docker 23.0.2 ...
▪ Generating certificates and keys ...
▪ Booting up control plane ...
▪ Configuring RBAC rules ...
🔗 Configuring CNI (Container Networking Interface) ...
▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟 Enabled addons: storage-provisioner, default-storageclass
🔎 Verifying Kubernetes components...

👍 Starting worker node polar-m02 in cluster polar
🚜 Pulling base image ...
🔥 Creating docker container (CPUs=2, Memory=2200MB) ...
🌐 Found network options:
▪ NO_PROXY=192.168.49.2
🐳 Preparing Kubernetes v1.26.3 on Docker 23.0.2 ...
▪ env NO_PROXY=192.168.49.2
🔎 Verifying Kubernetes components...
🏄 Done! kubectl is now configured to use "polar" cluster and "default" namespace by default

But ingress addons still was not able to enable, I tried two methods, but both of them failed.

  1. enable directly(minikube addons enable ingress -p polar)
  2. load image manually then enable ingress:

this is partial log of the second method's result:


🔎 Verifying ingress addon...
I0711 11:33:25.830617 566631 kapi.go:75] Waiting for pod with label "app.kubernetes.io/name=ingress-nginx" in ns "ingress-nginx" ...
I0711 11:33:25.838897 566631 kapi.go:86] Found 3 Pods for label selector app.kubernetes.io/name=ingress-nginx
I0711 11:33:25.838918 566631 kapi.go:96] waiting for pod "app.kubernetes.io/name=ingress-nginx", current state: Pending: []

...

I0711 11:39:25.850399 566631 kapi.go:96] waiting for pod "app.kubernetes.io/name=ingress-nginx", current state: Pending: []
I0711 11:39:25.850439 566631 kapi.go:107] duration metric: took 6m0.019834494s to wait for app.kubernetes.io/name=ingress-nginx ...
I0711 11:39:25.850837 566631 out.go:177]

W0711 11:39:25.850936 566631 out.go:239] ❌ Exiting due to MK_ADDON_ENABLE: enable failed: run callbacks: running callbacks: [waiting for app.kubernetes.io
/name=ingress-nginx pods: timed out waiting for the condition]
❌ Exiting due to MK_ADDON_ENABLE: enable failed: run callbacks: running callbacks: [waiting for app.kubernetes.io/name=ingress-nginx pods: timed out waitin
g for the condition]


And I used minikube ssh -p polar login, then try pull image manually, it showed below:

docker@polar:~$ docker pull registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20230312-helm-chart-4.5.2-28-g66a760794@sha256:01d181618f270f2a96c04006f33b2699ad3ccb02da48d0f89b22abce084b292f
Error response from daemon: Get "https://us-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/ingress-nginx/kube-webhook-certgen/manifests/sha256:01d181618f270f2a96c04006f33b2699ad3ccb02da48d0f89b22abce084b292f": dial tcp 142.250.101.82:443: i/o timeout

So It looks more like a problem with the network proxy?

@spowelljr
Copy link
Member

What do you get if you run dig registry.k8s.io?

@spowelljr spowelljr added kind/support Categorizes issue or PR as a support question. addon/ingress labels Jul 19, 2023
@ongiant
Copy link
Author

ongiant commented Jul 20, 2023

dig registry.k8s.io

  1. In my local machine is this:
dig registry.k8s.io

➜ ~ dig registry.k8s.io

; <<>> DiG 9.18.16 <<>> registry.k8s.io
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 54802
;; flags: qr rd ra cd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;registry.k8s.io. IN A

;; ANSWER SECTION:
registry.k8s.io. 3600 IN A 34.96.108.209

;; Query time: 806 msec
;; SERVER: 127.0.0.1#53(127.0.0.1) (UDP)
;; WHEN: Thu Jul 20 10:06:43 CST 2023
;; MSG SIZE rcvd: 60

In minikube, the output is:


➜ ~ minikube ssh -p polar
docker@polar:$
docker@polar:
$
docker@polar:~$ dig registry.k8s.io

; <<>> DiG 9.16.1-Ubuntu <<>> registry.k8s.io
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 11219
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;registry.k8s.io. IN A

;; ANSWER SECTION:
registry.k8s.io. 1589 IN A 34.96.108.209

;; Query time: 0 msec
;; SERVER: 192.168.49.1#53(192.168.49.1)
;; WHEN: Thu Jul 20 02:40:13 UTC 2023
;; MSG SIZE rcvd: 60

@spowelljr
Copy link
Member

So the DNS is resolving fine, are you behind a firewall perhaps? Does running the docker pull command work fine on your host?

@ongiant
Copy link
Author

ongiant commented Jul 30, 2023

So the DNS is resolving fine, are you behind a firewall perhaps? Does running the docker pull command work fine on your host?

My host does't have firewall like UFW. And I can use docker to pull the images hinting in error message.

  1. My iptables rules:
➜  ~ sudo iptables -L                                                                                                                                        
[sudo] password for zhongshiang: 
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
  1. Images on my host
    (I have modified the tag of the ingress-nginx image on my host, and these changes are reflected in the images.):
➜  ~ docker images 
REPOSITORY                                           TAG                                        IMAGE ID       CREATED        SIZE
paketobuildpacks/run                                 base-cnb                                   f2e5000af0cb   4 weeks ago    87MB
registry.k8s.io/ingress-nginx/kube-webhook-certgen   v20230407                                  7e7451bb7042   3 months ago   47.2MB
kicbase/stable                                       v0.0.39                                    67a4b1138d2d   3 months ago   1.05GB
gcr.io/k8s-minikube/kicbase                          v0.0.39                                    67a4b1138d2d   3 months ago   1.05GB
registry.k8s.io/ingress-nginx/controller             v1.7.0                                     0d4c0564c465   4 months ago   283MB
registry.k8s.io/ingress-nginx/kube-webhook-certgen   v20230312-helm-chart-4.5.2-28-g66a760794   5a86b03a88d2   4 months ago   47.1MB
config-service                                       latest                                     9ccc5cc0068a   43 years ago   289MB
edge-service                                         latest                                     068862751604   43 years ago   297MB
paketobuildpacks/builder                             base                                       99ec7fb86b9d   43 years ago   1.34GB
catalog-service                                      latest                                     595998cf4202   43 years ago   293MB
order-service                                        latest                                     b91d8d94ccb3   43 years ago   298MB

@ongiant
Copy link
Author

ongiant commented Aug 1, 2023

I still failed after upgrading minikube to v1.31.1......
I have configured a proxy for the Docker daemon on my local machine. Will this affect the Docker daemon in minikube when pulling images? The current situation is that the Docker daemon on my local machine is working fine, but the Docker daemon in minikube is not.


Some new findings:

  • I used minikube addons enable ingress -p polar --alsologtostderr --v=8 command, and I have noticed a recurring log that shows it is constantly requesting https://192.168.49.2:8443/api/v1/namespaces/ingress-nginx/pods?labelSelector=app.kubernetes.io%2Fname%3Dingress-nginx.
  • Then I ran the command http GET https://192.168.49.2:8443/api/v1/namespaces/ingress-nginx/pods\?labelSelector\=app.kubernetes.io%2Fname%3Dingress-nginx in the terminal of my local host to request that address, and the result showed a SSL certificate issue. (The tool that I used is HTTPie)
  1. repetitive stdout log
    
    I0801 16:18:34.225970  228629 kapi.go:96] waiting for pod "app.kubernetes.io/name=ingress-nginx", current state: Pending: []                            
    I0801 16:18:34.720299  228629 round_trippers.go:463] GET https://192.168.49.2:8443/api/v1/namespaces/ingress-nginx/pods?labelSelector=app.kubernetes.io%2Fname%3Dingress-nginx                                                                                                                                            
    I0801 16:18:34.720333  228629 round_trippers.go:469] Request Headers:                                                                                        
    I0801 16:18:34.720352  228629 round_trippers.go:473]     Accept: application/json, */*                                                                       
    I0801 16:18:34.720372  228629 round_trippers.go:473]     User-Agent: minikube/v0.0.0 (linux/amd64) kubernetes/$Format                                        
    I0801 16:18:34.724175  228629 round_trippers.go:574] Response Status: 200 OK in 3 milliseconds                                                               
    I0801 16:18:34.724216  228629 round_trippers.go:577] Response Headers:                                                                                       
    I0801 16:18:34.724238  228629 round_trippers.go:580]     Audit-Id: 58519d88-8c3d-438a-8ab3-47a353ea8a9d                                                      
    I0801 16:18:34.724257  228629 round_trippers.go:580]     Cache-Control: no-cache, private                                                                    
    I0801 16:18:34.724275  228629 round_trippers.go:580]     Content-Type: application/json                                                                      
    I0801 16:18:34.724291  228629 round_trippers.go:580]     X-Kubernetes-Pf-Flowschema-Uid: b09b0b91-e0c5-4c54-b128-e7701b570ab2                                
    I0801 16:18:34.724313  228629 round_trippers.go:580]     X-Kubernetes-Pf-Prioritylevel-Uid: d6a4000b-12d3-4395-9974-6e3f88b9a39d
    I0801 16:18:34.724332  228629 round_trippers.go:580]     Date: Tue, 01 Aug 2023 08:18:34 GMT
    I0801 16:18:34.724772  228629 request.go:1188] Response Body: {"kind":"PodList","apiVersion":"v1","metadata":{"resourceVersion":"991"},"items":[{"metadata":{
    "name":"ingress-nginx-admission-create-dqc67","generateName":"ingress-nginx-admission-create-","namespace":"ingress-nginx","uid":"06d9b48f-a856-41db-8c44-daa
    d2afa1632","resourceVersion":"977","creationTimestamp":"2023-08-01T08:12:35Z","labels":{"app.kubernetes.io/component":"admission-webhook","app.kubernetes.io/
    instance":"ingress-nginx","app.kubernetes.io/name":"ingress-nginx","batch.kubernetes.io/controller-uid":"e8941a34-c61c-477d-a0f5-b182062f9241","batch.kuberne
    tes.io/job-name":"ingress-nginx-admission-create","controller-uid":"e8941a34-c61c-477d-a0f5-b182062f9241","job-name":"ingress-nginx-admission-create"},"owner
    References":[{"apiVersion":"batch/v1","kind":"Job","name":"ingress-nginx-admission-create","uid":"e8941a34-c61c-477d-a0f5-b182062f9241","controller":true,"bl
    ockOwnerDeletion":true}],"finalizers":["batch.kubernetes.io/job-tracking"],"managedFields":[{"manager":"kube-controller-manager","operation":"Up [truncated 1
    8863 chars]
    ...
    

    I0801 16:18:34.726306 228629 kapi.go:96] waiting for pod "app.kubernetes.io/name=ingress-nginx", current state: Pending: []
    I0801 16:18:35.220203 228629 round_trippers.go:463] GET https://192.168.49.2:8443/api/v1/namespaces/ingress-nginx/pods?labelSelector=app.kubernetes.io%2Fname%3Dingress-nginx
    I0801 16:18:35.220237 228629 round_trippers.go:469] Request Headers:
    I0801 16:18:35.220262 228629 round_trippers.go:473] Accept: application/json, /
    I0801 16:18:35.220287 228629 round_trippers.go:473] User-Agent: minikube/v0.0.0 (linux/amd64) kubernetes/$Format
    I0801 16:18:35.224303 228629 round_trippers.go:574] Response Status: 200 OK in 3 milliseconds
    I0801 16:18:35.224348 228629 round_trippers.go:577] Response Headers:
    I0801 16:18:35.224383 228629 round_trippers.go:580] Content-Type: application/json
    I0801 16:18:35.224400 228629 round_trippers.go:580] X-Kubernetes-Pf-Flowschema-Uid: b09b0b91-e0c5-4c54-b128-e7701b570ab2
    I0801 16:18:35.224419 228629 round_trippers.go:580] X-Kubernetes-Pf-Prioritylevel-Uid: d6a4000b-12d3-4395-9974-6e3f88b9a39d
    I0801 16:18:35.224437 228629 round_trippers.go:580] Date: Tue, 01 Aug 2023 08:18:35 GMT
    I0801 16:18:35.224455 228629 round_trippers.go:580] Audit-Id: ac86df85-8618-43c5-8fb9-8109b02473a0
    I0801 16:18:35.224473 228629 round_trippers.go:580] Cache-Control: no-cache, private
    I0801 16:18:35.224807 228629 request.go:1188] Response Body: {"kind":"PodList","apiVersion":"v1","metadata":{"resourceVersion":"992"},"items":[{"metadata":{
    "name":"ingress-nginx-admission-create-dqc67","generateName":"ingress-nginx-admission-create-","namespace":"ingress-nginx","uid":"06d9b48f-a856-41db-8c44-daa
    d2afa1632","resourceVersion":"992","creationTimestamp":"2023-08-01T08:12:35Z","labels":{"app.kubernetes.io/component":"admission-webhook","app.kubernetes.io/
    instance":"ingress-nginx","app.kubernetes.io/name":"ingress-nginx","batch.kubernetes.io/controller-uid":"e8941a34-c61c-477d-a0f5-b182062f9241","batch.kuberne
    tes.io/job-name":"ingress-nginx-admission-create","controller-uid":"e8941a34-c61c-477d-a0f5-b182062f9241","job-name":"ingress-nginx-admission-create"},"owner
    References":[{"apiVersion":"batch/v1","kind":"Job","name":"ingress-nginx-admission-create","uid":"e8941a34-c61c-477d-a0f5-b182062f9241","controller":true,"bl
    ockOwnerDeletion":true}],"finalizers":["batch.kubernetes.io/job-tracking"],"managedFields":[{"manager":"kube-controller-manager","operation":"Up [truncated 1
    8736 chars]
    I0801 16:18:35.226344 228629 kapi.go:96] waiting for pod "app.kubernetes.io/name=ingress-nginx", current state: Pending: []
    I0801 16:18:35.698161 228629 kapi.go:107] duration metric: took 6m0.001013739s to wait for app.kubernetes.io/name=ingress-nginx ...
    I0801 16:18:35.698557 228629 out.go:177]

    W0801 16:18:35.698624 228629 out.go:239] ❌ Exiting due to MK_ADDON_ENABLE: enable failed: run callbacks: running callbacks: [waiting for app.kubernetes.io/name=ingress-nginx pods: context deadline exceeded]
    ❌ Exiting due to MK_ADDON_ENABLE: enable failed: run callbacks: running callbacks: [waiting for app.kubernetes.io/name=ingress-nginx pods: context deadline
    exceeded]

  2. The JSON format response body of the repetitive stdout
    
    {
        "Response Body": {
            "kind": "PodList",
            "apiVersion": "v1",
            "metadata": {
                "resourceVersion": "932"
            },
            "items": [
                {
                    "metadata": {
                        "name": "ingress-nginx-admission-create-5w2hh",
                        "generateName": "ingress-nginx-admission-create-",
                        "namespace": "ingress-nginx",
                        "uid": "8d963181-79f4-43c8-b7b8-c18e615ee90c",
                        "resourceVersion": "852",
                        "creationTimestamp": "2023-09-06T19: 22: 47Z",
                        "labels": {
                            "app.kubernetes.io/component": "admission-webhook",
                            "app.kubernetes.io/instance": "ingress-nginx",
                            "app.kubernetes.io/name": "ingress-nginx",
                            "batch.kubernetes.io/controller-uid": "fc68911a-52c6-41c8-bbb4-d941b05bc756",
                            "batch.kubernetes.io/job-name": "ingress-nginx-admission-create",
                            "controller-uid": "fc68911a-52c6-41c8-bbb4-d941b05bc756",
                            "job-name": "ingress-nginx-admission-create"
                        },
                        "ownerReferences": [
                            {
                                "apiVersion": "batch/v1",
                                "kind": "Job",
                                "name": "ingress-nginx-admission-create",
                                "uid": "fc68911a-52c6-41c8-bbb4-d941b05bc756",
                                "controller": true,
                                "blockOwnerDeletion": true
                            }
                        ],
                        "finalizers": [
                            "batch.kubernetes.io/job-tracking"
                        ],
                        "managedFields": [
                            {
                                "manager": "kube-controller-manager",
                                "operation": "Update",
                                "apiVersion": "v1",
                                "time": "2023-09-06T19: 22: 47Z",
                                "fieldsType": "FieldsV1",
                                "fieldsV1": {
                                    "f:metadata": {
                                        "f:finalizers": {
                                            ".": {},
                                            "v:\"batch.kubernetes.io/job-tracking\"": {}
                                        },
                                        "f:generateName": {},
                                        "f:labels": {
                                            ".": {},
                                            "f:app.kubernetes.io/component": {},
                                            "f:app.kubernetes.io/instance": {},
                                            "f:app.kubernetes.io/name": {},
                                            "f:batch.kubernetes.io/controller-uid": {},
                                            "f:batch.kubernetes.io/job-name": {},
                                            "f:controller-uid": {},
                                            "f:job-name": {}
                                        },
                                        "f:ownerReferences": {
                                            ".": {},
                                            "k: {\"uid\":\"fc68911a-52c6-41c8-bbb4-d941b05bc756\"}": {}
                                        }
                                    },
                                    "f:spec": {
                                        "f:containers": {
                                            "k: {\"name\":\"create\"}": {
                                                ".": {},
                                                "f:args": {},
                                                "f:env": {
                                                    ".": {},
                                                    "k:{\"name\":\"POD_NAMESPACE\"}": {
                                                        ".": {},
                                                        "f:name": {},
                                                        "f:valueFrom": {
                                                            ".": {},
                                                            "f:fieldRef": {}
                                                        }
                                                    }
                                                },
                                                "f:image": {},
                                                "f:imagePullPolicy": {},
                                                "f:name": {},
                                                "f:resources": {},
                                                "f:securityContext": {
                                                    ".": {},
                                                    "f:allowPrivilegeEscalation": {}
                                                },
                                                "f:terminationMessagePath": {},
                                                "f:terminationMessagePolicy": {}
                                            }
                                        },
                                        "f:dnsPolicy": {},
                                        "f:enableServiceLinks": {},
                                        "f:nodeSelector": {},
                                        "f:restartPolicy": {},
                                        "f:schedulerName": {},
                                        "f:securityContext": {
                                            ".": {},
                                            "f:runAsNonRoot": {},
                                            "f:runAsUser": {}
                                        },
                                        "f:serviceAccount": {},
                                        "f:serviceAccountName": {},
                                        "f:terminationGracePeriodSeconds": {}
                                    }
                                }
                            },
                            {
                                "manager": "kubelet",
                                "operation": "Update",
                                "apiVersion": "v1",
                                "time": "2023-09-06T19: 27: 16Z",
                                "fieldsType": "FieldsV1",
                                "fieldsV1": {
                                    "f:status": {
                                        "f:conditions": {
                                            "k: {\"type\":\"ContainersReady\"}": {
                                                ".": {},
                                                "f:lastProbeTime": {},
                                                "f:lastTransitionTime": {},
                                                "f:message": {},
                                                "f:reason": {},
                                                "f:status": {},
                                                "f:type": {}
                                            },
                                            "k: {\"type\":\"Initialized\"}": {
                                                ".": {},
                                                "f:lastProbeTime": {},
                                                "f:lastTransitionTime": {},
                                                "f:status": {},
                                                "f:type": {}
                                            },
                                            "k:{\"type\":\"Ready\"}": {
                                                ".": {},
                                                "f:lastProbeTime": {},
                                                "f:lastTransitionTime": {},
                                                "f:message": {},
                                                "f:reason": {},
                                                "f:status": {},
                                                "f:type": {}
                                            }
                                        },
                                        "f:containerStatuses": {},
                                        "f:hostIP": {},
                                        "f:podIP": {},
                                        "f:podIPs": {
                                            ".": {},
                                            "k: {\"ip\":\"10.244.0.3\"}": {
                                                ".": {},
                                                "f:ip": {}
                                            }
                                        },
                                        "f:startTime": {}
                                    }
                                },
                                "subresource": "status"
                            }
                        ]
                    },
                    "spec": {
                        "volumes": [
                            {
                                "name": "kube-api-access-6wfnl",
                                "projected": {
                                    "sources": [
                                        {
                                            "serviceAccountToken": {
                                                "expirationSeconds": 3607,
                                                "path": "token"
                                            }
                                        },
                                        {
                                            "configMap": {
                                                "name": "kube-root-ca.crt",
                                                "items": [
                                                    {
                                                        "key": "ca.crt",
                                                        "path": "ca.crt"
                                                    }
                                                ]
                                            }
                                        },
                                        {
                                            "downwardAPI": {
                                                "items": [
                                                    {
                                                        "path": "namespace",
                                                        "fieldRef": {
                                                            "apiVersion": "v1",
                                                            "fieldPath": "metadata.namespace"
                                                        }
                                                    }
                                                ]
                                            }
                                        }
                                    ],
                                    "defaultMode": 420
                                }
                            }
                        ],
                        "containers": [
                            {
                                "name": "create",
                                "image": "registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20230407@sha256: 543c40fd093964bc9ab509d3e791f9989963021f1e9e4c9c7b6700b02bfb227b",
                                "args": [
                                    "create",
                                    "--host=ingress-nginx-controller-admission,ingress-nginx-controller-admission.$(POD_NAMESPACE).svc",
                                    "--namespace=$(POD_NAMESPACE)",
                                    "--secret-name=ingress-nginx-admission"
                                ],
                                "env": [
                                    {
                                        "name": "POD_NAMESPACE",
                                        "valueFrom": {
                                            "fieldRef": {
                                                "apiVersion": "v1",
                                                "fieldPath": "metadata.namespace"
                                            }
                                        }
                                    }
                                ],
                                "resources": {},
                                "volumeMounts": [
                                    {
                                        "name": "kube-api-access-6wfnl",
                                        "readOnly": true,
                                        "mountPath": "/var/run/secrets/kubernetes.io/serviceaccount"
                                    }
                                ],
                                "terminationMessagePath": "/dev/termination-log",
                                "terminationMessagePolicy": "File",
                                "imagePullPolicy": "IfNotPresent",
                                "securityContext": {
                                    "allowPrivilegeEscalation": false
                                }
                            }
                        ],
                        "restartPolicy": "OnFailure",
                        "terminationGracePeriodSeconds": 30,
                        "dnsPolicy": "ClusterFirst",
                        "nodeSelector": {
                            "kubernetes.io/os": "linux",
                            "minikube.k8s.io/primary": "true"
                        },
                        "serviceAccountName": "ingress-nginx-admission",
                        "serviceAccount": "ingress-nginx-admission",
                        "nodeName": "polar",
                        "securityContext": {
                            "runAsUser": 2000,
                            "runAsNonRoot": true
                        },
                        "schedulerName": "default-scheduler",
                        "tolerations": [
                            {
                                "key": "node.kubernetes.io/not-ready",
                                "operator": "Exists",
                                "effect": "NoExecute",
                                "tolerationSeconds": 300
                            },
                            {
                                "key": "node.kubernetes.io/unreachable",
                                "operator": "Exists",
                                "effect": "NoExecute",
                                "tolerationSeconds": 300
                            }
                        ],
                        "priority": 0,
                        "enableServiceLinks": true,
                        "preemptionPolicy": "PreemptLowerPriority"
                    },
                    "status": {
                        "phase": "Pending",
                        "conditions": [
                            {
                                "type": "Initialized",
                                "status": "True",
                                "lastProbeTime": null,
                                "lastTransitionTime": "2023-09-06T19: 22: 47Z"
                            },
                            {
                                "type": "Ready",
                                "status": "False",
                                "lastProbeTime": null,
                                "lastTransitionTime": "2023-09-06T19: 22: 47Z",
                                "reason": "ContainersNotReady",
                                "message": "containers with unready status: [create]"
                            },
                            {
                                "type": "ContainersReady",
                                "status": "False",
                                "lastProbeTime": null,
                                "lastTransitionTime": "2023-09-06T19: 22: 47Z",
                                "reason": "ContainersNotReady",
                                "message": "containers with unready status: [create]"
                            },
                            {
                                "type": "PodScheduled",
                                "status": "True",
                                "lastProbeTime": null,
                                "lastTransitionTime": "2023-09-06T19: 22: 47Z"
                            }
                        ],
                        "hostIP": "192.168.49.2",
                        "podIP": "10.244.0.3",
                        "podIPs": [
                            {
                                "ip": "10.244.0.3"
                            }
                        ],
                        "startTime": "2023-09-06T19: 22: 47Z",
                        "containerStatuses": [
                            {
                                "name": "create",
                                "state": {
                                    "waiting": {
                                        "reason": "ImagePullBackOff",
                                        "message": "Back-off pulling image \"registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20230407@sha256: 543c40fd093964bc9ab509d3e791f9989963021f1e9e4c9c7b6700b02bfb227b\""
                                    }
                                },
                                "lastState": {},
                                "ready": false,
                                "restartCount": 0,
                                "image": "registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20230407@sha256:543c40fd093964bc9ab509d3e791f9989963021f1e9e4c9c7b6700b02bfb227b",
                                "imageID": "",
                                "started": false
                            }
                        ],
                        "qosClass": "BestEffort"
                    }
                },
                {
                    "metadata": {
                        "name": "ingress-nginx-admission-patch-rvph8",
                        "generateName": "ingress-nginx-admission-patch-",
                        "namespace": "ingress-nginx",
                        "uid": "05b9c313-5560-4f21-b3ac-30f8b905dc41",
                        "resourceVersion": "880",
                        "creationTimestamp": "2023-09-06T19: 22: 47Z",
                        "labels": {
                            "app.kubernetes.io/component": "admission-webhook",
                            "app.kubernetes.io/instance": "ingress-nginx",
                            "app.kubernetes.io/name": "ingress-nginx",
                            "batch.kubernetes.io/controller-uid": "cb946edc-fa77-44bf-b27b-c59426966d09",
                            "batch.kubernetes.io/job-name": "ingress-nginx-admission-patch",
                            "controller-uid": "cb946edc-fa77-44bf-b27b-c59426966d09",
                            "job-name": "ingress-nginx-admission-patch"
                        },
                        "ownerReferences": [
                            {
                                "apiVersion": "batch/v1",
                                "kind": "Job",
                                "name": "ingress-nginx-admission-patch",
                                "uid": "cb946edc-fa77-44bf-b27b-c59426966d09",
                                "controller": true,
                                "blockOwnerDeletion": true
                            }
                        ],
                        "finalizers": [
                            "batch.kubernetes.io/job-tracking"
                        ],
                        "managedFields": [
                            {
                                "manager": "kube-controller-manager",
                                "operation": "Update",
                                "apiVersion": "v1",
                                "time": "2023-09-06T19: 22: 47Z",
                                "fieldsType": "FieldsV1",
                                "fieldsV1": {
                                    "f:metadata": {
                                        "f:finalizers": {
                                            ".": {},
                                            "v:\"batch.kubernetes.io/job-tracking\"": {}
                                        },
                                        "f:generateName": {},
                                        "f:labels": {
                                            ".": {},
                                            "f:app.kubernetes.io/component": {},
                                            "f:app.kubernetes.io/instance": {},
                                            "f:app.kubernetes.io/name": {},
                                            "f:batch.kubernetes.io/controller-uid": {},
                                            "f:batch.kubernetes.io/job-name": {},
                                            "f:controller-uid": {},
                                            "f:job-name": {}
                                        },
                                        "f:ownerReferences": {
                                            ".": {},
                                            "k: {\"uid\":\"cb946edc-fa77-44bf-b27b-c59426966d09\"}": {}
                                        }
                                    },
                                    "f:spec": {
                                        "f:containers": {
                                            "k:{\"name\":\"patch\"}": {
                                                ".": {},
                                                "f:args": {},
                                                "f:env": {
                                                    ".": {},
                                                    "k:{\"name\":\"POD_NAMESPACE\"}": {
                                                        ".": {},
                                                        "f:name": {},
                                                        "f:valueFrom": {
                                                            ".": {},
                                                            "f:fieldRef": {}
                                                        }
                                                    }
                                                },
                                                "f:image": {},
                                                "f:imagePullPolicy": {},
                                                "f:name": {},
                                                "f:resources": {},
                                                "f:securityContext": {
                                                    ".": {},
                                                    "f:allowPrivilegeEscalation": {}
                                                },
                                                "f:terminationMessagePath": {},
                                                "f:terminationMessagePolicy": {}
                                            }
                                        },
                                        "f:dnsPolicy": {},
                                        "f:enableServiceLinks": {},
                                        "f:nodeSelector": {},
                                        "f:restartPolicy": {},
                                        "f:schedulerName": {},
                                        "f:securityContext": {
                                            ".": {},
                                            "f:runAsNonRoot": {},
                                            "f:runAsUser": {}
                                        },
                                        "f:serviceAccount": {},
                                        "f:serviceAccountName": {},
                                        "f:terminationGracePeriodSeconds": {}
                                    }
                                }
                            },
                            {
                                "manager": "kubelet",
                                "operation": "Update",
                                "apiVersion": "v1",
                                "time": "2023-09-06T19: 27: 43Z",
                                "fieldsType": "FieldsV1",
                                "fieldsV1": {
                                    "f:status": {
                                        "f:conditions": {
                                            "k: {\"type\":\"ContainersReady\"}": {
                                                ".": {},
                                                "f:lastProbeTime": {},
                                                "f:lastTransitionTime": {},
                                                "f:message": {},
                                                "f:reason": {},
                                                "f:status": {},
                                                "f:type": {}
                                            },
                                            "k: {\"type\":\"Initialized\"}": {
                                                ".": {},
                                                "f:lastProbeTime": {},
                                                "f:lastTransitionTime": {},
                                                "f:status": {},
                                                "f:type": {}
                                            },
                                            "k:{\"type\":\"Ready\"}": {
                                                ".": {},
                                                "f:lastProbeTime": {},
                                                "f:lastTransitionTime": {},
                                                "f:message": {},
                                                "f:reason": {},
                                                "f:status": {},
                                                "f:type": {}
                                            }
                                        },
                                        "f:containerStatuses": {},
                                        "f:hostIP": {},
                                        "f:podIP": {},
                                        "f:podIPs": {
                                            ".": {},
                                            "k:{\"ip\":\"10.244.0.4\"}": {
                                                ".": {},
                                                "f:ip": {}
                                            }
                                        },
                                        "f:startTime": {}
                                    }
                                },
                                "subresource": "status"
                            }
                        ]
                    },
                    "spec": {
                        "volumes": [
                            {
                                "name": "kube-api-access-fdjbl",
                                "projected": {
                                    "sources": [
                                        {
                                            "serviceAccountToken": {
                                                "expirationSeconds": 3607,
                                                "path": "token"
                                            }
                                        },
                                        {
                                            "configMap": {
                                                "name": "kube-root-ca.crt",
                                                "items": [
                                                    {
                                                        "key": "ca.crt",
                                                        "path": "ca.crt"
                                                    }
                                                ]
                                            }
                                        },
                                        {
                                            "downwardAPI": {
                                                "items": [
                                                    {
                                                        "path": "namespace",
                                                        "fieldRef": {
                                                            "apiVersion": "v1",
                                                            "fieldPath": "metadata.namespace"
                                                        }
                                                    }
                                                ]
                                            }
                                        }
                                    ],
                                    "defaultMode": 420
                                }
                            }
                        ],
                        "containers": [
                            {
                                "name": "patch",
                                "image": "registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20230407@sha256: 543c40fd093964bc9ab509d3e791f9989963021f1e9e4c9c7b6700b02bfb227b",
                                "args": [
                                    "patch",
                                    "--webhook-name=ingress-nginx-admission",
                                    "--namespace=$(POD_NAMESPACE)",
                                    "--patch-mutating=false",
                                    "--secret-name=ingress-nginx-admission",
                                    "--patch-failure-policy=Fail"
                                ],
                                "env": [
                                    {
                                        "name": "POD_NAMESPACE",
                                        "valueFrom": {
                                            "fieldRef": {
                                                "apiVersion": "v1",
                                                "fieldPath": "metadata.namespace"
                                            }
                                        }
                                    }
                                ],
                                "resources": {},
                                "volumeMounts": [
                                    {
                                        "name": "kube-api-access-fdjbl",
                                        "readOnly": true,
                                        "mountPath": "/var/run/secrets/kubernetes.io/serviceaccount"
                                    }
                                ],
                                "terminationMessagePath": "/dev/termination-log",
                                "terminationMessagePolicy": "File",
                                "imagePullPolicy": "IfNotPresent",
                                "securityContext": {
                                    "allowPrivilegeEscalation": false
                                }
                            }
                        ],
                        "restartPolicy": "OnFailure",
                        "terminationGracePeriodSeconds": 30,
                        "dnsPolicy": "ClusterFirst",
                        "nodeSelector": {
                            "kubernetes.io/os": "linux",
                            "minikube.k8s.io/primary": "true"
                        },
                        "serviceAccountName": "ingress-nginx-admission",
                        "serviceAccount": "ingress-nginx-admission",
                        "nodeName": "polar",
                        "securityContext": {
                            "runAsUser": 2000,
                            "runAsNonRoot": true
                        },
                        "schedulerName": "default-scheduler",
                        "tolerations": [
                            {
                                "key": "node.kubernetes.io/not-ready",
                                "operator": "Exists",
                                "effect": "NoExecute",
                                "tolerationSeconds": 300
                            },
                            {
                                "key": "node.kubernetes.io/unreachable",
                                "operator": "Exists",
                                "effect": "NoExecute",
                                "tolerationSeconds": 300
                            }
                        ],
                        "priority": 0,
                        "enableServiceLinks": true,
                        "preemptionPolicy": "PreemptLowerPriority"
                    },
                    "status": {
                        "phase": "Pending",
                        "conditions": [
                            {
                                "type": "Initialized",
                                "status": "True",
                                "lastProbeTime": null,
                                "lastTransitionTime": "2023-09-06T19: 22: 47Z"
                            },
                            {
                                "type": "Ready",
                                "status": "False",
                                "lastProbeTime": null,
                                "lastTransitionTime": "2023-09-06T19: 22: 47Z",
                                "reason": "ContainersNotReady",
                                "message": "containers with unready status: [patch]"
                            },
                            {
                                "type": "ContainersReady",
                                "status": "False",
                                "lastProbeTime": null,
                                "lastTransitionTime": "2023-09-06T19: 22: 47Z",
                                "reason": "ContainersNotReady",
                                "message": "containers with unready status: [patch]"
                            },
                            {
                                "type": "PodScheduled",
                                "status": "True",
                                "lastProbeTime": null,
                                "lastTransitionTime": "2023-09-06T19: 22: 47Z"
                            }
                        ],
                        "hostIP": "192.168.49.2",
                        "podIP": "10.244.0.4",
                        "podIPs": [
                            {
                                "ip": "10.244.0.4"
                            }
                        ],
                        "startTime": "2023-09-06T19: 22: 47Z",
                        "containerStatuses": [
                            {
                                "name": "patch",
                                "state": {
                                    "waiting": {
                                        "reason": "ImagePullBackOff",
                                        "message": "Back-off pulling image \"registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20230407@sha256:543c40fd093964bc9ab509d3e791f9989963021f1e9e4c9c7b6700b02bfb227b\""
                                    }
                                },
                                "lastState": {},
                                "ready": false,
                                "restartCount": 0,
                                "image": "registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20230407@sha256:543c40fd093964bc9ab509d3e791f9989963021f1e9e4c9c7b6700b02bfb227b",
                                "imageID": "",
                                "started": false
                            }
                        ],
                        "qosClass": "BestEffort"
                    }
                },
                {
                    "metadata": {
                        "name": "ingress-nginx-controller-7799c6795f-7gzx6",
                        "generateName": "ingress-nginx-controller-7799c6795f-",
                        "namespace": "ingress-nginx",
                        "uid": "b1dfcbd1-f5df-4585-b127-3f81b58dcf4e",
                        "resourceVersion": "557",
                        "creationTimestamp": "2023-09-06T19: 22: 47Z",
                        "labels": {
                            "app.kubernetes.io/component": "controller",
                            "app.kubernetes.io/instance": "ingress-nginx",
                            "app.kubernetes.io/name": "ingress-nginx",
                            "gcp-auth-skip-secret": "true",
                            "pod-template-hash": "7799c6795f"
                        },
                        "ownerReferences": [
                            {
                                "apiVersion": "apps/v1",
                                "kind": "ReplicaSet",
                                "name": "ingress-nginx-controller-7799c6795f",
                                "uid": "2b1350e0-f0a1-4679-995f-06d53c243b2e",
                                "controller": true,
                                "blockOwnerDeletion": true
                            }
                        ],
                        "managedFields": [
                            {
                                "manager": "kube-controller-manager",
                                "operation": "Update",
                                "apiVersion": "v1",
                                "time": "2023-09-06T19: 22: 47Z",
                                "fieldsType": "FieldsV1",
                                "fieldsV1": {
                                    "f:metadata": {
                                        "f:generateName": {},
                                        "f:labels": {
                                            ".": {},
                                            "f:app.kubernetes.io/component": {},
                                            "f:app.kubernetes.io/instance": {},
                                            "f:app.kubernetes.io/name": {},
                                            "f:gcp-auth-skip-secret": {},
                                            "f:pod-template-hash": {}
                                        },
                                        "f:ownerReferences": {
                                            ".": {},
                                            "k: {\"uid\":\"2b1350e0-f0a1-4679-995f-06d53c243b2e\"}": {}
                                        }
                                    },
                                    "f:spec": {
                                        "f:containers": {
                                            "k:{\"name\":\"controller\"}": {
                                                ".": {},
                                                "f:args": {},
                                                "f:env": {
                                                    ".": {},
                                                    "k:{\"name\":\"LD_PRELOAD\"}": {
                                                        ".": {},
                                                        "f:name": {},
                                                        "f:value": {}
                                                    },
                                                    "k:{\"name\":\"POD_NAME\"}": {
                                                        ".": {},
                                                        "f:name": {},
                                                        "f:valueFrom": {
                                                            ".": {},
                                                            "f:fieldRef": {}
                                                        }
                                                    },
                                                    "k:{\"name\":\"POD_NAMESPACE\"}": {
                                                        ".": {},
                                                        "f:name": {},
                                                        "f:valueFrom": {
                                                            ".": {},
                                                            "f:fieldRef": {}
                                                        }
                                                    }
                                                },
                                                "f:image": {},
                                                "f:imagePullPolicy": {},
                                                "f:lifecycle": {
                                                    ".": {},
                                                    "f:preStop": {
                                                        ".": {},
                                                        "f:exec": {
                                                            ".": {},
                                                            "f:command": {}
                                                        }
                                                    }
                                                },
                                                "f:livenessProbe": {
                                                    ".": {},
                                                    "f:failureThreshold": {},
                                                    "f:httpGet": {
                                                        ".": {},
                                                        "f:path": {},
                                                        "f:port": {},
                                                        "f:scheme": {}
                                                    },
                                                    "f:initialDelaySeconds": {},
                                                    "f:periodSeconds": {},
                                                    "f:successThreshold": {},
                                                    "f:timeoutSeconds": {}
                                                },
                                                "f:name": {},
                                                "f:ports": {
                                                    ".": {},
                                                    "k: {\"containerPort\":80,\"protocol\":\"TCP\"}": {
                                                        ".": {},
                                                        "f:containerPort": {},
                                                        "f:hostPort": {},
                                                        "f:name": {},
                                                        "f:protocol": {}
                                                    },
                                                    "k: {\"containerPort\":443,\"protocol\":\"TCP\"}": {
                                                        ".": {},
                                                        "f:containerPort": {},
                                                        "f:hostPort": {},
                                                        "f:name": {},
                                                        "f:protocol": {}
                                                    },
                                                    "k:{\"containerPort\":8443,\"protocol\":\"TCP\"}": {
                                                        ".": {},
                                                        "f:containerPort": {},
                                                        "f:name": {},
                                                        "f:protocol": {}
                                                    }
                                                },
                                                "f:readinessProbe": {
                                                    ".": {},
                                                    "f:failureThreshold": {},
                                                    "f:httpGet": {
                                                        ".": {},
                                                        "f:path": {},
                                                        "f:port": {},
                                                        "f:scheme": {}
                                                    },
                                                    "f:initialDelaySeconds": {},
                                                    "f:periodSeconds": {},
                                                    "f:successThreshold": {},
                                                    "f:timeoutSeconds": {}
                                                },
                                                "f:resources": {
                                                    ".": {},
                                                    "f:requests": {
                                                        ".": {},
                                                        "f:cpu": {},
                                                        "f:memory": {}
                                                    }
                                                },
                                                "f:securityContext": {
                                                    ".": {},
                                                    "f:allowPrivilegeEscalation": {},
                                                    "f:capabilities": {
                                                        ".": {},
                                                        "f:add": {},
                                                        "f:drop": {}
                                                    },
                                                    "f:runAsUser": {}
                                                },
                                                "f:terminationMessagePath": {},
                                                "f:terminationMessagePolicy": {},
                                                "f:volumeMounts": {
                                                    ".": {},
                                                    "k: {\"mountPath\":\"/usr/local/certificates/\"}": {
                                                        ".": {},
                                                        "f:mountPath": {},
                                                        "f:name": {},
                                                        "f:readOnly": {}
                                                    }
                                                }
                                            }
                                        },
                                        "f:dnsPolicy": {},
                                        "f:enableServiceLinks": {},
                                        "f:nodeSelector": {},
                                        "f:restartPolicy": {},
                                        "f:schedulerName": {},
                                        "f:securityContext": {},
                                        "f:serviceAccount": {},
                                        "f:serviceAccountName": {},
                                        "f:terminationGracePeriodSeconds": {},
                                        "f:tolerations": {},
                                        "f:volumes": {
                                            ".": {},
                                            "k: {\"name\":\"webhook-cert\"}": {
                                                ".": {},
                                                "f:name": {},
                                                "f:secret": {
                                                    ".": {},
                                                    "f:defaultMode": {},
                                                    "f:secretName": {}
                                                }
                                            }
                                        }
                                    }
                                }
                            },
                            {
                                "manager": "kubelet",
                                "operation": "Update",
                                "apiVersion": "v1",
                                "time": "2023-09-06T19:22:47Z",
                                "fieldsType": "FieldsV1",
                                "fieldsV1": {
                                    "f:status": {
                                        "f:conditions": {
                                            "k: {\"type\":\"ContainersReady\"}": {
                                                ".": {},
                                                "f:lastProbeTime": {},
                                                "f:lastTransitionTime": {},
                                                "f:message": {},
                                                "f:reason": {},
                                                "f:status": {},
                                                "f:type": {}
                                            },
                                            "k:{\"type\":\"Initialized\"}": {
                                                ".": {},
                                                "f:lastProbeTime": {},
                                                "f:lastTransitionTime": {},
                                                "f:status": {},
                                                "f:type": {}
                                            },
                                            "k:{\"type\":\"Ready\"}": {
                                                ".": {},
                                                "f:lastProbeTime": {},
                                                "f:lastTransitionTime": {},
                                                "f:message": {},
                                                "f:reason": {},
                                                "f:status": {},
                                                "f:type": {}
                                            }
                                        },
                                        "f:containerStatuses": {},
                                        "f:hostIP": {},
                                        "f:startTime": {}
                                    }
                                },
                                "subresource": "status"
                            }
                        ]
                    },
                    "spec": {
                        "volumes": [
                            {
                                "name": "webhook-cert",
                                "secret": {
                                    "secretName": "ingress-nginx-admission",
                                    "defaultMode": 420
                                }
                            },
                            {
                                "name": "kube-api-access-cwljk",
                                "projected": {
                                    "sources": [
                                        {
                                            "serviceAccountToken": {
                                                "expirationSeconds": 3607,
                                                "path": "token"
                                            }
                                        },
                                        {
                                            "configMap": {
                                                "name": "kube-root-ca.crt",
                                                "items": [
                                                    {
                                                        "key": "ca.crt",
                                                        "path": "ca.crt"
                                                    }
                                                ]
                                            }
                                        },
                                        {
                                            "downwardAPI": {
                                                "items": [
                                                    {
                                                        "path": "namespace",
                                                        "fieldRef": {
                                                            "apiVersion": "v1",
                                                            "fieldPath": "metadata.namespace"
                                                        }
                                                    }
                                                ]
                                            }
                                        }
                                    ],
                                    "defaultMode": 420
                                }
                            }
                        ],
                        "containers": [
                            {
                                "name": "controller",
                                "image": "registry.k8s.io/ingress-nginx/controller:v1.8.1@sha256:e5c4824e7375fcf2a393e1c03c293b69759af37a9ca6abdb91b13d78a93da8bd",
                                "args": [
                                    "/nginx-ingress-controller",
                                    "--election-id=ingress-nginx-leader",
                                    "--controller-class=k8s.io/ingress-nginx",
                                    "--watch-ingress-without-class=true",
                                    "--configmap=$(POD_NAMESPACE)/ingress-nginx-controller",
                                    "--tcp-services-configmap=$(POD_NAMESPACE)/tcp-services",
                                    "--udp-services-configmap=$(POD_NAMESPACE)/udp-services",
                                    "--validating-webhook=: 8443",
                                    "--validating-webhook-certificate=/usr/local/certificates/cert",
                                    "--validating-webhook-key=/usr/local/certificates/key"
                                ],
                                "ports": [
                                    {
                                        "name": "http",
                                        "hostPort": 80,
                                        "containerPort": 80,
                                        "protocol": "TCP"
                                    },
                                    {
                                        "name": "https",
                                        "hostPort": 443,
                                        "containerPort": 443,
                                        "protocol": "TCP"
                                    },
                                    {
                                        "name": "webhook",
                                        "containerPort": 8443,
                                        "protocol": "TCP"
                                    }
                                ],
                                "env": [
                                    {
                                        "name": "POD_NAME",
                                        "valueFrom": {
                                            "fieldRef": {
                                                "apiVersion": "v1",
                                                "fieldPath": "metadata.name"
                                            }
                                        }
                                    },
                                    {
                                        "name": "POD_NAMESPACE",
                                        "valueFrom": {
                                            "fieldRef": {
                                                "apiVersion": "v1",
                                                "fieldPath": "metadata.namespace"
                                            }
                                        }
                                    },
                                    {
                                        "name": "LD_PRELOAD",
                                        "value": "/usr/local/lib/libmimalloc.so"
                                    }
                                ],
                                "resources": {
                                    "requests": {
                                        "cpu": "100m",
                                        "memory": "90Mi"
                                    }
                                },
                                "volumeMounts": [
                                    {
                                        "name": "webhook-cert",
                                        "readOnly": true,
                                        "mountPath": "/usr/local/certificates/"
                                    },
                                    {
                                        "name": "kube-api-access-cwljk",
                                        "readOnly": true,
                                        "mountPath": "/var/run/secrets/kubernetes.io/serviceaccount"
                                    }
                                ],
                                "livenessProbe": {
                                    "httpGet": {
                                        "path": "/healthz",
                                        "port": 10254,
                                        "scheme": "HTTP"
                                    },
                                    "initialDelaySeconds": 10,
                                    "timeoutSeconds": 1,
                                    "periodSeconds": 10,
                                    "successThreshold": 1,
                                    "failureThreshold": 5
                                },
                                "readinessProbe": {
                                    "httpGet": {
                                        "path": "/healthz",
                                        "port": 10254,
                                        "scheme": "HTTP"
                                    },
                                    "initialDelaySeconds": 10,
                                    "timeoutSeconds": 1,
                                    "periodSeconds": 10,
                                    "successThreshold": 1,
                                    "failureThreshold": 3
                                },
                                "lifecycle": {
                                    "preStop": {
                                        "exec": {
                                            "command": [
                                                "/wait-shutdown"
                                            ]
                                        }
                                    }
                                },
                                "terminationMessagePath": "/dev/termination-log",
                                "terminationMessagePolicy": "File",
                                "imagePullPolicy": "IfNotPresent",
                                "securityContext": {
                                    "capabilities": {
                                        "add": [
                                            "NET_BIND_SERVICE"
                                        ],
                                        "drop": [
                                            "ALL"
                                        ]
                                    },
                                    "runAsUser": 101,
                                    "allowPrivilegeEscalation": true
                                }
                            }
                        ],
                        "restartPolicy": "Always",
                        "terminationGracePeriodSeconds": 0,
                        "dnsPolicy": "ClusterFirst",
                        "nodeSelector": {
                            "kubernetes.io/os": "linux",
                            "minikube.k8s.io/primary": "true"
                        },
                        "serviceAccountName": "ingress-nginx",
                        "serviceAccount": "ingress-nginx",
                        "nodeName": "polar",
                        "securityContext": {},
                        "schedulerName": "default-scheduler",
                        "tolerations": [
                            {
                                "key": "node-role.kubernetes.io/master",
                                "operator": "Equal",
                                "effect": "NoSchedule"
                            },
                            {
                                "key": "node.kubernetes.io/not-ready",
                                "operator": "Exists",
                                "effect": "NoExecute",
                                "tolerationSeconds": 300
                            },
                            {
                                "key": "node.kubernetes.io/unreachable",
                                "operator": "Exists",
                                "effect": "NoExecute",
                                "tolerationSeconds": 300
                            }
                        ],
                        "priority": 0,
                        "enableServiceLinks": true,
                        "preemptionPolicy": "PreemptLowerPriority"
                    },
                    "status": {
                        "phase": "Pending",
                        "conditions": [
                            {
                                "type": "Initialized",
                                "status": "True",
                                "lastProbeTime": null,
                                "lastTransitionTime": "2023-09-06T19:22:47Z"
                            },
                            {
                                "type": "Ready",
                                "status": "False",
                                "lastProbeTime": null,
                                "lastTransitionTime": "2023-09-06T19:22:47Z",
                                "reason": "ContainersNotReady",
                                "message": "containers with unready status: [controller]"
                            },
                            {
                                "type": "ContainersReady",
                                "status": "False",
                                "lastProbeTime": null,
                                "lastTransitionTime": "2023-09-06T19: 22: 47Z",
                                "reason": "ContainersNotReady",
                                "message": "containers with unready status: [controller]"
                            },
                            {
                                "type": "PodScheduled",
                                "status": "True",
                                "lastProbeTime": null,
                                "lastTransitionTime": "2023-09-06T19: 22: 47Z"
                            }
                        ],
                        "hostIP": "192.168.49.2",
                        "startTime": "2023-09-06T19: 22: 47Z",
                        "containerStatuses": [
                            {
                                "name": "controller",
                                "state": {
                                    "waiting": {
                                        "reason": "ContainerCreating"
                                    }
                                },
                                "lastState": {},
                                "ready": false,
                                "restartCount": 0,
                                "image": "registry.k8s.io/ingress-nginx/controller:v1.8.1@sha256:e5c4824e7375fcf2a393e1c03c293b69759af37a9ca6abdb91b13d78a93da8bd",
                                "imageID": "",
                                "started": false
                            }
                        ],
                        "qosClass": "Burstable"
                    }
                }
            ]
        }
    }
    
  3. output of command runned in my host's terminal
    
    ➜  ~ http GET https://192.168.49.2:8443/api/v1/namespaces/ingress-nginx/pods\?labelSelector\=app.kubernetes.io%2Fname%3Dingress-nginx
    

    http: LogLevel.ERROR: SSLError: HTTPSConnectionPool(host='192.168.49.2', port=8443): Max retries exceeded with url: /api/v1/namespaces/ingress-nginx/pods?labelSelector=app.kubernetes.io%2Fname%3Dingress-nginx (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1002)'))) while doing a GET request to URL: https://192.168.49.2:8443/api/v1/namespaces/ingress-nginx/pods?labelSelector=app.kubernetes.io%2Fname%3Dingress-nginx

@ongiant
Copy link
Author

ongiant commented Sep 12, 2023

Cause & Solution

Cause

The cause is that image cannot be pulled. It works well after I change the image registry.

Solution

minikube version: v1.31.2

Note:

  1. remove externalTrafficPolicy: Local in ingress-nginx-controller Service; $\quad$ (unnecessary)
  2. add hostNetwork: true in ingress-nginx-controller Deployment; $\quad$ (necessary)

This is my modified deploy file (source: https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.8.1/deploy/static/provider/cloud/deploy.yaml, I find this link on ingress nginx controller Installation Guide):

  1. deploy.yml
    
    apiVersion: v1
    kind: Namespace
    metadata:
      labels:
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/name: ingress-nginx
      name: ingress-nginx
    ---
    apiVersion: v1
    automountServiceAccountToken: true
    kind: ServiceAccount
    metadata:
      labels:
        app.kubernetes.io/component: controller
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
        app.kubernetes.io/version: 1.8.1
      name: ingress-nginx
      namespace: ingress-nginx
    ---
    apiVersion: v1
    kind: ServiceAccount
    metadata:
      labels:
        app.kubernetes.io/component: admission-webhook
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
        app.kubernetes.io/version: 1.8.1
      name: ingress-nginx-admission
      namespace: ingress-nginx
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: Role
    metadata:
      labels:
        app.kubernetes.io/component: controller
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
        app.kubernetes.io/version: 1.8.1
      name: ingress-nginx
      namespace: ingress-nginx
    rules:
      - apiGroups:
          - ""
        resources:
          - namespaces
        verbs:
          - get
      - apiGroups:
          - ""
        resources:
          - configmaps
          - pods
          - secrets
          - endpoints
        verbs:
          - get
          - list
          - watch
      - apiGroups:
          - ""
        resources:
          - services
        verbs:
          - get
          - list
          - watch
      - apiGroups:
          - networking.k8s.io
        resources:
          - ingresses
        verbs:
          - get
          - list
          - watch
      - apiGroups:
          - networking.k8s.io
        resources:
          - ingresses/status
        verbs:
          - update
      - apiGroups:
          - networking.k8s.io
        resources:
          - ingressclasses
        verbs:
          - get
          - list
          - watch
      - apiGroups:
          - coordination.k8s.io
        resourceNames:
          - ingress-nginx-leader
        resources:
          - leases
        verbs:
          - get
          - update
      - apiGroups:
          - coordination.k8s.io
        resources:
          - leases
        verbs:
          - create
      - apiGroups:
          - ""
        resources:
          - events
        verbs:
          - create
          - patch
      - apiGroups:
          - discovery.k8s.io
        resources:
          - endpointslices
        verbs:
          - list
          - watch
          - get
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: Role
    metadata:
      labels:
        app.kubernetes.io/component: admission-webhook
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
        app.kubernetes.io/version: 1.8.1
      name: ingress-nginx-admission
      namespace: ingress-nginx
    rules:
      - apiGroups:
          - ""
        resources:
          - secrets
        verbs:
          - get
          - create
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRole
    metadata:
      labels:
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
        app.kubernetes.io/version: 1.8.1
      name: ingress-nginx
    rules:
      - apiGroups:
          - ""
        resources:
          - configmaps
          - endpoints
          - nodes
          - pods
          - secrets
          - namespaces
        verbs:
          - list
          - watch
      - apiGroups:
          - coordination.k8s.io
        resources:
          - leases
        verbs:
          - list
          - watch
      - apiGroups:
          - ""
        resources:
          - nodes
        verbs:
          - get
      - apiGroups:
          - ""
        resources:
          - services
        verbs:
          - get
          - list
          - watch
      - apiGroups:
          - networking.k8s.io
        resources:
          - ingresses
        verbs:
          - get
          - list
          - watch
      - apiGroups:
          - ""
        resources:
          - events
        verbs:
          - create
          - patch
      - apiGroups:
          - networking.k8s.io
        resources:
          - ingresses/status
        verbs:
          - update
      - apiGroups:
          - networking.k8s.io
        resources:
          - ingressclasses
        verbs:
          - get
          - list
          - watch
      - apiGroups:
          - discovery.k8s.io
        resources:
          - endpointslices
        verbs:
          - list
          - watch
          - get
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRole
    metadata:
      labels:
        app.kubernetes.io/component: admission-webhook
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
        app.kubernetes.io/version: 1.8.1
      name: ingress-nginx-admission
    rules:
      - apiGroups:
          - admissionregistration.k8s.io
        resources:
          - validatingwebhookconfigurations
        verbs:
          - get
          - update
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: RoleBinding
    metadata:
      labels:
        app.kubernetes.io/component: controller
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
        app.kubernetes.io/version: 1.8.1
      name: ingress-nginx
      namespace: ingress-nginx
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: Role
      name: ingress-nginx
    subjects:
      - kind: ServiceAccount
        name: ingress-nginx
        namespace: ingress-nginx
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: RoleBinding
    metadata:
      labels:
        app.kubernetes.io/component: admission-webhook
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
        app.kubernetes.io/version: 1.8.1
      name: ingress-nginx-admission
      namespace: ingress-nginx
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: Role
      name: ingress-nginx-admission
    subjects:
      - kind: ServiceAccount
        name: ingress-nginx-admission
        namespace: ingress-nginx
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRoleBinding
    metadata:
      labels:
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
        app.kubernetes.io/version: 1.8.1
      name: ingress-nginx
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: ClusterRole
      name: ingress-nginx
    subjects:
      - kind: ServiceAccount
        name: ingress-nginx
        namespace: ingress-nginx
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRoleBinding
    metadata:
      labels:
        app.kubernetes.io/component: admission-webhook
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
        app.kubernetes.io/version: 1.8.1
      name: ingress-nginx-admission
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: ClusterRole
      name: ingress-nginx-admission
    subjects:
      - kind: ServiceAccount
        name: ingress-nginx-admission
        namespace: ingress-nginx
    ---
    apiVersion: v1
    data:
      allow-snippet-annotations: "true"
    kind: ConfigMap
    metadata:
      labels:
        app.kubernetes.io/component: controller
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
        app.kubernetes.io/version: 1.8.1
      name: ingress-nginx-controller
      namespace: ingress-nginx
    ---
    apiVersion: v1
    kind: Service
    metadata:
      labels:
        app.kubernetes.io/component: controller
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
        app.kubernetes.io/version: 1.8.1
      name: ingress-nginx-controller
      namespace: ingress-nginx
    spec:
      #externalTrafficPolicy: Local
      ipFamilies:
        - IPv4
      ipFamilyPolicy: SingleStack
      ports:
        - appProtocol: http
          name: http
          port: 80
          protocol: TCP
          targetPort: http
        - appProtocol: https
          name: https
          port: 443
          protocol: TCP
          targetPort: https
      selector:
        app.kubernetes.io/component: controller
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/name: ingress-nginx
      type: LoadBalancer
    ---
    apiVersion: v1
    kind: Service
    metadata:
      labels:
        app.kubernetes.io/component: controller
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
        app.kubernetes.io/version: 1.8.1
      name: ingress-nginx-controller-admission
      namespace: ingress-nginx
    spec:
      ports:
        - appProtocol: https
          name: https-webhook
          port: 443
          targetPort: webhook
      selector:
        app.kubernetes.io/component: controller
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/name: ingress-nginx
      type: ClusterIP
    ---
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      labels:
        app.kubernetes.io/component: controller
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
        app.kubernetes.io/version: 1.8.1
      name: ingress-nginx-controller
      namespace: ingress-nginx
    spec:
      minReadySeconds: 0
      revisionHistoryLimit: 10
      selector:
        matchLabels:
          app.kubernetes.io/component: controller
          app.kubernetes.io/instance: ingress-nginx
          app.kubernetes.io/name: ingress-nginx
      template:
        metadata:
          labels:
            app.kubernetes.io/component: controller
            app.kubernetes.io/instance: ingress-nginx
            app.kubernetes.io/name: ingress-nginx
            app.kubernetes.io/part-of: ingress-nginx
            app.kubernetes.io/version: 1.8.1
        spec:
          hostNetwork: true
          containers:
            - args:
                - /nginx-ingress-controller
                - --publish-service=$(POD_NAMESPACE)/ingress-nginx-controller
                - --election-id=ingress-nginx-leader
                - --controller-class=k8s.io/ingress-nginx
                - --ingress-class=nginx
                - --configmap=$(POD_NAMESPACE)/ingress-nginx-controller
                - --validating-webhook=:8443
                - --validating-webhook-certificate=/usr/local/certificates/cert
                - --validating-webhook-key=/usr/local/certificates/key
              env:
                - name: POD_NAME
                  valueFrom:
                    fieldRef:
                      fieldPath: metadata.name
                - name: POD_NAMESPACE
                  valueFrom:
                    fieldRef:
                      fieldPath: metadata.namespace
                - name: LD_PRELOAD
                  value: /usr/local/lib/libmimalloc.so
              image: bitnami/nginx-ingress-controller:1.8.1
              imagePullPolicy: IfNotPresent
              lifecycle:
                preStop:
                  exec:
                    command:
                      - /wait-shutdown
              livenessProbe:
                failureThreshold: 5
                httpGet:
                  path: /healthz
                  port: 10254
                  scheme: HTTP
                initialDelaySeconds: 10
                periodSeconds: 10
                successThreshold: 1
                timeoutSeconds: 1
              name: controller
              ports:
                - containerPort: 80
                  name: http
                  protocol: TCP
                - containerPort: 443
                  name: https
                  protocol: TCP
                - containerPort: 8443
                  name: webhook
                  protocol: TCP
              readinessProbe:
                failureThreshold: 3
                httpGet:
                  path: /healthz
                  port: 10254
                  scheme: HTTP
                initialDelaySeconds: 10
                periodSeconds: 10
                successThreshold: 1
                timeoutSeconds: 1
              resources:
                requests:
                  cpu: 100m
                  memory: 90Mi
              securityContext:
                allowPrivilegeEscalation: true
                capabilities:
                  add:
                    - NET_BIND_SERVICE
                  drop:
                    - ALL
                runAsUser: 101
              volumeMounts:
                - mountPath: /usr/local/certificates/
                  name: webhook-cert
                  readOnly: true
          dnsPolicy: ClusterFirst
          nodeSelector:
            kubernetes.io/os: linux
          serviceAccountName: ingress-nginx
          terminationGracePeriodSeconds: 300
          volumes:
            - name: webhook-cert
              secret:
                secretName: ingress-nginx-admission
    ---
    apiVersion: batch/v1
    kind: Job
    metadata:
      labels:
        app.kubernetes.io/component: admission-webhook
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
        app.kubernetes.io/version: 1.8.1
      name: ingress-nginx-admission-create
      namespace: ingress-nginx
    spec:
      template:
        metadata:
          labels:
            app.kubernetes.io/component: admission-webhook
            app.kubernetes.io/instance: ingress-nginx
            app.kubernetes.io/name: ingress-nginx
            app.kubernetes.io/part-of: ingress-nginx
            app.kubernetes.io/version: 1.8.1
          name: ingress-nginx-admission-create
        spec:
          containers:
            - args:
                - create
                - --host=ingress-nginx-controller-admission,ingress-nginx-controller-admission.$(POD_NAMESPACE).svc
                - --namespace=$(POD_NAMESPACE)
                - --secret-name=ingress-nginx-admission
              env:
                - name: POD_NAMESPACE
                  valueFrom:
                    fieldRef:
                      fieldPath: metadata.namespace
              image: anjia0532/google-containers.ingress-nginx.kube-webhook-certgen:v20230407
              imagePullPolicy: IfNotPresent
              name: create
              securityContext:
                allowPrivilegeEscalation: false
          nodeSelector:
            kubernetes.io/os: linux
          restartPolicy: OnFailure
          securityContext:
            fsGroup: 2000
            runAsNonRoot: true
            runAsUser: 2000
          serviceAccountName: ingress-nginx-admission
    ---
    apiVersion: batch/v1
    kind: Job
    metadata:
      labels:
        app.kubernetes.io/component: admission-webhook
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
        app.kubernetes.io/version: 1.8.1
      name: ingress-nginx-admission-patch
      namespace: ingress-nginx
    spec:
      template:
        metadata:
          labels:
            app.kubernetes.io/component: admission-webhook
            app.kubernetes.io/instance: ingress-nginx
            app.kubernetes.io/name: ingress-nginx
            app.kubernetes.io/part-of: ingress-nginx
            app.kubernetes.io/version: 1.8.1
          name: ingress-nginx-admission-patch
        spec:
          containers:
            - args:
                - patch
                - --webhook-name=ingress-nginx-admission
                - --namespace=$(POD_NAMESPACE)
                - --patch-mutating=false
                - --secret-name=ingress-nginx-admission
                - --patch-failure-policy=Fail
              env:
                - name: POD_NAMESPACE
                  valueFrom:
                    fieldRef:
                      fieldPath: metadata.namespace
              image: anjia0532/google-containers.ingress-nginx.kube-webhook-certgen:v20230407
              imagePullPolicy: IfNotPresent
              name: patch
              securityContext:
                allowPrivilegeEscalation: false
          nodeSelector:
            kubernetes.io/os: linux
          restartPolicy: OnFailure
          securityContext:
            fsGroup: 2000
            runAsNonRoot: true
            runAsUser: 2000
          serviceAccountName: ingress-nginx-admission
    ---
    apiVersion: networking.k8s.io/v1
    kind: IngressClass
    metadata:
      labels:
        app.kubernetes.io/component: controller
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
        app.kubernetes.io/version: 1.8.1
      name: nginx
    spec:
      controller: k8s.io/ingress-nginx
    ---
    apiVersion: admissionregistration.k8s.io/v1
    kind: ValidatingWebhookConfiguration
    metadata:
      labels:
        app.kubernetes.io/component: admission-webhook
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
        app.kubernetes.io/version: 1.8.1
      name: ingress-nginx-admission
    webhooks:
      - admissionReviewVersions:
          - v1
        clientConfig:
          service:
            name: ingress-nginx-controller-admission
            namespace: ingress-nginx
            path: /networking/v1/ingresses
        failurePolicy: Fail
        matchPolicy: Equivalent
        name: validate.nginx.ingress.kubernetes.io
        rules:
          - apiGroups:
              - networking.k8s.io
            apiVersions:
              - v1
            operations:
              - CREATE
              - UPDATE
            resources:
              - ingresses
        sideEffects: None
    

  2. aliyuncs_deploy.yml
    
    apiVersion: v1
    kind: Namespace
    metadata:
      labels:
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/name: ingress-nginx
      name: ingress-nginx
    ---
    apiVersion: v1
    automountServiceAccountToken: true
    kind: ServiceAccount
    metadata:
      labels:
        app.kubernetes.io/component: controller
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
        app.kubernetes.io/version: 1.8.1
      name: ingress-nginx
      namespace: ingress-nginx
    ---
    apiVersion: v1
    kind: ServiceAccount
    metadata:
      labels:
        app.kubernetes.io/component: admission-webhook
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
        app.kubernetes.io/version: 1.8.1
      name: ingress-nginx-admission
      namespace: ingress-nginx
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: Role
    metadata:
      labels:
        app.kubernetes.io/component: controller
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
        app.kubernetes.io/version: 1.8.1
      name: ingress-nginx
      namespace: ingress-nginx
    rules:
      - apiGroups:
          - ""
        resources:
          - namespaces
        verbs:
          - get
      - apiGroups:
          - ""
        resources:
          - configmaps
          - pods
          - secrets
          - endpoints
        verbs:
          - get
          - list
          - watch
      - apiGroups:
          - ""
        resources:
          - services
        verbs:
          - get
          - list
          - watch
      - apiGroups:
          - networking.k8s.io
        resources:
          - ingresses
        verbs:
          - get
          - list
          - watch
      - apiGroups:
          - networking.k8s.io
        resources:
          - ingresses/status
        verbs:
          - update
      - apiGroups:
          - networking.k8s.io
        resources:
          - ingressclasses
        verbs:
          - get
          - list
          - watch
      - apiGroups:
          - coordination.k8s.io
        resourceNames:
          - ingress-nginx-leader
        resources:
          - leases
        verbs:
          - get
          - update
      - apiGroups:
          - coordination.k8s.io
        resources:
          - leases
        verbs:
          - create
      - apiGroups:
          - ""
        resources:
          - events
        verbs:
          - create
          - patch
      - apiGroups:
          - discovery.k8s.io
        resources:
          - endpointslices
        verbs:
          - list
          - watch
          - get
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: Role
    metadata:
      labels:
        app.kubernetes.io/component: admission-webhook
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
        app.kubernetes.io/version: 1.8.1
      name: ingress-nginx-admission
      namespace: ingress-nginx
    rules:
      - apiGroups:
          - ""
        resources:
          - secrets
        verbs:
          - get
          - create
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRole
    metadata:
      labels:
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
        app.kubernetes.io/version: 1.8.1
      name: ingress-nginx
    rules:
      - apiGroups:
          - ""
        resources:
          - configmaps
          - endpoints
          - nodes
          - pods
          - secrets
          - namespaces
        verbs:
          - list
          - watch
      - apiGroups:
          - coordination.k8s.io
        resources:
          - leases
        verbs:
          - list
          - watch
      - apiGroups:
          - ""
        resources:
          - nodes
        verbs:
          - get
      - apiGroups:
          - ""
        resources:
          - services
        verbs:
          - get
          - list
          - watch
      - apiGroups:
          - networking.k8s.io
        resources:
          - ingresses
        verbs:
          - get
          - list
          - watch
      - apiGroups:
          - ""
        resources:
          - events
        verbs:
          - create
          - patch
      - apiGroups:
          - networking.k8s.io
        resources:
          - ingresses/status
        verbs:
          - update
      - apiGroups:
          - networking.k8s.io
        resources:
          - ingressclasses
        verbs:
          - get
          - list
          - watch
      - apiGroups:
          - discovery.k8s.io
        resources:
          - endpointslices
        verbs:
          - list
          - watch
          - get
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRole
    metadata:
      labels:
        app.kubernetes.io/component: admission-webhook
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
        app.kubernetes.io/version: 1.8.1
      name: ingress-nginx-admission
    rules:
      - apiGroups:
          - admissionregistration.k8s.io
        resources:
          - validatingwebhookconfigurations
        verbs:
          - get
          - update
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: RoleBinding
    metadata:
      labels:
        app.kubernetes.io/component: controller
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
        app.kubernetes.io/version: 1.8.1
      name: ingress-nginx
      namespace: ingress-nginx
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: Role
      name: ingress-nginx
    subjects:
      - kind: ServiceAccount
        name: ingress-nginx
        namespace: ingress-nginx
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: RoleBinding
    metadata:
      labels:
        app.kubernetes.io/component: admission-webhook
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
        app.kubernetes.io/version: 1.8.1
      name: ingress-nginx-admission
      namespace: ingress-nginx
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: Role
      name: ingress-nginx-admission
    subjects:
      - kind: ServiceAccount
        name: ingress-nginx-admission
        namespace: ingress-nginx
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRoleBinding
    metadata:
      labels:
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
        app.kubernetes.io/version: 1.8.1
      name: ingress-nginx
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: ClusterRole
      name: ingress-nginx
    subjects:
      - kind: ServiceAccount
        name: ingress-nginx
        namespace: ingress-nginx
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRoleBinding
    metadata:
      labels:
        app.kubernetes.io/component: admission-webhook
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
        app.kubernetes.io/version: 1.8.1
      name: ingress-nginx-admission
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: ClusterRole
      name: ingress-nginx-admission
    subjects:
      - kind: ServiceAccount
        name: ingress-nginx-admission
        namespace: ingress-nginx
    ---
    apiVersion: v1
    data:
      allow-snippet-annotations: "true"
    kind: ConfigMap
    metadata:
      labels:
        app.kubernetes.io/component: controller
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
        app.kubernetes.io/version: 1.8.1
      name: ingress-nginx-controller
      namespace: ingress-nginx
    ---
    apiVersion: v1
    kind: Service
    metadata:
      labels:
        app.kubernetes.io/component: controller
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
        app.kubernetes.io/version: 1.8.1
      name: ingress-nginx-controller
      namespace: ingress-nginx
    spec:
      # externalTrafficPolicy: Local
      ipFamilies:
        - IPv4
      ipFamilyPolicy: SingleStack
      ports:
        - appProtocol: http
          name: http
          port: 80
          protocol: TCP
          targetPort: http
        - appProtocol: https
          name: https
          port: 443
          protocol: TCP
          targetPort: https
      selector:
        app.kubernetes.io/component: controller
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/name: ingress-nginx
      type: LoadBalancer
    ---
    apiVersion: v1
    kind: Service
    metadata:
      labels:
        app.kubernetes.io/component: controller
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
        app.kubernetes.io/version: 1.8.1
      name: ingress-nginx-controller-admission
      namespace: ingress-nginx
    spec:
      ports:
        - appProtocol: https
          name: https-webhook
          port: 443
          targetPort: webhook
      selector:
        app.kubernetes.io/component: controller
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/name: ingress-nginx
      type: ClusterIP
    ---
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      labels:
        app.kubernetes.io/component: controller
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
        app.kubernetes.io/version: 1.8.1
      name: ingress-nginx-controller
      namespace: ingress-nginx
    spec:
      minReadySeconds: 0
      revisionHistoryLimit: 10
      selector:
        matchLabels:
          app.kubernetes.io/component: controller
          app.kubernetes.io/instance: ingress-nginx
          app.kubernetes.io/name: ingress-nginx
      template:
        metadata:
          labels:
            app.kubernetes.io/component: controller
            app.kubernetes.io/instance: ingress-nginx
            app.kubernetes.io/name: ingress-nginx
            app.kubernetes.io/part-of: ingress-nginx
            app.kubernetes.io/version: 1.8.1
        spec:
          hostNetwork: true
          containers:
            - args:
                - /nginx-ingress-controller
                - --publish-service=$(POD_NAMESPACE)/ingress-nginx-controller
                - --election-id=ingress-nginx-leader
                - --controller-class=k8s.io/ingress-nginx
                - --ingress-class=nginx
                - --configmap=$(POD_NAMESPACE)/ingress-nginx-controller
                - --validating-webhook=:8443
                - --validating-webhook-certificate=/usr/local/certificates/cert
                - --validating-webhook-key=/usr/local/certificates/key
              env:
                - name: POD_NAME
                  valueFrom:
                    fieldRef:
                      fieldPath: metadata.name
                - name: POD_NAMESPACE
                  valueFrom:
                    fieldRef:
                      fieldPath: metadata.namespace
                - name: LD_PRELOAD
                  value: /usr/local/lib/libmimalloc.so
              image: registry.aliyuncs.com/google_containers/nginx-ingress-controller:v1.8.1@sha256:e5c4824e7375fcf2a393e1c03c293b69759af37a9ca6abdb91b13d78a93da8bd
              imagePullPolicy: IfNotPresent
              lifecycle:
                preStop:
                  exec:
                    command:
                      - /wait-shutdown
              livenessProbe:
                failureThreshold: 5
                httpGet:
                  path: /healthz
                  port: 10254
                  scheme: HTTP
                initialDelaySeconds: 10
                periodSeconds: 10
                successThreshold: 1
                timeoutSeconds: 1
              name: controller
              ports:
                - containerPort: 80
                  name: http
                  protocol: TCP
                - containerPort: 443
                  name: https
                  protocol: TCP
                - containerPort: 8443
                  name: webhook
                  protocol: TCP
              readinessProbe:
                failureThreshold: 3
                httpGet:
                  path: /healthz
                  port: 10254
                  scheme: HTTP
                initialDelaySeconds: 10
                periodSeconds: 10
                successThreshold: 1
                timeoutSeconds: 1
              resources:
                requests:
                  cpu: 100m
                  memory: 90Mi
              securityContext:
                allowPrivilegeEscalation: true
                capabilities:
                  add:
                    - NET_BIND_SERVICE
                  drop:
                    - ALL
                runAsUser: 101
              volumeMounts:
                - mountPath: /usr/local/certificates/
                  name: webhook-cert
                  readOnly: true
          dnsPolicy: ClusterFirst
          nodeSelector:
            kubernetes.io/os: linux
          serviceAccountName: ingress-nginx
          terminationGracePeriodSeconds: 300
          volumes:
            - name: webhook-cert
              secret:
                secretName: ingress-nginx-admission
    ---
    apiVersion: batch/v1
    kind: Job
    metadata:
      labels:
        app.kubernetes.io/component: admission-webhook
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
        app.kubernetes.io/version: 1.8.1
      name: ingress-nginx-admission-create
      namespace: ingress-nginx
    spec:
      template:
        metadata:
          labels:
            app.kubernetes.io/component: admission-webhook
            app.kubernetes.io/instance: ingress-nginx
            app.kubernetes.io/name: ingress-nginx
            app.kubernetes.io/part-of: ingress-nginx
            app.kubernetes.io/version: 1.8.1
          name: ingress-nginx-admission-create
        spec:
          containers:
            - args:
                - create
                - --host=ingress-nginx-controller-admission,ingress-nginx-controller-admission.$(POD_NAMESPACE).svc
                - --namespace=$(POD_NAMESPACE)
                - --secret-name=ingress-nginx-admission
              env:
                - name: POD_NAMESPACE
                  valueFrom:
                    fieldRef:
                      fieldPath: metadata.namespace
              image: registry.aliyuncs.com/google_containers/kube-webhook-certgen:v20230407@sha256:543c40fd093964bc9ab509d3e791f9989963021f1e9e4c9c7b6700b02bfb227b
              imagePullPolicy: IfNotPresent
              name: create
              securityContext:
                allowPrivilegeEscalation: false
          nodeSelector:
            kubernetes.io/os: linux
          restartPolicy: OnFailure
          securityContext:
            fsGroup: 2000
            runAsNonRoot: true
            runAsUser: 2000
          serviceAccountName: ingress-nginx-admission
    ---
    apiVersion: batch/v1
    kind: Job
    metadata:
      labels:
        app.kubernetes.io/component: admission-webhook
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
        app.kubernetes.io/version: 1.8.1
      name: ingress-nginx-admission-patch
      namespace: ingress-nginx
    spec:
      template:
        metadata:
          labels:
            app.kubernetes.io/component: admission-webhook
            app.kubernetes.io/instance: ingress-nginx
            app.kubernetes.io/name: ingress-nginx
            app.kubernetes.io/part-of: ingress-nginx
            app.kubernetes.io/version: 1.8.1
          name: ingress-nginx-admission-patch
        spec:
          containers:
            - args:
                - patch
                - --webhook-name=ingress-nginx-admission
                - --namespace=$(POD_NAMESPACE)
                - --patch-mutating=false
                - --secret-name=ingress-nginx-admission
                - --patch-failure-policy=Fail
              env:
                - name: POD_NAMESPACE
                  valueFrom:
                    fieldRef:
                      fieldPath: metadata.namespace
              image: registry.aliyuncs.com/google_containers/kube-webhook-certgen:v20230407@sha256:543c40fd093964bc9ab509d3e791f9989963021f1e9e4c9c7b6700b02bfb227b
              imagePullPolicy: IfNotPresent
              name: patch
              securityContext:
                allowPrivilegeEscalation: false
          nodeSelector:
            kubernetes.io/os: linux
          restartPolicy: OnFailure
          securityContext:
            fsGroup: 2000
            runAsNonRoot: true
            runAsUser: 2000
          serviceAccountName: ingress-nginx-admission
    ---
    apiVersion: networking.k8s.io/v1
    kind: IngressClass
    metadata:
      labels:
        app.kubernetes.io/component: controller
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
        app.kubernetes.io/version: 1.8.1
      name: nginx
    spec:
      controller: k8s.io/ingress-nginx
    ---
    apiVersion: admissionregistration.k8s.io/v1
    kind: ValidatingWebhookConfiguration
    metadata:
      labels:
        app.kubernetes.io/component: admission-webhook
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
        app.kubernetes.io/version: 1.8.1
      name: ingress-nginx-admission
    webhooks:
      - admissionReviewVersions:
          - v1
        clientConfig:
          service:
            name: ingress-nginx-controller-admission
            namespace: ingress-nginx
            path: /networking/v1/ingresses
        failurePolicy: Fail
        matchPolicy: Equivalent
        name: validate.nginx.ingress.kubernetes.io
        rules:
          - apiGroups:
              - networking.k8s.io
            apiVersions:
              - v1
            operations:
              - CREATE
              - UPDATE
            resources:
              - ingresses
        sideEffects: None
    

  3. cn-hangzhou_aliyuncs_deploy.yml
    
    apiVersion: v1
    kind: Namespace
    metadata:
      labels:
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/name: ingress-nginx
      name: ingress-nginx
    ---
    apiVersion: v1
    automountServiceAccountToken: true
    kind: ServiceAccount
    metadata:
      labels:
        app.kubernetes.io/component: controller
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
        app.kubernetes.io/version: 1.8.1
      name: ingress-nginx
      namespace: ingress-nginx
    ---
    apiVersion: v1
    kind: ServiceAccount
    metadata:
      labels:
        app.kubernetes.io/component: admission-webhook
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
        app.kubernetes.io/version: 1.8.1
      name: ingress-nginx-admission
      namespace: ingress-nginx
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: Role
    metadata:
      labels:
        app.kubernetes.io/component: controller
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
        app.kubernetes.io/version: 1.8.1
      name: ingress-nginx
      namespace: ingress-nginx
    rules:
      - apiGroups:
          - ""
        resources:
          - namespaces
        verbs:
          - get
      - apiGroups:
          - ""
        resources:
          - configmaps
          - pods
          - secrets
          - endpoints
        verbs:
          - get
          - list
          - watch
      - apiGroups:
          - ""
        resources:
          - services
        verbs:
          - get
          - list
          - watch
      - apiGroups:
          - networking.k8s.io
        resources:
          - ingresses
        verbs:
          - get
          - list
          - watch
      - apiGroups:
          - networking.k8s.io
        resources:
          - ingresses/status
        verbs:
          - update
      - apiGroups:
          - networking.k8s.io
        resources:
          - ingressclasses
        verbs:
          - get
          - list
          - watch
      - apiGroups:
          - coordination.k8s.io
        resourceNames:
          - ingress-nginx-leader
        resources:
          - leases
        verbs:
          - get
          - update
      - apiGroups:
          - coordination.k8s.io
        resources:
          - leases
        verbs:
          - create
      - apiGroups:
          - ""
        resources:
          - events
        verbs:
          - create
          - patch
      - apiGroups:
          - discovery.k8s.io
        resources:
          - endpointslices
        verbs:
          - list
          - watch
          - get
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: Role
    metadata:
      labels:
        app.kubernetes.io/component: admission-webhook
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
        app.kubernetes.io/version: 1.8.1
      name: ingress-nginx-admission
      namespace: ingress-nginx
    rules:
      - apiGroups:
          - ""
        resources:
          - secrets
        verbs:
          - get
          - create
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRole
    metadata:
      labels:
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
        app.kubernetes.io/version: 1.8.1
      name: ingress-nginx
    rules:
      - apiGroups:
          - ""
        resources:
          - configmaps
          - endpoints
          - nodes
          - pods
          - secrets
          - namespaces
        verbs:
          - list
          - watch
      - apiGroups:
          - coordination.k8s.io
        resources:
          - leases
        verbs:
          - list
          - watch
      - apiGroups:
          - ""
        resources:
          - nodes
        verbs:
          - get
      - apiGroups:
          - ""
        resources:
          - services
        verbs:
          - get
          - list
          - watch
      - apiGroups:
          - networking.k8s.io
        resources:
          - ingresses
        verbs:
          - get
          - list
          - watch
      - apiGroups:
          - ""
        resources:
          - events
        verbs:
          - create
          - patch
      - apiGroups:
          - networking.k8s.io
        resources:
          - ingresses/status
        verbs:
          - update
      - apiGroups:
          - networking.k8s.io
        resources:
          - ingressclasses
        verbs:
          - get
          - list
          - watch
      - apiGroups:
          - discovery.k8s.io
        resources:
          - endpointslices
        verbs:
          - list
          - watch
          - get
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRole
    metadata:
      labels:
        app.kubernetes.io/component: admission-webhook
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
        app.kubernetes.io/version: 1.8.1
      name: ingress-nginx-admission
    rules:
      - apiGroups:
          - admissionregistration.k8s.io
        resources:
          - validatingwebhookconfigurations
        verbs:
          - get
          - update
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: RoleBinding
    metadata:
      labels:
        app.kubernetes.io/component: controller
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
        app.kubernetes.io/version: 1.8.1
      name: ingress-nginx
      namespace: ingress-nginx
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: Role
      name: ingress-nginx
    subjects:
      - kind: ServiceAccount
        name: ingress-nginx
        namespace: ingress-nginx
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: RoleBinding
    metadata:
      labels:
        app.kubernetes.io/component: admission-webhook
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
        app.kubernetes.io/version: 1.8.1
      name: ingress-nginx-admission
      namespace: ingress-nginx
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: Role
      name: ingress-nginx-admission
    subjects:
      - kind: ServiceAccount
        name: ingress-nginx-admission
        namespace: ingress-nginx
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRoleBinding
    metadata:
      labels:
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
        app.kubernetes.io/version: 1.8.1
      name: ingress-nginx
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: ClusterRole
      name: ingress-nginx
    subjects:
      - kind: ServiceAccount
        name: ingress-nginx
        namespace: ingress-nginx
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRoleBinding
    metadata:
      labels:
        app.kubernetes.io/component: admission-webhook
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
        app.kubernetes.io/version: 1.8.1
      name: ingress-nginx-admission
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: ClusterRole
      name: ingress-nginx-admission
    subjects:
      - kind: ServiceAccount
        name: ingress-nginx-admission
        namespace: ingress-nginx
    ---
    apiVersion: v1
    data:
      allow-snippet-annotations: "true"
    kind: ConfigMap
    metadata:
      labels:
        app.kubernetes.io/component: controller
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
        app.kubernetes.io/version: 1.8.1
      name: ingress-nginx-controller
      namespace: ingress-nginx
    ---
    apiVersion: v1
    kind: Service
    metadata:
      labels:
        app.kubernetes.io/component: controller
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
        app.kubernetes.io/version: 1.8.1
      name: ingress-nginx-controller
      namespace: ingress-nginx
    spec:
      # externalTrafficPolicy: Local
      ipFamilies:
        - IPv4
      ipFamilyPolicy: SingleStack
      ports:
        - appProtocol: http
          name: http
          port: 80
          protocol: TCP
          targetPort: http
        - appProtocol: https
          name: https
          port: 443
          protocol: TCP
          targetPort: https
      selector:
        app.kubernetes.io/component: controller
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/name: ingress-nginx
      type: LoadBalancer
    ---
    apiVersion: v1
    kind: Service
    metadata:
      labels:
        app.kubernetes.io/component: controller
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
        app.kubernetes.io/version: 1.8.1
      name: ingress-nginx-controller-admission
      namespace: ingress-nginx
    spec:
      ports:
        - appProtocol: https
          name: https-webhook
          port: 443
          targetPort: webhook
      selector:
        app.kubernetes.io/component: controller
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/name: ingress-nginx
      type: ClusterIP
    ---
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      labels:
        app.kubernetes.io/component: controller
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
        app.kubernetes.io/version: 1.8.1
      name: ingress-nginx-controller
      namespace: ingress-nginx
    spec:
      minReadySeconds: 0
      revisionHistoryLimit: 10
      selector:
        matchLabels:
          app.kubernetes.io/component: controller
          app.kubernetes.io/instance: ingress-nginx
          app.kubernetes.io/name: ingress-nginx
      template:
        metadata:
          labels:
            app.kubernetes.io/component: controller
            app.kubernetes.io/instance: ingress-nginx
            app.kubernetes.io/name: ingress-nginx
            app.kubernetes.io/part-of: ingress-nginx
            app.kubernetes.io/version: 1.8.1
        spec:
          hostNetwork: true
          containers:
            - args:
                - /nginx-ingress-controller
                - --publish-service=$(POD_NAMESPACE)/ingress-nginx-controller
                - --election-id=ingress-nginx-leader
                - --controller-class=k8s.io/ingress-nginx
                - --ingress-class=nginx
                - --configmap=$(POD_NAMESPACE)/ingress-nginx-controller
                - --validating-webhook=:8443
                - --validating-webhook-certificate=/usr/local/certificates/cert
                - --validating-webhook-key=/usr/local/certificates/key
              env:
                - name: POD_NAME
                  valueFrom:
                    fieldRef:
                      fieldPath: metadata.name
                - name: POD_NAMESPACE
                  valueFrom:
                    fieldRef:
                      fieldPath: metadata.namespace
                - name: LD_PRELOAD
                  value: /usr/local/lib/libmimalloc.so
              image: registry.cn-hangzhou.aliyuncs.com/google_containers/nginx-ingress-controller:v1.8.1@sha256:e5c4824e7375fcf2a393e1c03c293b69759af37a9ca6abdb91b13d78a93da8bd
              imagePullPolicy: IfNotPresent
              lifecycle:
                preStop:
                  exec:
                    command:
                      - /wait-shutdown
              livenessProbe:
                failureThreshold: 5
                httpGet:
                  path: /healthz
                  port: 10254
                  scheme: HTTP
                initialDelaySeconds: 10
                periodSeconds: 10
                successThreshold: 1
                timeoutSeconds: 1
              name: controller
              ports:
                - containerPort: 80
                  name: http
                  protocol: TCP
                - containerPort: 443
                  name: https
                  protocol: TCP
                - containerPort: 8443
                  name: webhook
                  protocol: TCP
              readinessProbe:
                failureThreshold: 3
                httpGet:
                  path: /healthz
                  port: 10254
                  scheme: HTTP
                initialDelaySeconds: 10
                periodSeconds: 10
                successThreshold: 1
                timeoutSeconds: 1
              resources:
                requests:
                  cpu: 100m
                  memory: 90Mi
              securityContext:
                allowPrivilegeEscalation: true
                capabilities:
                  add:
                    - NET_BIND_SERVICE
                  drop:
                    - ALL
                runAsUser: 101
              volumeMounts:
                - mountPath: /usr/local/certificates/
                  name: webhook-cert
                  readOnly: true
          dnsPolicy: ClusterFirst
          nodeSelector:
            kubernetes.io/os: linux
          serviceAccountName: ingress-nginx
          terminationGracePeriodSeconds: 300
          volumes:
            - name: webhook-cert
              secret:
                secretName: ingress-nginx-admission
    ---
    apiVersion: batch/v1
    kind: Job
    metadata:
      labels:
        app.kubernetes.io/component: admission-webhook
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
        app.kubernetes.io/version: 1.8.1
      name: ingress-nginx-admission-create
      namespace: ingress-nginx
    spec:
      template:
        metadata:
          labels:
            app.kubernetes.io/component: admission-webhook
            app.kubernetes.io/instance: ingress-nginx
            app.kubernetes.io/name: ingress-nginx
            app.kubernetes.io/part-of: ingress-nginx
            app.kubernetes.io/version: 1.8.1
          name: ingress-nginx-admission-create
        spec:
          containers:
            - args:
                - create
                - --host=ingress-nginx-controller-admission,ingress-nginx-controller-admission.$(POD_NAMESPACE).svc
                - --namespace=$(POD_NAMESPACE)
                - --secret-name=ingress-nginx-admission
              env:
                - name: POD_NAMESPACE
                  valueFrom:
                    fieldRef:
                      fieldPath: metadata.namespace
              image: registry.cn-hangzhou.aliyuncs.com/google_containers/kube-webhook-certgen:v20230407@sha256:543c40fd093964bc9ab509d3e791f9989963021f1e9e4c9c7b6700b02bfb227b
              imagePullPolicy: IfNotPresent
              name: create
              securityContext:
                allowPrivilegeEscalation: false
          nodeSelector:
            kubernetes.io/os: linux
          restartPolicy: OnFailure
          securityContext:
            fsGroup: 2000
            runAsNonRoot: true
            runAsUser: 2000
          serviceAccountName: ingress-nginx-admission
    ---
    apiVersion: batch/v1
    kind: Job
    metadata:
      labels:
        app.kubernetes.io/component: admission-webhook
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
        app.kubernetes.io/version: 1.8.1
      name: ingress-nginx-admission-patch
      namespace: ingress-nginx
    spec:
      template:
        metadata:
          labels:
            app.kubernetes.io/component: admission-webhook
            app.kubernetes.io/instance: ingress-nginx
            app.kubernetes.io/name: ingress-nginx
            app.kubernetes.io/part-of: ingress-nginx
            app.kubernetes.io/version: 1.8.1
          name: ingress-nginx-admission-patch
        spec:
          containers:
            - args:
                - patch
                - --webhook-name=ingress-nginx-admission
                - --namespace=$(POD_NAMESPACE)
                - --patch-mutating=false
                - --secret-name=ingress-nginx-admission
                - --patch-failure-policy=Fail
              env:
                - name: POD_NAMESPACE
                  valueFrom:
                    fieldRef:
                      fieldPath: metadata.namespace
              image: registry.cn-hangzhou.aliyuncs.com/google_containers/kube-webhook-certgen:v20230407@sha256:543c40fd093964bc9ab509d3e791f9989963021f1e9e4c9c7b6700b02bfb227b
              imagePullPolicy: IfNotPresent
              name: patch
              securityContext:
                allowPrivilegeEscalation: false
          nodeSelector:
            kubernetes.io/os: linux
          restartPolicy: OnFailure
          securityContext:
            fsGroup: 2000
            runAsNonRoot: true
            runAsUser: 2000
          serviceAccountName: ingress-nginx-admission
    ---
    apiVersion: networking.k8s.io/v1
    kind: IngressClass
    metadata:
      labels:
        app.kubernetes.io/component: controller
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
        app.kubernetes.io/version: 1.8.1
      name: nginx
    spec:
      controller: k8s.io/ingress-nginx
    ---
    apiVersion: admissionregistration.k8s.io/v1
    kind: ValidatingWebhookConfiguration
    metadata:
      labels:
        app.kubernetes.io/component: admission-webhook
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
        app.kubernetes.io/version: 1.8.1
      name: ingress-nginx-admission
    webhooks:
      - admissionReviewVersions:
          - v1
        clientConfig:
          service:
            name: ingress-nginx-controller-admission
            namespace: ingress-nginx
            path: /networking/v1/ingresses
        failurePolicy: Fail
        matchPolicy: Equivalent
        name: validate.nginx.ingress.kubernetes.io
        rules:
          - apiGroups:
              - networking.k8s.io
            apiVersions:
              - v1
            operations:
              - CREATE
              - UPDATE
            resources:
              - ingresses
        sideEffects: None
    

If you use the aliyuncs_deploy.yml or cn-hangzhou_aliyuncs_deploy.yml, you maybe need to use kubectl delete -A ValidatingWebhookConfiguration ingress-nginx-admission command (and rerun kubectl apply -f xxx) when it occurs following error:

Error from server (InternalError): error when creating "/home/zhongshiang/Project/cloud-native-spring-in-action/Chapter09/09-end/edge-service/k8s/ingress.yml": Internal er
ror occurred: failed calling webhook "validate.nginx.ingress.kubernetes.io": failed to call webhook: Post "https://ingress-nginx-controller-admission.ingress-nginx.svc:443
/networking/v1/ingresses?timeout=10s": tls: failed to verify certificate: x509: certificate is valid for minikubeCA, control-plane.minikube.internal, kubernetes.default.sv
c.cluster.local, kubernetes.default.svc, kubernetes.default, kubernetes, localhost, not ingress-nginx-controller-admission.ingress-nginx.svc

Reference

  1. how to pull the image from China network? ingress-nginx#6335 (comment)
  2. how to pull the image from China network? ingress-nginx#6335 (comment)
  3. https://blog.csdn.net/weixin_38797137/article/details/124251698#commentBox
  4. apply ingress rule error after install ingress-nginx: x509 certificate is not valid ingress-nginx-controller-admission.ingress-nginx.svc  ingress-nginx#5968 (comment) (If you don't remove externalTrafficPolicy: Local, you need this; and then rerun kubectl apply -f deploy.yml command)
  5. https://docs.k0sproject.io/v1.25.2+k0s.0/examples/nginx-ingress/#install-nginx-using-host-network
  6. https://kubernetes.github.io/ingress-nginx/deploy/#webhook-network-access
  7. https://kind.sigs.k8s.io/docs/user/ingress/#ingress-nginx
  8. https://kubernetes.io/docs/tasks/access-application-cluster/ingress-minikube/

@ongiant
Copy link
Author

ongiant commented Sep 12, 2023

update 1:2023-07-10 20:40

I guess that this problem only happens on the ingress-nginx addon, because I can enable other addons maintained by Kubernetes . Especially, metrics-server addon's DEFAULT REGISTRY is same as ingress, both are registry.k8s.io . see :

➜  ~ minikube addons enable default-storageclass -p polar                                                      
💡  default-storageclass is an addon maintained by Kubernetes. For any concerns contact minikube on GitHub.
You can view the list of minikube maintainers at: https://github.com/kubernetes/minikube/blob/master/OWNERS
🌟  The 'default-storageclass' addon is enabled
➜  ~ 
➜  ~ 
➜  ~ 
➜  ~ minikube addons enable metrics-server -p polar                            
💡  metrics-server is an addon maintained by Kubernetes. For any concerns contact minikube on GitHub.
You can view the list of minikube maintainers at: https://github.com/kubernetes/minikube/blob/master/OWNERS
    ▪ Using image registry.k8s.io/metrics-server/metrics-server:v0.6.3
🌟  The 'metrics-server' addon is enabled

Question

I believe that if I have correctly set up the network for my local machine and Minikube, I should not encounter issues with pulling images. Then, I have some questions:

  1. How can I utilize the original official image(e.g. registry.k8s.io/ingress-nginx/controller:v1.8.1@sha256:e5c4824e7375fcf2a393e1c03c293b69759af37a9ca6abdb91b13d78a93da8bd and registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20230407@sha256:543c40fd093964bc9ab509d3e791f9989963021f1e9e4c9c7b6700b02bfb227b) written in the official file (e.g. https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.8.1/deploy/static/provider/cloud/deploy.yaml) by setting up the network?
  2. However, the content I quoted above suggests that this issue seems to only occur with the ingress-nginx plugin, meaning that only the images of this plugin have difficulties being pulled. Could you please ask the relevant developers to investigate and address this problem?

@k8s-triage-robot
Copy link

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue as fresh with /remove-lifecycle stale
  • Close this issue with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jan 28, 2024
@k8s-triage-robot
Copy link

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue as fresh with /remove-lifecycle rotten
  • Close this issue with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

@k8s-ci-robot k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Feb 27, 2024
@k8s-triage-robot
Copy link

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Reopen this issue with /reopen
  • Mark this issue as fresh with /remove-lifecycle rotten
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

@k8s-ci-robot k8s-ci-robot closed this as not planned Won't fix, can't repro, duplicate, stale Mar 28, 2024
@k8s-ci-robot
Copy link
Contributor

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Reopen this issue with /reopen
  • Mark this issue as fresh with /remove-lifecycle rotten
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
addon/ingress kind/support Categorizes issue or PR as a support question. lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed.
Projects
None yet
Development

No branches or pull requests

6 participants
@torenware @k8s-ci-robot @ongiant @spowelljr @k8s-triage-robot and others