An exception occurred while deploying Milvus Operator using VNet

[rm-rfjiangxin]

Is there an existing issue for this?

• I have searched the existing issues

Environment

- Milvus version:2.4.x
- Deployment mode(standalone or cluster):no
- MQ type(rocksmq, pulsar or kafka):    no
- SDK version(e.g. pymilvus v2.0.0rc2):no
- OS(Ubuntu or CentOS): ubuntu22.04.04
- CPU/Memory: 4G
- GPU: no
- Others: no

Current Behavior

kubectl apply -f https://raw.githubusercontent.com/zilliztech/milvus-operator/main/deploy/manifests/deployment.yaml

output：
root@k8s-master:/home/jx# kubectl get pods -n milvus-operator
NAME READY STATUS RESTARTS AGE
milvus-operator-545894474-6t8qh 0/1 ContainerCreating 0 17h
milvus-operator-checker-5764r 0/1 ImagePullBackOff 0 17h

Expected Behavior

Why do you still prompt me that I don't have a mirror image while downloading it in advance. And there are also two milvus operator pods

Steps To Reproduce

kubectl  apply -f deployment.yaml



root@k8s-master:/home/jx/Milvus# kubectl get pods -n milvus-operator
NAME                              READY   STATUS              RESTARTS   AGE
milvus-operator-545894474-6t8qh   0/1     ContainerCreating   0          17h
milvus-operator-checker-5764r     0/1     ImagePullBackOff    0          17h

Milvus Log

No response

Anything else?

No response

[rm-rfjiangxin]

Supplement k8s version to 1.28.2

[rm-rfjiangxin]

root@k8s-master:/home/jx/Milvus# kubectl get sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
default-storageclass k8s.io/minikube-hostpath Delete Immediate false 19h

root@k8s-master:/home/jx/Milvus# kubectl get pods -n cert-manager
NAME READY STATUS RESTARTS AGE
cert-manager-679895f5-6xvxc 1/1 Running 1 (117m ago) 19h
cert-manager-cainjector-b78544fd4-c7tkz 1/1 Running 2 (99m ago) 19h
cert-manager-webhook-cd8bbbd67-8wvjl 1/1 Running 1 (117m ago) 19h

root@k8s-master:/home/jx/Milvus# kubectl get pods -n milvus-operator
NAME READY STATUS RESTARTS AGE
milvus-operator-545894474-6t8qh 0/1 ContainerCreating 0 17h
milvus-operator-checker-5764r 0/1 ImagePullBackOff 0 17h

root@k8s-master:/home/jx/Milvus# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
milvusdb/milvus-operator v1.1.2 57e7ab5dd7b8 6 days ago 179MB
calico/kube-controllers v3.27.3 3e4fd05c0c1c 7 months ago 75.7MB
calico/cni v3.27.3 6527a3558140 7 months ago 196MB
calico/pod2daemon-flexvol v3.27.3 ab5b4f1ca289 7 months ago 15.4MB
calico/node v3.27.3 5c6ffd2b2a1d 7 months ago 338MB
registry.aliyuncs.com/google_containers/kube-apiserver v1.28.2 cdcab12b2dd1 14 months ago 126MB
registry.aliyuncs.com/google_containers/kube-proxy v1.28.2 c120fed2beb8 14 months ago 73.1MB
registry.aliyuncs.com/google_containers/kube-scheduler v1.28.2 7a5d9d67a13f 14 months ago 60.1MB
registry.aliyuncs.com/google_containers/kube-controller-manager v1.28.2 55f13c92defb 14 months ago 122MB
registry.aliyuncs.com/google_containers/etcd 3.5.9-0 73deb9a3f702 18 months ago 294MB
registry.aliyuncs.com/google_containers/coredns v1.10.1 ead0a4a53df8 21 months ago 53.6MB
registry.aliyuncs.com/google_containers/pause 3.9 e6f181688397 2 years ago 744kB
quay.io/jetstack/cert-manager-cainjector v1.5.3 94c91157f1c5 3 years ago 42.2MB

Thank you, sir

[yanliang567]

/assign @haorenfsa
please help to take a look

/unassign

[haorenfsa]

milvus-operator-checker-5764r 0/1 ImagePullBackOff 0 17h

@rm-rfjiangxin It's due to the checker pod failed to pull its image. Could you please try delete the deployment & install it again. And if it still fails, provide the output of kubectl -n milvus-operator describe pod?

[rm-rfjiangxin]

milvus-operator-checker-5764r 0/1 ImagePullBackOff 0 17h

@rm-rfjiangxin这是因为checkerpod 无法拉取其镜像。您可以尝试删除部署并重新安装吗？如果仍然失败，请提供输出kubectl -n milvus-operator describe pod？

Hello sir!
I tried Docker and have already downloaded the image

But during the execution of 'kubectl apply - f YAML', the pod logs still need to be downloaded from the internet, but I am in an internal network environment I am very confused

[rm-rfjiangxin]

milvus-operator-checker-5764r 0/1 ImagePullBackOff 0 17h

@rm-rfjiangxin这是因为checkerpod 无法拉取其镜像。您可以尝试删除部署并重新安装吗？如果仍然失败，请提供输出kubectl -n milvus-operator describe pod？

#37663 (comment)

[haorenfsa]

@rm-rfjiangxin I'm not sure how you configured your kubernetes. But usually kubernetees uses containerd instead of docker, and containerd has a concept called namespace, so kubelet usually uses namespace like k8s.io. You can use command like ctr or crictl to check it.
https://devops.stackexchange.com/questions/18534/why-the-crictl-command-did-not-list-the-images

[rm-rfjiangxin]

@rm-rfjiangxin我不确定你是如何配置 kubernetes 的。但通常 kubernetees 使用 containerd 而不是 docker，而 containerd 有一个称为命名空间的概念，因此 kubelet 通常使用命名空间k8s.io。你可以使用类似ctr或的命令crictl来检查它。https ://devops.stackexchange.com/questions/18534/why-the-crictl-command-did-not-list-the-images

Thank you sir, I will test the namespace imported into k8s.io I used import, but I didn't test importing into the k8s.io namespace

[rm-rfjiangxin]

@rm-rfjiangxin我不确定你是如何配置 kubernetes 的。但通常 kubernetees 使用 containerd 而不是 docker，而 containerd 有一个称为命名空间的概念，因此 kubelet 通常使用命名空间k8s.io。你可以使用类似ctr或的命令crictl来检查它。https ://devops.stackexchange.com/questions/18534/why-the-crictl-command-did-not-list-the-images

Hello, may I ask!
Will Milvus Operator download etcd, Pulsar, and MinIO plugins during initialization?
What should I do if I want to download the dependent image in advance and then execute Milvus Operator
Or do these services run within the pod and do not need to be downloaded separately

[rm-rfjiangxin]

@rm-rfjiangxin我不确定你是如何配置 kubernetes 的。但通常 kubernetees 使用 containerd 而不是 docker，而 containerd 有一个称为命名空间的概念，因此 kubelet 通常使用命名空间k8s.io。你可以使用类似ctr或的命令crictl来检查它。https ://devops.stackexchange.com/questions/18534/why-the-crictl-command-did-not-list-the-images

Hello, I have resolved these issues through offline installation of version 2.3. x
However, we encountered many pitfalls
For example: I installed a default nfs (sc) myself
PVC not starting and other issues
There are also issues with the inability to download and incomplete image packages

thanks sir!

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.