[Bug]: Unable to create a Milvus Cluster in K8s

[knowitall12]

Is there an existing issue for this?

• I have searched the existing issues

Environment

- Milvus version: 2.4
- Deployment mode(standalone or cluster): Cluster
- MQ type(rocksmq, pulsar or kafka): Kafka   
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): Kubernetes
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

We have deployed Cert-manager, milvus-operator using the documentation at: https://milvus.io/docs/v2.3.x/install_cluster-milvusoperator.md. Now while deploying the milvus cluster using below script we are getting error.

apiVersion: milvus.io/v1beta1
kind: Milvus
metadata:
  name: milvus-cluster-min
  namespace: milvus-system
  labels:
    app: milvus
spec:
  mode: cluster
  config:
    minio:
      bucketName: milvus-bucket-name
      rootPath: milvus/my-release
      useSSL: true
      useIAM: true
  components: 
    enableRollingUpdate: true
    imageUpdateMode: rollingUpgrade
    proxy:
      replicas: 1
      serviceType: LoadBalancer
    dataNode:
      replicas: 1
    indexNode:
      replicas: 1
    queryNode:
      replicas: 1
    mixCoord:
      replicas: 1
  dependencies:
    etcd:
      inCluster:
        values:
          replicaCount: 1
        deletionPolicy: Delete
        pvcDeletion: true
    storage:
      external: true
      type: S3
      endpoint: s3.us-east-1.amazonaws.com:443
    msgStreamType: kafka
    kafka:
      inCluster:
        values:
          defaultReplicationFactor: 1
          offsetsTopicReplicationFactor: 1
          replicaCount: 1
          zookeeper:
            replicaCount: 1
        deletionPolicy: Delete
        pvcDeletion: true

Error: Error from server (InternalError): error when creating "milvus-cluster.yaml": Internal error occurred: failed calling webhook "mmilvus.kb.io": failed to call webhook: Post "https://milvus-operator-webhook-service.milvus-operator.svc:443/mutate-milvus-io-v1beta1-milvus?timeout=10s": Address is not allowed

Cert-manager and milvus-operator pods are also in running status.

kubectl get pods -n cert-manager
NAME                                       READY   STATUS    RESTARTS   AGE
cert-manager-674d9979f7-9zdkh              1/1     Running   0          5d4h
cert-manager-cainjector-7546cb5c6b-n6rdv   1/1     Running   0          5d4h
cert-manager-webhook-7dff4cddc7-j6fcd      1/1     Running   0          5d4h
kubectl get pods -n milvus-operator
NAME                               READY   STATUS    RESTARTS   AGE
milvus-operator-78b8d956bd-x6gl5   1/1     Running   0          99m

Expected Behavior

Should be able to deploy the cluster.

Steps To Reproduce

No response

Milvus Log

Error from server (InternalError): error when creating "milvus-cluster.yaml": Internal error occurred: failed calling webhook "mmilvus.kb.io": failed to call webhook: Post "https://milvus-operator-webhook-service.milvus-operator.svc:443/mutate-milvus-io-v1beta1-milvus?timeout=10s": Address is not allowed

Anything else?

No response

[yanliang567]

/assign @haorenfsa
please help to take a look
/unassign

[haorenfsa]

Hi @knowitall12, could you also provide me with the log of milvus-operator when the error happened?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.

[xpnuo]

你好@knowitall12，你能提供一下 milvus-operator 发生错误时的日志吗？
I got a same question.
this is milvus-operator of log
I have seen "webhook,enable:false" in the log .
Is it related to this?

version: 1.1.4
milvus-helm version: milvus-4.2.13
{"level":"info","ts":"2024-12-27T02:01:13Z","logger":"controller-runtime.metrics","msg":"Metrics server is starting to listen","addr":":8080"}
{"level":"info","ts":"2024-12-27T02:01:13Z","logger":"controller","msg":"enable webhook","enable":false}
{"level":"info","ts":"2024-12-27T02:01:13Z","logger":"setup","msg":"starting manager","version":"1.1.4","milvus-helm version":"milvus-4.2.13"}
{"level":"info","ts":"2024-12-27T02:01:13Z","msg":"Starting server","path":"/metrics","kind":"metrics","addr":"[::]:8080"}
{"level":"info","ts":"2024-12-27T02:01:13Z","msg":"Starting server","kind":"health probe","addr":"[::]:8081"}
I1227 02:01:13.068729 1 leaderelection.go:248] attempting to acquire leader lease milvus-operator/71808ec5.milvus.io...
I1227 02:01:31.083952 1 leaderelection.go:258] successfully acquired lease milvus-operator/71808ec5.milvus.io
{"level":"info","ts":"2024-12-27T02:01:31Z","msg":"Starting EventSource","controller":"milvus","controllerGroup":"milvus.io","controllerKind":"Milvus","source":"kind source: *v1beta1.Milvus"}
{"level":"info","ts":"2024-12-27T02:01:31Z","msg":"Starting Controller","controller":"milvus","controllerGroup":"milvus.io","controllerKind":"Milvus"}
{"level":"info","ts":"2024-12-27T02:01:31Z","msg":"Starting EventSource","controller":"milvuscluster","controllerGroup":"milvus.io","controllerKind":"MilvusCluster","source":"kind source: *v1alpha1.MilvusCluster"}
{"level":"info","ts":"2024-12-27T02:01:31Z","msg":"Starting Controller","controller":"milvuscluster","controllerGroup":"milvus.io","controllerKind":"MilvusCluster"}
{"level":"info","ts":"2024-12-27T02:01:31Z","msg":"Starting EventSource","controller":"milvusupgrade","controllerGroup":"milvus.io","controllerKind":"MilvusUpgrade","source":"kind source: *v1beta1.MilvusUpgrade"}
{"level":"info","ts":"2024-12-27T02:01:31Z","msg":"Starting EventSource","controller":"milvusupgrade","controllerGroup":"milvus.io","controllerKind":"MilvusUpgrade","source":"kind source: *v1.Pod"}
{"level":"info","ts":"2024-12-27T02:01:31Z","msg":"Starting Controller","controller":"milvusupgrade","controllerGroup":"milvus.io","controllerKind":"MilvusUpgrade"}
{"level":"info","ts":"2024-12-27T02:01:32Z","msg":"Starting workers","controller":"milvus","controllerGroup":"milvus.io","controllerKind":"Milvus","worker count":10}
{"level":"info","ts":"2024-12-27T02:01:32Z","msg":"Starting workers","controller":"milvusupgrade","controllerGroup":"milvus.io","controllerKind":"MilvusUpgrade","worker count":1}
{"level":"info","ts":"2024-12-27T02:01:32Z","msg":"Starting workers","controller":"milvuscluster","controllerGroup":"milvus.io","controllerKind":"MilvusCluster","worker count":1}