[Bug]: Proxy not healthy on Docker Swarm #25032
Assignees
Labels
Comments
Nik96i
added
kind/bug
yanliang567
added
help wanted
|
I run it multiple times and it doesn't work anymore. |
|
i am not an expert in docker, but usually rootcoord does not turn to ready because it cannot connect to etcd or minio. Could you please double check if they are working well? |
|
I change the docker-compose file to this: the minio container is accessible from the standalone container, I test it with curl inside the standalone like this: HTTP://minio:9000 |
|
/assign @LoveEachDay |
|
Hi again. I created an overlay docker network with the "attachable" option set to true.
now when I run Milvus with docker-compose and use the external network I made before it works fine, but when I run Milvus with the docker stack it doesn't work. Milvus doesn't work on swarm mode. the containers see each other but don't work. it returns a "proxy not healthy" error. Full log: |
|
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. |
same situation. maybe cause each service uses different local ip in swarm mode but milvus has it's own service discovery. |
|
Hi @Nik96i, I also found this issue when running milvus standalone using docker swam. I fixed it by changing the address of indexCoord, queryCoord, rootCoord, and dataCoord from |
|
@farizap Hi, thanks for your reply, but it still not work by following your solution. Could you please drop out a docker-compose.yaml using swarm mode. |
|
I think this is related to: #17191 (auto-closed, not fixed). The issue IMO is the fact that proxy component (maybe others) does not allow you to specify an address for proxy . It is very important to allow people to customize the addresses the app listens on. For convenience, most use IPV4 "0.0.0.0" which binds to all network interface. We also have an issue with this. We can't run milvus in Docker swarm because it randomly picks up an ip address to bind to and in swarm, you can have multiple interfaces. See issue #17191 for code references. |
|
I don't think we ever tried on docker swarm. @LoveEachDay could you help on investigating this issue? |
|
anyone else want to fix this issue is also highly welcomed |
1 task
|
Hi @EeyoreLee, sorry for the late reply. Here my docker-compose I start the service by using this command |
|
Hello, I found some time to look at this again and I believe there is a bug here that affects milvus 2.3.0 as well and manifests Milvus failed to start and it prints out TONS of logs. I included also the network interfaces from inside the container and the listening ports. As you can see from those logs, the app starts and binds to IPV6 on all interfaces to the port 53100 . Could it be that milvus does not work when IPV6 networks are involved ? To include network information use |
|
After some time if finaly failed with error: |
|
I tested version 2.3.1 and it works fine. Does anybody still have this issue? |
|
@Nik96i : Can you share your deployment template ? |
|
Hi @ieugen . First create a swarm overlay network: IP Addr: No. I don't use IPv6, I guess. ? |
|
Perhaps this is the cause ? That would be curios since binding to tcp6 will bind to tcp4 address as well. Not sure I can disable ipv6 on that server. I do think the code that selects the IP address is not the best: The code seems to take into account only ipv4 addresses (See to4 bellow and https://pkg.go.dev/net#IP.To4 ). |
|
@ieugen I don't understand. Milvus is in a swarm service, completely separate from the server. why does it matter if you have IP6 or not? |
|
@Nik96i The issue is that milvus is a unch of servers that need to talk to each other and for some reason they fail to do that in docker swarm networks, in my case. |
|
I did make some progress on this issue and I would like to confirm it. I added healthchecks which I was missing. @Nik96i (or anyone else) : In your swarm deployment, does milvus start right away or it fails for a few number of times and then it starts? In my experience, milvus restarts a couple of times until it gets the IP address that works: The tasks are ~2 minutes apart - the exact time it takes to go through all healthchecks and re-create the container. |
Aha, so the reason why I fail to run milvus on swarm mode is only I limited the retry time to 5 ? |
|
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. |
|
This is not done. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Is there an existing issue for this?
Environment
Current Behavior
When I run this docker-compose with
docker compose up -dit works fine. but when I run it withdocker stack deploy -c docker-compose.yml milvusit's not working. first it gives "Proxy not healthy" then after about 10-15 minutes it works fine.`
version: "3.9"
services:
etcd:
image: quay.io/coreos/etcd:v3.5.5
networks:
- frs-public
environment:
- ETCD_AUTO_COMPACTION_MODE=revision
- ETCD_AUTO_COMPACTION_RETENTION=1000
- ETCD_QUOTA_BACKEND_BYTES=4294967296
- ETCD_SNAPSHOT_COUNT=50000
volumes:
- type: bind
source: /home/frs/milvus/etcd
target: /etcd
command: etcd -advertise-client-urls=http://127.0.0.1:2379 -listen-client-urls http://0.0.0.0:2379 --data-dir /etcd
deploy:
placement:
constraints:
- "node.role==manager"
minio:
image: minio/minio:RELEASE.2023-03-20T20-16-18Z
networks:
- frs-public
volumes:
- type: bind
source: /home/frs/milvus/minio
target: /minio_data
environment:
MINIO_ACCESS_KEY: minioadmin
MINIO_SECRET_KEY: minioadmin
command: minio server /minio_data
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
interval: 30s
timeout: 20s
retries: 3
deploy:
placement:
constraints:
- "node.role==manager"
standalone:
image: milvusdb/milvus:v2.2.10
networks:
- frs-public
command: ["milvus", "run", "standalone"]
environment:
ETCD_ENDPOINTS: etcd:2379
MINIO_ADDRESS: minio:9000
volumes:
- type: bind
source: /home/frs/milvus/milvus
target: /var/lib/milvus
ports:
- "19530:19530"
- "9091:9091"
depends_on:
- etcd
- minio
deploy:
placement:
constraints:
- "node.role==manager"
networks:
frs-public:
`
Expected Behavior
No response
Steps To Reproduce
No response
Milvus Log
Complete log:
https://file.io/vxEe999zcntN
[2023/06/20 09:10:12.537 +00:00] [WARN] [rootcoord/proxy_client_manager.go:239] ["proxy client is empty, GetMetrics will not send to any client"]
[2023/06/20 09:10:12.538 +00:00] [WARN] [client/client.go:98] ["DataCoordClient, not existed in msess "] [key=datacoord] ["len of msess"=0]
[2023/06/20 09:10:12.538 +00:00] [ERROR] [grpcclient/client.go:158] ["failed to get client address"] [error="find no available datacoord, check datacoord state"] [stack="github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).connect\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:158\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).GetGrpcClient\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:131\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).callOnce\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:256\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:312\ngithub.com/milvus-io/milvus/internal/distributed/datacoord/client.(*Client).GetMetrics\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/client/client.go:527\ngithub.com/milvus-io/milvus/internal/rootcoord.(*QuotaCenter).syncMetrics.func2\n\t/go/src/github.com/milvus-io/milvus/internal/rootcoord/quota_center.go:213\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75"]
[2023/06/20 09:10:12.538 +00:00] [WARN] [grpcclient/client.go:318] ["ClientBase ReCall grpc first call get error"] [role=datacoord] [error="err: find no available datacoord, check datacoord state\n, /go/src/github.com/milvus-io/milvus/internal/util/trace/stack_trace.go:51 github.com/milvus-io/milvus/internal/util/trace.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:317 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/client/client.go:527 github.com/milvus-io/milvus/internal/distributed/datacoord/client.(*Client).GetMetrics\n/go/src/github.com/milvus-io/milvus/internal/rootcoord/quota_center.go:213 github.com/milvus-io/milvus/internal/rootcoord.(*QuotaCenter).syncMetrics.func2\n/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 golang.org/x/sync/errgroup.(*Group).Go.func1\n/usr/local/go/src/runtime/asm_amd64.s:1571 runtime.goexit\n"]
[2023/06/20 09:10:12.538 +00:00] [WARN] [client/client.go:94] ["QueryCoordClient msess key not existed"] [key=querycoord]
[2023/06/20 09:10:12.538 +00:00] [ERROR] [grpcclient/client.go:158] ["failed to get client address"] [error="find no available querycoord, check querycoord state"] [stack="github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).connect\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:158\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).GetGrpcClient\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:131\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).callOnce\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:256\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:312\ngithub.com/milvus-io/milvus/internal/distributed/querycoord/client.(*Client).GetMetrics\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/client/client.go:359\ngithub.com/milvus-io/milvus/internal/rootcoord.(*QuotaCenter).syncMetrics.func1\n\t/go/src/github.com/milvus-io/milvus/internal/rootcoord/quota_center.go:188\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75"]
[2023/06/20 09:10:12.539 +00:00] [WARN] [grpcclient/client.go:318] ["ClientBase ReCall grpc first call get error"] [role=querycoord] [error="err: find no available querycoord, check querycoord state\n, /go/src/github.com/milvus-io/milvus/internal/util/trace/stack_trace.go:51 github.com/milvus-io/milvus/internal/util/trace.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:317 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/client/client.go:359 github.com/milvus-io/milvus/internal/distributed/querycoord/client.(*Client).GetMetrics\n/go/src/github.com/milvus-io/milvus/internal/rootcoord/quota_center.go:188 github.com/milvus-io/milvus/internal/rootcoord.(*QuotaCenter).syncMetrics.func1\n/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 golang.org/x/sync/errgroup.(*Group).Go.func1\n/usr/local/go/src/runtime/asm_amd64.s:1571 runtime.goexit\n"]
[2023/06/20 09:10:12.539 +00:00] [WARN] [client/client.go:98] ["DataCoordClient, not existed in msess "] [key=datacoord] ["len of msess"=0]
[2023/06/20 09:10:12.539 +00:00] [ERROR] [grpcclient/client.go:158] ["failed to get client address"] [error="find no available datacoord, check datacoord state"] [stack="github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).connect\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:158\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).GetGrpcClient\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:131\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).callOnce\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:256\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:327\ngithub.com/milvus-io/milvus/internal/distributed/datacoord/client.(*Client).GetMetrics\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/client/client.go:527\ngithub.com/milvus-io/milvus/internal/rootcoord.(*QuotaCenter).syncMetrics.func2\n\t/go/src/github.com/milvus-io/milvus/internal/rootcoord/quota_center.go:213\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75"]
[2023/06/20 09:10:12.540 +00:00] [ERROR] [grpcclient/client.go:330] ["ClientBase ReCall grpc second call get error"] [role=datacoord] [error="err: find no available datacoord, check datacoord state\n, /go/src/github.com/milvus-io/milvus/internal/util/trace/stack_trace.go:51 github.com/milvus-io/milvus/internal/util/trace.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:329 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/client/client.go:527 github.com/milvus-io/milvus/internal/distributed/datacoord/client.(*Client).GetMetrics\n/go/src/github.com/milvus-io/milvus/internal/rootcoord/quota_center.go:213 github.com/milvus-io/milvus/internal/rootcoord.(*QuotaCenter).syncMetrics.func2\n/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 golang.org/x/sync/errgroup.(*Group).Go.func1\n/usr/local/go/src/runtime/asm_amd64.s:1571 runtime.goexit\n"] [stack="github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:330\ngithub.com/milvus-io/milvus/internal/distributed/datacoord/client.(*Client).GetMetrics\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/client/client.go:527\ngithub.com/milvus-io/milvus/internal/rootcoord.(*QuotaCenter).syncMetrics.func2\n\t/go/src/github.com/milvus-io/milvus/internal/rootcoord/quota_center.go:213\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75"]
[2023/06/20 09:10:12.540 +00:00] [WARN] [client/client.go:94] ["QueryCoordClient msess key not existed"] [key=querycoord]
[2023/06/20 09:10:12.540 +00:00] [ERROR] [grpcclient/client.go:158] ["failed to get client address"] [error="find no available querycoord, check querycoord state"] [stack="github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).connect\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:158\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).GetGrpcClient\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:131\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).callOnce\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:256\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:327\ngithub.com/milvus-io/milvus/internal/distributed/querycoord/client.(*Client).GetMetrics\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/client/client.go:359\ngithub.com/milvus-io/milvus/internal/rootcoord.(*QuotaCenter).syncMetrics.func1\n\t/go/src/github.com/milvus-io/milvus/internal/rootcoord/quota_center.go:188\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75"]
[2023/06/20 09:10:12.540 +00:00] [ERROR] [grpcclient/client.go:330] ["ClientBase ReCall grpc second call get error"] [role=querycoord] [error="err: find no available querycoord, check querycoord state\n, /go/src/github.com/milvus-io/milvus/internal/util/trace/stack_trace.go:51 github.com/milvus-io/milvus/internal/util/trace.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:329 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/client/client.go:359 github.com/milvus-io/milvus/internal/distributed/querycoord/client.(*Client).GetMetrics\n/go/src/github.com/milvus-io/milvus/internal/rootcoord/quota_center.go:188 github.com/milvus-io/milvus/internal/rootcoord.(*QuotaCenter).syncMetrics.func1\n/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 golang.org/x/sync/errgroup.(*Group).Go.func1\n/usr/local/go/src/runtime/asm_amd64.s:1571 runtime.goexit\n"] [stack="github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:330\ngithub.com/milvus-io/milvus/internal/distributed/querycoord/client.(*Client).GetMetrics\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/client/client.go:359\ngithub.com/milvus-io/milvus/internal/rootcoord.(*QuotaCenter).syncMetrics.func1\n\t/go/src/github.com/milvus-io/milvus/internal/rootcoord/quota_center.go:188\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75"]
[2023/06/20 09:10:12.540 +00:00] [WARN] [rootcoord/quota_center.go:143] ["quotaCenter sync metrics failed"] [error="err: find no available datacoord, check datacoord state\n, /go/src/github.com/milvus-io/milvus/internal/util/trace/stack_trace.go:51 github.com/milvus-io/milvus/internal/util/trace.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:329 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/client/client.go:527 github.com/milvus-io/milvus/internal/distributed/datacoord/client.(*Client).GetMetrics\n/go/src/github.com/milvus-io/milvus/internal/rootcoord/quota_center.go:213 github.com/milvus-io/milvus/internal/rootcoord.(*QuotaCenter).syncMetrics.func2\n/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 golang.org/x/sync/errgroup.(*Group).Go.func1\n/usr/local/go/src/runtime/asm_amd64.s:1571 runtime.goexit\n"]
[2023/06/20 09:10:13.083 +00:00] [INFO] [sessionutil/session_util.go:477] ["SessionUtil GetSessions "] [prefix=rootcoord] [key=rootcoord] [address=10.0.0.61:53100]
[2023/06/20 09:10:13.084 +00:00] [INFO] [client/client.go:109] ["RootCoordClient GetSessions success"] [address=10.0.0.61:53100]
[2023/06/20 09:10:13.093 +00:00] [INFO] [sessionutil/session_util.go:477] ["SessionUtil GetSessions "] [prefix=rootcoord] [key=rootcoord] [address=10.0.0.61:53100]
[2023/06/20 09:10:13.093 +00:00] [INFO] [client/client.go:109] ["RootCoordClient GetSessions success"] [address=10.0.0.61:53100]
[2023/06/20 09:10:13.284 +00:00] [WARN] [grpclog/grpclog.go:46] ["[core][Channel #753 SubChannel #754] grpc: addrConn.createTransport failed to connect to {\n "Addr": "10.0.0.61:53100",\n "ServerName": "10.0.0.61:53100",\n "Attributes": null,\n "BalancerAttributes": null,\n "Type": 0,\n "Metadata": null\n}. Err: connection error: desc = "transport: Error while dialing dial tcp 10.0.0.61:53100: i/o timeout""]
[2023/06/20 09:10:13.284 +00:00] [WARN] [grpcclient/client.go:318] ["ClientBase ReCall grpc first call get error"] [role=rootcoord] [error="err: failed to connect 10.0.0.61:53100, reason: context deadline exceeded\n, /go/src/github.com/milvus-io/milvus/internal/util/trace/stack_trace.go:51 github.com/milvus-io/milvus/internal/util/trace.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:317 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:130 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:65 github.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentStates.func1\n/go/src/github.com/milvus-io/milvus/internal/util/retry/retry.go:42 github.com/milvus-io/milvus/internal/util/retry.Do\n/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:89 github.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:104 github.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentHealthy\n/go/src/github.com/milvus-io/milvus/internal/distributed/indexcoord/service.go:148 github.com/milvus-io/milvus/internal/distributed/indexcoord.(*Server).init\n/go/src/github.com/milvus-io/milvus/internal/distributed/indexcoord/service.go:81 github.com/milvus-io/milvus/internal/distributed/indexcoord.(*Server).Run\n/go/src/github.com/milvus-io/milvus/cmd/components/index_coord.go:51 github.com/milvus-io/milvus/cmd/components.(*IndexCoord).Run\n"]
[2023/06/20 09:10:13.285 +00:00] [INFO] [sessionutil/session_util.go:477] ["SessionUtil GetSessions "] [prefix=rootcoord] [key=rootcoord] [address=10.0.0.61:53100]
[2023/06/20 09:10:13.285 +00:00] [INFO] [client/client.go:109] ["RootCoordClient GetSessions success"] [address=10.0.0.61:53100]
[2023/06/20 09:10:13.294 +00:00] [WARN] [grpcclient/client.go:318] ["ClientBase ReCall grpc first call get error"] [role=rootcoord] [error="err: failed to connect 10.0.0.61:53100, reason: context deadline exceeded\n, /go/src/github.com/milvus-io/milvus/internal/util/trace/stack_trace.go:51 github.com/milvus-io/milvus/internal/util/trace.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:317 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:234 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).DescribeCollection\n/go/src/github.com/milvus-io/milvus/internal/datacoord/server.go:974 github.com/milvus-io/milvus/internal/datacoord.(*Server).hasCollection\n/go/src/github.com/milvus-io/milvus/internal/datacoord/handler.go:361 github.com/milvus-io/milvus/internal/datacoord.(*ServerHandler).HasCollection.func1\n/go/src/github.com/milvus-io/milvus/internal/util/retry/retry.go:42 github.com/milvus-io/milvus/internal/util/retry.Do\n/go/src/github.com/milvus-io/milvus/internal/datacoord/handler.go:360 github.com/milvus-io/milvus/internal/datacoord.(*ServerHandler).HasCollection\n/go/src/github.com/milvus-io/milvus/internal/datacoord/handler.go:396 github.com/milvus-io/milvus/internal/datacoord.(*ServerHandler).CheckShouldDropChannel\n/go/src/github.com/milvus-io/milvus/internal/datacoord/channel_manager.go:246 github.com/milvus-io/milvus/internal/datacoord.(*ChannelManager).unwatchDroppedChannels\n/go/src/github.com/milvus-io/milvus/internal/datacoord/channel_manager.go:163 github.com/milvus-io/milvus/internal/datacoord.(*ChannelManager).Startup\n"]
[2023/06/20 09:10:13.294 +00:00] [WARN] [grpclog/grpclog.go:46] ["[core][Channel #755 SubChannel #756] grpc: addrConn.createTransport failed to connect to {\n "Addr": "10.0.0.61:53100",\n "ServerName": "10.0.0.61:53100",\n "Attributes": null,\n "BalancerAttributes": null,\n "Type": 0,\n "Metadata": null\n}. Err: connection error: desc = "transport: Error while dialing dial tcp 10.0.0.61:53100: operation was canceled""]
[2023/06/20 09:10:13.295 +00:00] [INFO] [sessionutil/session_util.go:477] ["SessionUtil GetSessions "] [prefix=rootcoord] [key=rootcoord] [address=10.0.0.61:53100]
[2023/06/20 09:10:13.295 +00:00] [INFO] [client/client.go:109] ["RootCoordClient GetSessions success"] [address=10.0.0.61:53100]
[2023/06/20 09:10:13.486 +00:00] [WARN] [grpclog/grpclog.go:46] ["[core][Channel #757 SubChannel #758] grpc: addrConn.createTransport failed to connect to {\n "Addr": "10.0.0.61:53100",\n "ServerName": "10.0.0.61:53100",\n "Attributes": null,\n "BalancerAttributes": null,\n "Type": 0,\n "Metadata": null\n}. Err: connection error: desc = "transport: Error while dialing dial tcp 10.0.0.61:53100: i/o timeout""]
[2023/06/20 09:10:13.486 +00:00] [ERROR] [grpcclient/client.go:330] ["ClientBase ReCall grpc second call get error"] [role=rootcoord] [error="err: failed to connect 10.0.0.61:53100, reason: context deadline exceeded\n, /go/src/github.com/milvus-io/milvus/internal/util/trace/stack_trace.go:51 github.com/milvus-io/milvus/internal/util/trace.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:329 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:130 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:65 github.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentStates.func1\n/go/src/github.com/milvus-io/milvus/internal/util/retry/retry.go:42 github.com/milvus-io/milvus/internal/util/retry.Do\n/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:89 github.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:104 github.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentHealthy\n/go/src/github.com/milvus-io/milvus/internal/distributed/indexcoord/service.go:148 github.com/milvus-io/milvus/internal/distributed/indexcoord.(*Server).init\n/go/src/github.com/milvus-io/milvus/internal/distributed/indexcoord/service.go:81 github.com/milvus-io/milvus/internal/distributed/indexcoord.(*Server).Run\n/go/src/github.com/milvus-io/milvus/cmd/components/index_coord.go:51 github.com/milvus-io/milvus/cmd/components.(*IndexCoord).Run\n"] [stack="github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:330\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:130\ngithub.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentStates.func1\n\t/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:65\ngithub.com/milvus-io/milvus/internal/util/retry.Do\n\t/go/src/github.com/milvus-io/milvus/internal/util/retry/retry.go:42\ngithub.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:89\ngithub.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentHealthy\n\t/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:104\ngithub.com/milvus-io/milvus/internal/distributed/indexcoord.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/indexcoord/service.go:148\ngithub.com/milvus-io/milvus/internal/distributed/indexcoord.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/indexcoord/service.go:81\ngithub.com/milvus-io/milvus/cmd/components.(*IndexCoord).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/index_coord.go:51\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:120"]
[2023/06/20 09:10:13.496 +00:00] [WARN] [grpclog/grpclog.go:46] ["[core][Channel #759 SubChannel #760] grpc: addrConn.createTransport failed to connect to {\n "Addr": "10.0.0.61:53100",\n "ServerName": "10.0.0.61:53100",\n "Attributes": null,\n "BalancerAttributes": null,\n "Type": 0,\n "Metadata": null\n}. Err: connection error: desc = "transport: Error while dialing dial tcp 10.0.0.61:53100: i/o timeout""]
[2023/06/20 09:10:13.496 +00:00] [ERROR] [grpcclient/client.go:330] ["ClientBase ReCall grpc second call get error"] [role=rootcoord] [error="err: failed to connect 10.0.0.61:53100, reason: context deadline exceeded\n, /go/src/github.com/milvus-io/milvus/internal/util/trace/stack_trace.go:51 github.com/milvus-io/milvus/internal/util/trace.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:329 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:234 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).DescribeCollection\n/go/src/github.com/milvus-io/milvus/internal/datacoord/server.go:974 github.com/milvus-io/milvus/internal/datacoord.(*Server).hasCollection\n/go/src/github.com/milvus-io/milvus/internal/datacoord/handler.go:361 github.com/milvus-io/milvus/internal/datacoord.(*ServerHandler).HasCollection.func1\n/go/src/github.com/milvus-io/milvus/internal/util/retry/retry.go:42 github.com/milvus-io/milvus/internal/util/retry.Do\n/go/src/github.com/milvus-io/milvus/internal/datacoord/handler.go:360 github.com/milvus-io/milvus/internal/datacoord.(*ServerHandler).HasCollection\n/go/src/github.com/milvus-io/milvus/internal/datacoord/handler.go:396 github.com/milvus-io/milvus/internal/datacoord.(*ServerHandler).CheckShouldDropChannel\n/go/src/github.com/milvus-io/milvus/internal/datacoord/channel_manager.go:246 github.com/milvus-io/milvus/internal/datacoord.(*ChannelManager).unwatchDroppedChannels\n/go/src/github.com/milvus-io/milvus/internal/datacoord/channel_manager.go:163 github.com/milvus-io/milvus/internal/datacoord.(*ChannelManager).Startup\n"] [stack="github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:330\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).DescribeCollection\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:234\ngithub.com/milvus-io/milvus/internal/datacoord.(*Server).hasCollection\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/server.go:974\ngithub.com/milvus-io/milvus/internal/datacoord.(*ServerHandler).HasCollection.func1\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/handler.go:361\ngithub.com/milvus-io/milvus/internal/util/retry.Do\n\t/go/src/github.com/milvus-io/milvus/internal/util/retry/retry.go:42\ngithub.com/milvus-io/milvus/internal/datacoord.(*ServerHandler).HasCollection\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/handler.go:360\ngithub.com/milvus-io/milvus/internal/datacoord.(*ServerHandler).CheckShouldDropChannel\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/handler.go:396\ngithub.com/milvus-io/milvus/internal/datacoord.(*ChannelManager).unwatchDroppedChannels\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/channel_manager.go:246\ngithub.com/milvus-io/milvus/internal/datacoord.(*ChannelManager).Startup\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/channel_manager.go:163\ngithub.com/milvus-io/milvus/internal/datacoord.(*Cluster).Startup\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/cluster.go:57\ngithub.com/milvus-io/milvus/internal/datacoord.(*Server).initServiceDiscovery\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/server.go:441\ngithub.com/milvus-io/milvus/internal/datacoord.(*Server).initDataCoord\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/server.go:310\ngithub.com/milvus-io/milvus/internal/datacoord.(*Server).Init\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/server.go:283\ngithub.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/service.go:135\ngithub.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/service.go:241\ngithub.com/milvus-io/milvus/cmd/components.(*DataCoord).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/data_coord.go:49\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:120"]
[2023/06/20 09:10:14.070 +00:00] [INFO] [sessionutil/session_util.go:477] ["SessionUtil GetSessions "] [prefix=rootcoord] [key=rootcoord] [address=10.0.0.61:53100]
[2023/06/20 09:10:14.070 +00:00] [INFO] [client/client.go:109] ["RootCoordClient GetSessions success"] [address=10.0.0.61:53100]
[2023/06/20 09:10:14.164 +00:00] [INFO] [sessionutil/session_util.go:477] ["SessionUtil GetSessions "] [prefix=rootcoord] [key=rootcoord] [address=10.0.0.61:53100]
[2023/06/20 09:10:14.164 +00:00] [INFO] [client/client.go:109] ["RootCoordClient GetSessions success"] [address=10.0.0.61:53100]
[2023/06/20 09:10:14.172 +00:00] [INFO] [sessionutil/session_util.go:477] ["SessionUtil GetSessions "] [prefix=rootcoord] [key=rootcoord] [address=10.0.0.61:53100]
[2023/06/20 09:10:14.172 +00:00] [INFO] [client/client.go:109] ["RootCoordClient GetSessions success"] [address=10.0.0.61:53100]
[2023/06/20 09:10:14.271 +00:00] [WARN] [grpclog/grpclog.go:46] ["[core][Channel #761 SubChannel #762] grpc: addrConn.createTransport failed to connect to {\n "Addr": "10.0.0.61:53100",\n "ServerName": "10.0.0.61:53100",\n "Attributes": null,\n "BalancerAttributes": null,\n "Type": 0,\n "Metadata": null\n}. Err: connection error: desc = "transport: Error while dialing dial tcp 10.0.0.61:53100: i/o timeout""]
[2023/06/20 09:10:14.271 +00:00] [WARN] [grpcclient/client.go:318] ["ClientBase ReCall grpc first call get error"] [role=rootcoord] [error="err: failed to connect 10.0.0.61:53100, reason: context deadline exceeded\n, /go/src/github.com/milvus-io/milvus/internal/util/trace/stack_trace.go:51 github.com/milvus-io/milvus/internal/util/trace.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:317 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:130 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:65 github.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentStates.func1\n/go/src/github.com/milvus-io/milvus/internal/util/retry/retry.go:42 github.com/milvus-io/milvus/internal/util/retry.Do\n/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:89 github.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:104 github.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentHealthy\n/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:387 github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).init\n/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:295 github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).Run\n/go/src/github.com/milvus-io/milvus/cmd/components/proxy.go:51 github.com/milvus-io/milvus/cmd/components.(*Proxy).Run\n"]
[2023/06/20 09:10:14.272 +00:00] [INFO] [sessionutil/session_util.go:477] ["SessionUtil GetSessions "] [prefix=rootcoord] [key=rootcoord] [address=10.0.0.61:53100]
[2023/06/20 09:10:14.272 +00:00] [INFO] [client/client.go:109] ["RootCoordClient GetSessions success"] [address=10.0.0.61:53100]
[2023/06/20 09:10:14.365 +00:00] [WARN] [grpclog/grpclog.go:46] ["[core][Channel #763 SubChannel #764] grpc: addrConn.createTransport failed to connect to {\n "Addr": "10.0.0.61:53100",\n "ServerName": "10.0.0.61:53100",\n "Attributes": null,\n "BalancerAttributes": null,\n "Type": 0,\n "Metadata": null\n}. Err: connection error: desc = "transport: Error while dialing dial tcp 10.0.0.61:53100: i/o timeout""]
[2023/06/20 09:10:14.366 +00:00] [WARN] [grpcclient/client.go:318] ["ClientBase ReCall grpc first call get error"] [role=rootcoord] [error="err: failed to connect 10.0.0.61:53100, reason: context deadline exceeded\n, /go/src/github.com/milvus-io/milvus/internal/util/trace/stack_trace.go:51 github.com/milvus-io/milvus/internal/util/trace.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:317 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:130 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:65 github.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentStates.func1\n/go/src/github.com/milvus-io/milvus/internal/util/retry/retry.go:42 github.com/milvus-io/milvus/internal/util/retry.Do\n/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:89 github.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:104 github.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentHealthy\n/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/service.go:164 github.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).init\n/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/service.go:98 github.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).Run\n/go/src/github.com/milvus-io/milvus/cmd/components/query_coord.go:52 github.com/milvus-io/milvus/cmd/components.(*QueryCoord).Run\n"]
[2023/06/20 09:10:14.367 +00:00] [INFO] [sessionutil/session_util.go:477] ["SessionUtil GetSessions "] [prefix=rootcoord] [key=rootcoord] [address=10.0.0.61:53100]
[2023/06/20 09:10:14.367 +00:00] [INFO] [client/client.go:109] ["RootCoordClient GetSessions success"] [address=10.0.0.61:53100]
[2023/06/20 09:10:14.372 +00:00] [WARN] [grpcclient/client.go:318] ["ClientBase ReCall grpc first call get error"] [role=rootcoord] [error="err: failed to connect 10.0.0.61:53100, reason: context deadline exceeded\n, /go/src/github.com/milvus-io/milvus/internal/util/trace/stack_trace.go:51 github.com/milvus-io/milvus/internal/util/trace.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:317 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:130 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:65 github.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentStates.func1\n/go/src/github.com/milvus-io/milvus/internal/util/retry/retry.go:42 github.com/milvus-io/milvus/internal/util/retry.Do\n/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:89 github.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:104 github.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentHealthy\n/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:275 github.com/milvus-io/milvus/internal/distributed/datanode.(*Server).init\n/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:172 github.com/milvus-io/milvus/internal/distributed/datanode.(*Server).Run\n/go/src/github.com/milvus-io/milvus/cmd/components/data_node.go:52 github.com/milvus-io/milvus/cmd/components.(*DataNode).Run\n"]
[2023/06/20 09:10:14.372 +00:00] [WARN] [grpclog/grpclog.go:46] ["[core][Channel #765 SubChannel #766] grpc: addrConn.createTransport failed to connect to {\n "Addr": "10.0.0.61:53100",\n "ServerName": "10.0.0.61:53100",\n "Attributes": null,\n "BalancerAttributes": null,\n "Type": 0,\n "Metadata": null\n}. Err: connection error: desc = "transport: Error while dialing dial tcp 10.0.0.61:53100: operation was canceled""]
[2023/06/20 09:10:14.373 +00:00] [INFO] [sessionutil/session_util.go:477] ["SessionUtil GetSessions "] [prefix=rootcoord] [key=rootcoord] [address=10.0.0.61:53100]
[2023/06/20 09:10:14.373 +00:00] [INFO] [client/client.go:109] ["RootCoordClient GetSessions success"] [address=10.0.0.61:53100]
[2023/06/20 09:10:14.472 +00:00] [WARN] [grpclog/grpclog.go:46] ["[core][Channel #767 SubChannel #768] grpc: addrConn.createTransport failed to connect to {\n "Addr": "10.0.0.61:53100",\n "ServerName": "10.0.0.61:53100",\n "Attributes": null,\n "BalancerAttributes": null,\n "Type": 0,\n "Metadata": null\n}. Err: connection error: desc = "transport: Error while dialing dial tcp 10.0.0.61:53100: operation was canceled""]
[2023/06/20 09:10:14.472 +00:00] [ERROR] [grpcclient/client.go:330] ["ClientBase ReCall grpc second call get error"] [role=rootcoord] [error="err: failed to connect 10.0.0.61:53100, reason: context deadline exceeded\n, /go/src/github.com/milvus-io/milvus/internal/util/trace/stack_trace.go:51 github.com/milvus-io/milvus/internal/util/trace.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:329 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:130 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:65 github.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentStates.func1\n/go/src/github.com/milvus-io/milvus/internal/util/retry/retry.go:42 github.com/milvus-io/milvus/internal/util/retry.Do\n/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:89 github.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:104 github.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentHealthy\n/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:387 github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).init\n/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:295 github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).Run\n/go/src/github.com/milvus-io/milvus/cmd/components/proxy.go:51 github.com/milvus-io/milvus/cmd/components.(*Proxy).Run\n"] [stack="github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:330\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:130\ngithub.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentStates.func1\n\t/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:65\ngithub.com/milvus-io/milvus/internal/util/retry.Do\n\t/go/src/github.com/milvus-io/milvus/internal/util/retry/retry.go:42\ngithub.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:89\ngithub.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentHealthy\n\t/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:104\ngithub.com/milvus-io/milvus/internal/distributed/proxy.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:387\ngithub.com/milvus-io/milvus/internal/distributed/proxy.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:295\ngithub.com/milvus-io/milvus/cmd/components.(*Proxy).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/proxy.go:51\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:120"]
[2023/06/20 09:10:14.568 +00:00] [WARN] [grpclog/grpclog.go:46] ["[core][Channel #769 SubChannel #770] grpc: addrConn.createTransport failed to connect to {\n "Addr": "10.0.0.61:53100",\n "ServerName": "10.0.0.61:53100",\n "Attributes": null,\n "BalancerAttributes": null,\n "Type": 0,\n "Metadata": null\n}. Err: connection error: desc = "transport: Error while dialing dial tcp 10.0.0.61:53100: i/o timeout""]
[2023/06/20 09:10:14.568 +00:00] [ERROR] [grpcclient/client.go:330] ["ClientBase ReCall grpc second call get error"] [role=rootcoord] [error="err: failed to connect 10.0.0.61:53100, reason: context deadline exceeded\n, /go/src/github.com/milvus-io/milvus/internal/util/trace/stack_trace.go:51 github.com/milvus-io/milvus/internal/util/trace.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:329 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:130 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:65 github.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentStates.func1\n/go/src/github.com/milvus-io/milvus/internal/util/retry/retry.go:42 github.com/milvus-io/milvus/internal/util/retry.Do\n/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:89 github.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:104 github.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentHealthy\n/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/service.go:164 github.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).init\n/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/service.go:98 github.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).Run\n/go/src/github.com/milvus-io/milvus/cmd/components/query_coord.go:52 github.com/milvus-io/milvus/cmd/components.(*QueryCoord).Run\n"] [stack="github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:330\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:130\ngithub.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentStates.func1\n\t/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:65\ngithub.com/milvus-io/milvus/internal/util/retry.Do\n\t/go/src/github.com/milvus-io/milvus/internal/util/retry/retry.go:42\ngithub.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:89\ngithub.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentHealthy\n\t/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:104\ngithub.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/service.go:164\ngithub.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/service.go:98\ngithub.com/milvus-io/milvus/cmd/components.(*QueryCoord).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/query_coord.go:52\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:120"]
[2023/06/20 09:10:14.574 +00:00] [WARN] [grpclog/grpclog.go:46] ["[core][Channel #771 SubChannel #772] grpc: addrConn.createTransport failed to connect to {\n "Addr": "10.0.0.61:53100",\n "ServerName": "10.0.0.61:53100",\n "Attributes": null,\n "BalancerAttributes": null,\n "Type": 0,\n "Metadata": null\n}. Err: connection error: desc = "transport: Error while dialing dial tcp 10.0.0.61:53100: i/o timeout""]
[2023/06/20 09:10:14.574 +00:00] [ERROR] [grpcclient/client.go:330] ["ClientBase ReCall grpc second call get error"] [role=rootcoord] [error="err: failed to connect 10.0.0.61:53100, reason: context deadline exceeded\n, /go/src/github.com/milvus-io/milvus/internal/util/trace/stack_trace.go:51 github.com/milvus-io/milvus/internal/util/trace.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:329 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:130 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:65 github.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentStates.func1\n/go/src/github.com/milvus-io/milvus/internal/util/retry/retry.go:42 github.com/milvus-io/milvus/internal/util/retry.Do\n/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:89 github.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:104 github.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentHealthy\n/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:275 github.com/milvus-io/milvus/internal/distributed/datanode.(*Server).init\n/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:172 github.com/milvus-io/milvus/internal/distributed/datanode.(*Server).Run\n/go/src/github.com/milvus-io/milvus/cmd/components/data_node.go:52 github.com/milvus-io/milvus/cmd/components.(*DataNode).Run\n"] [stack="github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:330\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:130\ngithub.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentStates.func1\n\t/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:65\ngithub.com/milvus-io/milvus/internal/util/retry.Do\n\t/go/src/github.com/milvus-io/milvus/internal/util/retry/retry.go:42\ngithub.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:89\ngithub.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentHealthy\n\t/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:104\ngithub.com/milvus-io/milvus/internal/distributed/datanode.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:275\ngithub.com/milvus-io/milvus/internal/distributed/datanode.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:172\ngithub.com/milvus-io/milvus/cmd/components.(*DataNode).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/data_node.go:52\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:120"]
[2023/06/20 09:10:15.537 +00:00] [WARN] [rootcoord/proxy_client_manager.go:239] ["proxy client is empty, GetMetrics will not send to any client"]
[2023/06/20 09:10:15.539 +00:00] [WARN] [client/client.go:94] ["QueryCoordClient msess key not existed"] [key=querycoord]
[2023/06/20 09:10:15.539 +00:00] [WARN] [client/client.go:98] ["DataCoordClient, not existed in msess "] [key=datacoord] ["len of msess"=0]
[2023/06/20 09:10:15.539 +00:00] [ERROR] [grpcclient/client.go:158] ["failed to get client address"] [error="find no available querycoord, check querycoord state"] [stack="github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).connect\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:158\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).GetGrpcClient\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:131\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).callOnce\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:256\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:312\ngithub.com/milvus-io/milvus/internal/distributed/querycoord/client.(*Client).GetMetrics\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/client/client.go:359\ngithub.com/milvus-io/milvus/internal/rootcoord.(*QuotaCenter).syncMetrics.func1\n\t/go/src/github.com/milvus-io/milvus/internal/rootcoord/quota_center.go:188\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75"]
[2023/06/20 09:10:15.539 +00:00] [ERROR] [grpcclient/client.go:158] ["failed to get client address"] [error="find no available datacoord, check datacoord state"] [stack="github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).connect\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:158\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).GetGrpcClient\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:131\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).callOnce\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:256\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:312\ngithub.com/milvus-io/milvus/internal/distributed/datacoord/client.(*Client).GetMetrics\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/client/client.go:527\ngithub.com/milvus-io/milvus/internal/rootcoord.(*QuotaCenter).syncMetrics.func2\n\t/go/src/github.com/milvus-io/milvus/internal/rootcoord/quota_center.go:213\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75"]
[2023/06/20 09:10:15.539 +00:00] [WARN] [grpcclient/client.go:318] ["ClientBase ReCall grpc first call get error"] [role=querycoord] [error="err: find no available querycoord, check querycoord state\n, /go/src/github.com/milvus-io/milvus/internal/util/trace/stack_trace.go:51 github.com/milvus-io/milvus/internal/util/trace.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:317 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/client/client.go:359 github.com/milvus-io/milvus/internal/distributed/querycoord/client.(*Client).GetMetrics\n/go/src/github.com/milvus-io/milvus/internal/rootcoord/quota_center.go:188 github.com/milvus-io/milvus/internal/rootcoord.(*QuotaCenter).syncMetrics.func1\n/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 golang.org/x/sync/errgroup.(*Group).Go.func1\n/usr/local/go/src/runtime/asm_amd64.s:1571 runtime.goexit\n"]
[2023/06/20 09:10:15.539 +00:00] [WARN] [grpcclient/client.go:318] ["ClientBase ReCall grpc first call get error"] [role=datacoord] [error="err: find no available datacoord, check datacoord state\n, /go/src/github.com/milvus-io/milvus/internal/util/trace/stack_trace.go:51 github.com/milvus-io/milvus/internal/util/trace.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:317 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/client/client.go:527 github.com/milvus-io/milvus/internal/distributed/datacoord/client.(*Client).GetMetrics\n/go/src/github.com/milvus-io/milvus/internal/rootcoord/quota_center.go:213 github.com/milvus-io/milvus/internal/rootcoord.(*QuotaCenter).syncMetrics.func2\n/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 golang.org/x/sync/errgroup.(*Group).Go.func1\n/usr/local/go/src/runtime/asm_amd64.s:1571 runtime.goexit\n"]
[2023/06/20 09:10:15.540 +00:00] [WARN] [client/client.go:98] ["DataCoordClient, not existed in msess "] [key=datacoord] ["len of msess"=0]
[2023/06/20 09:10:15.540 +00:00] [WARN] [client/client.go:94] ["QueryCoordClient msess key not existed"] [key=querycoord]
[2023/06/20 09:10:15.540 +00:00] [ERROR] [grpcclient/client.go:158] ["failed to get client address"] [error="find no available datacoord, check datacoord state"] [stack="github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).connect\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:158\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).GetGrpcClient\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:131\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).callOnce\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:256\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:327\ngithub.com/milvus-io/milvus/internal/distributed/datacoord/client.(*Client).GetMetrics\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/client/client.go:527\ngithub.com/milvus-io/milvus/internal/rootcoord.(*QuotaCenter).syncMetrics.func2\n\t/go/src/github.com/milvus-io/milvus/internal/rootcoord/quota_center.go:213\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75"]
[2023/06/20 09:10:15.540 +00:00] [ERROR] [grpcclient/client.go:158] ["failed to get client address"] [error="find no available querycoord, check querycoord state"] [stack="github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).connect\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:158\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).GetGrpcClient\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:131\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).callOnce\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:256\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:327\ngithub.com/milvus-io/milvus/internal/distributed/querycoord/client.(*Client).GetMetrics\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/client/client.go:359\ngithub.com/milvus-io/milvus/internal/rootcoord.(*QuotaCenter).syncMetrics.func1\n\t/go/src/github.com/milvus-io/milvus/internal/rootcoord/quota_center.go:188\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75"]
[2023/06/20 09:10:15.540 +00:00] [ERROR] [grpcclient/client.go:330] ["ClientBase ReCall grpc second call get error"] [role=datacoord] [error="err: find no available datacoord, check datacoord state\n, /go/src/github.com/milvus-io/milvus/internal/util/trace/stack_trace.go:51 github.com/milvus-io/milvus/internal/util/trace.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:329 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/client/client.go:527 github.com/milvus-io/milvus/internal/distributed/datacoord/client.(*Client).GetMetrics\n/go/src/github.com/milvus-io/milvus/internal/rootcoord/quota_center.go:213 github.com/milvus-io/milvus/internal/rootcoord.(*QuotaCenter).syncMetrics.func2\n/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 golang.org/x/sync/errgroup.(*Group).Go.func1\n/usr/local/go/src/runtime/asm_amd64.s:1571 runtime.goexit\n"] [stack="github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:330\ngithub.com/milvus-io/milvus/internal/distributed/datacoord/client.(*Client).GetMetrics\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/client/client.go:527\ngithub.com/milvus-io/milvus/internal/rootcoord.(*QuotaCenter).syncMetrics.func2\n\t/go/src/github.com/milvus-io/milvus/internal/rootcoord/quota_center.go:213\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75"]
[2023/06/20 09:10:15.540 +00:00] [ERROR] [grpcclient/client.go:330] ["ClientBase ReCall grpc second call get error"] [role=querycoord] [error="err: find no available querycoord, check querycoord state\n, /go/src/github.com/milvus-io/milvus/internal/util/trace/stack_trace.go:51 github.com/milvus-io/milvus/internal/util/trace.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:329 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/client/client.go:359 github.com/milvus-io/milvus/internal/distributed/querycoord/client.(*Client).GetMetrics\n/go/src/github.com/milvus-io/milvus/internal/rootcoord/quota_center.go:188 github.com/milvus-io/milvus/internal/rootcoord.(*QuotaCenter).syncMetrics.func1\n/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 golang.org/x/sync/errgroup.(*Group).Go.func1\n/usr/local/go/src/runtime/asm_amd64.s:1571 runtime.goexit\n"] [stack="github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:330\ngithub.com/milvus-io/milvus/internal/distributed/querycoord/client.(*Client).GetMetrics\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/client/client.go:359\ngithub.com/milvus-io/milvus/internal/rootcoord.(*QuotaCenter).syncMetrics.func1\n\t/go/src/github.com/milvus-io/milvus/internal/rootcoord/quota_center.go:188\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75"]
[2023/06/20 09:10:15.540 +00:00] [WARN] [rootcoord/quota_center.go:143] ["quotaCenter sync metrics failed"] [error="err: find no available datacoord, check datacoord state\n, /go/src/github.com/milvus-io/milvus/internal/util/trace/stack_trace.go:51 github.com/milvus-io/milvus/internal/util/trace.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:329 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/client/client.go:527 github.com/milvus-io/milvus/internal/distributed/datacoord/client.(*Client).GetMetrics\n/go/src/github.com/milvus-io/milvus/internal/rootcoord/quota_center.go:213 github.com/milvus-io/milvus/internal/rootcoord.(*QuotaCenter).syncMetrics.func2\n/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 golang.org/x/sync/errgroup.(*Group).Go.func1\n/usr/local/go/src/runtime/asm_amd64.s:1571 runtime.goexit\n"]
Anything else?
No response
The text was updated successfully, but these errors were encountered: