[Bug]: Milvus fails connecting to server, timeout

[sivanola]

Is there an existing issue for this?

• I have searched the existing issues

Environment

- Milvus version: 2.2.3 and 2.2.4
- Deployment mode(standalone or cluster): standalone
- MQ type(rocksmq, pulsar or kafka):    
- SDK version(e.g. pymilvus v2.0.0rc2): pymilvus v2.2.2
- OS(Ubuntu or CentOS): Mac OS M1
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

Milvus doesn't connect to server. Everything worked two days ago, yesterday it wouldn't start. I tried updating it (from v. 2.2.3 to 2.2.4), which didn't help. I restarted my computer, Milvus started, but when I tried creating a collection I kept getting this error: Error: <MilvusException: (code=2, message=Fail connecting to server on 127.0.0.1:19530. Timeout)> or Error: <MilvusException: (code=2, message=Fail connecting to server on localhost:19530. Timeout)>

After running netstat -an | grep 19530:
tcp4 0 0 *.19530 . LISTEN

Expected Behavior

I get the error when running the following:
from pymilvus import connections
connections.connect("default", host="localhost", port="19530")
or
connections.connect("default", host="127.0.0.1", port="19530")

Steps To Reproduce

No response

Milvus Log

milvus.log

Anything else?

No response

[yanliang567]

not see any connection requests in the attached logs, @czs007 could you please double check it?
/assign @czs007
/unassign

[christy]

I am also seeing this today. First time trying to connect.

• Milvus version: 2.2.4
• Deployment mode(standalone or cluster): standalone
• MQ type(rocksmq, pulsar or kafka):
• SDK version(e.g. pymilvus v2.0.0rc2): pymilvus v2.2.4
• OS(Ubuntu or CentOS): Mac OS M1
• CPU/Memory:
• GPU:
• Others: Python 3.8.15, grpc 1.47.5

Steps to reproduce:

1. git clone[email protected]:zilliztech/cloud-vectordb-examples.git
2. pip3 install pymilvus
3. cd cloud-vectordb-examples
   cd python
4. On Databases > click Name, mine was "Test" > scroll down to see Summary > click "Network Address" > click "+ IP Address" > click "Use My Current IP Address"
5. Replace the endpoint URI and the database authentication credentials in the configuration file (config.ini).
6. python3 hello_zilliz_vectordb.py

`Traceback (most recent call last):
File "/Users/christy/miniconda3/envs/py38/lib/python3.8/site-packages/pymilvus/client/grpc_handler.py", line 114, in _wait_for_channel_ready
grpc.channel_ready_future(self._channel).result(timeout=timeout)
File "/Users/christy/miniconda3/envs/py38/lib/python3.8/site-packages/grpc/_utilities.py", line 139, in result
self._block(timeout)
File "/Users/christy/miniconda3/envs/py38/lib/python3.8/site-packages/grpc/_utilities.py", line 85, in _block
raise grpc.FutureTimeoutError()
grpc.FutureTimeoutError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "hello_zilliz_vectordb.py", line 16, in
connections.connect("default",
File "/Users/christy/miniconda3/envs/py38/lib/python3.8/site-packages/pymilvus/orm/connections.py", line 303, in connect
connect_milvus(**kwargs, user=user, password=password)
File "/Users/christy/miniconda3/envs/py38/lib/python3.8/site-packages/pymilvus/orm/connections.py", line 274, in connect_milvus
gh._wait_for_channel_ready(timeout=timeout)
File "/Users/christy/miniconda3/envs/py38/lib/python3.8/site-packages/pymilvus/client/grpc_handler.py", line 117, in _wait_for_channel_ready
raise MilvusException(Status.CONNECT_FAILED,
pymilvus.exceptions.MilvusException: <MilvusException: (code=2, message=Fail connecting to server on in01-1225359410ed01c.aws-us-west-2.vectordb.zillizcloud.com:19531. Timeout)>`

[yanliang567]

@christy sounds like you are running on zilliz cloud, please try to add more timeout in connect() method. if it does not help, please try to contact cloud support for help.

[christy]

Thanks, I did that, contacted [email protected].

What finally worked for me was just to move to a different WiFi connection. Mine was behind a VPN, so I was not getting the correct IP Address to add to the white list.

[ToBlueSky]

Same issue:

----> 5 connections.connect(host="localhost", port="19530")
...
--> 123 raise MilvusException(Status.CONNECT_FAILED,
124 f'Fail connecting to server on {self._address}. Timeout') from e
126 raise MilvusException(Status.CONNECT_FAILED, 'No channel in handler, please setup grpc channel first')

I have checked the server, it is OK, below:
355e8efedbac milvusdb/milvus:v2.0.2 "/tini -- milvus run…" 22 hours ago Up 16 minutes 0.0.0.0:19530->19530/tcp, :::19530->19530/tcp milvus-standalone
f59e78670cc8 minio/minio:RELEASE.2020-12-03T00-03-10Z "/usr/bin/docker-ent…" 22 hours ago Up 16 minutes (healthy) 9000/tcp milvus-minio
033d9e8d7779 quay.io/coreos/etcd:v3.5.0 "etcd -advertise-cli…" 22 hours ago Up 16 minutes 2379-2380/tcp milvus-etcd

[xiaofan-luan]

Same issue:

----> 5 connections.connect(host="localhost", port="19530") ... --> 123 raise MilvusException(Status.CONNECT_FAILED, 124 f'Fail connecting to server on {self._address}. Timeout') from e 126 raise MilvusException(Status.CONNECT_FAILED, 'No channel in handler, please setup grpc channel first')

I have checked the server, it is OK, below: 355e8efedbac milvusdb/milvus:v2.0.2 "/tini -- milvus run…" 22 hours ago Up 16 minutes 0.0.0.0:19530->19530/tcp, :::19530->19530/tcp milvus-standalone f59e78670cc8 minio/minio:RELEASE.2020-12-03T00-03-10Z "/usr/bin/docker-ent…" 22 hours ago Up 16 minutes (healthy) 9000/tcp milvus-minio 033d9e8d7779 quay.io/coreos/etcd:v3.5.0 "etcd -advertise-cli…" 22 hours ago Up 16 minutes 2379-2380/tcp milvus-etcd

you should try to ping 19530 see if it can be connected.
if you are using docker compose, this is what configs you are looking for
# expose 19530 port so we can directly access milvus inside build container
# ports:
# - "19530:19530"

[ToBlueSky]

The root cause is I forgot to configure "no proxy 127.0.0.1 " in network proxy

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.

[Devendra748]

My code is working fine yesterday evening but when I open it today it gives me "[WARNING] PyMilvus:
<class 'list'>
/home/h/.local/lib/python3.10/site-packages/langchain_core/prompts/prompt.py:200: DeprecationWarning: `input_variables' is deprecated and ignored.
warnings.warn(
1
Failed to create new connection using: 5b84859b5fc04430830c796fded4e1ac
Traceback (most recent call last):
File "/home/h/Documents/relic-Gen-ai/genai-madgical/milvus-vector-db/genai/load_csv.py", line 84, in
update_weviate_with_csv("/home/h/Documents/relic-Gen-ai/genai-madgical/milvus-vector-db/genai/ramayan_data/c_rajagopal copy.csv","Ram_bot_one_3",Milvus(MILVUS_SERVER_URL))
File "/home/h/Documents/relic-Gen-ai/genai-madgical/milvus-vector-db/genai/load_csv.py", line 56, in update_weviate_with_csv
vector_store = MilvusVectorStore(
File "/home/h/.local/lib/python3.10/site-packages/llama_index/vector_stores/milvus.py", line 123, in init
self.milvusclient = MilvusClient(
File "/home/h/.local/lib/python3.10/site-packages/pymilvus/milvus_client/milvus_client.py", line 59, in init
self._using = self._create_connection(
File "/home/h/.local/lib/python3.10/site-packages/pymilvus/milvus_client/milvus_client.py", line 551, in _create_connection
raise ex from ex
File "/home/h/.local/lib/python3.10/site-packages/pymilvus/milvus_client/milvus_client.py", line 548, in _create_connection
connections.connect(using, user, password, db_name, token, uri=uri, **kwargs)
File "/home/h/.local/lib/python3.10/site-packages/pymilvus/orm/connections.py", line 414, in connect
connect_milvus(**kwargs, user=user, password=password, token=token, db_name=db_name)
File "/home/h/.local/lib/python3.10/site-packages/pymilvus/orm/connections.py", line 365, in connect_milvus
gh._wait_for_channel_ready(timeout=timeout)
File "/home/h/.local/lib/python3.10/site-packages/pymilvus/client/grpc_handler.py", line 146, in _wait_for_channel_ready
raise MilvusException(
pymilvus.exceptions.MilvusException: <MilvusException: (code=2, message=Fail connecting to server on localhost:19530. Timeout)>" error

[afzalIbnSH]

What helped me was use Google's public DNS servers