[Bug]: querynode often restart when loading diskann index to local

[wangqia0309]

Is there an existing issue for this?

• I have searched the existing issues

Environment

- Milvus version:2.4.1
- Deployment mode(standalone or cluster):k8s
- MQ type(rocksmq, pulsar or kafka):    pulsar
- SDK version(e.g. pymilvus v2.0.0rc2): 2.4
- OS(Ubuntu or CentOS): ubuntu
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

the querynode pod often restart casually when loading index from minio, all of the querynodes use the same share nfs directory(mounted as local directory) as localStorage to store diskann index.
the total index data is dozens of tb, corrspond with 10 querynode pod with 50 cpu,800gb memory.
the log attach a file when restart occured
please review the error
error.log

Expected Behavior

No response

Steps To Reproduce

No response

Milvus Log

No response

Anything else?

No response

[xiaofan-luan]

Is there an existing issue for this?

• I have searched the existing issues

Environment

- Milvus version:2.4.1
- Deployment mode(standalone or cluster):k8s
- MQ type(rocksmq, pulsar or kafka):    pulsar
- SDK version(e.g. pymilvus v2.0.0rc2): 2.4
- OS(Ubuntu or CentOS): ubuntu
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

the querynode pod often restart casually when loading index from minio, all of the querynodes use the same share nfs directory(mounted as local directory) as localStorage to store diskann index. the total index data is dozens of tb, corrspond with 10 querynode pod with 50 cpu,800gb memory. the log attach a file when restart occured please review the error error.log

Expected Behavior

No response

Steps To Reproduce

No response

Milvus Log

No response

Anything else?

No response

you can not use nfs for diskann index. DiskANN requires at least 100K iops and nfs usually not performant enought.

You need local nvme ssd for caching

[wangqia0309]

Is there an existing issue for this?

• I have searched the existing issues

Environment

- Milvus version:2.4.1
- Deployment mode(standalone or cluster):k8s
- MQ type(rocksmq, pulsar or kafka):    pulsar
- SDK version(e.g. pymilvus v2.0.0rc2): 2.4
- OS(Ubuntu or CentOS): ubuntu
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

the querynode pod often restart casually when loading index from minio, all of the querynodes use the same share nfs directory(mounted as local directory) as localStorage to store diskann index. the total index data is dozens of tb, corrspond with 10 querynode pod with 50 cpu,800gb memory. the log attach a file when restart occured please review the error error.log

Expected Behavior

No response

Steps To Reproduce

No response

Milvus Log

No response

Anything else?

No response

you can not use nfs for diskann index. DiskANN requires at least 100K iops and nfs usually not performant enought.

You need local nvme ssd for caching

can you review the error log , and explain the relationship between the error and nfs more in detail

[xiaofan-luan]

-- | --
  |   | 2024-07-26 19:03:06 | PC=0x7fe92f73103b m=55 sigcode=18446744073709551610
  |   | 2024-07-26 19:03:06 | SIGABRT: abort
  |   | 2024-07-26 19:03:06 | what(): Error:RemoveDir:No such file or directory
  |   | 2024-07-26 19:03:06 | terminate called after throwing an instance of 'milvus::SegcoreError'

[xiaofan-luan]

you don't have a directory specified

[yanliang567]

/assign @wangqia0309
/unassign

[wangqia0309]

does it means that milvus querynode actually does not support to configure the local storage dir with mounted share-nfs directory?

[xiaofan-luan]

share nfs is just too slow for milvus

[xiaofan-luan]

you can configure it to local storage, but usually we will see nfs/nas has super long latency or reported error when io issues

[rohan-puri]

@xiaofan-luan where does the 100k IOPS requirement comes from?

Can you point me out to resources on how milvus makes use of nvme ssds ? I am exploring milvus's use and optimizations for nvme drives (mainly what optimizations milvus has done for storage?)

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.

[xiaofan-luan]

@xiaofan-luan where does the 100k IOPS requirement comes from?

Can you point me out to resources on how milvus makes use of nvme ssds ? I am exploring milvus's use and optimizations for nvme drives (mainly what optimizations milvus has done for storage?)

We need SSD mainly on querynode(if you use mmap or diskann), and also etcd.