[Feat]: Milvus cannot connect to kafka&minio with ssl

[chenraoCR]

Is there an existing issue for this?

• I have searched the existing issues

Environment

- Milvus version: latest
- Deployment mode(standalone or cluster): standalone
- MQ type(rocksmq, pulsar or kafka): kafka with ssl
- Metadata storage: etcd with ssl
- S3: minio with ssl

Current Behavior

milvus cannot connect external services kafka & minio which configued with ssl

Expected Behavior

milvus could connect kafka & minio with ssl

Steps To Reproduce

1. Startup external services etcd, minio and kafka with ssl 
2. Configure milvus

.yaml
etcd:
  endpoints: [external-etcd-address]
  ......
  ssl:
    enabled: false # Whether to support ETCD secure connection mode
    tlsCert: /path/to/etcd-client.pem # path to your cert file
    tlsKey: /path/to/etcd-client-key.pem # path to your key file
    tlsCACert: /path/to/ca.pem # path to your CACert file
    # TLS min version
    # Optional values: 1.0, 1.1, 1.2, 1.3。
    # We recommend using version 1.2 and above
    tlsMinVersion: 1.3
  ......
......
minio:
  address: external-minio-address
  ......
  useSSL: true
  ......
......
kafka:
  brokerList: [external-kafka-brokers-address]
  ......
  securityProtocol: SSL
  ......
......
3. startup milvus, cannot connect to kafka & minio, ssl handshake will fail

Milvus Log

No response

Anything else?

should handle kafka & minio ssl just like etcd

[chenraoCR]

donot know why in milvus.yaml, unlike ectd, can enable kafka & minio ssl, but cannot configure kafka&minio ssl certs

[xiaofan-luan]

for kafka you have to use 2.3.1.
I believe S3 can support ssl, did ever try to config?

[chenraoCR]

@xiaofan-luan yes, but in the milvus.yaml, can only config minio.useSSL to true, but how to config the certs path? when just set minio.useSSL to true, cannot connect minio, SSL handshake error

[xiaofan-luan]

@xiaofan-luan yes, but in the milvus.yaml, can only config minio.useSSL to true, but how to config the certs path? when just set minio.useSSL to true, cannot connect minio, SSL handshake error

If you already enable minio tls, you should have the cert file already right?

[xiaofan-luan]

https://min.io/docs/minio/linux/operations/network-encryption.html
this document might help you?

[chenraoCR]

@xiaofan-luan yes, but in the milvus.yaml, can only config minio.useSSL to true, but how to config the certs path? when just set minio.useSSL to true, cannot connect minio, SSL handshake error

If you already enable minio tls, you should have the cert file already right?

sure, for minio tls, i tested with other app to connect to minio with tls, can
besides, i want to say that when minio enabled with tls with minio server certs, how does milvus connect to minio without client certs

[yanliang567]

/assign @LoveEachDay
/unassign

[LoveEachDay]

For minio and kafka, we only support tls signed by public ca. We don't have a config options to pass self-signed tls certs.

[yanliang567]

sounds like a new feature for milvus, @xiaofan-luan any plan for it?

/assign @chenraoCR
/unassign @LoveEachDay

[chenraoCR]

@LoveEachDay ah, got it and thanks, but hope it could have a config options to pass self-signed tls certs, as milvus already has the option to use TLS to connect minio&kafka while minio&kafka are possible to be configured with self-signed tls certs

[xiaofan-luan]

For minio and kafka, we only support tls signed by public ca. We don't have a config options to pass self-signed tls certs.

I think this is nice to have.
Anyone want to contribute on it?

[chenraoCR]

@xiaofan-luan so this feature that supports kafka&minio self certs will be added or not in the future and when will be released, thanks

[xiaofan-luan]

feel free to contribute on it but yes this is not our priority.

I think for kafka and minio we already support ssl but manage certs it too much for most users.

[xiaofan-luan]

we are open to any contribution on this and it shouldn't be that hard to implement

[chenraoCR]

okay, will try

[LinLeng]

public ca.

Can we disable cert check, like --insecure flag in mc client? thanks.

[chenraoCR]

in fact, i already added kafka tls settings, and can well work, but for minio, a little complex, and tried a long time, but failed, the only way i found is to set config.verifySSL = false in MinioChunkManager.cpp even when minio.useSsl is true

[LinLeng]

feel free to contribute on it but yes this is not our priority.

I think for kafka and minio we already support ssl but manage certs it too much for most users.

Our minio requires company CA cert to connect, is it possible to add/replace exisitng cert in milvus image so we can get this to work? thanks.

[chenraoCR]

feel free to contribute on it but yes this is not our priority.
I think for kafka and minio we already support ssl but manage certs it too much for most users.

Our minio requires company CA cert to connect, is it possible to add/replace exisitng cert in milvus image so we can get this to work? thanks.

no, tried, but cannot work