[Enhancement]: Uprade etcd to 3.5.13, pulsar to 3.0.4 and minio to latest

[xiaofan-luan]

Is there an existing issue for this?

• I have searched the existing issues

What would you like to be added?

The dependency for milvus is pretty old and there are critical issues to fixed

We we seen some of the critical issues related to dependency crash or data loss.

It's could help milvus service to be more stable and might also help the performance of insertion.

Why is this needed?

No response

Anything else?

No response

[xiaofan-luan]

/assign @LoveEachDay

[LoveEachDay]

@xiaofan-luan Our Pulsar 3.0.2 tests have been satisfactory, particularly in terms of broker load balancing. Is there any specific reason for upgrading to pulsar 3.0.4, which rolls out only ten days ago?

[xiaofan-luan]

@xiaofan-luan Our Pulsar 3.0.2 tests have been satisfactory, particularly in terms of broker load balancing. Is there any specific reason for upgrading to pulsar 3.0.4, which rolls out only ten days ago?

Since 3.0.4 is the latest version of 3.0.x, I'm assuming this is gonna to be more stable compare to 3.0.2

[lynnssi]

Is there an ETA on releasing the upgrades to etcd/pulsar/minio?

If I upgraded these images on my own Milvus implementation before an official release, would Milvus continue to work as expected, or should I wait until the upgrade is formally released?

[xiaofan-luan]

it should be.
@LoveEachDay any comments on this?

[lynnssi]

@LoveEachDay Any follow up here?

[lynnssi]

Can you confirm whether the upgraded dependencies are compatible with the latest version of Milvus?

If not, is there an ETA on when Milvus will be compatible with the latest versions of etcd/pulsar/minio?

[xiaofan-luan]

I believe upgrade etcd/minio/pulsar dependencies should not be aware of the dependency version.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.

[lynnssi]

/reopen
I have tried upgrading pulsar to version 3.0.4, but it appears that I need to also upgrade the helm chart in order for it to work. I have pulled the template for pulsar 3.0.0 from https://github.com/apache/pulsar-helm-chart/tree/pulsar-3.0.0, but am having some trouble.

@LoveEachDay could you share the pulsar helm chart that you used to do your testing?

It would be helpful to have this in a future Milvus release, as I am sure other users would like to use a more recent version of pulsar.

[xiaofan-luan]

/reopen I have tried upgrading pulsar to version 3.0.4, but it appears that I need to also upgrade the helm chart in order for it to work. I have pulled the template for pulsar 3.0.0 from https://github.com/apache/pulsar-helm-chart/tree/pulsar-3.0.0, but am having some trouble.

@LoveEachDay could you share the pulsar helm chart that you used to do your testing?

It would be helpful to have this in a future Milvus release, as I am sure other users would like to use a more recent version of pulsar.

the issue here is pulsar 2.9 and 3.0 is not compatible.
all the existing customer need to stop service before the upgrade.

[xiaofan-luan]

We decide to upgrade to pulsar 3.0 at Milvus 2.5

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.

[lynnssi]

@xiaofan-luan When can I expect to see Milvus 2.5 released?

[xiaofan-luan]

@xiaofan-luan When can I expect to see Milvus 2.5 released?

Currently the 2.5 code is almost there. The Target release date is mid September.

I think we hit some issue on can not rolling upgrade pulsar to 3.0.x smoothly? @LoveEachDay

[xiaofan-luan]

do we need to stop all service for user to upgrade?

If that's the case, my suggestion is to use 2.9.x by default on 2.5 (Given user an config option to use 3.0.x)
And upgrade pulsar to 3.0.2 by default in 3.0

[lynnssi]

If pulsar 2.9.x is the default on 2.5, a user option to use 3.0.x would be really useful! I am keen to upgrade my pulsar version and would much prefer to do it through a user option in the helm chart.

[xiaofan-luan]

If pulsar 2.9.x is the default on 2.5, a user option to use 3.0.x would be really useful! I am keen to upgrade my pulsar version and would much prefer to do it through a user option in the helm chart.

That is definitely doable.

The reason is you will need to stop pulsar and made some change on the helm chart.

[xiaofan-luan]

so the upgrade to pulsar 3.0.x will be left to milvus 3.0.
We will have a recommened setup for pulsar 3.0.0x and how you can upgrade from 2.8/2.9 to 3.0