fix: rocksmq data race on consumers modification

[chyezh]

Related Issue: #29101

• Fix data race by add new lock and CopyOnWrite
• Add new unittest to verify it

[sre-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: chyezh
To complete the pull request process, please assign congqixia after the PR has been reviewed.
You can assign the PR to them by writing /assign @congqixia in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[chyezh]

Data race detected on old implementation by unittest

Write at 0x00c0005e5470 by goroutine 141:
  github.com/milvus-io/milvus/internal/mq/mqimpl/rocksmq/server.(*rocksmq).RegisterConsumer()
      /Users/zilliz/repo/github/chyezh/milvus/internal/mq/mqimpl/rocksmq/server/rocksmq_impl.go:550 +0x54c
  github.com/milvus-io/milvus/internal/mq/mqimpl/rocksmq/server.TestRocksmq_RegisterConsumer.func2()
      /Users/zilliz/repo/github/chyezh/milvus/internal/mq/mqimpl/rocksmq/server/rocksmq_impl_test.go:246 +0x1c8
  github.com/milvus-io/milvus/internal/mq/mqimpl/rocksmq/server.TestRocksmq_RegisterConsumer.func10()
      /Users/zilliz/repo/github/chyezh/milvus/internal/mq/mqimpl/rocksmq/server/rocksmq_impl_test.go:249 +0x44

Previous write at 0x00c0005e5470 by goroutine 143:
  github.com/milvus-io/milvus/internal/mq/mqimpl/rocksmq/server.(*rocksmq).RegisterConsumer()
      /Users/zilliz/repo/github/chyezh/milvus/internal/mq/mqimpl/rocksmq/server/rocksmq_impl.go:550 +0x54c
  github.com/milvus-io/milvus/internal/mq/mqimpl/rocksmq/server.TestRocksmq_RegisterConsumer.func2()
      /Users/zilliz/repo/github/chyezh/milvus/internal/mq/mqimpl/rocksmq/server/rocksmq_impl_test.go:246 +0x1c8
  github.com/milvus-io/milvus/internal/mq/mqimpl/rocksmq/server.TestRocksmq_RegisterConsumer.func10()
      /Users/zilliz/repo/github/chyezh/milvus/internal/mq/mqimpl/rocksmq/server/rocksmq_impl_test.go:249 +0x44

Goroutine 141 (running) created at:
  github.com/milvus-io/milvus/internal/mq/mqimpl/rocksmq/server.TestRocksmq_RegisterConsumer()
      /Users/zilliz/repo/github/chyezh/milvus/internal/mq/mqimpl/rocksmq/server/rocksmq_impl_test.go:238 +0xa84
  testing.tRunner()
      /opt/homebrew/Cellar/go/1.21.5/libexec/src/testing/testing.go:1595 +0x1b0
  testing.(*T).Run.func1()
      /opt/homebrew/Cellar/go/1.21.5/libexec/src/testing/testing.go:1648 +0x40

[chyezh]

rerun ut

[mergify]

@chyezh E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

[chyezh]

/run-cpu-e2e

[chyezh]

/run-cpu-e2e

[chyezh]

rerun ut

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.

[mergify]

@chyezh Thanks for your contribution. Please submit with DCO, see the contributing guide https://github.com/milvus-io/milvus/blob/master/CONTRIBUTING.md#developer-certificate-of-origin-dco.