[Enhancement]: Support Growing Search With Querynode Delegator And Streaming Service

[chyezh]

Is there an existing issue for this?

• I have searched the existing issues

What would you like to be added?

Also see the parent issue #33285.
Detail coming soon.

Why is this needed?

In order to quickly access growing query at streaming architecture in 2.6.
We are now implementing an intermediate version of StreamingService by the current delegator implemetation.
Since 2.6:

• Every Streaming Node will embed a Query Node in it directly (Standalone mode will also embed an extra querynode in it).
• The new query node embedded in streaming node will be different with legacy querynode at coord view by assign a special server label QUERYNODE_STREAMING-EMBEDED, named streaming query node, aka. sqn.
• QueryCoord will assign the delegator(channel) only on the sqn, and assign the sealed segment only on legacy querynode.
• The querycoord will assign the channel on the sqn which the vchannel related wal located as much as possible.
• The replica manager will see the sqn, but the resource group manager will not. To avoid break the resource group api.

Here's the TODO list, part of them is extra enhancement for 2.6:

• Make the coord client lazy initialization on streaming node to make streaming node startup fast. enhance: use lazy initializing client for streaming node #38400
• Make querynode embedded in streamingnode at cluster and standalone mode.
• Change the replica manager to manage the sqn assignement for replicas, change the balance logic to achieve the channel/segment assignment on sqn/qn respectively.
• Remove the rpc layer when consume wal at same physical node.
• Remove the extra vchannel level wal consumer into wal level. fix: use new path for streamingnode recovery info #38516, enhance: use lazy initializing client for streaming node #38400
• Enable Kafka for streaming service. enhance: implement kafka for wal #38598
• Enable Rocksmq for streaming service. enhance: enable rmq for streaming #38669

Anything else?

No response