Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add spawn_dedicated_client to v2 group subscriber consumer config #3

Draft
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

urmastalimaa
Copy link
Member

When spawn_dedicated_client is set to {true, Bootstrap}, a separate client is spawned for each partition subscriber, allowing the subscribers themselves to block the fetch loop with setting min_bytes > 0, without affecting one another.

When `spawn_dedicated_client` is set to `{true, Bootstrap}`, a separate
client is spawned for each partition subscriber, allowing the
subscribers themselves to block the fetch loop with setting min_bytes >
0, without affecting one another.
Client;
{true, {Endpoints, ClientConfig} = _Bootstrap} ->
ClientId = list_to_atom(atom_to_list(Client) ++ binary_to_list(<<"-group-subscriber-v2-">>) ++ binary_to_list(Topic) ++ integer_to_list(Partition)),
{ok, _Pid} = brod_client:start_link(Endpoints, ClientId, ClientConfig),
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This does not work as desired, as the client is kept alive even when workers are terminated, e.g. when new assignments are received. See the PR comment for possible approaches

@urmastalimaa
Copy link
Member Author

urmastalimaa commented Jun 30, 2024

Options for dedicated clients in v2 group subscriber

Topic subscriber initiates a client itself

  • Change brod_topic_subscriber to accept a bootstrap in addition to a client ID
  • Start and link to the client in topic subscriber init

Pros: Relatively simple to implement
Cons: Significant change to topic subscriber start_link interface. When a group subscriber receives new assignments, all clients for the previously assigned partitions are disconnected and reconnected - a significant waste when assignment change.

v2 group subscriber maintains a pool of clients

  • Add to v2 group subscriber state a pool of clients, where each client may be associated with a worker
  • When starting a new worker, find an unassigned client from the pool or instantiate a new client
  • Disassociate the client and worker when a worker terminates (either by seeing the EXIT or in terminate_worker)
  • [Resource optimization] Disconnect a client when it has been unassigned for X time

Pros: Client connections remain intact when assignments change
Cons: Significant changes to v2 group subscriber internal logic

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

1 participant