Home
brod_consumer is essentially a poller, constantly asking for more data from the partition leader.
By subscribing to brod_consumer a process should receive the polled message sets (not individual messages) into its mailbox.
In brod, we have so far implemented two different subscribers (brod_topic_subscriber and brod_group_subscriber), hopefully to fulfil most of the common use cases.
For maximum flexibility, an application may implement its own per-partition subscriber.
Below diagrams illustrate how subscriber processes may organized
This gives the best flexibility as the per-partition subscribers work directly with per-partition pollers.
The messages are delivered to subscribers in message sets, not individual messages, (however the subscribers are allowed to ack individual offsets).
brod_topic_subscriber provides the easiest way to receive and process all messages from all partitions of a given topic.
Users may choose to implement the behaviour callbacks in a module, or simply provide an anonymous callback function to have the individual messages processed.
Similar to brod_topic_subscriber the behaviour callbacks should be implemented to process individual messages.
It is started by giving a set of topics, some (maybe none, or maybe all) of the partitions in the topic set should be assigned to it for subscription.
Users may also choose to implement the brod_group_member behaviour (callbacks for brod_group_coordinator) for a different group subscriber (e.g. spawn one subscriber per partition). see brucke for example.