Stream Grids 6.How to lookup from external data inside a stream workflow

Stream Grids provides for a way to broadcast data from external stores into Stream workflows. This will help in cases when one has to lookup these data and enrich the stream data. Different sources for broadcast are supported like HBase, Flat file etc. Here I'll explain how to broadcast data from HBase into the workflow.

First click on the broadcast button inside the 'Create Workflow' page.

Then input the following details: HBase connection name (persistent store conn. name), table name, column family name, column qualifier name, and a broadcast identifier that uniquely identifies this broadcast information (which can be later used in transformations like Enricher)

To add more broadcast data, click on the + button and repeat the same procedure. Finally click save.

Note: The framework internally constructs a hashmap of Row Key (in case of HBase) and the value of the column qualifier selected above and broadcasts this information. So for Enricher transformation, it is assumed that the field to be enriched must be the row key of its corresponding HBase lookup table.