-
Notifications
You must be signed in to change notification settings - Fork 125
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
* pinot connector first draft * linter, enable validator * rebase with main * linter, test * fix comparison by dim * pinot dsn * docs * comments * better type support lint * remove drop method from pinot driver * unsupported date time name * docs Co-authored-by: Benjamin Egelund-Müller <[email protected]> * review comments * fix lint and test error * merge with main * fix lint * review comments * mv dim support docs * update image --------- Co-authored-by: Benjamin Egelund-Müller <[email protected]>
- Loading branch information
1 parent
1592884
commit cbd0810
Showing
27 changed files
with
1,040 additions
and
32 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,99 @@ | ||
| --- | ||
| title: Pinot | ||
| description: Power Rill dashboards using Pinot | ||
| sidebar_label: Pinot | ||
| sidebar_position: 4 | ||
| --- | ||
|
|
||
| ## Overview | ||
|
|
||
| [Apache Pinot](https://docs.pinot.apache.org/) is a real-time distributed OLAP datastore purpose-built for low-latency, high-throughput analytics, and perfect for user-facing analytical workloads. | ||
|
|
||
| Rill supports connecting to an existing Pinot cluster and using it as an OLAP engine to power Rill dashboards built against [external tables](../../build/olap/olap.md#external-olap-tables). | ||
|
|
||
| ## Connection string (DSN) | ||
|
|
||
| Rill connects to Pinot using the [Pinot Golang Client](https://docs.pinot.apache.org/users/clients/golang) and requires a connection string of the following format: `http://<user>:<password>@<host>:<port>`. | ||
| `host`and `port` should be of the Pinot Controller server. If `user` or `password` contain special characters they should be URL encoded (i.e. `p@ssword` -> `p%40ssword`). This should be set in the `connector.pinot.dsn` property in Rill. | ||
|
|
||
| As an example, this typically looks something like: | ||
|
|
||
| ```bash | ||
| connector.pinot.dsn="https://username:password@localhost:9000" | ||
| ``` | ||
|
|
||
| :::info Need help connecting to Pinot? | ||
|
|
||
| If you would like to connect Rill to an existing Pinot instance, please don't hesitate to [contact us](../../contact.md). We'd love to help! | ||
|
|
||
| ::: | ||
|
|
||
| ## Setting the default OLAP connection | ||
|
|
||
| You'll also need to update the `olap_connector` property in your project's `rill.yaml` to change the default OLAP engine to Pinot: | ||
|
|
||
| ```yaml | ||
|
|
||
| olap_connector: pinot | ||
|
|
||
| ``` | ||
| :::note | ||
| For more information about available properties in `rill.yaml`, see our [project YAML](../project-files/rill-yaml.md) documentation. | ||
|
|
||
| ::: | ||
|
|
||
| :::info Interested in using multiple OLAP engines in the same project? | ||
|
|
||
| Please see our [Using Multiple OLAP Engines](multiple-olap.md) page. | ||
|
|
||
| ::: | ||
|
|
||
| ## Configuring Rill Developer | ||
|
|
||
| When using Rill for local development, there are two options to configure Rill to enable Pinot as an OLAP engine: | ||
| - You can set `connector.pinot.dsn` in your project's `.env` file or try pulling existing credentials locally using `rill env pull` if the project has already been deployed to Rill Cloud | ||
| - You can pass in `connector.pinot.dsn` as a variable to `rill start` directly (e.g. `rill start --var connector.pinot.dsn=...`) | ||
|
|
||
| :::tip Getting DSN errors in dashboards after setting `.env`? | ||
|
|
||
| If you are facing issues related to DSN connection errors in your dashboards even after setting the connection string via the project's `.env` file, try restarting Rill using the `rill start --reset` command. | ||
|
|
||
| ::: | ||
|
|
||
| ## Configuring Rill Cloud | ||
|
|
||
| When deploying a Pinot-backed project to Rill Cloud, you have the following options to pass the appropriate connection string to Rill Cloud: | ||
| - Use the `rill env configure` command to set `connector.pinot.dsn` after deploying the project | ||
| - If `connector.pinot.dsn` has already been set in your project `.env`, you can push and update these variables directly in your cloud deployment by using the `rill env push` command | ||
|
|
||
| :::info | ||
|
|
||
| Note that you must `cd` into the Git repository that your project was deployed from before running `rill env configure`. | ||
|
|
||
| ::: | ||
|
|
||
| ## Support for Multi-Valued dimensions | ||
|
|
||
| Multi-Valued dimensions needed to be defined in the dashboard yaml as expressions using `arrayToMv` function. For example if `RandomAirports` is a multi-valued column in a Pinot table then the dimension definition will look like: | ||
|
|
||
| ```yaml | ||
| - label: RandomAirports | ||
| expression: arrayToMv(RandomAirports) | ||
| name: RandomAirports | ||
| description: "Random Airports" | ||
| ignore: false | ||
| ``` | ||
| Refer to the [Dashboard YAML](../project-files/dashboards) reference page for all dimension properties detail. | ||
|
|
||
| :::note | ||
|
|
||
| Pinot does not support unnest function so don't set `unnest` property to true in the dimension definition of dashboard yaml. | ||
|
|
||
| ::: | ||
|
|
||
| ## Additional Notes | ||
|
|
||
| - At the moment, we do not support modeling with Pinot. If this is something you're interested in, please [contact us](../../contact.md). | ||
| - For dashboards powered by Pinot, [measure definitions](../../build/dashboards/dashboards.md#measures) are required to follow [Pinot SQL](https://docs.pinot.apache.org/users/user-guide-query/querying-pinot) syntax. |
Binary file added
BIN
+480 KB
docs/static/img/build/connect/external-tables/external-pinot-table.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
cbd0810There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🎉 Published on https://ui.rilldata.in as production
🚀 Deployed on https://662fb54292f2c035e5b019ca--rill-ui-dev.netlify.app