- List all dbt models and tests
- Get details on dbt models and tests like:
- Documentations
- Stats
- Columns
- Dependency graph
- Code source (raw and compiled)
As of now, the plugin only support the following backends:
- Google Cloud Storage
- AWS S3
- Azure Blob Storage
Note: catalog and manifest come from https://github.com/fivetran/dbt_shopify/tree/main
- Install packages:
# From your Backstage root directory
yarn --cwd packages/app add @iiben_orgii/backstage-plugin-dbt
yarn --cwd packages/backend add @iiben_orgii/backstage-plugin-dbt-backend
- Add a new dbt tab to the entity page.
packages/app/src/components/catalog/EntityPage.tsx
// packages/app/src/components/catalog/EntityPage.tsx
import { DbtPage, isDBTAvailable } from "@iiben_orgii/backstage-plugin-dbt";
// Farther down at the serviceEntityPage declaration
const serviceEntityPage = (
<EntityLayout>
{/* Place the following section where you want the tab to appear */}
<EntityLayout.Route if={isDBTAvailable} path="/dbt" title="dbt">
<DbtPage />
</EntityLayout.Route>
</EntityLayout>
);
- Add the
dbt
route by creating the filepackages/backend/src/plugins/dbt.ts
:
packages/backend/src/plugins/dbt.ts
import {
createRouter,
} from "@iiben_orgii/backstage-plugin-dbt-backend";
import { Router } from "express";
import { PluginEnvironment } from "../types";
export default async function createPlugin(
env: PluginEnvironment,
): Promise<Router> {
return await createRouter({
logger: env.logger,
config: env.config,
});
}
then you have to add the route as follows:
packages/backend/src/index.ts
// packages/backend/src/index.ts
import dbt from "./plugins/dbt";
async function main() {
//...
const dbtEnv = useHotMemoize(module, () => createEnv("dbt"));
//...
apiRouter.use("/dbt", await dbt(dbtEnv));
//...
}
In the file: packages/backend/src/index.ts
import { dbtPlugin } from '@iiben_orgii/backstage-plugin-dbt-backend';
//[...]
const backend = createBackend();
//[...]
backend.add(dbtPlugin);
//[...]
backend.start();
You can define one bucket with all your manifest and catalog files.
Add a file application/packages/app/config.d.ts
:
export interface Config {
dbtdoc: {
/**
* Frontend root URL
* @visibility frontend
*/
bucket: string;
};
}
Update the file application/packages/app/package.json
with
"files": [
"dist",
"config.d.ts"
],
"configSchema": "config.d.ts"
Then you can add to your app-config.yaml
:
dbtdoc:
bucket: your-bucket-123
backend: GoogleStorage # or S3
Limitation: all dbt docs must be saved on same backend type (GoogleStorage or S3)
Add dbtdoc-bucket
as annotation in catalog-info.yaml
. Optionally, add the dbtdoc-path
annotation if your GCS bucket structure does not conform to the expected structure.
apiVersion: backstage.io/v1alpha1
kind: Component
spec:
type: service
owner: user:guest
lifecycle: experimental
metadata:
name: "test"
annotations:
dbtdoc-bucket: "my-bucket"
dbtdoc-path: "optional/override/path" # Optional
Then you can add to your app-config.yaml
:
dbtdoc:
backend: GoogleStorage # or S3
Following path must be respect regardless your bucket setup (single or multi).
If using the multi setup, you can override the {kind}/{name}
portion of the path using the dbtdoc-path
annotation.
You can upload your manifest.json
and catalog.json
to a GCS Bucket as follow:
{dbtdoc-bucket}/{kind}/{name}/manifest.json
{dbtdoc-bucket}/{kind}/{name}/catalog.json
For authentification to GCS Bucket, the plugin use ADC credentials https://cloud.google.com/docs/authentication/provide-credentials-adc.
Update the app-config.yaml
as follow:
dbtdoc:
bucket: your-bucket-123
backend: GoogleStorage
or
dbtdoc:
bucket: your-bucket-123
backend: S3
Apply the following changes only if you use the old backend system.
In v1: packages/backend/src/plugins/dbt.ts
// packages/backend/src/plugins/dbt.ts
import {
createRouter,
GoogleStorageProvider,
} from "@iiben_orgii/backstage-plugin-dbt-backend";
import { Router } from "express";
import { PluginEnvironment } from "../types";
const storageProvider = new GoogleStorageProvider();
export default async function createPlugin(
env: PluginEnvironment,
): Promise<Router> {
return await createRouter({
logger: env.logger,
storageProvider: storageProvider,
});
}
In v2: packages/backend/src/plugins/dbt.ts
// packages/backend/src/plugins/dbt.ts
import {
createRouter,
} from "@iiben_orgii/backstage-plugin-dbt-backend";
import { Router } from "express";
import { PluginEnvironment } from "../types";
export default async function createPlugin(
env: PluginEnvironment,
): Promise<Router> {
return await createRouter({
logger: env.logger,
config: env.config,
});
}