Skip to content

IIBenII/backstage-plugin-dbt

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Backstage Plugin dbt

Backstage plugins to view dbt doc.

npm (scoped) latest workflow pr latest workflow release latest workflow prerelease Servier Inspired

Table of contents

Features

  • List all dbt models and tests
  • Get details on dbt models and tests like:
    • Documentations
    • Stats
    • Columns
    • Dependency graph
    • Code source (raw and compiled)

Limitations

As of now, the plugin only support the following backends:

  • Google Cloud Storage
  • AWS S3
  • Azure Blob Storage

Screenshots

Landing page: Landing page

Model details:

Test details:

Note: catalog and manifest come from https://github.com/fivetran/dbt_shopify/tree/main

Setup

  1. Install packages:
# From your Backstage root directory
yarn --cwd packages/app add @iiben_orgii/backstage-plugin-dbt
yarn --cwd packages/backend add @iiben_orgii/backstage-plugin-dbt-backend
  1. Add a new dbt tab to the entity page.

packages/app/src/components/catalog/EntityPage.tsx

// packages/app/src/components/catalog/EntityPage.tsx

import { DbtPage, isDBTAvailable } from "@iiben_orgii/backstage-plugin-dbt";

// Farther down at the serviceEntityPage declaration
const serviceEntityPage = (
  <EntityLayout>
    {/* Place the following section where you want the tab to appear */}
    <EntityLayout.Route if={isDBTAvailable} path="/dbt" title="dbt">
      <DbtPage />
    </EntityLayout.Route>
  </EntityLayout>
);

Legacy backend system

  1. Add the dbt route by creating the file packages/backend/src/plugins/dbt.ts:

packages/backend/src/plugins/dbt.ts

import {
  createRouter,
} from "@iiben_orgii/backstage-plugin-dbt-backend";

import { Router } from "express";
import { PluginEnvironment } from "../types";

export default async function createPlugin(
  env: PluginEnvironment,
): Promise<Router> {
  return await createRouter({
    logger: env.logger,
    config: env.config,
  });
}

then you have to add the route as follows:

packages/backend/src/index.ts

// packages/backend/src/index.ts
import dbt from "./plugins/dbt";

async function main() {
  //...
  const dbtEnv = useHotMemoize(module, () => createEnv("dbt"));
  //...
  apiRouter.use("/dbt", await dbt(dbtEnv));
  //...
}

New backend system

In the file: packages/backend/src/index.ts

import { dbtPlugin } from '@iiben_orgii/backstage-plugin-dbt-backend';

//[...]

const backend = createBackend();
//[...]

backend.add(dbtPlugin);

//[...]
backend.start();

Usage

Single bucket model

You can define one bucket with all your manifest and catalog files.

Add a file application/packages/app/config.d.ts:

export interface Config {
  dbtdoc: {
    /**
     * Frontend root URL
     * @visibility frontend
     */
    bucket: string;
  };
}

Update the file application/packages/app/package.json with

  "files": [
    "dist",
    "config.d.ts"
  ],
  "configSchema": "config.d.ts"

Then you can add to your app-config.yaml:

dbtdoc:
  bucket: your-bucket-123
  backend: GoogleStorage # or S3

One bucket for each application or overwrite the global config

Limitation: all dbt docs must be saved on same backend type (GoogleStorage or S3)

Add dbtdoc-bucket as annotation in catalog-info.yaml. Optionally, add the dbtdoc-path annotation if your GCS bucket structure does not conform to the expected structure.

apiVersion: backstage.io/v1alpha1
kind: Component
spec:
  type: service
  owner: user:guest
  lifecycle: experimental
metadata:
  name: "test"
  annotations:
    dbtdoc-bucket: "my-bucket"
    dbtdoc-path: "optional/override/path" # Optional

Then you can add to your app-config.yaml:

dbtdoc:
  backend: GoogleStorage # or S3

Files path in the bucket

Following path must be respect regardless your bucket setup (single or multi).

If using the multi setup, you can override the {kind}/{name} portion of the path using the dbtdoc-path annotation.

You can upload your manifest.json and catalog.json to a GCS Bucket as follow:

  • {dbtdoc-bucket}/{kind}/{name}/manifest.json
  • {dbtdoc-bucket}/{kind}/{name}/catalog.json

Authentication

For authentification to GCS Bucket, the plugin use ADC credentials https://cloud.google.com/docs/authentication/provide-credentials-adc.

Update from v1 to v2

Update the app-config.yaml as follow:

dbtdoc:
  bucket: your-bucket-123
  backend: GoogleStorage

or

dbtdoc:
  bucket: your-bucket-123
  backend: S3

Apply the following changes only if you use the old backend system.

In v1: packages/backend/src/plugins/dbt.ts

// packages/backend/src/plugins/dbt.ts
import {
  createRouter,
  GoogleStorageProvider,
} from "@iiben_orgii/backstage-plugin-dbt-backend";
import { Router } from "express";
import { PluginEnvironment } from "../types";

const storageProvider = new GoogleStorageProvider();

export default async function createPlugin(
  env: PluginEnvironment,
): Promise<Router> {
  return await createRouter({
    logger: env.logger,
    storageProvider: storageProvider,
  });
}

In v2: packages/backend/src/plugins/dbt.ts

// packages/backend/src/plugins/dbt.ts
import {
  createRouter,
} from "@iiben_orgii/backstage-plugin-dbt-backend";

import { Router } from "express";
import { PluginEnvironment } from "../types";

export default async function createPlugin(
  env: PluginEnvironment,
): Promise<Router> {
  return await createRouter({
    logger: env.logger,
    config: env.config,
  });
}