adding_lsif_to_workflows.md

Adding Precise Code Intelligence to CI/CD workflows

Language-specific guides

We are working on creating language specific guides for use with LSIF indexers, so make sure to check for the documentation for your language! If there isn't a guide for your language, this general guide will help you through the precise code intelligence setup process.

NOTE: First make sure to complete the how-to guides on indexing.

Benefits of CI integration

Setting up a source code indexing job in your CI build provides you with fast code intelligence that gives you more control on when source code gets indexed, and ensures accuracy of your code intelligence by keeping in sync with changes in your repository. Due to the large number of CI frameworks and languages we may not have specific documentation for your use case. Feel free to file an issue if you're having difficulties and we can help troubleshoot your setup.

Index

• Using indexer containers
• CI from scratch
  • GitHub Action Examples
  • Circle CI Examples
  • Travis CI Examples
• Recommended upload frequency
• Uploading LSIF data to Sourcegraph.com

Using indexer containers

We're working on creating containers that bundle indexers and the Sourcegraph CLI (src). All the languages in our language-specific guides are supported, and this section provides a general guide to using those containers. We've pinned the containers to the latest tag in these samples so they stay up to date, but make sure you pin them before going live to ensure reproducibility.

Basic usage

Some indexers will work out of the box by just running them in your project root. Here's an example using GitHub actions and Go:

jobs:
  lsif-go:
    # this line will prevent forks of this repo from uploading lsif indexes
    if: github.repository == '<insert your repo name>'
    runs-on: ubuntu-latest
    container: sourcegraph/lsif-go:latest
    steps:
      - uses: actions/checkout@v1
      - name: Generate LSIF data
        run: lsif-go
      - name: Upload LSIF data
        # this will upload to Sourcegraph.com, you may need to substitute a different command
        # by default, we ignore failures to avoid disrupting CI pipelines with non-critical errors.
        run: src lsif upload -github-token=${{ secrets.GITHUB_TOKEN }} -ignore-upload-failure

Sub-projects

If your repository contains multiple code projects in different folders, your CI framework likely provides a "working directory" option so that you can do something like:

jobs:
  lsif-go:
    # this line will prevent forks of this repo from uploading lsif indexes
    if: github.repository == '<insert your repo name>'
    runs-on: ubuntu-latest
    container: sourcegraph/lsif-go:latest
    steps:
      - uses: actions/checkout@v1
      - name: Generate LSIF data
        working-directory: backend/
        run: lsif-go
      - name: Upload LSIF data
        # note that the upload command also needs to happen in the same directory!
        working-directory: backend/
        # this will upload to Sourcegraph.com, you may need to substitute a different command
        # by default, we ignore failures to avoid disrupting CI pipelines with non-critical errors.
        run: src lsif upload -github-token=${{ secrets.GITHUB_TOKEN }} -ignore-upload-failure

Custom build environments

Depending on which language you're using, you may need to perform some pre-indexing steps in a custom build environment. You have a couple options for this:

• Add the indexer and Sourcegraph CLI (src) to your environment. To make this easier, you could either directly build a Docker image from our indexer container, or use its Dockerfile as inspiration for getting the tools installed in your environment.
• Do the pre-indexing steps in your environment, and then make those artifacts available to our container.

This second step is easy in GitHub actions because our container can be used as an action. Here's an example for a TypeScript project:

jobs:
  scip-typescript:
    # this line will prevent forks of this repo from uploading lsif indexes
    if: github.repository == '<insert your repo name>'
    runs-on: ubuntu-latest
    container: my-awesome-container
    steps:
      - uses: actions/checkout@v1
      - name: Install dependencies
        run: <install dependencies>
      - name: Generate LSIF data
        uses: docker://sourcegraph/scip-typescript:latest
        with:
          args: scip-typescript index
      - name: Upload LSIF data
        uses: docker://sourcegraph/src-cli:latest
        with:
          # this will upload to Sourcegraph.com, you may need to substitute a different command
          # by default, we ignore failures to avoid disrupting CI pipelines with non-critical errors.
          args: lsif upload -github-token=${{ secrets.GITHUB_TOKEN }} -ignore-upload-failure

GitHub Action Examples

The following projects have example GitHub Action workflows to generate and upload LSIF indexes.

• elastic/kibana
• golang/go
• kubernetes/kubernetes
• lodash/lodash
• moby/moby
• ReactiveX/IxJS

Circle CI Examples

Certain frameworks like Circle CI may require you to explicitly cache artifacts between jobs. In CircleCI this might look like the following:

version: 2.1

jobs:
  install-deps:
    docker:
      - image: my-awesome-container
    steps:
      - checkout
      - <install dependencies>
      - save_cache:
          paths:
            - node_modules
          key: dependencies

jobs:
  scip-typescript:
    docker:
      - image: sourcegraph/scip-typescript:latest
    steps:
      - checkout
      - restore_cache:
          keys:
            - dependencies
      - run: scip-typescript index
        # this will upload to Sourcegraph.com, you may need to substitute a different command
        # by default, we ignore failures to avoid disrupting CI pipelines with non-critical errors.
      - run: src lsif upload -github-token=<<parameters.github-token>> -ignore-upload-failure

workflows:
  scip-typescript:
    jobs:
      - install-deps
      - scip-typescript:
          requires:
            - install-deps

The following projects have example CircleCI configurations to generate and upload LSIF indexes.

• angular/angular
• facebook/jest
• facebook/react
• grafana
• helm
• prometheus/prometheus
• ReactiveX/rxjs

Travis CI Examples

The following projects have example Travis CI configurations to generate and upload LSIF indexes.

• aws/aws-sdk-go
• etcd-io/etcd
• expressjs/express
• hugo/hugo
• Microsoft/TypeScript
• moment/moment
• sindresorhus/got

CI from scratch

If you're indexing a language we haven't documented yet in our language-specific guides, follow the instructions in this section. We want to have containers available for every language with a robust indexer, so please also consider filing an issue to let us know we're missing one.

Set up your CI machines

Your CI machines will need two command-line tools installed. Depending on your build system setup, you can do this as part of the CI step, or you can add it directly to your CI machines for use by the build.

1. The Sourcegraph CLI (src).
2. The LSIF indexer for your language.

Add steps to your CI

1. Generate the LSIF file for a project within your repository by running the LSIF indexer in the project directory (see docs for your LSIF indexer).
2. Upload that generated LSIF file to your Sourcegraph instance.

Recommended upload frequency

We recommend that you start with a CI job that runs on every commit (including branches).

If you see too much load on your CI, your Sourcegraph instance, or a rapid decrease in free disk space on your Sourcegraph instance, you can instead index only the default branch, or set up a periodic job (e.g. daily) in CI that indexes your default branch.

With periodic jobs, you should still receive precise code intelligence on non-indexed commits on lines that are unchanged since the nearest indexed commit. This requires that the indexed commit be a direct ancestor or descendant no more than 100 commits away. If your commit frequency is too high and your index frequency is too low, you may find commits with no precise code intelligence at all. In this case, we recommend you try to increase your index frequency if possible.

Uploading LSIF data to Sourcegraph.com

LSIF data can be uploaded to a self-hosted Sourcegraph instance or to Sourcegraph.com. Using the Sourcegraph.com endpoint will surface code intelligence for your public repositories directly on GitHub via the Sourcegraph browser extension and at https://sourcegraph.com/github.com/<your-username>/<your-repo>.

Using the Sourcegraph.com endpoint is free and your LSIF data is treated as User-Generated Content (you own it, as covered in our Sourcegraph.com terms of service). If you run into trouble, or a situation arises where you need all of your LSIF data expunged, please reach out to us at [email protected].