This is a Singer tap that produces JSON-formatted data following the Singer spec.
This tap:
- Pulls raw data from Circle CI
- Extracts the following resources:
- Outputs the schema for each resource
- Incrementally pulls data based on the input state
-
Install
git clone [email protected]:sisudata/tap-circle-ci.git && cd tap-circle-ci && pip install -e .
-
Create a Circle CI access token
Login to your Circle CI account, go to the Personal API Tokens page, and generate a new token. Copy the token and save it somewhere safe.
-
Create the config file
Create a JSON file containing the token you just created as well as the project slug to the project you want to extract data from. Retrieve the project slug from the url for a workflow - it should be the VCS your project uses (
gh
for Github orbb
for Bitbucket), followed by the owner or organization, followed by the repository name ex.gh/singer-io/singer-python
. You can enter multiple project slugs separated by spaces to pull data from multiple projects.{ "token": "your-access-token", "project_slugs": "gh/singer-io/singer-python gh/singer-io/getting-started" }
-
Run the tap in discovery mode to get catalog.json file
tap-circle-ci --config config.json --discover > catalog.json
-
In the catalog.json file, select the streams to sync
Each stream in the properties.json file has a "metadata" entry. To select a stream to sync, add
{"breadcrumb": [], "metadata": {"selected": true}}
to that stream's "metadata" entry.
For example, to sync the pipelines stream:... "type": [ "null", "object" ], "additionalProperties": false }, "stream": "pipelines", "metadata": [{"breadcrumb": [], "metadata": {"selected": true}}] }, ...
Another way to select a stream to sync is to add
"selected": true
into that stream's schema:... "tap_stream_id": "workflows", "key_properties": [], "schema": { "selected": true, "properties": { "_pipeline_id": { "type": [ "null", "string" ] ...
Either way is acceptable, but the first way is preferred.
-
Run the application (will print records and other messages to the console)
tap-circle-ci
can be run with:tap-circle-ci --config config.json --catalog catalog.json
To save output to a file:
tap-circle-ci --config config.json --catalog catalog.json > output.txt
It is our intention that this singer tap gets used with a singer target, which will load the output into a database. More information on singer targets here.
Copyright © 2020 Stitch