Skip to content

Commit

Permalink
Add Salesforce Connector (#3747)
Browse files Browse the repository at this point in the history
* Add Salesforce Connector

Add Salesforce connector supporting the FileIterator interface using the
Bulk API with CSV jobs.

Each source requires a SOQL Query and SObject name.

Credentials can be provided for the source or the connector.
Username/password and JWT authentication is supported.

Each bulk job batch result is retrieved as a single CSV file for rill to
ingest.

* Improve Readability With Whitespace

* Lowercase Error Strings

Update error strings to be lowercase for consistency with existing
convention.

* Error If No Client Id When Authenticating

Update salesforce.authenticate method to return an error if there is no
Connected App client id defined.

* Remove Authenticable Interface

Remove Authenticable interface and forceProvider implementation.  For
force functions directly.

* Fix Call to force.JwtAssertionForEndpoint

JwtAssertionForEndpoint takes a file containing the signing key for
creating a JWT assertion.  Write the configured key to a temp file.

* Simplify Missing Username Error Handling

* Add Tests for endpoint Function

* Remove Forceable Interface

Use force.Force directly.

* Replace Deprecated iotuil.TempFile with os.CreateTemp

* Simplify endpoint()

Short-circuit and return default endpoint if no endpoint is defined.

Make code clearer when scheme is missing from the endpoint.

* Return Error If Bulk Query Fails

If the query is invalid and the initial PK chunking batch fails, return
an error.

* Add Documentation For Configuring Salesforce Credentials
  • Loading branch information
cwarden authored Jan 15, 2024
1 parent e38be3f commit 2a71b2b
Show file tree
Hide file tree
Showing 14 changed files with 1,018 additions and 39 deletions.
1 change: 1 addition & 0 deletions cli/cmd/runtime/start.go
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@ import (
_ "github.com/rilldata/rill/runtime/drivers/https"
_ "github.com/rilldata/rill/runtime/drivers/postgres"
_ "github.com/rilldata/rill/runtime/drivers/s3"
_ "github.com/rilldata/rill/runtime/drivers/salesforce"
_ "github.com/rilldata/rill/runtime/drivers/snowflake"
_ "github.com/rilldata/rill/runtime/drivers/sqlite"
_ "github.com/rilldata/rill/runtime/reconcilers"
Expand Down
1 change: 1 addition & 0 deletions docs/docs/deploy/credentials/credentials.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,3 +18,4 @@ For instructions on how to create a service account and set credentials in Rill
- [Amazon Athena](./athena.md)
- [BigQuery](./bigquery.md)
- [Snowflake](./snowflake.md)
- [Salesforce](./salesforce.md)
50 changes: 50 additions & 0 deletions docs/docs/deploy/credentials/salesforce.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
---
title: Salesforce
description: Connect to data in a Salesforce org using the Bulk API
sidebar_label: Salesforce
sidebar_position: 80
---

<!-- WARNING: There are links to this page in source code. If you move it, find and replace the links and consider adding a redirect in docusaurus.config.js. -->

## How to configure credentials in Rill

Rill utilizes a Salesforce username along with a password (and token, depending
on org configuration) to authenticate against a Salesforce org.

### Configure credentials for local development

When working on a local project, you have the option to specify credentials when running Rill using the `--env` flag.
An example of using this syntax in terminal:
```
rill start --env connector.salesforce.username="[email protected]" --env connector.salesforce.password="MyPasswordMyToken"
```

Alternatively, you can include the credentials string directly in the source code by adding the `username` and `password` parameters.
An example of a source using this approach:
```
type: "salesforce"
endpoint: "login.salesforce.com"
username: "[email protected]"
password: "MyPasswordMyToken"
soql: "SELECT Id, Name, CreatedDate FROM Opportunity"
sobject: "Opportunity"
```
This approach is less recommended because it places the connection string (which may contain sensitive information like passwords) in the source file, which is committed to Git. For more information, please refer to the documentation on [sources](../../reference/project-files/index.md).

### Configure credentials for deployments on Rill Cloud

Once a project having a Salesforce source has been deployed using `rill deploy`, Rill requires you to explicitly provide the credentials using following command:
```
rill env configure
```
Note that you must `cd` into the Git repository that your project was deployed from before running `rill env configure`.

Leave `key` and `client_id` blank unless using JWT as described below.

### JWT

Authentication using JWT instead of a password is also supported by setting
`client_id` to the Client Id (also known as Consumer Key) of the Connected App
to use, and setting `key` to contain the PEM-formatted private key to use for
signing.
57 changes: 35 additions & 22 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ require (
github.com/Azure/azure-sdk-for-go/sdk/azcore v1.7.0
github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.3.0
github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.1.0
github.com/ForceCLI/force v1.0.5-0.20231227180521-1b251cf1a8b0
github.com/Masterminds/sprig/v3 v3.2.3
github.com/MicahParks/keyfunc v1.9.0
github.com/NYTimes/gziphandler v1.1.1
Expand Down Expand Up @@ -65,7 +66,7 @@ require (
github.com/robfig/cron/v3 v3.0.1
github.com/rs/cors v1.9.0
github.com/snowflakedb/gosnowflake v1.7.0
github.com/spf13/cobra v1.7.0
github.com/spf13/cobra v1.8.0
github.com/spf13/pflag v1.0.5
github.com/stretchr/testify v1.8.4
github.com/testcontainers/testcontainers-go v0.27.0
Expand All @@ -88,12 +89,12 @@ require (
gocloud.dev v0.34.0
golang.org/x/exp v0.0.0-20231006140011-7918f672742d
golang.org/x/oauth2 v0.13.0
golang.org/x/sync v0.4.0
golang.org/x/sync v0.5.0
golang.org/x/sys v0.15.0
google.golang.org/api v0.149.0
google.golang.org/genproto/googleapis/api v0.0.0-20231212172506-995d672761c0
google.golang.org/grpc v1.60.0
google.golang.org/protobuf v1.31.0
google.golang.org/grpc v1.60.1
google.golang.org/protobuf v1.32.0
gopkg.in/natefinch/lumberjack.v2 v2.2.1
gopkg.in/yaml.v2 v2.4.0
gopkg.in/yaml.v3 v3.0.1
Expand All @@ -114,6 +115,32 @@ require (
github.com/Azure/go-autorest v14.2.0+incompatible // indirect
github.com/Azure/go-autorest/autorest/to v0.4.0 // indirect
github.com/AzureAD/microsoft-authentication-library-for-go v1.0.0 // indirect
github.com/apache/arrow/go/v12 v12.0.0 // indirect
github.com/danieljoos/wincred v1.1.2 // indirect
github.com/distribution/reference v0.5.0 // indirect
github.com/dvsekhvalnov/jose2go v1.5.0 // indirect
github.com/form3tech-oss/jwt-go v3.2.5+incompatible // indirect
github.com/godbus/dbus v0.0.0-20190726142602-4481cbc300e2 // indirect
github.com/gsterjov/go-libsecret v0.0.0-20161001094733-a6f4afe4910c // indirect
github.com/jackc/pgtype v1.12.0 // indirect
github.com/jackc/puddle/v2 v2.2.1 // indirect
github.com/kylelemons/godebug v1.1.0 // indirect
github.com/matttproud/golang_protobuf_extensions/v2 v2.0.0 // indirect
github.com/mtibben/percent v0.2.1 // indirect
github.com/pkg/browser v0.0.0-20210911075715-681adbf594b8 // indirect
google.golang.org/genproto v0.0.0-20231211222908-989df2bf70f3 // indirect
)

require (
github.com/ForceCLI/inflect v0.0.0-20130829110746-cc00b5ad7a6a // indirect
github.com/ViViDboarder/gotifier v0.0.0-20140619195515-0f19f3d7c54c // indirect
github.com/dgrijalva/jwt-go v3.2.0+incompatible // indirect
github.com/mitchellh/go-homedir v1.1.0 // indirect
github.com/onsi/gomega v1.20.1 // indirect
)

require (
github.com/ForceCLI/config v0.0.0-20230217143549-9149d42a3c99 // indirect
github.com/JohnCGriffin/overflow v0.0.0-20211019200055-46fa312c352c // indirect
github.com/Masterminds/goutils v1.1.1 // indirect
github.com/Masterminds/semver/v3 v3.2.1 // indirect
Expand All @@ -123,7 +150,6 @@ require (
github.com/acomagu/bufpipe v1.0.4 // indirect
github.com/alicebob/gopher-json v0.0.0-20230218143504-906a9b012302 // indirect
github.com/andybalholm/brotli v1.0.5 // indirect
github.com/apache/arrow/go/v12 v12.0.0 // indirect
github.com/apache/thrift v0.18.1 // indirect
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.4.11 // indirect
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.13.7 // indirect
Expand All @@ -147,19 +173,15 @@ require (
github.com/containerd/containerd v1.7.11 // indirect
github.com/containerd/log v0.1.0 // indirect
github.com/cpuguy83/dockercfg v0.3.1 // indirect
github.com/danieljoos/wincred v1.1.2 // indirect
github.com/davecgh/go-spew v1.1.1 // indirect
github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f // indirect
github.com/distribution/reference v0.5.0 // indirect
github.com/docker/distribution v2.8.3+incompatible // indirect
github.com/docker/docker v24.0.7+incompatible // indirect
github.com/docker/go-connections v0.4.0 // indirect
github.com/docker/go-units v0.5.0 // indirect
github.com/dustin/go-humanize v1.0.1 // indirect
github.com/dvsekhvalnov/jose2go v1.5.0 // indirect
github.com/emirpasic/gods v1.18.1 // indirect
github.com/felixge/httpsnoop v1.0.4 // indirect
github.com/form3tech-oss/jwt-go v3.2.5+incompatible // indirect
github.com/gabriel-vasile/mimetype v1.4.2 // indirect
github.com/go-git/gcfg v1.5.1-0.20230307220236-3a3c6141e376 // indirect
github.com/go-git/go-billy/v5 v5.4.1 // indirect
Expand All @@ -169,7 +191,6 @@ require (
github.com/go-playground/locales v0.14.1 // indirect
github.com/go-playground/universal-translator v0.18.1 // indirect
github.com/goccy/go-json v0.10.2 // indirect
github.com/godbus/dbus v0.0.0-20190726142602-4481cbc300e2 // indirect
github.com/gogo/protobuf v1.3.2 // indirect
github.com/golang/glog v1.1.2 // indirect
github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da // indirect
Expand All @@ -183,19 +204,16 @@ require (
github.com/google/wire v0.5.0 // indirect
github.com/googleapis/enterprise-certificate-proxy v0.3.2 // indirect
github.com/googleapis/gax-go/v2 v2.12.0 // indirect
github.com/gsterjov/go-libsecret v0.0.0-20161001094733-a6f4afe4910c // indirect
github.com/hashicorp/go-uuid v1.0.3 // indirect
github.com/huandu/xstrings v1.4.0 // indirect
github.com/icholy/digest v0.1.22 // indirect
github.com/imdario/mergo v0.3.15 // indirect
github.com/imdario/mergo v0.3.16 // indirect
github.com/inconshreveable/mousetrap v1.1.0 // indirect
github.com/jackc/chunkreader/v2 v2.0.1 // indirect
github.com/jackc/pgio v1.0.0 // indirect
github.com/jackc/pgpassfile v1.0.0 // indirect
github.com/jackc/pgproto3/v2 v2.3.2 // indirect
github.com/jackc/pgservicefile v0.0.0-20221227161230-091c0ba34f0a // indirect
github.com/jackc/pgtype v1.12.0 // indirect
github.com/jackc/puddle/v2 v2.2.1 // indirect
github.com/jbenet/go-context v0.0.0-20150711004518-d14ea06fba99 // indirect
github.com/jcmturner/aescts/v2 v2.0.0 // indirect
github.com/jcmturner/dnsutils/v2 v2.0.0 // indirect
Expand All @@ -210,15 +228,13 @@ require (
github.com/klauspost/asmfmt v1.3.2 // indirect
github.com/klauspost/compress v1.16.7 // indirect
github.com/klauspost/cpuid/v2 v2.2.5 // indirect
github.com/kylelemons/godebug v1.1.0 // indirect
github.com/leodido/go-urn v1.2.4 // indirect
github.com/lufia/plan9stats v0.0.0-20211012122336-39d0f177ccd0 // indirect
github.com/magiconair/properties v1.8.7 // indirect
github.com/mattn/go-colorable v0.1.13 // indirect
github.com/mattn/go-isatty v0.0.19 // indirect
github.com/mattn/go-runewidth v0.0.14 // indirect
github.com/mattn/go-isatty v0.0.20 // indirect
github.com/mattn/go-runewidth v0.0.15 // indirect
github.com/mattn/go-sqlite3 v2.0.3+incompatible // indirect
github.com/matttproud/golang_protobuf_extensions/v2 v2.0.0 // indirect
github.com/mgutz/ansi v0.0.0-20200706080929-d51e80ef957d // indirect
github.com/minio/asm2plan9s v0.0.0-20200509001527-cdd76441f9d8 // indirect
github.com/minio/c2goasm v0.0.0-20190812172519-36a3d3bbc4f3 // indirect
Expand All @@ -229,13 +245,11 @@ require (
github.com/moby/term v0.5.0 // indirect
github.com/mohae/deepcopy v0.0.0-20170929034955-c48cc78d4826 // indirect
github.com/morikuni/aec v1.0.0 // indirect
github.com/mtibben/percent v0.2.1 // indirect
github.com/opencontainers/go-digest v1.0.0 // indirect
github.com/opencontainers/image-spec v1.1.0-rc5 // indirect
github.com/opencontainers/runc v1.1.7 // indirect
github.com/pierrec/lz4/v4 v4.1.18 // indirect
github.com/pjbgf/sha1cd v0.3.0 // indirect
github.com/pkg/browser v0.0.0-20210911075715-681adbf594b8 // indirect
github.com/pkg/errors v0.9.1 // indirect
github.com/pmezard/go-difflib v1.0.0 // indirect
github.com/power-devops/perfstat v0.0.0-20210106213030-5aafc221ea8c // indirect
Expand Down Expand Up @@ -263,15 +277,14 @@ require (
github.com/zeebo/xxh3 v1.0.2 // indirect
go.opencensus.io v0.24.0 // indirect
go.opentelemetry.io/proto/otlp v1.0.0 // indirect
golang.org/x/crypto v0.16.0 // indirect
golang.org/x/crypto v0.17.0 // indirect
golang.org/x/mod v0.13.0 // indirect
golang.org/x/net v0.19.0 // indirect
golang.org/x/term v0.15.0 // indirect
golang.org/x/text v0.14.0 // indirect
golang.org/x/tools v0.14.0 // indirect
golang.org/x/xerrors v0.0.0-20220907171357-04be3eba64a2 // indirect
google.golang.org/appengine v1.6.8 // indirect
google.golang.org/genproto v0.0.0-20231211222908-989df2bf70f3 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20231212172506-995d672761c0 // indirect
gopkg.in/warnings.v0 v0.1.2 // indirect
lukechampine.com/uint128 v1.3.0 // indirect
Expand Down
Loading

1 comment on commit 2a71b2b

@github-actions
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please sign in to comment.