ci360-s3-bulk-connector

CI360 S3 Bulk Connector

Overview

SAS Customer Intelligence 360 (CI360) integration to Amazon Web Services (AWS) S3 using CI360 Bulk Connector Framework. CI360 S3 Connector supports uploading customer data files generated by CI360 to external S3 bucket. This bulk connector is intended to be used with a CI360 Custom Task Type created to support sending data to S3. We can use data in AWS S3 to for customer activation through number of downstream AWS services (like ads or data clean rooms), or as data source for other internal processes.

Connector Architecture

Connector uses AWS Lambda function to to process the files staged by CI360. Lambda function is invoked via the webhook call of CI360 bulk connector.

Lambda function uses the following AWS features:

• Lambda function Environment Variables: for basic configuration of the function
• IAM: definition of role under which Lambda functions are being executed, and which gives access to other AWS components (like S3)
• API Gateway: exposes Lambda function as API endpoint, also secures it using API Keys
• Python code for Lambda function was written with expectation that function is integrated with API Gateway using Lambda proxy integration (if not using proxy integration, slight change to code will be required)

Prerequisites

This connector has been developed for AWS platform. Account needs to be set up for the AWS platform.

Installation

AWS Deployment

Steps required to install connector functions to AWS:

• Create Lambda function (s3BulkUpload)
• Adjust ephemeral storage for Lambda function as needed for your use case: needed for temporary file storage, Lambda ephemeral storage is mounted as /tmp and can be configured for 512MB to 10GB
• Add Environment Variables for the function
  • s3_bucket_name
• Modify the role assigned to Lambda function and add AmazonS3FullAccess policy
• Create API gateway (matchBulkUploadApi)
  • Add resources to API gateway
  • Configure POST methods, proxy lambda integration
  • Configure API Keys and Plans and associate with resource/stage (for authentication)

Using the Connector

Configuration

The following Lambda environment variables are used to configure connector behavior for outbound function:

• s3_bucket_name: name of AWS S3 bucket where file will be saved (Lambda function has to have S3 permissions)

CI360 Setup

Steps to set up new bulk connector for AWS S3:

• In CI360, go to General Settings -> External Access -> Connectors
• Create “New Connector” of type "Bulk"
• Name it “AWS Connector”
• Select "Bulk" as Connection Type
• Create “New Webhook Endpoint”
• Name it “S3 Upload”
  • URL is your deployed Lambda function (behind API Gateway in AWS)
  • Method is POST
  • Add x-api-key header if API Keys are configured

After the connector has been created and configured, create a new Custom Task Type using the webhook endpoint as connection. You can name the task type "S3", "AWS S3" or something similar and appropriate. The new custom task type is expected to have the following Send Parameters configured:

• s3_filename: The name of the file that will be placed in S3 bucket (the name of the bucket is configured as environment variable of the Lambda function, and not specified by marketing user) You can specify required outbound data fields, if appropriate for your use case, or leave it completely up to end users to configure desired data fields to be sent to S3 in each individual task, by selecting "Allow users to add attributes while creating the task".

Using Connector with Custom Task Types

Customer data is uploaded to S3 based on audience configured in CI360 (sourced from on-prem data source, uploaded cloud data or cloud audience). File uploaded to S3 will have columns specified in Outbound Data section of Delivery tab in new Custom Task Type. The name of the file will be specified by the user on the Send Parameters tab of the new task.