DE-101: Re-platform application to Python

.github/workflows/run-unit-tests.yml

@@ -0,0 +1,26 @@
+name: Run Python unit tests
+
+on: [ push ]
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout repo
+        uses: actions/checkout@v3
+
+      - name: Set up Python 3.9
+        uses: actions/setup-python@v4
+        with:
+          python-version: '3.9'
+          cache: 'pip'
+
+      - name: Install dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install -r requirements.txt
+
+      - name: Run linter and test suite
+        run: |
+          make lint
+          make test

.gitignore

@@ -2,7 +2,15 @@ node_modules
 
 coverage
 
+.aws-sam/
+.coverage
 .DS_Store
+.vscode/
+__pycache__/
 .terraform.lock.hcl
 .terraform
+.pytest_cache
 dist.zip
+*env/
+*.py[cod]
+*$py.class

.python-version

@@ -0,0 +1 @@
+3.9.16

Makefile

@@ -0,0 +1,21 @@
+.DEFAULT: help
+
+help:
+	@echo "make help"
+	@echo "    display this help statement"
+	@echo "make run"
+	@echo "    run the application in devel"
+	@echo "make test"
+	@echo "    run associated test suite with pytest"
+	@echo "make lint"
+	@echo "    lint project files using the black linter"
+
+run:
+	export ENVIRONMENT=devel; \
+	python -c 'import lambda_function; lambda_function.lambda_handler(None, None)'
+
+test:
+	pytest tests -W ignore::DeprecationWarning
+
+lint:
+	black ./ --check --exclude="(env/)|(tests/)"

README.md

@@ -1,68 +1,48 @@
 # Kinesis Firehose Avro to Json Transformer Lambda
 [![Build Status](https://travis-ci.org/NYPL/firehose-avro-to-json-transformer.svg?branch=main)](https://travis-ci.org/NYPL/firehose-avro-to-json-transformer)
 
-This app reads from Firehose Kinesis streams, decodes the records using the appropriate Avro schema based on the stream name, and returns the resulting records as either JSON or CSV (base64 encoded). This app is responsible for decoding records immediately before ingest into the [BIC](https://github.com/NYPL/BIC).
-
-## Version
-> v1.0.1
-
-## Installation
-
-Install all Node dependencies via NPM
-
-```console
-nvm use
-npm install
-```
+This Python application is responsible for Avro-decoding events immediately before ingestion into the [BIC](https://github.com/NYPL/BIC). Originally developed for the Data Warehouse, this is deployed as an AWS Lambda (["AvroToJsonTransformer-qa"](https://console.aws.amazon.com/lambda/home?region=us-east-1#/functions/AvroToJsonTransformer-qa?tab=configuration) and ["AvroToJsonTransformer-production"](https://console.aws.amazon.com/lambda/home?region=us-east-1#/functions/AvroToJsonTransformer-production?tab=configuration)). In essence, the code does the following:
+ - Decodes the incoming batch of records using the corresponding Avro schema, which is determined based on the name of the incoming Kinesis stream
+ - Converts said records into a hash with `recordId`, `result: 'Ok'`, and `data` containing a JSON or CSV serialization of the record, which is also base64 encoded
+ - Returns processed records in this format: `{ records: [ { recordId: '[record id]', result: 'Ok', data: 'eyJmb28iOiJiYXIifQ....' }, ... ] }`
 
 ## Running Locally
 
-Use the [sam cli](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-cli-install.html) to run the lambda on arbitrary firehose events. To process a firehose event containing 3 CircTrans records and print out the result:
+Use the [sam cli](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-cli-install.html) to run the Lambda on arbitrary Firehose events. To process a Firehose event containing 3 CircTrans records and print out the result:
 
 ```
-sam local invoke --profile nypl-digital-dev -t sam.qa.yml -e sample/firehose-CircTrans-3-records-encoded.json
+sam local invoke --profile nypl-digital-dev -t config/sam.qa.yml -e sample/firehose-CircTrans-3-records-encoded.json
 ```
 
-## Contributing
+The [sample](./sample) folder contains sample Firehose events and their expected outcomes after Lambda event handling, so you can test the efficacy of your code with various schemas.
+
+With Python, you also have the option of using the [python-lambda-local](https://pypi.org/project/python-lambda-local/) package for local development!
+
+## Contributing / Deployment
 
 This repo uses the ["PRs Target Main, Merge to Deployment Branches" git workflow](https://github.com/NYPL/engineering-general/blob/main/standards/git-workflow.md#prs-target-main-merge-to-deployment-branches):
  - Cut PRs from `main`
  - Merge `main` > `qa`
  - Merge `main` > `production`
 
-## Deployment
-
-This app is deployed via Travis-CI using terraform. Code in `qa` is pushed to AvroToJsonTransformer-qa. Code in `production` is pushed to AvroToJsonTransformer-production.
-
-## Tests
+This app is deployed via Travis-CI using Terraform. Code in `qa` is pushed to ["AvroToJsonTransformer-qa"](https://console.aws.amazon.com/lambda/home?region=us-east-1#/functions/AvroToJsonTransformer-qa?tab=configuration). Code in `production` is pushed to ["AvroToJsonTransformer-production"](https://console.aws.amazon.com/lambda/home?region=us-east-1#/functions/AvroToJsonTransformer-production?tab=configuration).
 
-To run all tests found in `./test/`:
-
-```console
-npm run test
+## Test Coverage
+Use the Python [coverage package](https://coverage.readthedocs.io/en/7.6.0/) to measure test coverage:
 ```
-
-To run a specific test for the given filename:
-
-```console
-npm run test [filename].test.js
+coverage run -m pytest
 ```
 
-### Test Coverage
-
-This repo uses c8 to compute test coverage (because [Istanbul](https://github.com/istanbuljs/nyc) doesn't appear to support ESM at writing). Coverage reports are included at the end of `npm test`. For a detailed line-by-line breakdown, view the HTML report:
-
-```console
-npm run coverage-report
-open coverage/index.html
+To see what exactly which lines are missing testing:
+```
+coverage report -m
 ```
 
-### Linting
-
-This codebase uses [Standard JS](https://www.npmjs.com/package/standard) as the JavaScript linter.
+## Linting
 
-To check for linting errors:
+This codebase uses [Black](https://github.com/psf/black) as the Python linter.
 
-```console
-npm run lint
+To format the codebase as a whole:
+```
+make lint
 ```

config/devel.yaml

@@ -0,0 +1,6 @@
+---
+PLAINTEXT_VARIABLES:
+    ENVIRONMENT: devel
+    NYPL_DATA_API_BASE_URL: https://qa-platform.nypl.org/api/v0.1/
+    SCHEMA_PATH: current-schemas/
+...

config/production.yaml

@@ -0,0 +1,6 @@
+---
+PLAINTEXT_VARIABLES:
+    ENVIRONMENT: production
+    NYPL_DATA_API_BASE_URL: https://platform.nypl.org/api/v0.1/
+    SCHEMA_PATH: current-schemas/
+...

config/qa.yaml

@@ -0,0 +1,6 @@
+---
+PLAINTEXT_VARIABLES:
+    ENVIRONMENT: qa
+    NYPL_DATA_API_BASE_URL: https://qa-platform.nypl.org/api/v0.1/
+    SCHEMA_PATH: current-schemas/
+...

sam.qa.yml → config/sam.qa.yml

@@ -6,8 +6,9 @@ Resources:
   AvroToJsonTransformer:
     Type: AWS::Serverless::Function
     Properties:
-      Handler: index.handler
-      Runtime: nodejs14.x
+      CodeUri: .
+      Handler: lambda_function.lambda_handler
+      Runtime: python3.9
       Timeout: 10
       Environment:
         Variables:

deployment_script.sh

@@ -0,0 +1,10 @@
+#!/bin/zsh
+
+rm -f -r ./package
+rm -f deployment-package.zip
+pip3.9 install --target ./package -r requirements.txt
+cd package
+zip -r ../deployment-package.zip .
+cd ..
+zip deployment-package.zip lambda_function.py
+zip deployment-package.zip record_processor.py