Skip to content

Commit

Permalink
Updating to version 4.3.0.
Browse files Browse the repository at this point in the history
  • Loading branch information
georgebearden committed Nov 20, 2019
1 parent 16bef0e commit 82f0d7b
Show file tree
Hide file tree
Showing 19 changed files with 520 additions and 666 deletions.
14 changes: 14 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
# Change Log
All notable changes to this project will be documented in this file.

The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [4.3.0] - 2019-11-01
### Added
- CHANGELOG file

### Changed
- Lambda functions (except for mediainfo) runtime to nodejs10.x
- Mediainfo lambda function to python3.7
- Mediainfo executable version (from v0.7.92.1 to v19.09)
4 changes: 2 additions & 2 deletions CODE_OF_CONDUCT.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
## Code of Conduct
This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct).
For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact
This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct).
For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact
[email protected] with any additional questions or comments.
6 changes: 3 additions & 3 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ information to effectively respond to your bug report or contribution.

We welcome you to use the GitHub issue tracker to report bugs or suggest features.

When filing an issue, please check [existing open](https://github.com/awslabs/media-analysis-solution/issues), or [recently closed](https://github.com/awslabs/media-analysis-solution/issues?utf8=%E2%9C%93&q=is%3Aissue%20is%3Aclosed%20), issues to make sure somebody else hasn't already
When filing an issue, please check [existing open](https://github.com/awslabs/%%SOLUTION_NAME%%/issues), or [recently closed](https://github.com/awslabs/%%SOLUTION_NAME%%/issues?utf8=%E2%9C%93&q=is%3Aissue%20is%3Aclosed%20), issues to make sure somebody else hasn't already
reported the issue. Please try to include as much information as you can. Details like these are incredibly useful:

* A reproducible test case or series of steps
Expand Down Expand Up @@ -41,7 +41,7 @@ GitHub provides additional document on [forking a repository](https://help.githu


## Finding contributions to work on
Looking at the existing issues is a great way to find something to contribute on. As our projects, by default, use the default GitHub issue labels ((enhancement/bug/duplicate/help wanted/invalid/question/wontfix), looking at any ['help wanted'](https://github.com/awslabs/media-analysis-solution/labels/help%20wanted) issues is a great place to start.
Looking at the existing issues is a great way to find something to contribute on. As our projects, by default, use the default GitHub issue labels ((enhancement/bug/duplicate/help wanted/invalid/question/wontfix), looking at any ['help wanted'](https://github.com/awslabs/%%SOLUTION_NAME%%/labels/help%20wanted) issues is a great place to start.


## Code of Conduct
Expand All @@ -56,6 +56,6 @@ If you discover a potential security issue in this project we ask that you notif

## Licensing

See the [LICENSE](https://github.com/awslabs/media-analysis-solution/blob/master/LICENSE) file for our project's licensing. We will ask you to confirm the licensing of your contribution.
See the [LICENSE](https://github.com/awslabs/%%SOLUTION_NAME%%/blob/master/LICENSE) file for our project's licensing. We will ask you to confirm the licensing of your contribution.

We may ask you to sign a [Contributor License Agreement (CLA)](http://en.wikipedia.org/wiki/Contributor_License_Agreement) for larger changes.
32 changes: 16 additions & 16 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -131,10 +131,9 @@ AWS MediaConvert Quality-defined Variable Bit-Rate (QVBR) control mode get the b

For more detail please see [QVBR and MediaConvert](https://docs.aws.amazon.com/mediaconvert/latest/ug/cbr-vbr-qvbr.html).

## Source code (Nodejs 8.10)
## Source code (Node.js 10)
* **archive-source:** Lambda function to tag the source video in s3 to enable the Glacier lifecycle policy.
* **encode:** Lambda function to submit an encoding job to Elemental MediaConvert
* **mediainfo:** Lambda function to run mediainfo on s3 signed url. https://mediaarea.net/en/MediaInfo. bin/mediainfo must be made executable before deploying to lambda.
* **encode:** Lambda function to submit an encoding job to Elemental MediaConvert
* **custom-resource:** Lambda backed CloudFormation custom resource to deploy MediaConvert templates configure S3
event notifications.
* **error-handler:** Lambda function to handler any errors created by the workflow or MediaConvert.
Expand All @@ -143,33 +142,35 @@ event notifications.
* **dynamo:** Lambda function to Update DynamoDB
* **input-validate:** Lambda function to parse S3 event notifications and define the workflow parameters.
* **profiler:** Lambda function used to send publish and/or error notifications.
* **media-package-assets:** Lambda function to ingest an asset into MediaPackage-VOD.

## Source code (Python 3.7)
> **Note**: The _mediainfo_ function uses the python3.7 runtime since the distributable was compiled on Amazon Linux, and the [Operating System for the node version 10 runtime](https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtimes.html) is Amazon Linux 2.
* **mediainfo:** Lambda function to run mediainfo on s3 signed url. https://mediaarea.net/en/MediaInfo. bin/mediainfo must be made executable before deploying to lambda.

## Creating a custom Build
The solution can be deployed through the CloudFormation template available on the solution home page: [Video on Demand on AWS](https://aws.amazon.com/answers/media-entertainment/video-on-demand-on-aws/).
To make changes to the solution, download or clone this repo, update the source code and then run the deployment/build-s3-dist.sh script to deploy the updated Lambda code to an S3 bucket in your account.

### Pre-requirements:
* [AWS Command Line Interface](https://aws.amazon.com/cli/)
* Node.js 8.x or Python 3.x
* Node.js 10.x and Python 3.x

### 1. Create an Amazon S3 Bucket.
The CloudFormation template is configured to pull the Lambda deployment packages from Amazon S3 bucket in the region the template is being launched in. Create a bucket in the desired region with the region name appended to the name of the bucket. eg: for us-east-1 create a bucket named: ```bucket-us-east-1```

### 2. Create the deployment packages:
Run the build-s3-dist.sh script, passing in 2 variables:
Run the build-s3-dist.sh script, passing in 3 variables:
* CODEBUCKET = the name of the S3 bucket (do NOT include the -region extension)
* CODEVERSION = this will be the subfolder containing the code (video-on-demand-on-aws/*codeversion*).

This will:
* copy the console files to ./deployment/dist/.
* copy the CloudFormation template to ./deployment/dist/ and updates the source code mappings.
* zip and copy the source code to ./deployment/dist/
* SOLUTIONNAME = name of the solution (video-on-demand-on-aws)
* v4.3.0 = this will be the subfolder containing the code

Example:
```
cd deployment/
./build-s3-dist.sh bucket 1.01
chmod +x ./build-s3-dist.sh
./build-s3-dist.sh bucket video-on-demand-on-aws 1.01
```
This will update the CloudFormation template mappings:
```
Expand All @@ -189,17 +190,16 @@ In the example for us-east-1 this would be:


### 3. Upload the Code to Amazon S3.

Use the AWS CLI to sync the lambda code and demo console files to amazon S3:

```
cd deployment/
aws s3 sync .dist/ s3://bucket-us-east-1/video-on-demand-on-aws/1.01/.
aws s3 sync ./regional-s3-assets/ s3://bucket-us-east-1/video-on-demand-on-aws/1.01/ --recursive --acl bucket-owner-full-control
```

### 4. Launch the CloudFormation template.

Launch the updated CloudFormation template from ```deployment/dist/``` folder.
* Get the link of the video-on-demand-on-aws.template uploaded to your Amazon S3 bucket.
* Deploy the Video on Demand to your account by launching a new AWS CloudFormation stack using the link of the video-on-demand-on-aws.template.


## Additional Resources
Expand Down
109 changes: 77 additions & 32 deletions deployment/build-s3-dist.sh
Original file line number Diff line number Diff line change
@@ -1,43 +1,88 @@
#!/bin/bash
#
# This assumes all of the OS-level configuration has been completed and git repo has already been cloned
#
# This script should be run from the repo's deployment directory
# cd deployment
# ./build-s3-dist.sh source-bucket-base-name trademarked-solution-name version-code
#
# Paramenters:
# - source-bucket-base-name: Name for the S3 bucket location where the template will source the Lambda
# code from. The template will append '-[region_name]' to this bucket name.
# For example: ./build-s3-dist.sh solutions my-solution v1.0.0
# The template will then expect the source code to be located in the solutions-[region_name] bucket
#
# - trademarked-solution-name: name of the solution for consistency
#
# - version-code: version of the package

# Check to see if input has been provided:
if [ -z "$1" ] || [ -z "$2" ]; then
echo "Please provide the base source bucket name and version (subfolder) where the lambda code will eventually reside."
echo "For example: ./build-s3-dist.sh solutions v1.0.0"
if [ -z "$1" ] || [ -z "$2" ] || [ -z "$3" ]; then
echo "Please provide the base source bucket name, trademark approved solution name and version where the lambda code will eventually reside."
echo "For example: ./build-s3-dist.sh solutions trademarked-solution-name v1.0.0"
exit 1
fi

[ -e dist ] && rm -r dist
echo "== mkdir -p dist"
mkdir -p dist
ls -lh
#TEMPALTE
echo "==cp video-on-demand-on-aws.yaml dist/video-on-demand-on-aws.template"
cp video-on-demand-on-aws.yaml dist/video-on-demand-on-aws.template
echo "==update CODEBUCKET in template with $1"
set -e

# Get reference for all important folders
template_dir="$PWD"
template_dist_dir="$template_dir/global-s3-assets"
build_dist_dir="$template_dir/regional-s3-assets"
source_dir="$template_dir/../source"

echo "------------------------------------------------------------------------------"
echo "Rebuild distribution"
echo "------------------------------------------------------------------------------"
rm -rf $template_dist_dir
mkdir -p $template_dist_dir
rm -rf $build_dist_dir
mkdir -p $build_dist_dir

echo "------------------------------------------------------------------------------"
echo "CloudFormation Template"
echo "------------------------------------------------------------------------------"
cp $template_dir/video-on-demand-on-aws.yaml $template_dist_dir/video-on-demand-on-aws.template

replace="s/CODEBUCKET/$1/g"
sed -i -e $replace dist/video-on-demand-on-aws.template
echo "==update CODEVERSION in template with $2"
replace="s/CODEVERSION/$2/g"
sed -i -e $replace dist/video-on-demand-on-aws.template
# remove tmp file for MACs
[ -e dist/video-on-demand-on-aws.template-e ] && rm -r dist/video-on-demand-on-aws.template-e

#SOURCE CODE
echo "== zip and copy lambda deployment pacages to dist/"
cd ../source/

echo "== chmod +x ./mediainfo/bin/mediainfo"
echo "sed -i -e $replace"
sed -i -e $replace $template_dist_dir/video-on-demand-on-aws.template

replace="s/SOLUTION_NAME/$2/g"
echo "sed -i -e $replace"
sed -i -e $replace $template_dist_dir/video-on-demand-on-aws.template

replace="s/CODEVERSION/$3/g"
echo "sed -i -e $replace"
sed -i -e $replace $template_dist_dir/video-on-demand-on-aws.template
sed -i -e $replace $template_dir/../README.md

cp $template_dist_dir/video-on-demand-on-aws.template $build_dist_dir/

echo "------------------------------------------------------------------------------"
echo "Lambda Functions"
echo "------------------------------------------------------------------------------"
cd $source_dir
chmod +x ./mediainfo/bin/mediainfo

for folder in */ ; do
cd "$folder"
echo "==creating deployment package ${PWD##*/}"
rm -rf node_modules/
npm install --production
rm package-lock.json
zip -q -r9 ../../deployment/dist/${PWD##*/}.zip *
cd ../
function_name=${PWD##*/}
echo "Creating deployment package for $function_name"

if [ -e "package.json" ]; then
rm -rf node_modules/
npm install --production
rm package-lock.json

zip -q -r9 "$build_dist_dir/$function_name.zip" .
else
python3 setup.py build_pkg --zip-path="$build_dist_dir/$function_name.zip"
fi

cd ..
done
cd ../deployment/
echo "== s3 files in dist/:"
ls -lh dist/

echo "------------------------------------------------------------------------------"
echo "S3 Packaging Complete"
echo "------------------------------------------------------------------------------"
37 changes: 37 additions & 0 deletions deployment/run-unit-tests.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
#!/bin/bash

cd ../source/archive-source
npm install --silent
npm test

cd ../custom-resource
npm install --silent
npm test

cd ../dynamo
npm install --silent
npm test

cd ../encode
npm install --silent
npm test

cd ../error-handler
npm install --silent
npm test

cd ../input-validate
npm install --silent
npm test

cd ../profiler
npm install --silent
npm test

cd ../sns-notification
npm install --silent
npm test

cd ../step-functions
npm install --silent
npm test
Loading

0 comments on commit 82f0d7b

Please sign in to comment.