Merge pull request #177 from asfadmin/devel

build.77 didn't include recent changes because of SNAFU.

NGAP-DEPLOY-README.MD

@@ -1,18 +1,24 @@
 # Deploying Thin Egress to NGAP
 
-The Thin Egress App is available as a [Terraform module](https://www.terraform.io/docs/configuration/modules.html). It can be added to your Terraform configuration using a `module` block. If your deployment does not yet use Terraform, [these instructions](https://nasa.github.io/cumulus/docs/deployment/components) should prove useful.
+### ⚠️ If you're deploying from Cumulus, most or all of this provided for you. Please consult current Cumulus Documentation ⚠️
 
-The lastest version of the module can be found at
-[https://s3.amazonaws.com/asf.public.code/index.html](https://s3.amazonaws.com/asf.public.code/index.html).
+Cumulus provides a baseline Bucket Map, but its often desirable to overwrite that bucket map with one that fits your particular and evolving deployment, data structure, and DAAC needs. For more information on creating a bucket map file, see the [**Create A Bucket Map**](https://github.com/asfadmin/thin-egress-app/blob/devel/NGAP-DEPLOY-README.MD#create-a-bucket-map-file) Section below. 
+
+### Using Terraform to deploy TEA 
+
+The Thin Egress App is available as a [Terraform module](https://www.terraform.io/docs/configuration/modules.html). It can be added to your Terraform configuration using a `module` block. 
+
+The latest version of TEA and the Terrafrom module can be found at
+[https://s3.amazonaws.com/asf.public.code/index.html](https://s3.amazonaws.com/asf.public.code/index.html). The below guide is intended to be a step-by-step set of instructions for deploying TEA into NGAP. 
 
 As of this writing, the lastest module source is
-`https://s3.amazonaws.com/asf.public.code/thin-egress-app/tea-terraform-build.16.zip`.
+`https://s3.amazonaws.com/asf.public.code/thin-egress-app/tea-terraform-build.74.zip`, The latest released version is ![Last Release](https://img.shields.io/endpoint.svg?url=https%3A%2F%2Fs3.amazonaws.com%2Fasf.public.code%2Fthin-egress-app%2Flastrelease.json)
 
 ```hcl
 # Example Thin Egress App configuration
 
 module "thin_egress_app" {
-  source = "https://s3.amazonaws.com/asf.public.code/thin-egress-app/tea-terraform-build.16.zip"
+  source = "https://s3.amazonaws.com/asf.public.code/thin-egress-app/tea-terraform-build.74.zip"
 
   bucketname_prefix          = ""
   config_bucket              = "my-config-bucket"
@@ -21,31 +27,32 @@ module "thin_egress_app" {
 }
 ```
 
+
+
 ## Input variables
 
 ### Required
 
-* **bucketname_prefix** (string) - all data buckets should have names prefixed with this. Must be compatible with S3 naming conventions (lower case only, etc). An empty string can be used to indicate no prefix.
+* **bucketname_prefix** (string) - all data buckets should have names prefixed with this. Must be compatible with S3 naming conventions (lower case only, etc). An empty string can be used to indicate no prefix
 * **config_bucket** (string) - the bucket where config files can be found
+* **jwt_secret_name** (string) - name of AWS secret where keys for JWT encode/decode are stored
 * **stack_name** (string) - the name of the Thin Egress App CloudFormation stack
-* **urs_auth_creds_secret_name** (string) - AWS Secrets Manager name of URS creds. Must consist of two rows, names 'UrsId' and 'UrsAuth'.
+* **urs_auth_creds_secret_name** (string) - AWS Secrets Manager name of URS creds. Must consist of two rows, names 'UrsId' and 'UrsAuth'
 
 ### Optional
 
 * **auth_base_url** (string) - the 'AUTH_BASE_URL' env var in the lambda. Defaults to "https://urs.earthdata.nasa.gov".
-* **bucket_map_file** (string) - path and file of bucketmap file's location in the ConfigBucket. Defaults to "bucket_map.yaml".
+* **bucket_map_file** (string) - path and file of bucketmap file's location in the config_bucket. Defaults to "bucket_map.yaml".
 * **domain_name** (string) - custom domain name used by redirect_url
 * **download_role_arn** (string) - ARN for reading of data buckets
 * **html_template_dir** (string) - directory in ConfigBucket where the lambda will look for html templates. Lambda will not look into subdirectories. Please put only html templates in this dir. Leave this field blank to use default templates that are included with the lambda code zip file.
+* **jwt_algo** (string) - Algorithm with which to encode the JWT cookie. Defautls to "RS256"
 * **lambda_code_s3_bucket** (string) - S3 bucket of packaged lambda egress code. Defaults to "asf.public.code"
 * **lambda_code_s3_key** (string) - S3 key of packaged lambda egress code. Defaults to "thin-egress-app/tea-code-<BUILD_ID>.zip", where "BUILD_ID" is determined when the Terraform module was pacakged.
 * **log_level** (string) - Python loglevel. Defaults to "DEBUG"
 * **maturity** (string) - maturity of deployment. Defaults to "DEV".
 * **permissions_boundary_name** (string) - PermissionsBoundary Policy name to be used when creating IAM roles
 * **private_vpc** (string) - internal VPC to deploy to
-* **private_buckets_file** (string) - path and file of private buckets file's location in the ConfigBucket
-* **public_buckets_file** (string) - path and file of public buckets file's location in the ConfigBucket
-* **session_store** (string) - "DB" for storing sessions in DynamoDB, "S3" for storing sessions in S3. Defaults to "DB".
 * **session_ttl** (number) - time to live for auth session, in hours. Defaults to 168.
 * **stage_name** (string) - this value will show up as the base of the url path as so: `https://xxxxxxxx.execute-api.us-east-1.amazonaws.com/<StageName>/and/so/on`. Defaults to "API".
 * **use_reverse_bucket_map** (bool) - standard bucketmaps are not reverse. Defaults to false.
@@ -56,19 +63,20 @@ module "thin_egress_app" {
 
 * **api_endpoint** (string) - the API Gateway endpoint of the deployment
 * **urs_redirect_uri** (string) - the URS redirect URI to be configured for the app in Earthdata Login
+* **egress_log_group** (string) - API Gateway Access logs LogGroup, if enabled
 
 ## Pre-deployment configuration steps
 
 ### Create a bucket map file
 
-Prefix is `sirc-ingest-dev-`, so  `public` maps to `sirc-ingest-dev-public`.
+In the example below, `bucketname_prefix` is `data-ingest-dev-`, so `public` maps to `data-ingest-dev-public`. If you have not supplied a prefix, you can instead use the full bucket name. Use a prefix can be desirable as it allows bucket maps to potentially be viable across maturies and projects
 
-Data in s3://sirc-ingest-dev-public will be addressable at https://endpoint/SIRC/BROWSE/{object}
+Data in s3://data-ingest-dev-public will be addressable at https://endpoint/DATA/BROWSE/{object}
 
 ```shell
 cat > /tmp/bucket_map.yaml <<EOS
 MAP:
-  SIRC:
+  DATA:
     BROWSE:   public
     SLC:      protected
     GRD:      protected
@@ -79,17 +87,22 @@ PUBLIC_BUCKETS:
 EOS
 ```
 
-### Upload bucketmap to internal bucket
+Data in the `protected` (aka `s3://data-ingest-dev-protected`) bucket will require URS Authentication to access. 
 
-```shell
-aws s3 $AWSENV cp /tmp/bucket_map.yaml s3://sirc-ingest-dev-internal/bucket_map.yaml
-```
+For more info on constructing a bucket Mmp, see [Buckets and Bucket Map](https://github.com/asfadmin/thin-egress-app#buckets-and-bucket-map)
 
-### Create URS Credentials
+### Upload bucket map to internal/config bucket
 
 ```shell
-aws secretsmanager create-secret $AWSENV \
-  --name tea-urs-creds-dev \
-  --secret-string '{"UrsId": "VXJzSWQ=","UrsAuth": "VXJzQXV0aA=="}' \
-  --description 'URS Credentials for TEA Distribution App'
+aws s3 $AWSENV cp /tmp/bucket_map.yaml s3://<your_config_bucket>/bucket_map.yaml
 ```
+
+### Create URS Credentials And JWT Secrets:
+
+You will need to provide your URS App credentails (`urs_auth_creds_secret_name`) and the JWT Cookie Keys (`jwt_secret_name`) as AWS Screts Manager items.
+
+See [AWS Secrets](https://github.com/asfadmin/thin-egress-app/#aws-secrets) for more information.  
+
+### Update URS redirect_uri:
+
+The TEA Terraform module has the output value `urs_redirect_uri`. This URL should be added to your URS App as a valid Redirect URI. If updating the app to supply CloudFront URL or Custom DNS records using the `domain_name` and `cookie_domain` fields, it will be important to update or replace the output `urs_redirect_uri` value in your URS App. 

README.MD

@@ -92,6 +92,9 @@ zip -r9 ./${CODE_ARCHIVE_FILENAME} ./lambda
 # Add the egress python code
 zip -g ./${CODE_ARCHIVE_FILENAME} ./app.py
 
+# Add the update lambda
+zip -g ./${CODE_ARCHIVE_FILENAME} ./update_lambda.py
+
 # Add the html templates
 zip -g -r2 ./${CODE_ARCHIVE_FILENAME} ./templates
 
@@ -163,7 +166,7 @@ profile_name=<aws_profile> aws_region=<region> bash setup_jwt_cookie.sh
 
 ### Buckets and Bucket map
 
-The bucket map allows the app to determine in which bucket to look when given the path from the URL. It's possible to separate the maps into bucket, public and private, but this functionality is deprecated and will be removed in a future version of TEA.
+The bucket map allows the app to determine in which bucket to look when given the path from the URL. It's possible to separate the maps into separate files for bucket, public and private, but this functionality is deprecated and will be removed in a future version of TEA.
 
 If a url for a product would look like:
 ```https://datapile.domain.com/STAGE/PROCESSING_TYPE_1/PLATFORM_A/datafile.dat```
@@ -176,6 +179,10 @@ And if we have a data bucket prefix of `prfx-d-` and our data bucket list looks
 - prfx-d-pb-pt1
 - prfx-d-pa-pt2
 - prfx-d-pb-pt2
+- prfx-d-pa-pt3a
+- prfx-d-pb-pt3b
+- prfx-d-pa-pt3a
+- prfx-d-pb-pt3b
 - prfx-d-private-x
 - prfx-d-private-y
 
@@ -188,9 +195,29 @@ MAP:
   PROCESSING_TYPE_1:
     PLATFORM_A:         pa-pt1
     PLATFORM_B:         pb-pt1
+
+  # Custom response headers for the redirect are supported:
   PROCESSING_TYPE_2:
-    PLATFORM_A:         pa-pt2
-    PLATFORM_B:         pb-pt2
+    PLATFORM_A:
+      bucket:           pa-pt2
+      headers:
+        x-custom-header-1: "custom header 1 value"
+        x-custom-header-2: "custom header 2 value"
+    PLATFORM_B:         
+      bucket:           pb-pt2
+      headers:
+        x-custom-header-1: "custom header 1 value"
+        x-custom-header-2: "custom header 2 value"
+
+  # arbritary number of "subdirectories" are supported:
+  PROCESSING_TYPE_3:
+    SUB_PROCESSING_TYPE_3A:
+      PLATFORM_A:       pa-pt3a
+      PLATFORM_B:       pb-pt3a
+    SUB_PROCESSING_TYPE_3B:
+      PLATFORM_A:       pa-pt3b
+      PLATFORM_B:       pb-pt3b
+
   THUMBNAIL:
     PLATFORM_A:         imgs
     PLATFORM_B:         imgs
@@ -199,12 +226,19 @@ PUBLIC_BUCKETS:
   imgs: 'BROWSE IMAGERY'
 
 PRIVATE_BUCKETS:
-  private-x:
+  pa-pt1:
   - urs_group_name_0
-  private-y:
+  pa-pt2:
   - urs_group_name_1
 
 ```
+
+Given this bucketmap and a bucket prefix of `prfx-d-`: 
+* A URL like `http://domain.com/PROCESSING_TYPE_1/PLATFORM_A/file.dat` would cause TEA to look for `file.dat` in a bucket named `prfx-d-pa-pt1`. Additionally, because that bucket is listed in the `PRIVATE_BUCKETS` section, it will only be available to users belonging to a URS group named `urs_group_name_0`.
+* A URL like `http://domain.com/PROCESSING_TYPE_2/PLATFORM_A/file.dat` would cause TEA to look for `file.dat` in a bucket named `prfx-d-pa-pt2`. It will send response headers defined in that section with the redirect. 
+* A URL like `http://domain.com/THUMBNAIL/PLATFORM_A/file.dat` would cause TEA to look for `file.dat` in a bucket named `prfx-d-imgs`. Because `imgs` is listed under `PUBLIC_BUCKETS`, the file would be public and available to all.
+
+
 ### Config bucket
 You will need a bucket for config and optionally the html templates. This should be in the same region as the stack.
 

build/Jenkinsfile

@@ -79,6 +79,7 @@ pipeline {
         sh """ cd ${WORKSPACE}/lambda  && \
                sed -i -e "s/<BUILD_ID>/${BUILDTAG}/" ./app.py && \
                zip -g ../${CODE_ARCHIVE_FILENAME} ./app.py && \
+               zip -g ../${CODE_ARCHIVE_FILENAME} ./update_lambda.py && \
                zip -g -r ../${CODE_ARCHIVE_FILENAME} ./templates && \
                cd .. && \
                cd rain-api-core  && \
@@ -131,9 +132,9 @@ pipeline {
                        URSAuthCredsSecretName=${URS_CREDS_SECRET_NAME} \
                        ConfigBucket=${BUCKETNAME_PREFIX}config \
                        PermissionsBoundaryName= \
-                       BucketMapFile=bucket_map.yaml \
-                       PublicBucketsFile=public_buckets.yaml \
-                       PrivateBucketsFile=private_buckets.yaml \
+                       BucketMapFile=bucket_map_customheaders.yaml \
+                       PublicBucketsFile="" \
+                       PrivateBucketsFile="" \
                        BucketnamePrefix=${BUCKETNAME_PREFIX} \
                        DownloadRoleArn="" \
                        DownloadRoleInRegionArn="" \
@@ -165,9 +166,9 @@ pipeline {
                        URSAuthCredsSecretName=${URS_CREDS_SECRET_NAME} \
                        ConfigBucket=rain-t-config \
                        PermissionsBoundaryName= \
-                       BucketMapFile=bucket_map_TEST_ngap2.yaml \
-                       PublicBucketsFile=public_buckets_TEA.yaml \
-                       PrivateBucketsFile=private_buckets_TEA.yaml \
+                       BucketMapFile=bucket_map_customheaders.yaml \
+                       PublicBucketsFile="" \
+                       PrivateBucketsFile="" \
                        BucketnamePrefix=${BUCKETNAME_PREFIX_SCND} \
                        DownloadRoleArn=${DOWNLOAD_ROLE_ARN} \
                        DownloadRoleInRegionArn=${DOWNLOAD_ROLE_ARN_INREGION} \

build/dependency_builder.sh

@@ -11,6 +11,7 @@ mkdir -p /depbuild/pkg/python
 
 cd /depbuild/pkg/python || exit
 
+pip3 install --upgrade setuptools
 pip3 install -r /depbuild/in/requirements_tea.txt --target .
 pip3 install -r /depbuild/in/requirements_rain-api-core.txt --target .
 

build/download_test.sh

@@ -13,6 +13,7 @@ if [[ -z $DOMAIN_NAME ]];  then API=$(aws apigateway get-rest-apis --query "item
 echo " >>> APIROOT is $APIROOT"
 
 METADATA_FILE=SA/METADATA_GRD_HS/S1A_EW_GRDM_1SDH_20190206T190846_20190206T190951_025813_02DF0B_781A.iso.xml
+METADATA_FILE_CH=SA/METADATA_GRD_HS_CH/S1A_EW_GRDM_1SDH_20190206T190846_20190206T190951_025813_02DF0B_781A.iso.xml
 METADATA_CHECK='<gco:CharacterString>S1A_EW_GRDM_1SDH_20190206T190846_20190206T190951_025813_02DF0B_781A.iso.xml</gco:CharacterString>'
 BROWSE_FILE=SA/BROWSE/S1A_EW_GRDM_1SDH_20190206T190846_20190206T190951_025813_02DF0B_781A.jpg
 
@@ -80,22 +81,40 @@ grep 'HTTP/1.1 403 Forbidden' /tmp/test8 && grep -q 'HTTP/1.1 403 Forbidden' /tm
 if [ $? -ne 0 ]; then echo; echo " >> Could not verify PRIVATE access was restricted (TEST 8) << "; echo; FC=$((FC+1)); else echo " >>> TEST 8 PASSED"; fi
 
 # Validating objects with prefix
-echo " >>> Validating accessing objects with prefix's"
+echo " >>> Validating accessing objects with prefixes"
 echo " > curl -s -L $APIROOT/SA/BROWSE/dir1/dir2/deepfile.txt | grep 'The file was successfully downloaded'"
 curl -s -L $APIROOT/SA/BROWSE/dir1/dir2/deepfile.txt 2>&1 &> /tmp/test9
 cat /tmp/test9 && grep -q 'The file was successfully downloaded' /tmp/test9
 if [ $? -ne 0 ]; then echo; echo " >> Could not verify prefixed file access (TEST9) << "; echo; FC=$((FC+1)); else echo " >>> TEST 9 PASSED"; fi
 
+
+# Validating custom headers
+echo " >>> Validating custom headers"
+echo " > curl -s -o /dev/null -b /tmp/urscookie.txt -c /tmp/urscookie.txt -D - $APIROOT/$METADATA_FILE_CH | grep 'x-rainheader'"
+curl -s -o /dev/null -b /tmp/urscookie.txt -c /tmp/urscookie.txt -D - $APIROOT/$METADATA_FILE_CH 2>&1 &> /tmp/test10
+cat /tmp/test10 && grep -q 'x-rainheader' /tmp/test10
+if [ $? -ne 0 ]; then echo; echo " >> Could not custom headers (TEST10) << "; echo; FC=$((FC+1)); else echo " >>> TEST 10 PASSED"; fi
+
+#curl -s -o /dev/null -b ./urscookie.txt -c ./urscookie.txt -D - $METADATA_FILE_CH
+
+# Validate /locate handles complex configuration keys
+# LOCATE_OUTPUT should be set e.g. '["SA/OCN", "SA/OCN_CH", "SB/OCN_CN", "SB/OCN_CH"]'
+# LOCATE_BUCKET should be set
+echo " >>> Validating /locate endpoint handles complex bucket map configuration"
+echo " > curl $APIROOT/locate?bucket_name=$LOCATE_BUCKET |grep -F \"$LOCATE_OUTPUT\""
+curl -sv $APIROOT/locate?bucket_name=$LOCATE_BUCKET |grep -F "$LOCATE_OUTPUT"
+if [ $? -ne 0 ]; then echo; echo " >> Could not validate $LOCATE_BUCKET mapping on /locate endpoint"; else echo " >>> TEST 11 PASSED"; fi
+
 # Build Summary
 if [ $FC -le 0 ]; then
    echo " >>> All Tests Passed!"
    echo '{ "schemaVersion": 1, "label": "Tests", "message": "All Tests Passed", "color": "success" }' > /tmp/testresults.json
 elif [ $FC -lt 3 ]; then
    echo " >>> Some Tests Failed"
-   echo '{ "schemaVersion": 1, "label": "Tests", "message": "'$FC'/9 Tests Failed ⚠️", "color": "important" }' > /tmp/testresults.json
+   echo '{ "schemaVersion": 1, "label": "Tests", "message": "'$FC'/10 Tests Failed ⚠️", "color": "important" }' > /tmp/testresults.json
 else
    echo " >>> TOO MANY TEST FAILURES! "
-   echo '{ "schemaVersion": 1, "label": "Tests", "message": "'$FC'/9 Tests Failed ☠", "color": "critical" }' > /tmp/testresults.json
+   echo '{ "schemaVersion": 1, "label": "Tests", "message": "'$FC'/10 Tests Failed ☠", "color": "critical" }' > /tmp/testresults.json
 fi
 
 # Upload test results