Feature: Persist forecasted grant opportunities to DynamoDB #926

TylerHendrickson · 2024-10-07T23:27:00Z

Ticket #861

Description

This PR modifies the PersistGrantsGovXMLDB pipeline step so that it supports persisting forecasted grant opportunities to DynamoDB. The process of taking XML data from an S3 object and persisting it to a DynamoDB item remains basically the same, except that all new records are persisted with an is_forecast boolean field, which can be used to discriminate between OpportunitySynopsisDetail_1_0 (is_forecast=false) and OpportunityForecastDetail_1_0 (is_forecast=true) source XML record types.

Summary of changes:

A new grantRecord interface type is defined, which is implemented by both opportunity and forecast struct types.
XML unmarshaling now uses an xml.Decoder instead of calling xml.Unmarshal(), in order to explicitly map OpportunitySynopsisDetail_1_0 and OpportunityForecastDetail_1_0 elements to their respective Go types.
The dynamodb.go helper functions have been tweaked to only deal with types provided by the DynamoDB SDK rather than custom types defined locally, which makes it easier and cleaner to write unit tests for this file.
Updated unit tests to reflect changes, with particular focus on ensuring that records are saved to DynamoDB with the correct is_forecast attribute value.
Renamed custom Datadog metrics to use more generic "record" terminology instead of "opportunity"
Added an S3-to-lambda bucket notification so that PersistGrantsGovXMLDB is invoked when forecasted grant XML records are uploaded to S3. Note that this has no effect unless forecasted grants are being uploaded by SplitGrantsGovXMLDB.

Testing

Start LocalStack using the included docker-compose.yml file
Deploy to LocalStack (hint: run Taskfile command task local:from-scratch)
Kick off the pipeline using the steps in README (hint: run task local:invoke-DownloadGrantsGovDB)
Check that all records saved to DynamoDB have an is_forecast attribute with the correct value

Automated and Unit Tests

Added Unit tests

Manual tests for Reviewer

Added steps to test feature/functionality manually

Checklist

Provided ticket and description
Provided testing information
Provided adequate test coverage for all new code
Added PR reviewers

github-actions · 2024-10-07T23:31:02Z

Terraform Summary

Step	Result
🖌 Terraform Format & Style	✅
⚙️ Terraform Initialization	✅
🤖 Terraform Validation	✅
📖 Terraform Plan	✅

Output

Validation Output

Success! The configuration is valid.

Plan Output

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  ~ update in-place
-   destroy
-/+ destroy and then create replacement

Terraform will perform the following actions:

  # aws_s3_bucket_notification.grant_prepared_data will be updated in-place
  ~ resource "aws_s3_bucket_notification" "grant_prepared_data" {
        id          = "grantsingest-grantsprepareddata-357150818708-us-west-2"
        # (2 unchanged attributes hidden)

      ~ lambda_function {
          ~ filter_suffix       = "/ffis.org/v1.json" -> "/grants.gov/v2.OpportunityForecastDetail_1_0.xml"
            id                  = "tf-s3-lambda-20240710180849875400000001"
          ~ lambda_function_arn = "arn:aws:lambda:us-west-2:357150818708:function:grants_ingest-PersistFFISData" -> "arn:aws:lambda:us-west-2:357150818708:function:grants_ingest-PersistGrantsGovXMLDB"
            # (1 unchanged attribute hidden)
        }
+       lambda_function {
+           events              = [
+               "s3:ObjectCreated:*",
            ]
+           filter_suffix       = "/ffis.org/v1.json"
+           lambda_function_arn = "arn:aws:lambda:us-west-2:357150818708:function:grants_ingest-PersistFFISData"
        }

        # (2 unchanged blocks hidden)
    }

  # datadog_dashboard.service_dashboard[0] will be updated in-place
  ~ resource "datadog_dashboard" "service_dashboard" {
        id                      = "4jj-dh7-k8z"
        tags                    = []
        # (9 unchanged attributes hidden)

      ~ widget {
            id = 6267115815800911

          ~ group_definition {
                # (3 unchanged attributes hidden)

              ~ widget {
                    id = 7818734294460690

                  ~ timeseries_definition {
                      ~ title          = "Opportunity Records Saved vs Failed" -> "Grant Records Saved vs Failed"
                        # (3 unchanged attributes hidden)

                      ~ request {
                            # (2 unchanged attributes hidden)

                          ~ formula {
                              ~ formula_expression = "records_saved" -> "opportunities_saved + records_saved"
                                # (1 unchanged attribute hidden)

                                # (1 unchanged block hidden)
                            }
                          ~ formula {
                              ~ formula_expression = "records_failed" -> "opportunities_failed + records_failed"
                                # (1 unchanged attribute hidden)

                                # (1 unchanged block hidden)
                            }

                          ~ query {
                              ~ metric_query {
                                  ~ name        = "records_saved" -> "opportunities_saved"
                                    # (2 unchanged attributes hidden)
                                }
                            }
                          ~ query {
                              ~ metric_query {
                                  ~ name        = "records_failed" -> "records_saved"
                                  ~ query       = "sum:grants_ingest.PersistGrantsGovXMLDB.opportunity.failed{$env,$service,$version}.as_count()" -> "sum:grants_ingest.PersistGrantsGovXMLDB.record.saved{$env,$service,$version}.as_count()"
                                    # (1 unchanged attribute hidden)
                                }
                            }
+                           query {
+                               metric_query {
+                                   data_source = "metrics"
+                                   name        = "opportunities_failed"
+                                   query       = "sum:grants_ingest.PersistGrantsGovXMLDB.opportunity.failed{$env,$service,$version}.as_count()"
                                }
                            }
+                           query {
+                               metric_query {
+                                   data_source = "metrics"
+                                   name        = "records_failed"
+                                   query       = "sum:grants_ingest.PersistGrantsGovXMLDB.record.failed{$env,$service,$version}.as_count()"
                                }
                            }
                        }

                        # (1 unchanged block hidden)
                    }

                    # (1 unchanged block hidden)
                }

                # (2 unchanged blocks hidden)
            }

            # (1 unchanged block hidden)
        }

        # (14 unchanged blocks hidden)
    }

  # datadog_metric_metadata.custom["grants_ingest.PersistGrantsGovXMLDB.opportunity.failed"] will be destroyed
  # (because key ["grants_ingest.PersistGrantsGovXMLDB.opportunity.failed"] is not in for_each map)
-   resource "datadog_metric_metadata" "custom" {
-       description     = "Count of opportunity records that failed to be persisted to DynamoDB with Grants.gov data." -> null
-       id              = "grants_ingest.PersistGrantsGovXMLDB.opportunity.failed" -> null
-       metric          = "grants_ingest.PersistGrantsGovXMLDB.opportunity.failed" -> null
-       short_name      = "Failed opportunities" -> null
-       statsd_interval = 0 -> null
-       type            = "gauge" -> null
-       unit            = "record" -> null
    }

  # datadog_metric_metadata.custom["grants_ingest.PersistGrantsGovXMLDB.opportunity.saved"] will be destroyed
  # (because key ["grants_ingest.PersistGrantsGovXMLDB.opportunity.saved"] is not in for_each map)
-   resource "datadog_metric_metadata" "custom" {
-       description     = "Count of opportunity records persisted to DynamoDB with Grants.gov data." -> null
-       id              = "grants_ingest.PersistGrantsGovXMLDB.opportunity.saved" -> null
-       metric          = "grants_ingest.PersistGrantsGovXMLDB.opportunity.saved" -> null
-       short_name      = "Saved opportunities" -> null
-       statsd_interval = 0 -> null
-       type            = "gauge" -> null
-       unit            = "record" -> null
    }

  # module.DownloadFFISSpreadsheet.module.lambda_function.aws_lambda_function.this[0] will be updated in-place
  ~ resource "aws_lambda_function" "this" {
        id                             = "grants_ingest-DownloadFFISSpreadsheet"
      ~ qualified_arn                  = "arn:aws:lambda:us-west-2:357150818708:function:grants_ingest-DownloadFFISSpreadsheet:65" -> (known after apply)
      ~ qualified_invoke_arn           = "arn:aws:apigateway:us-west-2:lambda:path/2015-03-31/functions/arn:aws:lambda:us-west-2:357150818708:function:grants_ingest-DownloadFFISSpreadsheet:65/invocations" -> (known after apply)
        tags                           = {}
      ~ version                        = "65" -> (known after apply)
        # (21 unchanged attributes hidden)

      ~ environment {
          ~ variables = {
              ~ "DD_TAGS"                      = "git.commit.sha:c31998d5a08abeb22c4d77db01e4939a1a581dbe,git.repository_url:github.com/usdigitalresponse/grants-ingest,handlername:downloadffisspreadsheet" -> "git.commit.sha:7215e75f60b97ea94caaec7abcb0652875020c60,git.repository_url:github.com/usdigitalresponse/grants-ingest,handlername:downloadffisspreadsheet"
              ~ "DD_VERSION"                   = "c31998d5a08abeb22c4d77db01e4939a1a581dbe" -> "7215e75f60b97ea94caaec7abcb0652875020c60"
                # (11 unchanged elements hidden)
            }
        }

        # (3 unchanged blocks hidden)
    }

  # module.DownloadFFISSpreadsheet.module.lambda_function.aws_lambda_permission.current_version_triggers["SQSQueueNotification"] must be replaced
-/+ resource "aws_lambda_permission" "current_version_triggers" {
      ~ id                  = "SQSQueueNotification" -> (known after apply)
      ~ qualifier           = "65" # forces replacement -> (known after apply) # forces replacement
+       statement_id_prefix = (known after apply)
        # (4 unchanged attributes hidden)
    }

  # module.DownloadGrantsGovDB.module.lambda_function.aws_lambda_function.this[0] will be updated in-place
  ~ resource "aws_lambda_function" "this" {
        id                             = "grants_ingest-DownloadGrantsGovDB"
      ~ qualified_arn                  = "arn:aws:lambda:us-west-2:357150818708:function:grants_ingest-DownloadGrantsGovDB:65" -> (known after apply)
      ~ qualified_invoke_arn           = "arn:aws:apigateway:us-west-2:lambda:path/2015-03-31/functions/arn:aws:lambda:us-west-2:357150818708:function:grants_ingest-DownloadGrantsGovDB:65/invocations" -> (known after apply)
        tags                           = {}
      ~ version                        = "65" -> (known after apply)
        # (21 unchanged attributes hidden)

      ~ environment {
          ~ variables = {
              ~ "DD_TAGS"                        = "git.commit.sha:c31998d5a08abeb22c4d77db01e4939a1a581dbe,git.repository_url:github.com/usdigitalresponse/grants-ingest,handlername:downloadgrantsgovdb" -> "git.commit.sha:7215e75f60b97ea94caaec7abcb0652875020c60,git.repository_url:github.com/usdigitalresponse/grants-ingest,handlername:downloadgrantsgovdb"
              ~ "DD_VERSION"                     = "c31998d5a08abeb22c4d77db01e4939a1a581dbe" -> "7215e75f60b97ea94caaec7abcb0652875020c60"
                # (13 unchanged elements hidden)
            }
        }

        # (3 unchanged blocks hidden)
    }

  # module.DownloadGrantsGovDB.module.lambda_function.aws_lambda_permission.current_version_triggers["Schedule"] must be replaced
-/+ resource "aws_lambda_permission" "current_version_triggers" {
      ~ id                  = "Schedule" -> (known after apply)
      ~ qualifier           = "65" # forces replacement -> (known after apply) # forces replacement
+       statement_id_prefix = (known after apply)
        # (5 unchanged attributes hidden)
    }

  # module.EnqueueFFISDownload.module.lambda_function.aws_lambda_function.this[0] will be updated in-place
  ~ resource "aws_lambda_function" "this" {
        id                             = "grants_ingest-EnqueueFFISDownload"
      ~ qualified_arn                  = "arn:aws:lambda:us-west-2:357150818708:function:grants_ingest-EnqueueFFISDownload:65" -> (known after apply)
      ~ qualified_invoke_arn           = "arn:aws:apigateway:us-west-2:lambda:path/2015-03-31/functions/arn:aws:lambda:us-west-2:357150818708:function:grants_ingest-EnqueueFFISDownload:65/invocations" -> (known after apply)
        tags                           = {}
      ~ version                        = "65" -> (known after apply)
        # (21 unchanged attributes hidden)

      ~ environment {
          ~ variables = {
              ~ "DD_TAGS"                      = "git.commit.sha:c31998d5a08abeb22c4d77db01e4939a1a581dbe,git.repository_url:github.com/usdigitalresponse/grants-ingest,handlername:enqueueffisdownload" -> "git.commit.sha:7215e75f60b97ea94caaec7abcb0652875020c60,git.repository_url:github.com/usdigitalresponse/grants-ingest,handlername:enqueueffisdownload"
              ~ "DD_VERSION"                   = "c31998d5a08abeb22c4d77db01e4939a1a581dbe" -> "7215e75f60b97ea94caaec7abcb0652875020c60"
                # (11 unchanged elements hidden)
            }
        }

        # (3 unchanged blocks hidden)
    }

  # module.EnqueueFFISDownload.module.lambda_function.aws_lambda_permission.current_version_triggers["S3BucketNotification"] must be replaced
-/+ resource "aws_lambda_permission" "current_version_triggers" {
      ~ id                  = "S3BucketNotification" -> (known after apply)
      ~ qualifier           = "65" # forces replacement -> (known after apply) # forces replacement
+       statement_id_prefix = (known after apply)
        # (5 unchanged attributes hidden)
    }

  # module.ExtractGrantsGovDBToXML.module.lambda_function.aws_lambda_function.this[0] will be updated in-place
  ~ resource "aws_lambda_function" "this" {
        id                             = "grants_ingest-ExtractGrantsGovDBToXML"
      ~ qualified_arn                  = "arn:aws:lambda:us-west-2:357150818708:function:grants_ingest-ExtractGrantsGovDBToXML:65" -> (known after apply)
      ~ qualified_invoke_arn           = "arn:aws:apigateway:us-west-2:lambda:path/2015-03-31/functions/arn:aws:lambda:us-west-2:357150818708:function:grants_ingest-ExtractGrantsGovDBToXML:65/invocations" -> (known after apply)
        tags                           = {}
      ~ version                        = "65" -> (known after apply)
        # (21 unchanged attributes hidden)

      ~ environment {
          ~ variables = {
              ~ "DD_TAGS"                      = "git.commit.sha:c31998d5a08abeb22c4d77db01e4939a1a581dbe,git.repository_url:github.com/usdigitalresponse/grants-ingest,handlername:extractgrantsgovdbtoxml" -> "git.commit.sha:7215e75f60b97ea94caaec7abcb0652875020c60,git.repository_url:github.com/usdigitalresponse/grants-ingest,handlername:extractgrantsgovdbtoxml"
              ~ "DD_VERSION"                   = "c31998d5a08abeb22c4d77db01e4939a1a581dbe" -> "7215e75f60b97ea94caaec7abcb0652875020c60"
                # (11 unchanged elements hidden)
            }
        }

        # (3 unchanged blocks hidden)
    }

  # module.ExtractGrantsGovDBToXML.module.lambda_function.aws_lambda_permission.current_version_triggers["S3BucketNotification"] must be replaced
-/+ resource "aws_lambda_permission" "current_version_triggers" {
      ~ id                  = "S3BucketNotification" -> (known after apply)
      ~ qualifier           = "65" # forces replacement -> (known after apply) # forces replacement
+       statement_id_prefix = (known after apply)
        # (5 unchanged attributes hidden)
    }

  # module.PersistFFISData.module.lambda_function.aws_lambda_function.this[0] will be updated in-place
  ~ resource "aws_lambda_function" "this" {
        id                             = "grants_ingest-PersistFFISData"
      ~ qualified_arn                  = "arn:aws:lambda:us-west-2:357150818708:function:grants_ingest-PersistFFISData:65" -> (known after apply)
      ~ qualified_invoke_arn           = "arn:aws:apigateway:us-west-2:lambda:path/2015-03-31/functions/arn:aws:lambda:us-west-2:357150818708:function:grants_ingest-PersistFFISData:65/invocations" -> (known after apply)
        tags                           = {}
      ~ version                        = "65" -> (known after apply)
        # (21 unchanged attributes hidden)

      ~ environment {
          ~ variables = {
              ~ "DD_TAGS"                       = "git.commit.sha:c31998d5a08abeb22c4d77db01e4939a1a581dbe,git.repository_url:github.com/usdigitalresponse/grants-ingest,handlername:persistffisdata" -> "git.commit.sha:7215e75f60b97ea94caaec7abcb0652875020c60,git.repository_url:github.com/usdigitalresponse/grants-ingest,handlername:persistffisdata"
              ~ "DD_VERSION"                    = "c31998d5a08abeb22c4d77db01e4939a1a581dbe" -> "7215e75f60b97ea94caaec7abcb0652875020c60"
                # (11 unchanged elements hidden)
            }
        }

        # (3 unchanged blocks hidden)
    }

  # module.PersistFFISData.module.lambda_function.aws_lambda_permission.current_version_triggers["S3BucketNotification"] must be replaced
-/+ resource "aws_lambda_permission" "current_version_triggers" {
      ~ id                  = "S3BucketNotification" -> (known after apply)
      ~ qualifier           = "65" # forces replacement -> (known after apply) # forces replacement
+       statement_id_prefix = (known after apply)
        # (5 unchanged attributes hidden)
    }

  # module.PersistGrantsGovXMLDB.module.lambda_artifact.aws_s3_object.lambda_function must be replaced
-/+ resource "aws_s3_object" "lambda_function" {
+       acl                    = (known after apply)
      ~ arn                    = "arn:aws:s3:::grantsingest-lambdaartifacts-357150818708-us-west-2/a30cca2909d3ff0253fabfe01f767ebe.zip" -> (known after apply)
      ~ bucket_key_enabled     = false -> (known after apply)
+       checksum_crc32         = (known after apply)
+       checksum_crc32c        = (known after apply)
+       checksum_sha1          = (known after apply)
+       checksum_sha256        = (known after apply)
      ~ content_type           = "binary/octet-stream" -> (known after apply)
      ~ etag                   = "4ee06784221fce78424a87da9bd9f52a-3" -> (known after apply)
      ~ id                     = "a30cca2909d3ff0253fabfe01f767ebe.zip" -> (known after apply)
      ~ key                    = "a30cca2909d3ff0253fabfe01f767ebe.zip" -> "d9f2e88adbe4b96e9d03fba38336b72a.zip" # forces replacement
+       kms_key_id             = (known after apply)
-       metadata               = {} -> null
      ~ storage_class          = "STANDARD" -> (known after apply)
-       tags                   = {} -> null
      ~ version_id             = "NwTHy.CjQzQpko8fmjCS0AaihcuNqip8" -> (known after apply)
        # (5 unchanged attributes hidden)
    }

  # module.PersistGrantsGovXMLDB.module.lambda_function.aws_lambda_function.this[0] will be updated in-place
  ~ resource "aws_lambda_function" "this" {
        id                             = "grants_ingest-PersistGrantsGovXMLDB"
      ~ last_modified                  = "2024-10-08T18:05:14.000+0000" -> (known after apply)
      ~ qualified_arn                  = "arn:aws:lambda:us-west-2:357150818708:function:grants_ingest-PersistGrantsGovXMLDB:65" -> (known after apply)
      ~ qualified_invoke_arn           = "arn:aws:apigateway:us-west-2:lambda:path/2015-03-31/functions/arn:aws:lambda:us-west-2:357150818708:function:grants_ingest-PersistGrantsGovXMLDB:65/invocations" -> (known after apply)
      ~ s3_key                         = "a30cca2909d3ff0253fabfe01f767ebe.zip" -> "d9f2e88adbe4b96e9d03fba38336b72a.zip"
        tags                           = {}
      ~ version                        = "65" -> (known after apply)
        # (19 unchanged attributes hidden)

      ~ environment {
          ~ variables = {
              ~ "DD_TAGS"                       = "git.commit.sha:c31998d5a08abeb22c4d77db01e4939a1a581dbe,git.repository_url:github.com/usdigitalresponse/grants-ingest,handlername:persistgrantsgovxmldb" -> "git.commit.sha:7215e75f60b97ea94caaec7abcb0652875020c60,git.repository_url:github.com/usdigitalresponse/grants-ingest,handlername:persistgrantsgovxmldb"
              ~ "DD_VERSION"                    = "c31998d5a08abeb22c4d77db01e4939a1a581dbe" -> "7215e75f60b97ea94caaec7abcb0652875020c60"
                # (11 unchanged elements hidden)
            }
        }

        # (3 unchanged blocks hidden)
    }

  # module.PersistGrantsGovXMLDB.module.lambda_function.aws_lambda_permission.current_version_triggers["S3BucketNotification"] must be replaced
-/+ resource "aws_lambda_permission" "current_version_triggers" {
      ~ id                  = "S3BucketNotification" -> (known after apply)
      ~ qualifier           = "65" # forces replacement -> (known after apply) # forces replacement
+       statement_id_prefix = (known after apply)
        # (5 unchanged attributes hidden)
    }

  # module.PublishGrantEvents.module.lambda_function.aws_lambda_function.this[0] will be updated in-place
  ~ resource "aws_lambda_function" "this" {
        id                             = "grants_ingest-PublishGrantEvents"
      ~ qualified_arn                  = "arn:aws:lambda:us-west-2:357150818708:function:grants_ingest-PublishGrantEvents:66" -> (known after apply)
      ~ qualified_invoke_arn           = "arn:aws:apigateway:us-west-2:lambda:path/2015-03-31/functions/arn:aws:lambda:us-west-2:357150818708:function:grants_ingest-PublishGrantEvents:66/invocations" -> (known after apply)
        tags                           = {}
      ~ version                        = "66" -> (known after apply)
        # (21 unchanged attributes hidden)

      ~ environment {
          ~ variables = {
              ~ "DD_TAGS"                      = "git.commit.sha:c31998d5a08abeb22c4d77db01e4939a1a581dbe,git.repository_url:github.com/usdigitalresponse/grants-ingest,handlername:publishgrantevents" -> "git.commit.sha:7215e75f60b97ea94caaec7abcb0652875020c60,git.repository_url:github.com/usdigitalresponse/grants-ingest,handlername:publishgrantevents"
              ~ "DD_VERSION"                   = "c31998d5a08abeb22c4d77db01e4939a1a581dbe" -> "7215e75f60b97ea94caaec7abcb0652875020c60"
                # (11 unchanged elements hidden)
            }
        }

        # (3 unchanged blocks hidden)
    }

  # module.PublishGrantEvents.module.lambda_function.aws_lambda_permission.current_version_triggers["dynamodb"] must be replaced
-/+ resource "aws_lambda_permission" "current_version_triggers" {
      ~ id                  = "dynamodb" -> (known after apply)
      ~ qualifier           = "66" # forces replacement -> (known after apply) # forces replacement
+       statement_id_prefix = (known after apply)
        # (5 unchanged attributes hidden)
    }

  # module.ReceiveFFISEmail.module.lambda_function.aws_lambda_function.this[0] will be updated in-place
  ~ resource "aws_lambda_function" "this" {
        id                             = "grants_ingest-ReceiveFFISEmail"
      ~ qualified_arn                  = "arn:aws:lambda:us-west-2:357150818708:function:grants_ingest-ReceiveFFISEmail:64" -> (known after apply)
      ~ qualified_invoke_arn           = "arn:aws:apigateway:us-west-2:lambda:path/2015-03-31/functions/arn:aws:lambda:us-west-2:357150818708:function:grants_ingest-ReceiveFFISEmail:64/invocations" -> (known after apply)
        tags                           = {}
      ~ version                        = "64" -> (known after apply)
        # (21 unchanged attributes hidden)

      ~ environment {
          ~ variables = {
              ~ "DD_TAGS"                        = "git.commit.sha:c31998d5a08abeb22c4d77db01e4939a1a581dbe,git.repository_url:github.com/usdigitalresponse/grants-ingest,handlername:receiveffisemail" -> "git.commit.sha:7215e75f60b97ea94caaec7abcb0652875020c60,git.repository_url:github.com/usdigitalresponse/grants-ingest,handlername:receiveffisemail"
              ~ "DD_VERSION"                     = "c31998d5a08abeb22c4d77db01e4939a1a581dbe" -> "7215e75f60b97ea94caaec7abcb0652875020c60"
                # (12 unchanged elements hidden)
            }
        }

        # (3 unchanged blocks hidden)
    }

  # module.ReceiveFFISEmail.module.lambda_function.aws_lambda_permission.current_version_triggers["S3BucketNotification"] must be replaced
-/+ resource "aws_lambda_permission" "current_version_triggers" {
      ~ id                  = "S3BucketNotification" -> (known after apply)
      ~ qualifier           = "64" # forces replacement -> (known after apply) # forces replacement
+       statement_id_prefix = (known after apply)
        # (5 unchanged attributes hidden)
    }

  # module.SplitFFISSpreadsheet.module.lambda_function.aws_lambda_function.this[0] will be updated in-place
  ~ resource "aws_lambda_function" "this" {
        id                             = "grants_ingest-SplitFFISSpreadsheet"
      ~ qualified_arn                  = "arn:aws:lambda:us-west-2:357150818708:function:grants_ingest-SplitFFISSpreadsheet:65" -> (known after apply)
      ~ qualified_invoke_arn           = "arn:aws:apigateway:us-west-2:lambda:path/2015-03-31/functions/arn:aws:lambda:us-west-2:357150818708:function:grants_ingest-SplitFFISSpreadsheet:65/invocations" -> (known after apply)
        tags                           = {}
      ~ version                        = "65" -> (known after apply)
        # (21 unchanged attributes hidden)

      ~ environment {
          ~ variables = {
              ~ "DD_TAGS"                          = "git.commit.sha:c31998d5a08abeb22c4d77db01e4939a1a581dbe,git.repository_url:github.com/usdigitalresponse/grants-ingest,handlername:splitffisspreadsheet" -> "git.commit.sha:7215e75f60b97ea94caaec7abcb0652875020c60,git.repository_url:github.com/usdigitalresponse/grants-ingest,handlername:splitffisspreadsheet"
              ~ "DD_VERSION"                       = "c31998d5a08abeb22c4d77db01e4939a1a581dbe" -> "7215e75f60b97ea94caaec7abcb0652875020c60"
                # (14 unchanged elements hidden)
            }
        }

        # (3 unchanged blocks hidden)
    }

  # module.SplitFFISSpreadsheet.module.lambda_function.aws_lambda_permission.current_version_triggers["S3BucketNotification"] must be replaced
-/+ resource "aws_lambda_permission" "current_version_triggers" {
      ~ id                  = "S3BucketNotification" -> (known after apply)
      ~ qualifier           = "65" # forces replacement -> (known after apply) # forces replacement
+       statement_id_prefix = (known after apply)
        # (5 unchanged attributes hidden)
    }

  # module.SplitGrantsGovXMLDB.module.lambda_artifact.aws_s3_object.lambda_function must be replaced
-/+ resource "aws_s3_object" "lambda_function" {
+       acl                    = (known after apply)
      ~ arn                    = "arn:aws:s3:::grantsingest-lambdaartifacts-357150818708-us-west-2/49619436296c4287a68f6bd8fc28c125.zip" -> (known after apply)
      ~ bucket_key_enabled     = false -> (known after apply)
+       checksum_crc32         = (known after apply)
+       checksum_crc32c        = (known after apply)
+       checksum_sha1          = (known after apply)
+       checksum_sha256        = (known after apply)
      ~ content_type           = "binary/octet-stream" -> (known after apply)
      ~ etag                   = "ec011397e317df42edce7ed2137cd498-3" -> (known after apply)
      ~ id                     = "49619436296c4287a68f6bd8fc28c125.zip" -> (known after apply)
      ~ key                    = "49619436296c4287a68f6bd8fc28c125.zip" -> "9393e888662690c9418e85eb134ae42d.zip" # forces replacement
+       kms_key_id             = (known after apply)
-       metadata               = {} -> null
      ~ storage_class          = "STANDARD" -> (known after apply)
-       tags                   = {} -> null
      ~ version_id             = "JnIIh_CzcmrwvJk1i0wnPR_K6gS7YGtZ" -> (known after apply)
        # (5 unchanged attributes hidden)
    }

  # module.SplitGrantsGovXMLDB.module.lambda_function.aws_lambda_function.this[0] will be updated in-place
  ~ resource "aws_lambda_function" "this" {
        id                             = "grants_ingest-SplitGrantsGovXMLDB"
      ~ last_modified                  = "2024-10-08T18:05:14.000+0000" -> (known after apply)
      ~ qualified_arn                  = "arn:aws:lambda:us-west-2:357150818708:function:grants_ingest-SplitGrantsGovXMLDB:65" -> (known after apply)
      ~ qualified_invoke_arn           = "arn:aws:apigateway:us-west-2:lambda:path/2015-03-31/functions/arn:aws:lambda:us-west-2:357150818708:function:grants_ingest-SplitGrantsGovXMLDB:65/invocations" -> (known after apply)
      ~ s3_key                         = "49619436296c4287a68f6bd8fc28c125.zip" -> "9393e888662690c9418e85eb134ae42d.zip"
        tags                           = {}
      ~ version                        = "65" -> (known after apply)
        # (19 unchanged attributes hidden)

      ~ environment {
          ~ variables = {
              ~ "DD_TAGS"                          = "git.commit.sha:c31998d5a08abeb22c4d77db01e4939a1a581dbe,git.repository_url:github.com/usdigitalresponse/grants-ingest,handlername:splitgrantsgovxmldb" -> "git.commit.sha:7215e75f60b97ea94caaec7abcb0652875020c60,git.repository_url:github.com/usdigitalresponse/grants-ingest,handlername:splitgrantsgovxmldb"
              ~ "DD_VERSION"                       = "c31998d5a08abeb22c4d77db01e4939a1a581dbe" -> "7215e75f60b97ea94caaec7abcb0652875020c60"
+               "MAX_SPLIT_FORECAST_RECORDS"       = "-1"
+               "MAX_SPLIT_OPPORTUNITY_RECORDS"    = "-1"
                # (17 unchanged elements hidden)
            }
        }

        # (3 unchanged blocks hidden)
    }

  # module.SplitGrantsGovXMLDB.module.lambda_function.aws_lambda_permission.current_version_triggers["S3BucketNotification"] must be replaced
-/+ resource "aws_lambda_permission" "current_version_triggers" {
      ~ id                  = "S3BucketNotification" -> (known after apply)
      ~ qualifier           = "65" # forces replacement -> (known after apply) # forces replacement
+       statement_id_prefix = (known after apply)
        # (5 unchanged attributes hidden)
    }

Plan: 12 to add, 12 to change, 14 to destroy.

Pusher: @TylerHendrickson, Action: pull_request_target, Workflow: Continuous Integration

TylerHendrickson and others added 8 commits October 3, 2024 00:37

Add types with grantRecord interface

98b4c00

Refactor handler to use grantRecord interface

9e743a3

Simplify DynamoDB helpers

3445502

Remove debugging panic

445f818

Updated tests for handler.go

587f865

Add helper function to simplify test body

eaf75ca

Update custom metric names in Datadog

d09bf62

Add tests for grantRecord-to-DDB conversions

cc592b7

TylerHendrickson self-assigned this Oct 7, 2024

github-actions bot added enhancement New feature or request go Pull requests that update Go code terraform Pull requests that update Terraform code labels Oct 7, 2024

TylerHendrickson requested review from as1729 and a team October 8, 2024 01:58

TylerHendrickson added 4 commits October 8, 2024 10:19

Merge branch 'main' into feat/861/persist-forecasts-to-ddb

36caa82

Add S3 trigger for forecasted grant persistence

12ae52a

Merge branch 'main' into feat/861/persist-forecasts-to-ddb

fdb2cfe

Merge branch 'main' into feat/861/persist-forecasts-to-ddb

7215e75

TylerHendrickson merged commit 962ac1d into main Oct 31, 2024
17 of 19 checks passed

TylerHendrickson deleted the feat/861/persist-forecasts-to-ddb branch October 31, 2024 17:42

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature: Persist forecasted grant opportunities to DynamoDB #926

Feature: Persist forecasted grant opportunities to DynamoDB #926

TylerHendrickson commented Oct 7, 2024 •

edited

Loading

github-actions bot commented Oct 7, 2024 •

edited

Loading

Feature: Persist forecasted grant opportunities to DynamoDB #926

Feature: Persist forecasted grant opportunities to DynamoDB #926

Conversation

TylerHendrickson commented Oct 7, 2024 • edited Loading

Ticket #861

Description

Testing

Automated and Unit Tests

Manual tests for Reviewer

Checklist

github-actions bot commented Oct 7, 2024 • edited Loading

Terraform Summary

Output

TylerHendrickson commented Oct 7, 2024 •

edited

Loading

github-actions bot commented Oct 7, 2024 •

edited

Loading