Errors while deploying egress backend app

[manics]

When I attempt to deploy the data egress backend I keep seeing an error:

17:47:44 | CREATE_FAILED        | Custom::EmailConfigurationSetEventDestination
| EmailConfiguration...d/Resource/Default
Received response status [FAILED] from custom resource. Message returned: Could
not publish message to SNS topic <arn:aws:sns:eu-west-2:<AWS_ACCOUNT_ID>:EgressAppB
ackend-SESMonitoringNotificationsEF268111-AOQ8X3J79LKB>. (RequestId: 8a2b7147-6
64c-4962-a641-1464070f9b38)

This matches the logwatch logs:

2022-05-25T17:47:38.362Z	0fdf8891-294b-4b1b-8d9a-36a40c6f5ad0	INFO	BadRequestException: Could not publish message to SNS topic <arn:aws:sns:eu-west-2:<AWS_ACCOUNT_ID>:EgressAppBackend-SESMonitoringNotificationsEF268111-AOQ8X3J79LKB>.
    at Object.extractError (/tmp/node_modules/aws-sdk/lib/protocol/json.js:52:27)
    at Request.extractError (/tmp/node_modules/aws-sdk/lib/protocol/rest_json.js:49:8)
    at Request.callListeners (/tmp/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
    at Request.emit (/tmp/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
    at Request.emit (/tmp/node_modules/aws-sdk/lib/request.js:686:14)
    at Request.transition (/tmp/node_modules/aws-sdk/lib/request.js:22:10)
    at AcceptorStateMachine.runTo (/tmp/node_modules/aws-sdk/lib/state_machine.js:14:12)
    at /tmp/node_modules/aws-sdk/lib/state_machine.js:26:10
    at Request.<anonymous> (/tmp/node_modules/aws-sdk/lib/request.js:38:9)
    at Request.<anonymous> (/tmp/node_modules/aws-sdk/lib/request.js:688:12) {
  code: 'BadRequestException',
  time: 2022-05-25T17:47:38.252Z,
  requestId: 'c69951d7-98a3-476a-90b8-530de635c781',
  statusCode: 400,
  retryable: false,
  retryDelay: 21.041289007606157
}

This was run using the deployment EC2 instance with these additional permissions: #72

[manics]

Is this a race condition- is the custom resource trying to notify SNS in the same stack?

[sylnkt]

While I have encountered race conditions with CDK apps before, I don't believe this is one.

There was a very recent change from the security review: 3de0a3d#diff-c28d34187f2bc27b26c2942353c46f2e7568c8316cdc79b83c86ddf62d207698R151 , which added encryption to that SNS topic. I think we also need to add some additional permissions (KMS key related) for the resource throwing the error for it to publish to that topic now.

We will prioritise fixing this in the morning to make the deployment work with this encryption enhancement too.

[sylnkt]

Adding

        sns_kms_key.add_to_resource_policy(
            iam.PolicyStatement(
                effect=iam.Effect.ALLOW,
                principals=[
                    iam.ServicePrincipal("ses.amazonaws.com"),
                ],
                resources=[
                    "*",
                ],
                actions=[
                  "kms:GenerateDataKey*",
                  "kms:Decrypt",
                ],
            )
        )

to the egress_backend_stack.py is one way to fix this (just tested cdk deploy and it completed all steps).

I don't know if there's a more permission restrictive way to do it, I'll let @awskaran advise.

[awskaran]

Yes the commit from security review broke it, i was under an impression, cdk would smartly grant the permission on the KMS. I will add the required permission explicitly.

[manics]

#74 fixes the original issue, but there's a follow-up problem that needs to be investigated:

09:29:40 | CREATE_FAILED        | Custom::EmailConfigurationSetEventDestination
| EmailConfiguration...nationProd5F0E58F0
Received response status [FAILED] from custom resource. Message returned: User:
arn:aws:sts::<AWS_ACCOUNT_ID>:assumed-role/EgressAppBackend-AWS679f53fac002430cb0da
5b7982bd22-769NXUX4VELD/EgressAppBackend-AWS679f53fac002430cb0da5b7982bd22-wIPV
RylhsDgl is not authorized to perform: ses:CreateConfigurationSetEventDestinati
on on resource: arn:aws:ses:eu-west-2:<AWS_ACCOUNT_ID>:configuration-set/egress_app
_configuration_set_Prod because no identity-based policy allows the ses:CreateC
onfigurationSetEventDestination action (RequestId: 05159908-c27e-43dd-88ab-75cf
b31786bc)

I made this change:

diff --git a/src/components/egress_app_backend/egress_backend/components/email_configuration_set_event_dest/email_configuration_set_event_dest_cr.py b/src/components/egress_app_backend/egress_backend/components/email_configuration_set_event_dest/email_configuration_set_event_dest_cr.py
index 96a1cbc..d06cfd1 100644
--- a/src/components/egress_app_backend/egress_backend/components/email_configuration_set_event_dest/email_configuration_set_event_dest_cr.py
+++ b/src/components/egress_app_backend/egress_backend/components/email_configuration_set_event_dest/email_configuration_set_event_dest_cr.py
@@ -46,6 +46,7 @@ class EmailConfigurationSetEventDestinationCustomResource(cdk.Construct):
                     "logs:CreateLogGroup",
                     "logs:CreateLogStream",
                     "logs:PutLogEvents",
+                    "ses:CreateConfigurationSetEventDestination",
                 ],
             )
         )