Merge pull request #325 from awslabs/master

RELEASE 0.8.3

README.rst

@@ -1,6 +1,6 @@
 rdk
 ===
-Rule Development Kit 
+Rule Development Kit
 We are greatly appreciated feedback and bug reports at [email protected]! You may also create an issue on this repo.
 
 The RDK is designed to support a "Compliance-as-Code" workflow that is intuitive and productive.  It abstracts away much of the undifferentiated heavy lifting associated with deploying AWS Config rules backed by custom lambda functions, and provides a streamlined develop-deploy-monitor iterative process.
@@ -65,6 +65,13 @@ Running ``init`` subsequent times will validate your AWS Config setup and re-cre
 
   $ rdk init --skip-code-bucket-creation
 
+- If you want rdk to create/update and upload the rdklib-layer for you, then use **--generate-lambda-layer** argument. In supported regions, rdk will deploy the layer using the Serverless Application Repository, otherwise it will build a local lambda layer archive and upload it for use:::
+
+  $ rdk init --generate-lambda-layer 
+- If you want rdk to give a custom name to the lambda layer for you, then use **--custom-layer-namer** argument. The Serverless Application Repository currently cannot be used for custom lambda layers.:::
+
+  $ rdk init --generate-lambda-layer --custom-layer-name <LAYER_NAME>
+
 Create Rules
 ------------
 In your working directory, use the ``create`` command to start creating a new custom rule.  You must specify the runtime for the lambda function that will back the Rule, and you can also specify a resource type (or comma-separated list of types) that the Rule will evaluate or a maximum frequency for a periodic rule.  This will add a new directory for the rule and populate it with several files, including a skeleton of your Lambda code.
@@ -158,12 +165,12 @@ Once you have completed your compliance validation code and set your Rule's conf
   Waiting for CloudFormation stack operation to complete...
   Config deploy complete.
 
-The exact output will vary depending on Lambda runtime.  You can use the --all flag to deploy all of the rules in your working directory.
+The exact output will vary depending on Lambda runtime.  You can use the --all flag to deploy all of the rules in your working directory.  If you used the --generate-lambda-layer flag in rdk init, use the --generated-lambda-layer flag for rdk deploy.
 
 Deploy Organization Rule
 ------------------------
 You can also deploy the Rule to your AWS Organization using the ``deploy-organization`` command.
-For successful evaluation of custom rules in child accounts, please make sure you do one of the following: 
+For successful evaluation of custom rules in child accounts, please make sure you do one of the following:
 
 1. Set ASSUME_ROLE_MODE in Lambda code to True, to get the lambda to assume the Role attached on the Config Service and confirm that the role trusts the master account where the Lambda function is going to be deployed.
 2. Set ASSUME_ROLE_MODE in Lambda code to True, to get the lambda to assume a custom role and define an optional parameter with key as ExecutionRoleName and set the value to your custom role name; confirm that the role trusts the master account of the organization where the Lambda function will be deployed.
@@ -179,7 +186,7 @@ For successful evaluation of custom rules in child accounts, please make sure yo
   ...
   Waiting for CloudFormation stack operation to complete...
   Config deploy complete.
-  
+
 The exact output will vary depending on Lambda runtime.  You can use the --all flag to deploy all of the rules in your working directory.
 This command uses 'PutOrganizationConfigRule' API for the rule deployment. If a new account joins an organization, the rule is deployed to that account. When an account leaves an organization, the rule is removed. Deployment of existing organizational AWS Config Rules will only be retried for 7 hours after an account is added to your organization if a recorder is not available. You are expected to create a recorder if one doesn't exist within 7 hours of adding an account to your organization.
 
@@ -239,10 +246,10 @@ It is now possible to define a resource type that is not yet supported by rdk. T
   Skip-Supported-Resource-Check Flag set (--skip-supported-resource-check), ignoring missing resource type error.
   Running create!
   Local Rule files created.
-  
+
 Custom Lambda Function Name
 ---------------------------
-As of version 0.7.14, instead of defaulting the lambda function names to 'RDK-Rule-Function-<RULE_NAME>' it is possible to customize the name for the Lambda function to any 64 characters string as per Lambda's naming standards using the optional '--custom-lambda-name' flag while performing rdk create. This opens up new features like : 
+As of version 0.7.14, instead of defaulting the lambda function names to 'RDK-Rule-Function-<RULE_NAME>' it is possible to customize the name for the Lambda function to any 64 characters string as per Lambda's naming standards using the optional '--custom-lambda-name' flag while performing rdk create. This opens up new features like :
 
 1. Longer config rule name.
 2. Custom lambda function naming as per personal or enterprise standards.
@@ -252,7 +259,7 @@ As of version 0.7.14, instead of defaulting the lambda function names to 'RDK-Ru
   $ rdk create MyLongerRuleName --runtime python3.8 --resource-types AWS::EC2::Instance --custom-lambda-name custom-prefix-for-MyLongerRuleName
   Running create!
   Local Rule files created.
-  
+
 The above example would create files with config rule name as 'MyLongerRuleName' and lambda function with the name 'custom-prefix-for-MyLongerRuleName' instead of 'RDK-Rule-Function-MyLongerRuleName'
 
 RuleSets
@@ -293,10 +300,26 @@ The RDK is able to deploy AWS Managed Rules.
 
 To do so, create a rule using "rdk create" and provide a valid SourceIdentifier via the --source-identifier CLI option. The list of Managed Rules can be found here: https://docs.aws.amazon.com/config/latest/developerguide/managed-rules-by-aws-config.html, and note that the Identifier can be obtained by replacing the dashes with underscores and using all capitals (for example, the "guardduty-enabled-centralized" rule has the SourceIdentifier "GUARDDUTY_ENABLED_CENTRALIZED").  Just like custom Rules you will need to specify source events and/or a maximum evaluation frequency, and also pass in any Rule parameters.  The resulting Rule directory will contain only the parameters.json file, but using `rdk deploy` or `rdk create-rule-template` can be used to deploy the Managed Rule like any other Custom Rule.
 
+Deploying Rules Across Multiple Regions
+---------------------------------------
+The RDK is able to run init/deploy/undeploy across multiple regions with a `rdk -f <region file> -t <region set>`
+
+If no region group is specified, rdk will deploy to the `default` region set
+
+To create a sample starter region group, run `rdk create-region-set` to specify the filename, add the `-o <region set output file name>` this will create a region set with the following tests and regions `"default":["us-east-1","us-west-1","eu-north-1","ap-east-1"],"aws-cn-region-set":["cn-north-1","cn-northwest-1"]`
+
+Using RDK to Generate a Lambda Layer in a region (Python3)
+----------------------------------------------------------
+By default `rdk init --generate-lambda-layer` will generate an rdklib lambda layer while running init in whatever region it is run, to force re-generation of the layer, run `rdk init --generate-lambda-layer` again over a region
+
+To use this generated lambda layer, add the flag `--generated-lambda-layer` when running `rdk deploy`. For example: `rdk -f regions.yaml deploy LP3_TestRule_P36_lib --generated-lambda-layer`
+
+If you created layer with a custom name (by running `rdk init --custom-lambda-layer`, add a similar `custom-lambda-layer` flag when running deploy.
+
 Contributing
 ============
 
-email us at [email protected] if you have any questions. We are happy to help and discuss. 
+email us at [email protected] if you have any questions. We are happy to help and discuss.
 
 Authors
 =======

docs/getting_started.rst

@@ -221,3 +221,42 @@ Rules can be added to or removed from RuleSets using the `add` and `remove` subc
   RSTest removed from RuleSet AnotherRuleSet
 
 RuleSets are a convenient way to maintain a single repository of Config Rules that may need to have subsets of them deployed to different environments.  For example your development environment may contain some of the Rules that you run in Production but not all of them; RuleSets gives you a way to identify and selectively deploy the appropriate Rules to each environment.
+
+
+Region Sets
+~~~~~~~~~~~
+`rdk init`, `rdk deploy`, and `rdk undeploy` subcommands now support running across multiple regions in parallel using region sets defined in a yaml file.
+
+To run a subcommand with a region set, pass in the region set yaml file and the specific region set to run through.
+
+::
+
+    $ rdk -f regions.yaml --region-set region-set-1 undeploy CUSTOM_RULE
+    Deleting rules in the following regions: ['sa-east-1', 'us-east-1'].
+    Delete specified Rules and Lambda Functions from your AWS Account? (y/N): y
+    [sa-east-1] Running un-deploy!
+    [us-east-1] Running un-deploy!
+    [us-east-1] Rule removal initiated. Waiting for Stack Deletion to complete.
+    [sa-east-1] Rule removal initiated. Waiting for Stack Deletion to complete.
+    [us-east-1] CloudFormation stack operation complete.
+    [us-east-1] Rule removal complete, but local files have been preserved.
+    [us-east-1] To re-deploy, use the 'deploy' command.
+    [sa-east-1] CloudFormation stack operation complete.
+    [sa-east-1] Rule removal complete, but local files have been preserved.
+    [sa-east-1] To re-deploy, use the 'deploy' command.
+
+Example region set file:
+
+::
+
+    default:
+      - us-west-1
+      - us-west-2
+    region-set-1:
+      - sa-east-1
+      - us-east-1
+    region-set-2:
+      - ap-southeast-1
+      - eu-central-1
+      - sa-east-1
+      - us-east-1

docs/requirements.txt

@@ -2,3 +2,5 @@ Sphinx==1.7.8
 sphinx-argparse==0.2.5
 sphinx-rtd-theme==0.4.3
 sphinxcontrib-websupport==1.1.0
+PyYAML==5.4.1
+

rdk/__init__.py

@@ -6,4 +6,4 @@
 #
 #    or in the "license" file accompanying this file. This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
 
-MY_VERSION = "0.8.2"
+MY_VERSION = "0.8.3"

rdk/cli.py

@@ -6,7 +6,10 @@
 #
 #    or in the "license" file accompanying this file. This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
 
+import concurrent.futures
+import copy
 import six
+import time
 
 if six.PY2:
     import rdk
@@ -19,5 +22,33 @@ def main():
     my_parser = rdk.get_command_parser()
     args = my_parser.parse_args()
     my_rdk = rdk.rdk(args)
+
+    if args.region_file:
+        if args.command in ['init', 'deploy', 'undeploy']:
+            regions = rdk.parse_region_file(args)
+            print(f"{args.command.capitalize()}ing rules in the following regions: {regions}.")
+            if args.command == 'undeploy' and "--force" not in args.command_args:
+                my_input = input("Delete specified Rules and Lambda Functions from your AWS Account? (y/N): ")
+                while my_input.lower() not in ["y", "n"]:
+                    my_input = input(f"Invalid input: {my_input}. Please enter either 'y' or 'n': ")
+                if my_input.lower() == "y":
+                    vars(args)["command_args"].append("--force")
+                elif my_input.lower() == "n" or my_input == "":
+                    exit(0)
+
+            args_list = []
+            for region in regions:
+                vars(args)['region'] = region
+                args_list.append(copy.copy(args))
+
+            data = []
+            with concurrent.futures.ProcessPoolExecutor(max_workers=16) as executor:
+                future_run_multi_region = {executor.submit(rdk.run_multi_region, args): args for args in args_list}
+                for future in concurrent.futures.as_completed(future_run_multi_region):
+                    data.append(future.result())
+            exit(0)
+        else:
+            my_parser.error("Command must be 'init', 'deploy', or 'undeploy' when --region-file argument is provided.")
+
     return_val = my_rdk.process_command()
     exit(return_val)