Compatibility

As of Module version v1.8.0, Plaintext Ports are no longer allowed through the module's security groups for both the broker and zookeeper

This means that from module v1.8.0 onwards the minimum supported Kafka version is 2.5.1

Should you require an older version of Kafka than you should use module version v1.7.x. However, the downside is that plaintext ports will be allowed on the older module version

Usage

MSK Cluster

module "msk_cluster" {
  source = "git::https://github.com/UKHomeOffice/acp-tf-msk-cluster?ref=master"

  name                   = "msktestcluster"
  msk_instance_type      = "kafka.m5.large"
  kafka_version          = "2.8.1"
  environment            = var.environment
  number_of_broker_nodes = "3"
  subnet_ids             = data.aws_subnet_ids.compute.ids
  vpc_id                 = var.vpc_id
  ebs_volume_size        = "50"
  cidr_blocks            = values(var.compute_cidrs)
  # certificateauthority = true (This will fail on merge the first time it's executed, this is expected. Install the CA in the AWS console then restart the merge.)
  # or
  # ca_arn               = [module.<existing_cert>.ca_certificate_arn]
}

MSK Cluster with config

module "msk_cluster_with_config" {
  source = "git::https://github.com/UKHomeOffice/acp-tf-msk-cluster?ref=master"

  name                        = "msktestclusterwithconfig"
  msk_instance_type           = "kafka.m5.large"
  kafka_version               = "2.8.1"
  environment                 = var.environment
  number_of_broker_nodes      = "3"
  subnet_ids                  = data.aws_subnet_ids.compute.ids
  vpc_id                      = var.vpc_id
  ebs_volume_size             = "50"
  cidr_blocks                 = values(var.compute_cidrs)
  # certificateauthority      = true (This will fail on merge the first time it's executed, this is expected. Install the CA in the AWS console then restart the merge.)
  # or
  # ca_arn                    = [module.<existing_cert>.ca_certificate_arn]
  config_name                 = "test-msk-config"
  config_kafka_versions       = ["2.8.1"]
  config_description          = "Test MSK configuration"

  config_server_properties = <<PROPERTIES
 auto.create.topics.enable = true
 delete.topic.enable = true
 PROPERTIES
}

Requirements

Name	Version
terraform	>= 1.0
aws	~> 3.0

Providers

Name	Version
aws	3.76.1

Modules

Name	Source	Version
self_serve_access_keys	git::https://github.com/UKHomeOffice/acp-tf-self-serve-access-keys	v0.1.0

Resources

Name	Type
aws_acmpca_certificate_authority.msk_kafka_ca_with_config	resource
aws_acmpca_certificate_authority.msk_kafka_with_ca	resource
aws_appautoscaling_policy.msk_appautoscaling_policy	resource
aws_appautoscaling_target.msk_appautoscaling_target	resource
aws_iam_policy.acmpca_policy_with_msk_policy	resource
aws_iam_policy.msk_iam_policy	resource
aws_iam_policy_attachment.msk_acmpca_iam_policy_attachment	resource
aws_iam_policy_attachment.msk_iam_policy_attachment	resource
aws_iam_user.msk_acmpca_iam_user	resource
aws_iam_user.msk_iam_user	resource
aws_iam_policy.msk_iam_authentication	resource
aws_iam_policy_attachment.msk_iam_authentication_policy	resource
aws_kms_alias.msk_cluster_kms_alias	resource
aws_kms_key.kms	resource
aws_msk_cluster.msk_kafka	resource
aws_msk_cluster.msk_kafka_with_config	resource
aws_msk_configuration.msk_kafka_config	resource
aws_security_group.sg_msk	resource
aws_caller_identity.current	data source
aws_iam_policy_document.kms_key_policy_document	data source

Inputs

Name	Description	Type	Default	Required
acmpca_iam_user_name	The name of the IAM user assigned to the created AWS Private CA	string	""	no
ca_arn	ARN of the AWS managed CA to attach to the MSK cluster	list(string)	[]	no
certificateauthority	Should a CA be created with the MSK cluster?	bool	false	no
cidr_blocks	The CIDR blocks that the MSK cluster allows ingress connections from	list(string)	[ "0.0.0.0/0" ]	no
config_arn	ARN of the MSK configuration to attach to the MSK cluster	string	""	no
config_description	The description of the MSK configuration	string	""	no
config_kafka_versions	A list of Kafka versions that the configuration supports	list(string)	[]	no
config_name	Name of the MSK configuration to attach to the MSK cluster	string	""	no
config_revision	The revision of the MSK configuration to use	string	""	no
config_server_properties	The properties to set on the MSK cluster. Omitted properties are set to a default value	string	""	no
ebs_volume_size	The MSK cluster EBS volume size for each broker	any	n/a	yes
email_addresses	A list of email addresses for key rotation notifications.	list(string)	[]	no
encryption_at_rest_kms_key_arn	Use to set custom KMS key to encrypt data written to EBS volume	any	null	no
enhanced_monitoring	The desired enhanced MSK CloudWatch monitoring level	string	"DEFAULT"	no
environment	The environment the MSK cluster is running in i.e. dev, prod etc	any	n/a	yes
iam_authentication	Enables IAM client authentication.	bool	"false"	no
kafka_version	The Kafka version for the AWS MSK cluster	string	"2.2.1"	no
key_rotation	Enable email notifications for old IAM keys.	string	"true"	no
logging_broker_s3	Configuration block for Broker Logs settings for s3.	object({ enabled = bool bucket = string prefix = string })	null	no
msk_instance_type	The MSK cluster instance type	any	n/a	yes
name	Name of the MSK cluster	any	n/a	yes
number_of_broker_nodes	The number of broker nodes running in the MSK cluster	any	n/a	yes
prometheus_jmx_exporter_enabled	Enable Prometheus open monitoring for the JMX exporter	bool	false	no
prometheus_node_exporter_enabled	Enable Prometheus open monitoring for the node exporter	bool	false	no
storage_autoscaling_max_capacity	The MSK cluster EBS maximum volume size for each broker. Value between 1 and 16384.	number	1	no
storage_autoscaling_threshold	The percentage threshold that needs to be exceeded to trigger a scale up. Value between 10 and 80.	number	65	no
subnet_ids	A list of subnets that the MSK cluster should run in	list(string)	n/a	yes
tags	A map of tags to add to all resources	map(string)	{}	no
type	The type of the certificate authority	string	""	no
vpc_id	The MSK cluster's VPC ID	any	n/a	yes

Outputs

Name	Description
bootstrap_brokers	Plaintext connection host:port pairs
bootstrap_brokers_tls	TLS connection host:port pairs
msk_cluster_arn	The MSK cluster arn
msk_sg_id	The MSK security group ID
zookeeper_connect_string	A comma separated list of one or more IP:port pairs to use to connect to the Apache Zookeeper cluster