Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(fluentd): Add support for Cloudwatch Logging #296

Merged
merged 7 commits into from
Oct 27, 2023
Merged

feat(fluentd): Add support for Cloudwatch Logging #296

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
14debaf
add support for CloudWatch
jrlonan-gt Oct 18, 2023
0c2a7ee
add cloudwatch IAM policy
jrlonan-gt Oct 23, 2023
9db37cb
add log group name variable
jrlonan-gt Oct 23, 2023
6a079fc
fix cloudwatch config
jrlonan-gt Oct 23, 2023
24cee14
run fmt & regenerate INOUT
jrlonan-gt Oct 23, 2023
647b420
add https only S3 bucket policy
jrlonan-gt Oct 25, 2023
ed4ce81
remove ssl transport policy from IAM policy
jrlonan-gt Oct 27, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion modules/fluentd/INOUT.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,13 +24,15 @@
| fluentd\_cpu | CPU resource assigned to the fluentd job | `number` | `3000` | no |
| fluentd\_force\_pull | Force pull an image. Useful if the tag is mutable. | `string` | `"false"` | no |
| fluentd\_image | Docker image for fluentd | `string` | `"govtechsg/fluentd-s3-elasticsearch"` | no |
| fluentd\_match | Tags that fluentd should output to S3 and Elasticsearch | `string` | `"@ERROR app.** docker.** services.** system.** vault**"` | no |
| fluentd\_match | Tags that fluentd should output to S3, CloudWatch and Elasticsearch | `string` | `"@ERROR app.** docker.** services.** system.** vault**"` | no |
| fluentd\_memory | Memory resource assigned to the fluentd job | `number` | `512` | no |
| fluentd\_port | Port on the Docker image in which the TCP interface is exposed | `number` | `4224` | no |
| fluentd\_tag | Tag for fluentd Docker image | `string` | `"1.2.5-latest"` | no |
| inject\_source\_host | Inject the log source host name and address into the logs | `bool` | `true` | no |
| log\_vault\_policy | Name of the Vault policy to allow creating AWS credentials to write to Elasticsearch and S3 | `string` | `"fluentd_logger"` | no |
| log\_vault\_role | Name of the Vault role in the AWS secrets engine to provide credentials for fluentd to write to Elasticsearch and S3 | `string` | `"fluentd_logger"` | no |
| logs\_cloudwatch\_enabled | Enable to log to CloudWatch | `bool` | `false` | no |
| logs\_log\_group\_name | Name of CloudWatch Log Group to store logs | `string` | `"/fluentd/logs"` | no |
| logs\_s3\_abort\_incomplete\_days | Specifies the number of days after initiating a multipart upload when the multipart upload must be completed. | `number` | `7` | no |
| logs\_s3\_bucket\_name | Name of S3 bucket to store logs for long term archival | `string` | `""` | no |
| logs\_s3\_enabled | Enable to log to S3 | `bool` | `true` | no |
Expand Down
9 changes: 9 additions & 0 deletions modules/fluentd/consul.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ locals {
file_logging_consul_key = "${var.consul_key_prefix}fluentd/log_to_file"
fluentd_match_consul_key = "${var.consul_key_prefix}fluentd/match"
s3_consul_key = "${var.consul_key_prefix}fluentd/log_to_s3"
cloudwatch_consul_key = "${var.consul_key_prefix}fluentd/log_to_cloudwatch"
inject_source_host = "${var.consul_key_prefix}fluentd/inject_source_host"
source_address_key = "${var.consul_key_prefix}fluentd/source_address_key"
source_hostname_key = "${var.consul_key_prefix}fluentd/source_hostname_key"
Expand Down Expand Up @@ -45,6 +46,14 @@ resource "consul_keys" "log_to_s3" {
}
}

resource "consul_keys" "log_to_cloudwatch" {
key {
path = local.cloudwatch_consul_key
value = var.logs_cloudwatch_enabled ? "true" : "false"
delete = true
}
}

resource "consul_keys" "inject_source_host" {
key {
path = local.inject_source_host
Expand Down
4 changes: 4 additions & 0 deletions modules/fluentd/main.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,8 +16,12 @@ data "template_file" "fluentd_tf_rendered_conf" {
file_logging_consul_key = local.file_logging_consul_key
fluentd_match_consul_key = local.fluentd_match_consul_key
s3_consul_key = local.s3_consul_key
cloudwatch_consul_key = local.cloudwatch_consul_key
weekly_index_enabled_consul_key = local.weekly_index_enabled_consul_key

log_group_name = var.logs_log_group_name
aws_region = var.aws_region

inject_source_host = local.inject_source_host
source_address_key = local.source_address_key
source_hostname_key = local.source_hostname_key
Expand Down
15 changes: 15 additions & 0 deletions modules/fluentd/s3.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,21 @@ data "aws_iam_policy_document" "logs_s3" {
"${aws_s3_bucket.logs[0].arn}/*",
]
}

statement {
effect = "Allow"

actions = [
"logs:PutLogEvents",
"logs:CreateLogGroup",
"logs:PutRetentionPolicy",
"logs:CreateLogStream",
"logs:DescribeLogGroups",
"logs:DescribeLogStreams"
]

resources = ["*"]
}
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion: could we include the ssl policy that tzeyang implemented into this s3 resource?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sure, i can add it

}

resource "aws_iam_policy" "logs_s3" {
Expand Down
12 changes: 12 additions & 0 deletions modules/fluentd/templates/fluent.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,9 +10,21 @@
@include /config/secrets/*.conf
@include /config/additional/*.conf


<match {{ key "${fluentd_match_consul_key}" }}>
@type copy

{{ if eq (keyOrDefault "${cloudwatch_consul_key}" "false") "true" }}
<store {{ key "${fluentd_match_consul_key}" }}>
@type cloudwatch_logs
region ${aws_region}
log_group_name ${log_group_name}
use_tag_as_stream true
message_keys log
auto_create_stream true
</store>
{{ end }}

{{ if eq (keyOrDefault "${file_logging_consul_key}" "false") "true" }}
<store ignore_error>
@type file
Expand Down
16 changes: 15 additions & 1 deletion modules/fluentd/variables.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -176,6 +176,20 @@ variable "tags" {
}
}

#############################
# CloudWatch Logging related
#############################
variable "logs_cloudwatch_enabled" {
description = "Enable to log to CloudWatch"
default = false
}

variable "logs_log_group_name" {
description = "Name of CloudWatch Log Group to store logs"
default = "/fluentd/logs"
}


# --------------------------------------------------------------------------------------------------
# CORE INTEGRATION SETTINGS
# --------------------------------------------------------------------------------------------------
Expand All @@ -195,6 +209,6 @@ variable "enable_file_logging" {
}

variable "fluentd_match" {
description = "Tags that fluentd should output to S3 and Elasticsearch"
description = "Tags that fluentd should output to S3, CloudWatch and Elasticsearch"
default = "@ERROR app.** docker.** services.** system.** vault**"
}