Home

Splunk AWS GDI Toolkit

The Splunk AWS GDI Toolkit is a set of resources designed to help organizations easily ingest data of AWS into Splunk to improve visibility, observability, and monitoring from AWS accounts. The toolkit consists of a series of CloudFormation templates to enable service and get the data from those services into Splunk. These templates can also serve as a reference or starting point for organizations looking to implement use-cases that the Splunk AWS GDI Toolkit doesn't exactly meet.

The goals of this toolkit are to:

• Augment the Splunk Cloud Data Manager with CloudFormation templates to enable pre-requisites for getting data in from single AWS accounts.
• Follow AWS best practices for getting data in from multiple AWS accounts utilizing AWS Organizations.
• Combine the best of Splunk Cloud Data Manager, Splunk Trumpet, and Grand Central.

PRs are open, and feel free to reach out to me over the Splunk Usergroups Slack if you have questions, comments, or concerns!

Use Case Library

• Send CloudTrail, ELB Access, VPC Flow , AWS Billing Cost and Usage Reports, Amazon Route 53 resolver query, S3 Server Access logs to Splunk - https://github.com/splunk/splunk-aws-gdi-toolkit/wiki/S3-SQS-Lambda-Firehose-Method
• Send events from a custom sourcetype from S3 buckets to Splunk - https://github.com/splunk/splunk-aws-gdi-toolkit/wiki/S3-SQS-Lambda-Firehose-Method
• Send events from Amazon GuardDuty and AWS Security Hub to Splunk - https://github.com/splunk/splunk-aws-gdi-toolkit/wiki/CloudWatch-Events-Firehose-Method
• Send CloudWatch Metrics to Splunk as either event-style or metric-style data - https://github.com/splunk/splunk-aws-gdi-toolkit/wiki/CloudWatch-Metrics-Firehose-Method
• Send CloudWatch Logs to Splunk as events - https://github.com/splunk/splunk-aws-gdi-toolkit/wiki/CloudWatchLogs-Firehose-Method
• Send VPC Flow logs to Splunk without the need for CloudWatch Logs or S3 - https://github.com/splunk/splunk-aws-gdi-toolkit/wiki/VPC-Flow-Logs-Firehose-Method
• Implement SCDM pre-requistes using CloudFormation - https://github.com/splunk/splunk-aws-gdi-toolkit/wiki/Single-AWS-Account-Ingestion-with-SCDM

When to use Splunk Data Manager or Splunk AWS GDI Toolkit

Both the Splunk Cloud Data Manager (SCDM) and the CloudFormation templates in this toolkit can be used to pull in some of the same events form AWS, like CloudTrail events. The SCDM prioritizes event latency, while the CloudFormation templates here prioritize following AWS best-practices and cost.

Mermaid code:

graph TD;
	start([Start here])
	usingOrg{Are you using AWS Organization and/or Landing Zones?}
	costOrLtency{Do you want to minimize lowering cost or data latency?}
	SCDM([Use Data Manager, referencing the CloudFormation Templates here for prerequisites])
	sAWSGDITK([Use the CloudFormation Templates here])
	start-->usingOrg
	usingOrg-->|No|costOrLtency
	costOrLtency-->|Minimize latency|SCDM
	costOrLtency-->|Minimize cost|sAWSGDITK
	usingOrg-->|Yes|sAWSGDITK