Merge pull request #1353 from splunk/urbiz-OD5655-aws-console-2

[OD5655]: Connect from AWS console
splunk · Oct 17, 2023 · 8ff3264 · 8ff3264
2 parents 98caa8a + 389157f
commit 8ff3264
Show file tree

Hide file tree

Showing 6 changed files with 145 additions and 31 deletions.
diff --git a/_images/gdi/aws-console-splunk.png b/_images/gdi/aws-console-splunk.png
diff --git a/gdi/get-data-in/connect/aws/aws-apiconfig.rst b/gdi/get-data-in/connect/aws/aws-apiconfig.rst
@@ -49,27 +49,29 @@ Your system response looks something like this:
   "authMethod" : "ExternalId",
   "enabled" : false,
   "externalId" : "<externalId>",
-  "id" : "<id>",
+  "id" : "<integrationId>",
   "importCloudWatch" : false,
   "name" : "AWS",
   "pollRate" : 300000,
   "regions" : [ ],
   "roleArn" : null,
   "services" : [ ],
+  "sfxAwsAccountArn" : "arn:aws:iam::<accountId>:root"
   "type" : "AWSCloudWatch"
   }
 
-In the system response, note the following:
+In the system response, note the following fields:
 
-- Values are displayed for the ``externalId`` and ``id`` fields.
+- The ``id`` field is the id of the integration you just created. 
+- ``externalId`` and ``accountId`` will be used when you create an IAM (Identity and Access Management) role in the AWS console later on.
 - The ``importCloudWatch`` value is set to ``false`` because CloudWatch Metrics collection has not been configured.
 
 .. _aws-api-create-policy-role:
 
 Create an AWS policy and IAM role
 ---------------------------------------------------------------------
 
-To create an AWS policy and an AWS IAM (Identity and Access Management) role with a unique Amazon Resource Name (ARN), go to the AWS console and follow the instructions in :ref:`aws-authentication`. Use the ``externalId`` value generated in the previous step.
+To create an AWS policy and an AWS IAM role with a unique Amazon Resource Name (ARN), go to the AWS console and follow the instructions in :ref:`aws-authentication`. Use the ``externalId`` and ``accountId`` values generated in the previous step.
 
 .. _review-aws-iam-policy:
 
@@ -409,7 +411,7 @@ The following example shows how to collect metrics from all regions and services
 
 .. code-block:: none
 
-  curl -X PUT 'https://app.<realm>.signalfx.com/v2/integration/<IntegrationID>' \
+  curl -X PUT 'https://app.<realm>.signalfx.com/v2/integration/<integrationId>' \
     -H 'accept: application/json, text/plain, */*' \
     -H 'x-sf-token: <USER_API_ACCESS_TOKEN>' \
     -H 'content-type: application/json' \
@@ -422,8 +424,8 @@ The following example shows how to collect metrics from all regions and services
       "enableAwsUsage" : false,
       "enableCheckLargeVolume" : false,
       "enabled" : true,
-      "externalId" : "bqvguakfajpzxgqobzvd",
-      "id" : "F2aURjcAAAI",
+      "externalId" : "<externalId>",
+      "id" : "<integrationId>",
       "importCloudWatch" : true,
       "largeVolume" : false,
       "lastUpdated" : 1690856052734,
@@ -434,7 +436,7 @@ The following example shows how to collect metrics from all regions and services
       "regions" : [ ],
       "roleArn" : "<your-aws-iam-role-arn>",
       "services" : [ ],
-      "sfxAwsAccountArn" : "arn:aws:iam::134183635603:root",
+      "sfxAwsAccountArn" : "arn:aws:iam::<accountId>:root",
       "syncCustomNamespacesOnly" : false,
       "syncLoadBalancerTargetGroupTags" : false,
       "type" : "AWSCloudWatch"}'
@@ -461,11 +463,11 @@ Activate CloudWatch Metric Streams (optional)
 
 To activate CloudWatch Metric Streams as an alternative to traditional API polling, follow these steps:
 
-#. Submit a GET request to ``https://api.<realm>.signalfx.com/v2/integration/<integration-id>`` to retrieve your current settings. Make sure to substitute your own realm and integration ID in the URL.
+#. Submit a GET request to ``https://api.<realm>.signalfx.com/v2/integration/<integrationId>`` to retrieve your current settings. Make sure to substitute your own realm and integration ID in the URL.
 #. Set the ``metricStreamsSyncState`` field to ``ENABLED``.
 #. Set the ``importCloudWatch`` field to ``true``.
 #. Set the ``enabled`` field to ``true``.
-#. Submit a PUT request to the ``https://api.<realm>.signalfx.com/v2/integration/<integration-id>`` endpoint to save your updated settings.
+#. Submit a PUT request to the ``https://api.<realm>.signalfx.com/v2/integration/<integrationId>`` endpoint to save your updated settings.
 
 .. caution:: CloudWatch Metric Streams supports filtering by namespace and metric name but doesn't support filtering based on resource tags.   
 
@@ -488,7 +490,7 @@ Deactivate Metric Streams
 
 To deactivate Metric Streams, follow these steps:
 
-#. Submit a GET request to ``https://api.<realm>.signalfx.com/v2/integration/<integration-id>`` to retrieve your current settings. Make sure to substitute your own realm and integration ID in the URL.
+#. Submit a GET request to ``https://api.<realm>.signalfx.com/v2/integration/<integrationId>`` to retrieve your current settings. Make sure to substitute your own realm and integration ID in the URL.
 #. Set the ``metricStreamsSyncState`` field to ``CANCELLING``.
 #. Wait for Splunk Observability Cloud to clean up. This can take up to 15 minutes. 
 

diff --git a/gdi/get-data-in/connect/aws/aws-compare-connect.rst b/gdi/get-data-in/connect/aws/aws-compare-connect.rst
@@ -0,0 +1,47 @@
+.. _aws-compare-connect:
+
+***********************************************************************************
+Compare AWS connection options 
+***********************************************************************************
+
+.. meta::
+  :description: Compare the different options to integrate and connect Splunk Observability Cloud with AWS
+
+Compare the different available options to connect Splunk Observability Cloud with AWS:
+
+.. list-table::
+  :header-rows: 1
+  :width: 100%
+  :widths: 10 30 30 30
+
+  * - :strong:`Connection option`
+    - :strong:`Sample use cases`
+    - :strong:`Advantages`
+    - :strong:`Disadvantages`
+
+  * - Polling (default)
+    - #. Your metrics are not time-sensitive, and delays are acceptable
+      #. You need to use tag filtering and/or poll rate to manage cost
+    - * Easiest method to set up
+      * Highest granularity to manage cost through tag filtering or by prolonging polling intervals
+      * Applies to selected AWS regions under the AWS account
+    - * Metrics might be delayed since they're are polled at intervals
+      * If polling intervals are short, this method might be more expensive
+
+  * - Streaming (Splunk-managed)
+    - #. You're looking for real-time metrics 
+      #. You need to sync metrics from multiple AWS regions
+    - * Access metrics faster and at scale through Kinese Data Firehose streams
+      * Less expensive than polling when you need close to real-time metrics
+      * Manage stream filters in Splunk
+      * Applies to all AWS regions under the AWS account
+    - * Harder to manage costs due to lack of tag filtering and control of streaming input
+      * You need to deploy :ref:`CloudFormation templates <aws-cloudformation>`
+
+  * - :ref:`Streaming (AWS-managed) <aws-console-ms>`
+    - You're looking for real-time metrics 
+    - * Access metrics faster and at scale through Kinese Data Firehose streams
+      * Less expensive than polling when you need close to real-time metrics
+      * Direct control over the metric stream set-up
+    - * Harder to manage costs due to lack of tag filtering and control of streaming input
+      * You need create metric streams through the CloudWatch console for each AWS region
diff --git a/gdi/get-data-in/connect/aws/aws-console-ms.rst b/gdi/get-data-in/connect/aws/aws-console-ms.rst
@@ -0,0 +1,54 @@
+.. _aws-console-ms:
+
+***********************************************************************************
+Connect Splunk Observability Cloud with Metric Streams from the AWS console
+***********************************************************************************
+
+.. meta::
+  :description: Connect to AWS from the AWS console using Metric Streams
+
+Amazon CloudWatch supports a quick setup experience for AWS Partner destinations in the CloudWatch Metric Streams console. With this simplified getting-started experience you can create a Metric Stream to Splunk Observability Cloud in a single step.
+
+Prerequisites
+======================================================
+
+Ensure you comply with the following requirements before you proceed to create your Metric Streams connection between your AWS and your Splunk Observability Cloud accounts:
+
+* Make sure you have an active AWS integration in your associated Splunk Observability Cloud account. To learn how to create an integration, see :ref:`get-started-aws`. 
+* Make sure Metric Streams is activated in the integration.
+* Make sure the AWS account you used to create the integration contains the required policies for Metric Streams, as described in :ref:`metricstreams_iampolicy`. 
+
+.. note:: You can only have one integration using Metric Streams externally managed from AWS at a time for each AWS account. All Metric Streams you created in different AWS regions will be sent to this integration. Editing included regions will affect metadata, and might impact how corresponding metric streams are located and displayed in Observability Cloud features such as dashboards.
+
+Integrate from the AWS console using the API 
+======================================================
+
+If you want to integrate from the AWS console, make sure the following fields are configured:
+
+.. code-block:: yaml
+
+  "importCloudWatch": true // fetch metrics
+  "metricStreamsSyncState": "ENABLED" // Metric Streams is activated
+  "metricStreamsManagedExternally": true // Metric Streams managed by AWS
+
+Learn more in our API reference guide at :new-page:`Integrations <https://dev.splunk.com/observability/reference/api/integrations/latest#endpoint-retrieve-integrations-query>`.
+
+Find Splunk Observability Cloud in AWS
+======================================================
+
+To connect Splunk Observability Cloud from the AWS console, follow these steps:
+
+1. In the AWS console, go to :guilabel:`CloudWatch > Metrics > Streams`, and select :guilabel:`Create metric stream`.
+
+2. Next, select :guilabel:`Quick AWS Partner setup` as the destination.
+
+3. In the drop down menu, select :guilabel:`Splunk Observability Cloud`.
+
+.. image:: /_images/gdi/aws-console-splunk.png
+  :width: 55%
+
+4. In the :guilabel:`Configure the AWS Partner destination` menu, select the Splunk Observability Cloud Ingest endpoints from the dropdown list. See :guilabel:`Profile > Organizations > Real-time Data Ingest Endpoint` in the Splunk Observability Cloud console for more information. 
+
+5. Fill in the access token by copying one of the access tokens with ``INGEST`` authorization scope from :guilabel:`Settings > Access Tokens` in the Splunk Observability Cloud console.
+
+6. You can leave the rest of the form as default, or customize the config according to your needs, such as filtering based on namespaces.
diff --git a/gdi/get-data-in/connect/aws/aws-prereqs.rst b/gdi/get-data-in/connect/aws/aws-prereqs.rst
@@ -56,8 +56,8 @@ Create an AWS IAM role
 After creating an AWS IAM policy, you need to assign that policy to a particular role by performing the following steps in the Amazon Web Services console:
 
 #. Go to :strong:`Roles`, then :strong:`Create Role`, and select :strong:`Another AWS account` as the type of trusted entity.
-#. Copy and paste the Account ID displayed in guided setup into the :strong:`Account ID` field.
-#. Select :strong:`Require external ID`. Copy and paste the External ID displayed in the guided setup into the :strong:`External ID` field.
+#. Copy and paste the Account ID displayed in either guided setup or the numeric value within ``sfxAwsAccountArn`` from API call's response into the :strong:`Account ID` field.
+#. Select :strong:`Require external ID`. Copy and paste the External ID displayed in the guided setup or ``externalId`` from API call's response into the :strong:`External ID` field.
 #. Continue with :strong:`Next: Permissions`. Under :strong:`Policy name`, select the policy you made in the previous step.
 #. Follow the instructions, and name and create your new AWS IAM role.  
 

diff --git a/gdi/get-data-in/connect/aws/get-awstoc.rst b/gdi/get-data-in/connect/aws/get-awstoc.rst
@@ -11,9 +11,11 @@ Connect to AWS and send data to Splunk Observability Cloud
   :hidden:
 
   AWS authentication and supported regions <aws-prereqs>
+  Compare connection options <aws-compare-connect>
   Connect to AWS with the UI guided setup <aws-wizardconfig>
   Connect to AWS with the API <aws-apiconfig>
   Connect to AWS with Terraform <aws-terraformconfig>
+  Connect Metric Streams from the AWS console <aws-console-ms>  
   Collect logs from AWS <aws-logs>
   CloudFormation templates <aws-cloudformation>
   Next steps <aws-post-install>
@@ -58,7 +60,7 @@ You can poll data from AWS at specified intervals using CloudWatch APIs. Due to
 .. raw:: html
 
   <embed>
-    <h4>High data volume warning</h4>
+    <h4>High data volume warning <a name="aws-data-limits" class="headerlink" href="#aws-data-limits" title="Permalink to this headline">¶</a></h4>
   </embed>
 
 After you create an AWS integration, if more than 100,000 metrics are retrieved from CloudWatch, Splunk Observability Cloud automatically deactivates the integration and sends you a warning email.  
@@ -72,7 +74,7 @@ You can deactivate this check by setting the ``enableCheckLargeVolume`` field in
 .. raw:: html
 
   <embed>
-    <h4>Tag filtering</h4>
+    <h4>Tag filtering<a name="tag-filtering-aws" class="headerlink" href="#tag-filtering-aws" title="Permalink to this headline">¶</a></h4>
   </embed>
 
 If you filter data based on tags, your costs for Amazon CloudWatch and Splunk Infrastructure Monitoring might decrease.
@@ -87,9 +89,19 @@ Be careful when choosing tag names: Splunk Observability Cloud only allows alpha
     <h3>Use Metric Streams to forward data to Splunk Observability Cloud<a name="aws-metricstreams" class="headerlink" href="#aws-metricstreams" title="Permalink to this headline">¶</a></h3>
   </embed>
 
-Rather than polling for data, Metric Streams continually stream Amazon CloudWatch metrics to Splunk Observability Cloud. You can activate this option in the UI wizard, or through the API.
+Rather than polling for data, Metric Streams continually stream Amazon CloudWatch metrics to Splunk Observability Cloud. Although they're more efficient than API polling, consider the :ref:`cost constraints <collection-interval-aws>`.
 
-Although they're more efficient than API polling, consider the constraints below.
+.. _aws-data-availability:
+
+.. raw:: html
+
+  <embed>
+    <h2>Data availability<a name="aws-data-availability" class="headerlink" href="#aws-data-availability" title="Permalink to this headline">¶</a></h2>
+  </embed>
+
+.. caution:: Splunk is not responsible for data availability, and it can take up to several minutes (or longer, depending on your configuration) from the time you connect until you start seeing valid data from your account.
+
+By default, Splunk Observability Cloud brings in data from all :ref:`supported AWS services <aws-integrations>` associated with your account. To limit the amount of data to import, see :ref:`specify-data-metadata`.
 
 .. _collection-interval-aws:
 
@@ -101,7 +113,7 @@ Although they're more efficient than API polling, consider the constraints below
 
 In most cases, metrics are reported every minute. However, some services use a different cadence: For example, selected S3 metrics are reported on a daily basis. Check AWS documentation to verify how often your services' metrics are reported.
 
-Collecting Amazon CloudWatch metrics via the polling APIs at the default polling rate of 300 seconds (5 minutes) is generally cheaper than using Metric Streams. On the other hand, if you set polling intervals to one minute, generally you'll see an increase in Amazon CloudWatch usage costs compared to Metric Streams.
+Collecting Amazon CloudWatch metrics via the polling APIs at the default polling rate of 300 seconds (5 minutes) is usually cheaper than using Metric Streams. On the other hand, if you set polling intervals to one minute, generally you'll see an increase in Amazon CloudWatch usage costs compared to Metric Streams.
 
 Learn more at :ref:`Amazon CloudWatch usage costs <aws-costs>`.
 
@@ -110,14 +122,10 @@ Learn more at :ref:`Amazon CloudWatch usage costs <aws-costs>`.
 .. raw:: html
 
   <embed>
-    <h2>Connect with AWS<a name="aws-connection-options" class="headerlink" href="#aws-connection-options" title="Permalink to this headline">¶</a></h2>
+    <h2>Connect with AWS: Available options<a name="aws-connection-options" class="headerlink" href="#aws-connection-options" title="Permalink to this headline">¶</a></h2>
   </embed>
 
-You can connect Splunk Observability Cloud to AWS in several ways. By default, Splunk Observability Cloud brings in data from all :ref:`supported AWS services <aws-integrations>` associated with your account. To limit the amount of data to import, see :ref:`specify-data-metadata`.
-
-.. caution:: Splunk is not responsible for data availability, and it can take up to several minutes (or longer, depending on your configuration) from the time you connect until you start seeing valid data from your account.
-
-Choose the connection method that best matches your needs:
+You can connect Splunk Observability Cloud to AWS in several ways. See a comparison of the connection options at :ref:`aws-compare-connect`, and choose the connection method that best matches your needs:
 
 .. list-table::
   :header-rows: 1
@@ -133,23 +141,26 @@ Choose the connection method that best matches your needs:
   * - :ref:`Splunk Observability Cloud API <get-configapi>`
     - Requires knowledge of POST and PUT call syntax, but includes options and automation that are not part of the guided setup. Choose this method if you want to configure many integrations at once. 
 
+  * - :ref:`Metric Streams from the AWS console <aws-console-ms>`
+    - Connect and manage Metric Streams from the AWS console.
+
   * - :ref:`Splunk Terraform <terraform-config>`
     - Use this connection method if you already manage your infrastructure as code by deploying through Terraform.
 
-If you can't connect AWS to Splunk Observability Cloud, see :ref:`Troubleshoot your AWS connection <aws-troubleshooting>`.
+  * - :new-page:`Splunk add-on for Amazon Kinesis Firehose <https://docs.splunk.com/Documentation/AddOns/latest/Firehose/ConfigureFirehose>`
+    - Use the Splunk add-on if you want to configure Amazon Kinesis Firehose to send data to the Splunk platform.
+
+.. note:: If you can't connect AWS to Splunk Observability Cloud, see :ref:`Troubleshoot your AWS connection <aws-troubleshooting>`.
 
 .. _aws-connection-options-more:
 
 .. raw:: html
 
   <embed>
-    <h3>More options to connect with AWS</h3>
+    <h3>Private connectivity<a name="aws-connection-options-more" class="headerlink" href="#aws-connection-options-more" title="Permalink to this headline">¶</a></h3>
   </embed>  
 
-Observability Cloud also offers you the following options to connect to AWS:
-
-* :ref:`aws-privatelink`.
-* The :new-page:`Splunk add-on for Amazon Kinesis Firehose <https://docs.splunk.com/Documentation/AddOns/latest/Firehose/ConfigureFirehose>`.  
+Splunk Observability Cloud also offers secured connectivity with AWS. For more information, see :ref:`aws-privatelink`.
 
 .. _aws-collector:
 
@@ -164,7 +175,7 @@ To take advantage of the full benefits of the Splunk Observability Cloud platfor
 You can track the degree of OpenTelemetry enablement in your AWS integrations by going to :guilabel:`Data Management > AWS`.
 
 ..  image:: /_images/gdi/aws-collector-insights.jpg
-  :width: 100%
+  :width: 80%
   :alt: Amount of AWS entities with the Collector installed.
 
 Select the :guilabel:`OpenTelemetry Enabled` button to see whether the Collector is installed on each AWS EC2 instance. This will help you identify the instances that still need to be instrumented. For instances that are successfully instrumented, you can see which version of the Collector is deployed.