Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Could not put metric data using the following endpoint - Exception: 403 Client Error: Forbidden for url #55

Open
ratneshamarnath opened this issue Oct 2, 2017 · 4 comments
Open

Could not put metric data using the following endpoint - Exception: 403 Client Error: Forbidden for url #55

ratneshamarnath opened this issue Oct 2, 2017 · 4 comments

Comments

@ratneshamarnath
Copy link

ratneshamarnath commented Oct 2, 2017 •
edited
Loading

Hi All,

I am trying to run collectd-cloudwatch on AWS instance.

I am seeing following error in while I run "sudo /etc/init.d/collectd restart" -

===============================================
[2017-10-02 06:38:11] [warning] [AmazonCloudWatchPlugin][cloudwatch.modules.client.putclient] Could not put metric data using the following endpoint: 'https://monitoring.us-east-2.amazonaws.com/'. [Exception: 403 Client Error: Forbidden for url: https://monitoring.us-east-2.amazonaws.com/?Action=PutMetricData&MetricData.member.1.Dimensions.member.1.Name=Host&MetricData.member.1.Dimensions.member.1.Value=i-0a780d545c7ad981c&MetricData.member.1.Dimensions.member.2.Name=PluginInstance&MetricData.member.1.Dimensions.member.2.Value=dev-shm&MetricData.member.1.MetricName=df.percent_bytes.used&MetricData.member.1.StatisticValues.Maximum=0.0&MetricData.member.1.StatisticValues.Minimum=0.0&MetricData.member.1.StatisticValues.SampleCount=6&MetricData.member.1.StatisticValues.Sum=0.0&MetricData.member.1.Timestamp=20171002T063711Z&MetricData.member.2.Dimensions.member.1.Name=Host&MetricData.member.2.Dimensions.member.1.Value=i-0a780d545c7ad981c&MetricData.member.2.Dimensions.member.2.Name=PluginInstance&MetricData.member.2.Dimensions.member.2.Value=de
[2017-10-02 06:38:11] [warning] [AmazonCloudWatchPlugin][cloudwatch.modules.client.putclient] Request details: 'Action=PutMetricData&MetricData.member.1.Dimensions.member.1.Name=Host&MetricData.member.1.Dimensions.member.1.Value=i-0a780d545c7ad981c&MetricData.member.1.Dimensions.member.2.Name=PluginInstance&MetricData.member.1.Dimensions.member.2.Value=dev-shm&MetricData.member.1.MetricName=df.percent_bytes.used&MetricData.member.1.StatisticValues.Maximum=0.0&MetricData.member.1.StatisticValues.Minimum=0.0&MetricData.member.1.StatisticValues.SampleCount=6&MetricData.member.1.StatisticValues.Sum=0.0&MetricData.member.1.Timestamp=20171002T063711Z&MetricData.member.2.Dimensions.member.1.Name=Host&MetricData.member.2.Dimensions.member.1.Value=i-0a780d545c7ad981c&MetricData.member.2.Dimensions.member.2.Name=PluginInstance&MetricData.member.2.Dimensions.member.2.Value=dev&MetricData.member.2.MetricName=df.percent_bytes.used&MetricData.member.2.StatisticValues.Maximum=0.0112053789198&MetricData.member.2.StatisticValues.Minimum=0.0112053789198&Met
============================================

Could someone provide me pointer to proceed further on it.

Thanks
Ratnesh
@ratneshamarnath ratneshamarnath changed the title Could not put metric data using the following endpoint: 'https://monitoring.us-east-2.amazonaws.com/'. [Exception: 403 Client Error: Forbidden for url: https://monitoring.us-east-2.amazonaws.com/?Action=PutMetricData&MetricData Could not put metric data using the following endpoint - Exception: 403 Client Error: Forbidden for url: Oct 2, 2017
@ratneshamarnath ratneshamarnath changed the title Could not put metric data using the following endpoint - Exception: 403 Client Error: Forbidden for url: Could not put metric data using the following endpoint - Exception: 403 Client Error: Forbidden for url Oct 2, 2017
@yimuniao
Copy link
Contributor

yimuniao commented Oct 3, 2017

Normally, it was caused by the invalid key, how do you setup your EC2 instance? do you attach IAM role to the EC2 instance? or do you put the credentials to /root/.aws/credentials?

@ratneshamarnath
Copy link
Author

Hi,

I have tried both using IAM role and credential steps.

===================================================================

[ec2-user@ip-172-31-30-191 src]$
[ec2-user@ip-172-31-30-191 src]$ sudo ./setup.py
Installing dependencies ... OK
Installing python dependencies ... OK
Downloading plugin ... OK
Extracting plugin ... OK
Moving to collectd plugins directory ... OK
Copying CloudWatch plugin include file ... OK
DEBUG:urllib3.util.retry:Converted retries value: 1 -> Retry(total=1, connect=None, read=None, redirect=None, status=None)
DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): 169.254.169.254
DEBUG:urllib3.connectionpool:http://169.254.169.254:80 "GET /latest/meta-data/placement/availability-zone/ HTTP/1.1" 200 10

Choose AWS region for published metrics:

  1. Automatic [us-east-2]
  2. Custom
    Enter choice [1]: 1
    DEBUG:urllib3.util.retry:Converted retries value: 1 -> Retry(total=1, connect=None, read=None, redirect=None, status=None)
    DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): 169.254.169.254
    DEBUG:urllib3.connectionpool:http://169.254.169.254:80 "GET /latest/meta-data/instance-id/ HTTP/1.1" 200 19

Choose hostname for published metrics:

  1. EC2 instance id [i-0a780d545c7ad981c]
  2. Custom
    Enter choice [1]: 1
    DEBUG:urllib3.util.retry:Converted retries value: 1 -> Retry(total=1, connect=None, read=None, redirect=None, status=None)
    DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): 169.254.169.254
    DEBUG:urllib3.connectionpool:http://169.254.169.254:80 "GET /latest/meta-data/iam/security-credentials/ HTTP/1.1" 200 10

Choose authentication method:

  1. IAM Role [cloudwatch]
  2. IAM User
    Enter choice [1]: 1

Enter proxy server name:

  1. None
  2. Custom
    Enter choice [1]: 1

Enter proxy server port:

  1. None
  2. Custom
    Enter choice [1]: 1

Include the Auto-Scaling Group name as a metric dimension:

  1. No
  2. Yes
    Enter choice [1]: 1

Include the FixedDimension as a metric dimension:

  1. No
  2. Yes
    Enter choice [1]: 1

Enable high resolution:

  1. Yes
  2. No
    Enter choice [2]: 2

Enter flush internal:

  1. Default 60s
  2. Custom
    Enter choice [1]: 1

Choose how to install CloudWatch plugin in collectd:

  1. Do not modify existing collectd configuration
  2. Add plugin to the existing configuration
  3. Use CloudWatch recommended configuration (4 metrics)
    Enter choice [3]: 3
    Plugin configuration written successfully.
    Creating backup of the original configuration ... OK
    Replacing collectd configuration ... OK
    Replacing whitelist configuration ... OK
    Stopping collectd process ... OK
    Starting collectd process ... OK
    [ec2-user@ip-172-31-30-191 src]$
    [ec2-user@ip-172-31-30-191 src]$
    [ec2-user@ip-172-31-30-191 src]$
    [ec2-user@ip-172-31-30-191 src]$ sudo /etc/init.d/collectd restart
    Stopping collectd: [FAILED]
    Starting collectd: [ OK ]
    [ec2-user@ip-172-31-30-191 src]$
    [ec2-user@ip-172-31-30-191 src]$ tail -f /var/log/collectd.log
    [2017-10-02 17:41:38] [info] plugin_load: plugin "df" successfully loaded.
    [2017-10-02 17:41:38] [info] plugin_load: plugin "memory" successfully loaded.
    [2017-10-02 17:41:38] [info] plugin_load: plugin "swap" successfully loaded.
    [2017-10-02 17:41:38] [info] plugin_load: plugin "python" successfully loaded.
    [2017-10-02 17:41:38] [warning] [AmazonCloudWatchPlugin][cloudwatch.modules.client.ec2getclient] Could not get the autoscalig group name using the following endpoint: 'https://ec2.us-east-2.amazonaws.com/'. [Exception: 401 Client Error: Unauthorized for url: https://ec2.us-east-2.amazonaws.com/?Action=DescribeTags&Filter.1.Name=key&Filter.1.Value.1=aws%3Aautoscaling%3AgroupName&Filter.2.Name=resource-id&Filter.2.Value.1=i-0a780d545c7ad981c&Version=2016-11-15&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAJYIZE2JHMZVBEDTA%2F20171002%2Fus-east-2%2Fec2%2Faws4_request&X-Amz-Date=20171002T174137Z&X-Amz-SignedHeaders=host&X-Amz-Signature=a48034203dfcacb8565c3aca234be4ba1ea230104512e36e2b14625d7eee6bec]
    [2017-10-02 17:41:38] [warning] [AmazonCloudWatchPlugin][cloudwatch.modules.client.ec2getclient] Request details: 'Action=DescribeTags&Filter.1.Name=key&Filter.1.Value.1=aws%3Aautoscaling%3AgroupName&Filter.2.Name=resource-id&Filter.2.Value.1=i-0a780d545c7ad981c&Version=2016-11-15&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAJYIZE2JHMZVBEDTA%2F20171002%2Fus-east-2%2Fec2%2Faws4_request&X-Amz-Date=20171002T174137Z&X-Amz-SignedHeaders=host&X-Amz-Signature=a48034203dfcacb8565c3aca234be4ba1ea230104512e36e2b14625d7eee6bec'
    [2017-10-02 17:41:38] [info] [AmazonCloudWatchPlugin][cloudwatch.modules.configuration.confighelper] Fetched asg name as NONE
    [2017-10-02 17:41:38] [info] [AmazonCloudWatchPlugin][cloudwatch.modules.client.putclient] No proxy server is in use
    [2017-10-02 17:41:38] [info] [AmazonCloudWatchPlugin][cloudwatch_writer] Initialization finished successfully.
    [2017-10-02 17:36:59] [warning] [AmazonCloudWatchPlugin][cloudwatch.modules.flusher] Adding Metric value is not numerical, key: swap--percent-used value: [nan]
    [2017-10-02 17:36:59] [info] Initialization complete, entering read-loop.
    [2017-10-02 17:37:53] [warning] [AmazonCloudWatchPlugin][cloudwatch.modules.client.putclient] Could not put metric data using the following endpoint: 'https://monitoring.us-east-2.amazonaws.com/'. [Exception: HTTPSConnectionPool(host='monitoring.us-east-2.amazonaws.com', port=443): Max retries exceeded with url: /?Action=PutMetricData&MetricData.member.1.Dimensions.member.1.Name=Host&MetricData.member.1.Dimensions.member.1.Value=i-0a780d545c7ad981c&MetricData.member.1.Dimensions.member.2.Name=PluginInstance&MetricData.member.1.Dimensions.member.2.Value=NONE&MetricData.member.1.MetricName=cpu.percent.active&MetricData.member.1.StatisticValues.Maximum=7.59109311741&MetricData.member.1.StatisticValues.Minimum=0.299102691924&MetricData.member.1.StatisticValues.SampleCount=5&MetricData.member.1.StatisticValues.Sum=9.28461990044&MetricData.member.1.Timestamp=20171002T173702Z&MetricData.member.2.Dimensions.member.1.Name=Host&MetricData.member.2.Dimensions.member.1.Value=i-0a780d545c7ad981c&MetricData.member.2.Dimensions.member.2.Name=PluginInstan
    [2017-10-02 17:37:53] [warning] [AmazonCloudWatchPlugin][cloudwatch.modules.client.putclient] Request details: 'Action=PutMetricData&MetricData.member.1.Dimensions.member.1.Name=Host&MetricData.member.1.Dimensions.member.1.Value=i-0a780d545c7ad981c&MetricData.member.1.Dimensions.member.2.Name=PluginInstance&MetricData.member.1.Dimensions.member.2.Value=NONE&MetricData.member.1.MetricName=cpu.percent.active&MetricData.member.1.StatisticValues.Maximum=7.59109311741&MetricData.member.1.StatisticValues.Minimum=0.299102691924&MetricData.member.1.StatisticValues.SampleCount=5&MetricData.member.1.StatisticValues.Sum=9.28461990044&MetricData.member.1.Timestamp=20171002T173702Z&MetricData.member.2.Dimensions.member.1.Name=Host&MetricData.member.2.Dimensions.member.1.Value=i-0a780d545c7ad981c&MetricData.member.2.Dimensions.member.2.Name=PluginInstance&MetricData.member.2.Dimensions.member.2.Value=NONE&MetricData.member.2.MetricName=memory.percent.used&MetricData.member.2.StatisticValues.Maximum=5.21932980267&MetricData.member.2.StatisticValues.Minim
    ==================================================================

From the error description given on url (http://docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.html), it seems that's problem is related to access -

  1. AccessDenied -- Access Denied -- 403 Forbidden Client
  2. AccountProblem -- There is a problem with your AWS account that prevents the operation from completing successfully. Please use Contact Us. -- 403 Forbidden Client

Please let me know if you need anything else to proceed further.

Thanks in Advance !!
Ratnesh

@yimuniao
Copy link
Contributor

yimuniao commented Oct 4, 2017

Could you enable "debug" flag in /opt/collectd-plugins/cloudwatch/config//plugin.conf?
Then restart collectd, then you will see there are http post trace in a temporary file like /tmp/collectd_plugin_request_trace_log

copies one of the commands, then run in terminal.
it will give you more failure information.
Like

[ec2-user@ip-172-31-34-234 config]$ curl -i -v -connect-timeout 1 -m 3 -w %{http_code}:%{http_connect}:%{content_type}:%{time_namelookup}:%{time_redirect}:%{time_pretransfer}:%{time_connect}:%{time_starttransfer}:%{time_total}:%{speed_download} -A "collectd/1.0" 'https://monitoring.eu-west-1.amazonaws.com/?Action=PutMetricData&MetricData.member.1.Dimensions.member.1.Name=Host&MetricData.member.1.Dimensions.member.1.Value=i-081dcbd2a7b7f0852&MetricData.member.1.Dimensions.member.2.Name=PluginInstance&MetricData.member.1.Dimensions.member.2.Value=NONE&MetricData.member.1.MetricName=cpu.percent.active&MetricData.member.1.StatisticValues.Maximum=0.5&MetricData.member.1.StatisticValues.Minimum=0.0&MetricData.member.1.StatisticValues.SampleCount=6&MetricData.member.1.StatisticValues.Sum=0.700300500902&MetricData.member.1.Timestamp=20171004T070624Z&MetricData.member.2.Dimensions.member.1.Name=Host&MetricData.member.2.Dimensions.member.1.Value=i-081dcbd2a7b7f0852&MetricData.member.2.Dimensions.member.2.Name=PluginInstance&MetricData.member.2.Dimensions.member.2.Value=NONE&MetricData.member.2.MetricName=memory.percent.used&MetricData.member.2.StatisticValues.Maximum=4.5757107528&MetricData.member.2.StatisticValues.Minimum=4.3323386136&MetricData.member.2.StatisticValues.SampleCount=6&MetricData.member.2.StatisticValues.Sum=26.7001647382&MetricData.member.2.Timestamp=20171004T070624Z&MetricData.member.3.Dimensions.member.1.Name=Host&MetricData.member.3.Dimensions.member.1.Value=i-081dcbd2a7b7f0852&MetricData.member.3.Dimensions.member.2.Name=PluginInstance&MetricData.member.3.Dimensions.member.2.Value=root&MetricData.member.3.MetricName=df.percent_bytes.used&MetricData.member.3.StatisticValues.Maximum=14.0054941177&MetricData.member.3.StatisticValues.Minimum=14.0054941177&MetricData.member.3.StatisticValues.SampleCount=6&MetricData.member.3.StatisticValues.Sum=84.0329647064&MetricData.member.3.Timestamp=20171004T070624Z&Namespace=collectd&Version=2010-08-01&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=accesskey%2F20171004%2Feu-west-1%2Fmonitoring%2Faws4_request&X-Amz-Date=20171004T070724Z&X-Amz-SignedHeaders=host&X-Amz-Signature=f4200b40a340830c887f2b32fd6cf880e8e3f33e20696ff3c805f72ba5f9d9fe'
* Rebuilt URL to: 1/
*   Trying 0.0.0.1...
* TCP_NODELAY set
* Immediate connect fail for 0.0.0.1: Invalid argument
* Closing connection 0
curl: (7) Couldn't connect to server
000:000::0.004:0.000:0.000:0.000:0.000:0.000:0.000*   Trying 52.94.219.139...
* TCP_NODELAY set
* Connected to monitoring.eu-west-1.amazonaws.com (52.94.219.139) port 443 (#1)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* ALPN/NPN, server did not agree to a protocol
* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
* Server certificate:
* 	subject: CN=monitoring.eu-west-1.amazonaws.com,O="Amazon.com, Inc.",L=Seattle,ST=Washington,C=US
* 	start date: Sep 26 00:00:00 2017 GMT
* 	expire date: Jun 26 23:59:59 2018 GMT
* 	common name: monitoring.eu-west-1.amazonaws.com
* 	issuer: CN=Symantec Class 3 Secure Server CA - G4,OU=Symantec Trust Network,O=Symantec Corporation,C=US
> GET /?Action=PutMetricData&MetricData.member.1.Dimensions.member.1.Name=Host&MetricData.member.1.Dimensions.member.1.Value=i-081dcbd2a7b7f0852&MetricData.member.1.Dimensions.member.2.Name=PluginInstance&MetricData.member.1.Dimensions.member.2.Value=NONE&MetricData.member.1.MetricName=cpu.percent.active&MetricData.member.1.StatisticValues.Maximum=0.5&MetricData.member.1.StatisticValues.Minimum=0.0&MetricData.member.1.StatisticValues.SampleCount=6&MetricData.member.1.StatisticValues.Sum=0.700300500902&MetricData.member.1.Timestamp=20171004T070624Z&MetricData.member.2.Dimensions.member.1.Name=Host&MetricData.member.2.Dimensions.member.1.Value=i-081dcbd2a7b7f0852&MetricData.member.2.Dimensions.member.2.Name=PluginInstance&MetricData.member.2.Dimensions.member.2.Value=NONE&MetricData.member.2.MetricName=memory.percent.used&MetricData.member.2.StatisticValues.Maximum=4.5757107528&MetricData.member.2.StatisticValues.Minimum=4.3323386136&MetricData.member.2.StatisticValues.SampleCount=6&MetricData.member.2.StatisticValues.Sum=26.7001647382&MetricData.member.2.Timestamp=20171004T070624Z&MetricData.member.3.Dimensions.member.1.Name=Host&MetricData.member.3.Dimensions.member.1.Value=i-081dcbd2a7b7f0852&MetricData.member.3.Dimensions.member.2.Name=PluginInstance&MetricData.member.3.Dimensions.member.2.Value=root&MetricData.member.3.MetricName=df.percent_bytes.used&MetricData.member.3.StatisticValues.Maximum=14.0054941177&MetricData.member.3.StatisticValues.Minimum=14.0054941177&MetricData.member.3.StatisticValues.SampleCount=6&MetricData.member.3.StatisticValues.Sum=84.0329647064&MetricData.member.3.Timestamp=20171004T070624Z&Namespace=collectd&Version=2010-08-01&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=accesskey%2F20171004%2Feu-west-1%2Fmonitoring%2Faws4_request&X-Amz-Date=20171004T070724Z&X-Amz-SignedHeaders=host&X-Amz-Signature=f4200b40a340830c887f2b32fd6cf880e8e3f33e20696ff3c805f72ba5f9d9fe HTTP/1.1
> Host: monitoring.eu-west-1.amazonaws.com
> User-Agent: collectd/1.0
> Accept: */*
> 
< HTTP/1.1 403 Forbidden
HTTP/1.1 403 Forbidden
< x-amzn-RequestId: dd2d4791-a8d2-11e7-a9a4-17012e56af42
x-amzn-RequestId: dd2d4791-a8d2-11e7-a9a4-17012e56af42
< Content-Type: text/xml
Content-Type: text/xml
< Content-Length: 312
Content-Length: 312
< Date: Wed, 04 Oct 2017 07:08:46 GMT
Date: Wed, 04 Oct 2017 07:08:46 GMT

< 
<ErrorResponse xmlns="http://monitoring.amazonaws.com/doc/2010-08-01/">
  <Error>
    <Type>Sender</Type>
    <Code>InvalidClientTokenId</Code>
    <Message>The security token included in the request is invalid.</Message>
  </Error>
  <RequestId>dd2d4791-a8d2-11e7-a9a4-17012e56af42</RequestId>
</ErrorResponse>
* Curl_http_done: called premature == 0
* Connection #1 to host monitoring.eu-west-1.amazonaws.com left intact

@ratneshamarnath
Copy link
Author

Hey it worked when I created new AWS instance with same Role.

so why its not working if I assign role to existing AWS instance ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@yimuniao @ratneshamarnath