Prometheus container access denied issue

[iamtejasmane]

Hi @santosh07bec , first of all, thanks for the efforts you have put into creating the ecs-prometheus repository, which I have found to be extremely helpful in my current project.

Issue: I have deployed all four containers successfully but facing this issue in the Prometheus container.

level=error ts=2023-04-13T02:55:13.45984975Z caller=ec2.go:184 component="discovery manager scrape" discovery=ec2 msg="Refresh failed" err="could not describe instances: AccessDenied: User: arn:aws:sts::9486709xxxxx:assumed-role/ecsTaskExecutionRole/a72e28e47203410faea0fc6f18c1f262 is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::9486709xxxxx:role/ecsInstanceRole\n\tstatus code: 403”

Steps were taken to resolve the issue:

1. The IAM permissions for both the ecsTaskExecutionRole and ecsInstanceRole were reviewed.
2. The DescribeInstances permission was assigned to both roles to ensure that they have the necessary access
3. Ensured that the ports are opened for communication.

Can please you please tell me how I could resolve this?

Thank you!

[akki93]

Hi Santosh,
Really thanks for sharing the piece of code for Prometheus and Grafana stack along with CAdvisor. I am also facing totally similar issue as @iamtejasmane
Can you please suggest the fix here, as I tried providing the ECS with EC2ReadOnly Access which contains the DescribeInstances permission but it still fails.