MongoDB connectivity issue

[mkrum001]

Hello,
I have issues with connecting to MongoDB Atlas replicas. I have followed this doccumentation I have 3 MongoDB instances ( : ) from the replica set, configured in my Kubernetes service. They were taken from the "dig" command output, port is the default 27017.

My current config looks like:

apiVersion: v1
kind: Service
metadata:
  name: datadog-mongodb
  labels:
    tags.datadoghq.com/env: 'dev'
    tags.datadoghq.com/service: 'mongodb-atlas'
  annotations:
    ad.datadoghq.com/service.checks: |
      {
        "mongo": {
          "init_config": {},
          "instances": [
            {
              "hosts": ["somecluster-shard-00-00.XXX.mongodb.net"],
              "username": "ENC[k8s_secret@monitoring/datadog-secrets/user]",
              "password": "ENC[k8s_secret@monitoring/datadog-secrets/password]",
              "options": {
                "authSource": "admin"
              },
              "dbm": true,
              "cluster_name": "dev",
              "additional_metrics": ["metrics.commands", "tcmalloc", "top", "collection"],
              "collections_indexes_stats": true,
              "database_autodiscovery": {
                "enabled": true
              }
            },
            {
              "hosts": ["somecluster-shard-00-01.XXX.mongodb.net"],
              "username": "ENC[k8s_secret@monitoring/datadog-secrets/user]",
              "password": "ENC[k8s_secret@monitoring/datadog-secrets/password]",
              "options": {
                "authSource": "admin"
              },
              "dbm": true,
              "cluster_name": "dev",
              "additional_metrics": ["metrics.commands", "tcmalloc", "top", "collection"],
              "collections_indexes_stats": true,
              "database_autodiscovery": {
                "enabled": true
              }
            },
            {
              "hosts": ["somecluster-shard-00-02.XXX.mongodb.net"],
              "username": "ENC[k8s_secret@monitoring/datadog-secrets/user]",
              "password": "ENC[k8s_secret@monitoring/datadog-secrets/password]",
              "options": {
                "authSource": "admin"
              },
              "dbm": true,
              "cluster_name": "dev",
              "additional_metrics": ["metrics.commands", "tcmalloc", "top", "collection"],
              "collections_indexes_stats": true,
              "database_autodiscovery": {
                "enabled": true
              }
            }
          ]
        }
      }
spec:
  ports:
    - port: 27017
      protocol: TCP
      targetPort: 27017
      name: mongodb
---
apiVersion: datadoghq.com/v2alpha1
kind: DatadogAgent
metadata:
  name: datadog
spec:
  global:
    site: {{ .Values.agent.spec.global.site }}
    credentials:
      apiSecret:
        secretName: {{ .Values.agent.spec.global.secret_apikey }}
        keyName: api-key
      appSecret:
        secretName: {{ .Values.agent.spec.global.secret_appkey }}
        keyName: app-key
  features:
    liveContainerCollection:
      enabled: true
    logCollection:
      enabled: true
      containerCollectAll: true
    clusterChecks:
      enabled: true
  override:
    nodeAgent:
      image:
        name: {{ .Values.agent.image }}
      tolerations:
        - operator: Exists
      containers:
        agent:
          env:
            - name: DD_SECRET_BACKEND_COMMAND
              value: "/readsecret_multiple_providers.sh"```

And this is the error from the "agent status" command:

root@datadog-agent-bsml4:/# agent status | grep mongo
    mongo (8.1.0)
      Instance ID: mongo:20835cded2c966e5 [ERROR]
      Configuration Source: kube_services:kube_service://monitoring/datadog-mongodb
      Error: dev-shard-00-02.XXX.mongodb.net:27017: connection closed (configured timeouts: socketTimeoutMS: 30000.0ms, connectTimeoutMS: 30000.0ms), Timeout: 29.999983082991093s, Topology Description: <TopologyDescription id: 6757043df9e935be6e8089b6, topology_type: Single, servers: [<ServerDescription ('dev-shard-00-02.XXX.mongodb.net', 27017) server_type: Unknown, rtt: None, error=AutoReconnect('dev-shard-00-02.XXX.mongodb.net:27017: connection closed (configured timeouts: socketTimeoutMS: 30000.0ms, connectTimeoutMS: 30000.0ms)')>]>
        File "/opt/datadog-agent/embedded/lib/python3.12/site-packages/datadog_checks/mongo/mongo.py", line 283, in check
        File "/opt/datadog-agent/embedded/lib/python3.12/site-packages/datadog_checks/mongo/mongo.py", line 270, in check
        File "/opt/datadog-agent/embedded/lib/python3.12/site-packages/datadog_checks/mongo/mongo.py", line 290, in _refresh_metadata
        File "/opt/datadog-agent/embedded/lib/python3.12/site-packages/datadog_checks/mongo/api.py", line 85, in server_info
        File "/opt/datadog-agent/embedded/lib/python3.12/site-packages/pymongo/mongo_client.py", line 1928, in server_info
        File "/opt/datadog-agent/embedded/lib/python3.12/site-packages/pymongo/_csot.py", line 107, in csot_wrapper
        File "/opt/datadog-agent/embedded/lib/python3.12/site-packages/pymongo/database.py", line 893, in command
        File "/opt/datadog-agent/embedded/lib/python3.12/site-packages/pymongo/mongo_client.py", line 1375, in _conn_for_reads
        File "/opt/datadog-agent/embedded/lib/python3.12/site-packages/pymongo/mongo_client.py", line 1322, in _select_server
        File "/opt/datadog-agent/embedded/lib/python3.12/site-packages/pymongo/topology.py", line 368, in select_server
        File "/opt/datadog-agent/embedded/lib/python3.12/site-packages/pymongo/topology.py", line 346, in _select_server
        File "/opt/datadog-agent/embedded/lib/python3.12/site-packages/pymongo/topology.py", line 253, in select_servers
        File "/opt/datadog-agent/embedded/lib/python3.12/site-packages/pymongo/topology.py", line 303, in _select_servers_loop
      pymongo.errors.ServerSelectionTimeoutError: dev-shard-00-02.XXX.mongodb.net:27017: connection closed (configured timeouts: socketTimeoutMS: 30000.0ms, connectTimeoutMS: 30000.0ms), Timeout: 29.999983082991093s, Topology Description: <TopologyDescription id: 6757043df9e935be6e8089b6, topology_type: Single, servers: [<ServerDescription ('dev-shard-00-02.XXX.mongodb.net', 27017) server_type: Unknown, rtt: None, error=AutoReconnect('dev-shard-00-02.XXX.mongodb.net:27017: connection closed (configured timeouts: socketTimeoutMS: 30000.0ms, connectTimeoutMS: 30000.0ms)')>]>

Auth info of the user that I'm using:

Atlas dev-shard-0 [primary] admin> db.runCommand({ connectionStatus: 1 }).authInfo
{
  authenticatedUsers: [ { user: 'datadog', db: 'admin' } ],
  authenticatedUserRoles: [
    { role: 'datadog', db: 'admin' },
    { role: 'read', db: 'config' },
    { role: 'read', db: 'local' },
    { role: 'read', db: 'mydb' },
    { role: 'read', db: 'clusterMonitor' },
    { role: 'read', db: 'admin' }
  ]
}

In the Mongo database network config I have 0.0.0.0/0 so it is not restricting me from connecting.
Telnet from datadog pod is able to connect on 27017 for any of the three hosts.

I tried to troubleshoot with the following commands:

mongosh "mongodb+srv://user:[email protected]/admin

and successfuly connecting to the cluster address, but as far as I understand it is not correct to use Cluster host. I even tried, but the agent can't resolve shards hosts when using cluster host address.

Then with this one for connecting to the shard host:

mongosh "mongodb://user:[email protected]"

I've got:

MongoServerSelectionError: read ECONNRESET. It looks like this is a MongoDB Atlas cluster. Please ensure that your Network Access List allows connections from your IP.