[BUG] unable to find valid certification path to requested target

[yanivNaor92]

Describe the bug

I'm installing opensearch in my Kubernetes cluster by using the helm-charts.
After upgrading opensearch from version 2.11 to 2.17.1, I started getting the following error in opensearch-cluster-master pod:

[WARN ][o.o.h.AbstractHttpServerTransport] [opensearch-cluster-master-2] caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=/100.96.7.135:9200, remoteAddress=/100.96.8.78:33458}
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

I omitted the stack trace for brevity.

This is my opensearch.yml file:

cluster.name: opensearch-cluster
compatibility.override_main_response_version: true
# Bind to all interfaces because we don't know what IP address Docker will assign to us.
network.host: 0.0.0.0
# Setting network.host to a non-loopback address enables the annoying bootstrap checks. "Single-node" mode disables them again.
# Implicitly done if ".singleNode" is set to "true".
# discovery.type: single-node
# Start OpenSearch Security Demo Configuration
# WARNING: revise all the lines below before you go into production
plugins:
  security:
    ssl:
      transport:
        pemcert_filepath: esnode.pem
        pemkey_filepath: esnode-key.pem
        pemtrustedcas_filepath: root-ca.pem
        enforce_hostname_verification: false
        enabled_protocols:
          - "TLSv1.2"
      http:
        enabled: true
        pemcert_filepath: certs/tls.crt
        pemkey_filepath: certs/tls.key
        pemtrustedcas_filepath: certs/ca.crt
        clientauth_mode: REQUIRE
        enabled_protocols:
          - "TLSv1.2"
    allow_unsafe_democertificates: true
    allow_default_init_securityindex: true
    authcz:
      admin_dn:
        - CN=kirk,OU=client,O=client,L=test,C=de
    audit.type: internal_opensearch
    enable_snapshot_restore_privilege: true
    check_snapshot_restore_write_privileges: true
    restapi:
      roles_enabled: ["all_access", "security_rest_api_access"]
    system_indices:
      enabled: true
      indices:
        [
          ".opendistro-alerting-config",
          ".opendistro-alerting-alert*",
          ".opendistro-anomaly-results*",
          ".opendistro-anomaly-detector*",
          ".opendistro-anomaly-checkpoints",
          ".opendistro-anomaly-detection-state",
          ".opendistro-reports-*",
          ".opendistro-notifications-*",
          ".opendistro-notebooks",
          ".opendistro-asynchronous-search-response*",
        ]
######## End OpenSearch Security Demo Configuration ########

######## Start OpenSearch Security Demo Configuration ########
# WARNING: revise all the lines below before you go into production
plugins.security.ssl.transport.pemcert_filepath: esnode.pem
plugins.security.ssl.transport.pemkey_filepath: esnode-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: root-ca.pem
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: esnode.pem
plugins.security.ssl.http.pemkey_filepath: esnode-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: root-ca.pem
plugins.security.allow_unsafe_democertificates: true
plugins.security.allow_default_init_securityindex: true
plugins.security.authcz.admin_dn: ['CN=kirk,OU=client,O=client,L=test,C=de']
plugins.security.audit.type: internal_opensearch
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.restapi.roles_enabled: [all_access, security_rest_api_access]
plugins.security.system_indices.enabled: true
plugins.security.system_indices.indices: [.plugins-ml-agent, .plugins-ml-config, .plugins-ml-connector,
  .plugins-ml-controller, .plugins-ml-model-group, .plugins-ml-model, .plugins-ml-task,
  .plugins-ml-conversation-meta, .plugins-ml-conversation-interactions, .plugins-ml-memory-meta,
  .plugins-ml-memory-message, .plugins-ml-stop-words, .opendistro-alerting-config,
  .opendistro-alerting-alert*, .opendistro-anomaly-results*, .opendistro-anomaly-detector*,
  .opendistro-anomaly-checkpoints, .opendistro-anomaly-detection-state, .opendistro-reports-*,
  .opensearch-notifications-*, .opensearch-notebooks, .opensearch-observability, .ql-datasources,
  .opendistro-asynchronous-search-response*, .replication-metadata-store, .opensearch-knn-models,
  .geospatial-ip2geo-data*, .plugins-flow-framework-config, .plugins-flow-framework-templates,
  .plugins-flow-framework-state]
node.max_local_storage_nodes: 3
######## End OpenSearch Security Demo Configuration ########

I already validated the following:

• The certs/tls.crt, certs/tls.key and certs/ca.crt files exists in the config folder and contain valid certificate, private key and CA respectively. I manually created them using OpenSSL.
• The error occurs only when mTLS communication is enabled (clientauth_mode: REQUIRE). When clientauth_mode is set to NONE, the communication between opensearch-cluster and opensearch-dashboards works fine.
• The same configuration works fine with OpenSearch v2.11.0.
• I found the following issue, which reported a similar error message, but the solution provided there is not applicable to me because I install OpenSearch via the helm charts.

Note: I enabled only TLS 1.2 due to the following issue: opensearch-project/security#3299

Your help is much appreciated.

Related component

Plugins

To Reproduce

1. Install OpenSearch helm chart with version 2.26.1 (app version: 2.17.1) with the configuration above in a K8S cluster.

Expected behavior

mTLS communication between opensearch-cluster and opensearch-dashboards pods should work without any errors.

[dancristiancecoi]

The issue is due to having duplicate security config in opensearch.yml which in turn is probably due to demo configuration script being run by default on Helm charts even if you made changes to the security config.

As a consequence, the Security Plugin ends up using the demo certs instead of the ones you've added.

To get past this issue you can disable the demo security configuration by setting DISABLE_INSTALL_DEMO_CONFIG to "true" in the extraEnvs section of your values.yaml file.

extraEnvs:
  - name: DISABLE_INSTALL_DEMO_CONFIG
    value: "true"

Here is a similar bug raised in the helm-charts repo: opensearch-project/helm-charts#564

@cwperks is this something that can be fixed on the Security Plugin side?

[cwperks]

@dancristiancecoi This should be resolved in 2.18: opensearch-project/security#4793

[yanivNaor92]

@dancristiancecoi, thank you for your response.
Setting DISABLE_INSTALL_DEMO_CONFIG to true solved my issue.
However, I had to make a few more adjustments to my configuration. I'll post them here if someone else faces a similar issue.

After disabling the demo config, the default cert files that I used for the transport ssl configurations (esnode.pem, esnode-key.pem, etc.) no longer exist, so I had to reference my custom certificates in the transport config as well.
Also, I had to mention my custom certificate's DN in the plugins.security.nodes_dn field.

@cwperks Do you have an estimation about the release date of v2.18?

[cwperks]

@yanivNaor92 I expect it to be released by November 5: https://opensearch.org/releases.html

[Nmishin]

It looks like this was fixed in version 2.18 🎉. I have just updated the versions, and everything works as expected without any configuration updates. (I also updated the opensearch-k8s-operator to the latest version.)

[cwperks]

Thank you for confirming @Nmishin! Closing this issue.