Unable to verify CA: SSL handshake failed

[ffissore]

Since confluent-kafka-python 2.0.2 (bundling librdkafka v2.0.2), we started seeing our clients failing with
SSL handshake failed: error:0A000086:SSL routines::certificate verify failed: broker certificate could not be verified, verify that ssl.ca.location is correctly configured or root CA certificates are installed (install ca-certificates package)

Our CA is a self-signed one.

Nothing has changed in our configuration, we are still using the same options that work fine with librdkafka 1.9.2

security.protocol: ssl
ssl.ca.location: /var/config/ca-cert
ssl.certificate.location: /var/config/client.pem
ssl.key.location: /var/config/client.key

Does anyone know what's going on?

[ffissore]

After lots of attempts and googling, this might be the culprit: Verification error: CA signature digest algorithm too weak

This is what I get when I try to run an openssl handshake with openssl s_client -connect kafka-broker:9092 -tls1 -servername kafka-broker -cert configs/kafka/client/client.pem -key configs/kafka/client/client.key -CAfile configs/kafka/ca-cert

Any hint on how to relax openssl? What other options do I have?

[emasab]

@ffissore it's better if you regenerate your certificates, OpenSSL 3.0 removed some legacy algorithms:

Algorithms in the legacy provider include MD2, MD4, MDC2, RMD160, CAST5, BF (Blowfish), IDEA, SEED, RC2, RC4, RC5 and DES (but not 3DES).

if it's not possible you can configure ssl.providers=default,legacy, check latest CHANGELOG. But be warned that it's a security regression.

[ffissore]

I tried that setting and it didn't help. We plan to upgrade our kafka cluster certificates, but I still have to gather the documentation to do that in a non-disruptive way (the cluster needs to work with both old and new CAs and certificates, or we'll create an outage)

[moritzmeister]

I am having a similar issue since last week, producers can't connect to the brokers:

%7|1678200724.581|FAIL|meister#producer-1| [thrd:ssl://3.142.251.253:9092/bootstrap]: ssl://kafka:9092/bootstrap: SSL handshake failed: ssl/statem/statem_clnt.c:1921:tls_process_server_certificate error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed: broker certificate could not be verified, verify that ssl.ca.location is correctly configured or root CA certificates are installed (brew install openssl) (after 116ms in state SSL_HANDSHAKE) (_SSL)

I don't see what you @ffissore are seeing when running a manual openssl handshake. However, I see another error:

4366501248:error:141A10F4:SSL routines:ossl_statem_client_read_transition:unexpected message:ssl/statem/statem_clnt.c:395:

[zhangwen-network]

I am having a similar issue, producer(1.8.2 update to 2.1.1) can't connect to the brokers:SSL handshake failed: ssl/statem/statem_clnt.c:1916:tls_process_server_certificate error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed: broker certificate could not be verified.
The version of libssl not changed,why 1.8.2 work fine, but 2.1.1 work failed?

[ffissore]

Because since librdkafka 2, the bundled version of openssl has changed. The version installed on your machine doesn't count: librdkafka uses its own bundled version

[zhangwen-network]

Thanks a lot. Have you find the way to work well with both old and new certificates?

[ffissore]

See my reply below

[ffissore]

I forgot to report the workaround that we found in the meantime: we lowered the default SECLEVEL of openssl

Our Dockerfiles now have this line on top: RUN sed -i 's/DEFAULT@SECLEVEL=2/DEFAULT@SECLEVEL=0/' /usr/lib/ssl/openssl.cnf (our images are debian-based).

Even though librdkafka bundles its own openssl, it still reads the usual configuration files: this lets us lower its expectation, until we'll be able to upgrade our cluster certificates

[zhangwen-network]

See #4349. With the config "ssl.endpoint.identification.algorithm=none", the producer works well.