TLJH upgrade causes SSLV3_ALERT_HANDSHAKE_FAILURE until ciphers explicitly configured

[fredcy]

Bug description

Upgrading TLJH breaks LDAP authentication with this error:

ldap3.core.exceptions.LDAPSocketOpenError: ("('socket ssl wrapping error: [SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:1007)',)",)

If I explicitly set the list of TLS ciphers in the TLJH config as below, then authentication starts working again. So maybe this is not a bug-report per se, but report of a workaround that took me a long time to discover.

    tls_strategy: "on_connect"
    tls_kwargs: { "ciphers": "TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_128_GCM_SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:DHE-RSA-AES256-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES256-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES128-SHA:DHE-RSA-AES128-SHA:RSA-PSK-AES256-GCM-SHA384:DHE-PSK-AES256-GCM-SHA384:RSA-PSK-CHACHA20-POLY1305:DHE-PSK-CHACHA20-POLY1305:ECDHE-PSK-CHACHA20-POLY1305:AES256-GCM-SHA384:PSK-AES256-GCM-SHA384:PSK-CHACHA20-POLY1305:RSA-PSK-AES128-GCM-SHA256:DHE-PSK-AES128-GCM-SHA256:AES128-GCM-SHA256:PSK-AES128-GCM-SHA256:AES256-SHA256:AES128-SHA256:ECDHE-PSK-AES256-CBC-SHA384:ECDHE-PSK-AES256-CBC-SHA:SRP-RSA-AES-256-CBC-SHA:SRP-AES-256-CBC-SHA:RSA-PSK-AES256-CBC-SHA384:DHE-PSK-AES256-CBC-SHA384:RSA-PSK-AES256-CBC-SHA:DHE-PSK-AES256-CBC-SHA:AES256-SHA:PSK-AES256-CBC-SHA384:PSK-AES256-CBC-SHA:ECDHE-PSK-AES128-CBC-SHA256:ECDHE-PSK-AES128-CBC-SHA:SRP-RSA-AES-128-CBC-SHA:SRP-AES-128-CBC-SHA:RSA-PSK-AES128-CBC-SHA256:DHE-PSK-AES128-CBC-SHA256:RSA-PSK-AES128-CBC-SHA:DHE-PSK-AES128-CBC-SHA:AES128-SHA:PSK-AES128-CBC-SHA256:PSK-AES128-CBC-SHA" }

How to reproduce

Start with a year-old TLJH instance that uses ldapauthenticator configured with use_ssl: True.

Update TLJH to the latest version, and thus ldapauthenticator also.

Try to login at the TLJH web home page:

Expected behaviour

I expected to be able to login as I did before the upgrade.

Actual behaviour

It fails with HTTP 500 error, and the internal logs show the above SSLV3_ALERT_HANDSHAKE_FAILURE error.

Your personal set up

I'm using the TLJH on Ubuntu 22.04.5 LTS.

We originally installed TLJH a year ago.

Our OpenLDAP server expects to connect with TLS 1.2 using TLS-over-SSL on port 636.

• Version(s):

Python 3.10.12

Full environment

aiohttp==3.8.5
aiosignal==1.3.1
alembic==1.12.0
annotated-types==0.7.0
arrow==1.3.0
async-generator==1.10
async-timeout==4.0.3
attrs==23.1.0
backoff==2.2.1
bcrypt==4.0.1
certifi==2023.7.22
certipy==0.1.3
cffi==1.15.1
charset-normalizer==3.2.0
cryptography==41.0.3
escapism==1.0.1
filelock==3.16.1
fqdn==1.5.1
frozenlist==1.4.0
greenlet==2.0.2
idna==3.4
isoduration==20.11.0
Jinja2==3.1.2
jsonpointer==3.0.0
jsonschema==4.19.0
jsonschema-specifications==2023.7.1
jupyter-events==0.10.0
jupyter-telemetry==0.1.0
jupyterhub==5.2.1
jupyterhub-firstuseauthenticator==1.1.0
jupyterhub-idle-culler==1.4.0
jupyterhub-ldapauthenticator==2.0.1
jupyterhub-nativeauthenticator==1.3.0
jupyterhub-systemdspawner==1.0.2
jupyterhub-tmpauthenticator==1.0.0
jupyterhub-traefik-proxy==2.0.0
ldap3==2.9.1
Mako==1.2.4
MarkupSafe==2.1.3
multidict==6.0.4
oauthenticator==17.1.0
oauthlib==3.2.2
onetimepass==1.0.1
packaging==23.1
pamela==1.1.0
passlib==1.7.4
pluggy==1.3.0
prometheus-client==0.17.1
pyasn1==0.5.0
pycparser==2.21
pycurl==7.45.3
pydantic==2.9.2
pydantic_core==2.23.4
PyJWT==2.9.0
pyOpenSSL==23.2.0
python-dateutil==2.8.2
python-json-logger==2.0.7
PyYAML==6.0.2
referencing==0.30.2
requests==2.31.0
rfc3339-validator==0.1.4
rfc3986-validator==0.1.1
rpds-py==0.10.2
ruamel.yaml==0.18.6
ruamel.yaml.clib==0.2.7
six==1.16.0
SQLAlchemy==2.0.20
the-littlest-jupyterhub @ git+https://github.com/jupyterhub/the-littlest-jupyterhub.git@a47a171850afc22d46f3b5b5bb0d09f2fae0937f
toml==0.10.2
tornado==6.3.3
traitlets==5.9.0
types-python-dateutil==2.9.0.20241003
typing_extensions==4.7.1
uri-template==1.3.0
urllib3==2.0.4
webcolors==24.8.0
yarl==1.9.2

[fredcy]

I got the cipher list from the output of openssl ciphers -tls1_2

[consideRatio]

I think you configured the following ciphers

['AES128-GCM-SHA256',
 'AES128-SHA',
 'AES128-SHA256',
 'AES256-GCM-SHA384',
 'AES256-SHA',
 'AES256-SHA256',
 'DHE-PSK-AES128-CBC-SHA',
 'DHE-PSK-AES128-CBC-SHA256',
 'DHE-PSK-AES128-GCM-SHA256',
 'DHE-PSK-AES256-CBC-SHA',
 'DHE-PSK-AES256-CBC-SHA384',
 'DHE-PSK-AES256-GCM-SHA384',
 'DHE-PSK-CHACHA20-POLY1305',
 'DHE-RSA-AES128-GCM-SHA256',
 'DHE-RSA-AES128-SHA',
 'DHE-RSA-AES128-SHA256',
 'DHE-RSA-AES256-GCM-SHA384',
 'DHE-RSA-AES256-SHA',
 'DHE-RSA-AES256-SHA256',
 'DHE-RSA-CHACHA20-POLY1305',
 'ECDHE-ECDSA-AES128-GCM-SHA256',
 'ECDHE-ECDSA-AES128-SHA',
 'ECDHE-ECDSA-AES128-SHA256',
 'ECDHE-ECDSA-AES256-GCM-SHA384',
 'ECDHE-ECDSA-AES256-SHA',
 'ECDHE-ECDSA-AES256-SHA384',
 'ECDHE-ECDSA-CHACHA20-POLY1305',
 'ECDHE-PSK-AES128-CBC-SHA',
 'ECDHE-PSK-AES128-CBC-SHA256',
 'ECDHE-PSK-AES256-CBC-SHA',
 'ECDHE-PSK-AES256-CBC-SHA384',
 'ECDHE-PSK-CHACHA20-POLY1305',
 'ECDHE-RSA-AES128-GCM-SHA256',
 'ECDHE-RSA-AES128-SHA',
 'ECDHE-RSA-AES128-SHA256',
 'ECDHE-RSA-AES256-GCM-SHA384',
 'ECDHE-RSA-AES256-SHA',
 'ECDHE-RSA-AES256-SHA384',
 'ECDHE-RSA-CHACHA20-POLY1305',
 'PSK-AES128-CBC-SHA',
 'PSK-AES128-CBC-SHA256',
 'PSK-AES128-GCM-SHA256',
 'PSK-AES256-CBC-SHA',
 'PSK-AES256-CBC-SHA384',
 'PSK-AES256-GCM-SHA384',
 'PSK-CHACHA20-POLY1305',
 'RSA-PSK-AES128-CBC-SHA',
 'RSA-PSK-AES128-CBC-SHA256',
 'RSA-PSK-AES128-GCM-SHA256',
 'RSA-PSK-AES256-CBC-SHA',
 'RSA-PSK-AES256-CBC-SHA384',
 'RSA-PSK-AES256-GCM-SHA384',
 'RSA-PSK-CHACHA20-POLY1305',
 'SRP-AES-128-CBC-SHA',
 'SRP-AES-256-CBC-SHA',
 'SRP-RSA-AES-128-CBC-SHA',
 'SRP-RSA-AES-256-CBC-SHA',
 'TLS_AES_128_GCM_SHA256',
 'TLS_AES_256_GCM_SHA384',
 'TLS_CHACHA20_POLY1305_SHA256']

where the default ciphers may have been:

['DHE-RSA-AES128-GCM-SHA256',
 'DHE-RSA-AES128-SHA256',
 'DHE-RSA-AES256-GCM-SHA384',
 'DHE-RSA-AES256-SHA256',
 'ECDHE-ECDSA-AES128-GCM-SHA256',
 'ECDHE-ECDSA-AES128-SHA256',
 'ECDHE-ECDSA-AES256-GCM-SHA384',
 'ECDHE-ECDSA-AES256-SHA384',
 'ECDHE-ECDSA-CHACHA20-POLY1305',
 'ECDHE-RSA-AES128-GCM-SHA256',
 'ECDHE-RSA-AES128-SHA256',
 'ECDHE-RSA-AES256-GCM-SHA384',
 'ECDHE-RSA-AES256-SHA384',
 'ECDHE-RSA-CHACHA20-POLY1305',
 'TLS_AES_128_GCM_SHA256',
 'TLS_AES_256_GCM_SHA384',
 'TLS_CHACHA20_POLY1305_SHA256']

Hmmm...

I wonder what cipher ended up being needed. Do you know what ciphers was supported via OpenLDAP? I observe that the default ciphers was all part of the larger list you specified.

[consideRatio]

In Python < 3.10, the following ciphers were part of the default ssl context:

['SSLv3__AES128-SHA',
 'SSLv3__AES256-SHA',
 'SSLv3__DHE-RSA-AES128-SHA',
 'SSLv3__DHE-RSA-AES256-SHA',
 'TLSv1.0__ECDHE-ECDSA-AES128-SHA',
 'TLSv1.0__ECDHE-ECDSA-AES256-SHA',
 'TLSv1.0__ECDHE-RSA-AES128-SHA',
 'TLSv1.0__ECDHE-RSA-AES256-SHA',
 'TLSv1.2__AES128-GCM-SHA256',
 'TLSv1.2__AES128-SHA256',
 'TLSv1.2__AES256-GCM-SHA384',
 'TLSv1.2__AES256-SHA256',
 'TLSv1.2__DHE-RSA-AES128-GCM-SHA256',
 'TLSv1.2__DHE-RSA-AES128-SHA256',
 'TLSv1.2__DHE-RSA-AES256-GCM-SHA384',
 'TLSv1.2__DHE-RSA-AES256-SHA256',
 'TLSv1.2__DHE-RSA-CHACHA20-POLY1305',
 'TLSv1.2__ECDHE-ECDSA-AES128-GCM-SHA256',
 'TLSv1.2__ECDHE-ECDSA-AES128-SHA256',
 'TLSv1.2__ECDHE-ECDSA-AES256-GCM-SHA384',
 'TLSv1.2__ECDHE-ECDSA-AES256-SHA384',
 'TLSv1.2__ECDHE-ECDSA-CHACHA20-POLY1305',
 'TLSv1.2__ECDHE-RSA-AES128-GCM-SHA256',
 'TLSv1.2__ECDHE-RSA-AES128-SHA256',
 'TLSv1.2__ECDHE-RSA-AES256-GCM-SHA384',
 'TLSv1.2__ECDHE-RSA-AES256-SHA384',
 'TLSv1.2__ECDHE-RSA-CHACHA20-POLY1305',
 'TLSv1.3__TLS_AES_128_GCM_SHA256',
 'TLSv1.3__TLS_AES_256_GCM_SHA384',
 'TLSv1.3__TLS_CHACHA20_POLY1305_SHA256']

In Python 3.10+, that became strictly narrower list with no new ciphers added:

['TLSv1.2__DHE-RSA-AES128-GCM-SHA256',
 'TLSv1.2__DHE-RSA-AES128-SHA256',
 'TLSv1.2__DHE-RSA-AES256-GCM-SHA384',
 'TLSv1.2__DHE-RSA-AES256-SHA256',
 'TLSv1.2__ECDHE-ECDSA-AES128-GCM-SHA256',
 'TLSv1.2__ECDHE-ECDSA-AES128-SHA256',
 'TLSv1.2__ECDHE-ECDSA-AES256-GCM-SHA384',
 'TLSv1.2__ECDHE-ECDSA-AES256-SHA384',
 'TLSv1.2__ECDHE-ECDSA-CHACHA20-POLY1305',
 'TLSv1.2__ECDHE-RSA-AES128-GCM-SHA256',
 'TLSv1.2__ECDHE-RSA-AES128-SHA256',
 'TLSv1.2__ECDHE-RSA-AES256-GCM-SHA384',
 'TLSv1.2__ECDHE-RSA-AES256-SHA384',
 'TLSv1.2__ECDHE-RSA-CHACHA20-POLY1305',
 'TLSv1.3__TLS_AES_128_GCM_SHA256',
 'TLSv1.3__TLS_AES_256_GCM_SHA384',
 'TLSv1.3__TLS_CHACHA20_POLY1305_SHA256']

[consideRatio]

I suspect upgrading to using a more modern version of Python led to the list of ciphers was too narrow for an agreement with the server, which doesn't sound bad security wise --- could it be that your LDAP server isn't allowing any cipher part of the python 3.10 list?

I figure its probably not suitable to expand the list of ciphers by default within this project.

[fredcy]

Using tcpdump and wireshark, I found that the TLJH setup that I've been using over the last year would settle on the TLS_RSA_WITH_AES_256_GCM_SHA384 cipher when connecting successfully to our LDAP server. (That's the name as displayed by wireshark, apparently encoded as 0x009d). When I got the connection to work again on the latest TLJH update and with the ciphers explicitly listed as above, the same cipher was chosen.

I don't have access to our LDAP server but it's running RHEL 7 and has correspondingly older versions of the openssl libs.

[fredcy]

In lieu of extending the list of default ciphers, perhaps the docs for ldapauthenticator could mention the symptoms and workaround for the case of cipher mismatch.

[consideRatio]

I think TLS_RSA_WITH_AES_256_GCM_SHA384 is what openssl calls AES256-GCM-SHA384 (TLS 1.2) which was indeed used with Python < 3.10, but not after Python >= 3.10.

Amazing debugging into this @fredcy, I fully agree on docs about this - possibly also a log message if this error ocurr to point users in the right direction.

[consideRatio]

@fredcy I opened #297 about this, thank you for digging in deep to this and reporting this -- I feel confident this will help others as well!