Certificate verification failing when Digital signature and TLS1.3 enabled (IDFGH-14097)

[msivanesancq]

Answers checklist.

• I have read the documentation ESP-IDF Programming Guide and the issue is not addressed there.
• I have updated my IDF branch (master or release) to the latest version and checked that the issue is present there.
• I have searched the issue tracker for a similar issue and not found a similar issue.

IDF version.

5.3.1

Espressif SoC revision.

ESP32-C3 QFN32(Rev v0.3)

Operating System used.

Windows

How did you build your project?

VS Code IDE

If you are using Windows, please specify command line type.

None

Development Kit.

ESP32-C3-MINI-1

Power Supply used.

USB

What is the expected behavior?

It looks like MBEDTLS_PK_RSA is the appropriate key type when digital signing peripheral is in use.
But device chooses MBEDTLS_PK_RSASSA_PSS option and finally failing to sign/verify in the MBEDTLS_SSL_CLIENT_CERTIFICATE_VERIFY state.

It seems ssl_preset_default_sig_algs is containing
MBEDTLS_TLS1_3_SIG_RSA_PSS_RSAE_SHA512
MBEDTLS_TLS1_3_SIG_RSA_PSS_RSAE_SHA384
and
MBEDTLS_TLS1_3_SIG_RSA_PSS_RSAE_SHA256
which is not really supported by DS peripheral.

What is the actual behavior?

See the logs below for more details:
The device is rejecting the certificate signature and sending alert to the server.
I tried by disabling MBEDTLS_X509_RSASSA_PSS_SUPPORT macro from mbedtls_config.h. But the error is still happening.

Steps to reproduce.

Perform the TLS 1.3 handshake with certificates(RootCA and temporary cert).

Debug Logs.

Standard setting
----------------
I (8944) mbedtls: IDF\components\mbedtls\mbedtls\library\ssl_tls.c:4525 client state: MBEDTLS_SSL_CLIENT_CERTIFICATE_VERIFY
 
I (8954) mbedtls: IDF\components\mbedtls\mbedtls\library\ssl_tls13_generic.c:1132 => write certificate verify

I (8964) mbedtls: IDF\components\mbedtls\mbedtls\library\ssl_tls13_generic.c:1093 CertificateVerify signature failed with rsa_pss_rsae_sha256
 
I (8984) mbedtls: IDF\components\mbedtls\mbedtls\library\ssl_tls13_generic.c:1095 mbedtls_pk_sign_ext() returned -16128 (-0x3f00)

I (8994) mbedtls: IDF\components\mbedtls\mbedtls\library\ssl_tls13_generic.c:1093 CertificateVerify signature failed with rsa_pss_rsae_sha384
 
I (9004) mbedtls: IDF\components\mbedtls\mbedtls\library\ssl_tls13_generic.c:1095 mbedtls_pk_sign_ext() returned -16128 (-0x3f00)
 
I (9024) mbedtls: IDF\components\mbedtls\mbedtls\library\ssl_tls13_generic.c:1093 CertificateVerify signature failed with rsa_pss_rsae_sha512
 
I (9034) mbedtls: IDF\components\mbedtls\mbedtls\library\ssl_tls13_generic.c:1095 mbedtls_pk_sign_ext() returned -16128 (-0x3f00)
 
W (9044) mbedtls: IDF\components\mbedtls\mbedtls\library\ssl_tls13_generic.c:1112 no suitable signature algorithm
 
I (9064) mbedtls: IDF\components\mbedtls\mbedtls\library\ssl_tls13_generic.c:1150 <= write certificate verify
 
I (9074) mbedtls: IDF\components\mbedtls\mbedtls\library\ssl_msg.c:5168 => send alert message
 
I (9084) mbedtls: IDF\components\mbedtls\mbedtls\library\ssl_msg.c:2943 => write record
 
I (9084) mbedtls: IDF\components\mbedtls\mbedtls\library\ssl_msg.c:935 => encrypt buf
 
I (9094) mbedtls: IDF\components\mbedtls\mbedtls\library\ssl_msg.c:1474 <= encrypt buf
 
I (9104) mbedtls: IDF\components\mbedtls\mbedtls\library\ssl_msg.c:2353 => flush output
 
I (9114) mbedtls: IDF\components\mbedtls\mbedtls\library\ssl_msg.c:2367 message length: 37, out_left: 37
 
I (9124) mbedtls: IDF\components\mbedtls\mbedtls\library\ssl_msg.c:2374 ssl->f_send() returned 37 (-0xffffffdb)
 
I (9134) mbedtls: IDF\components\mbedtls\mbedtls\library\ssl_msg.c:2401 <= flush output
 
I (9144) mbedtls: IDF\components\mbedtls\mbedtls\library\ssl_msg.c:3080 <= write record
 
I (9154) mbedtls: IDF\components\mbedtls\mbedtls\library\ssl_msg.c:5180 <= send alert message
 
I (9164) mbedtls: IDF\components\mbedtls\mbedtls\library\ssl_tls.c:4617 <= handshake
 
E (9174) esp-tls-mbedtls: mbedtls_ssl_handshake returned -0x6E00
I (9174) esp-tls-mbedtls: Certificate verified.
E (9184) esp-tls: Failed to open new connection
E (9194) transport_base: Failed to open a new connection
E (9194) HTTP_CLIENT: Connection failed, sock < 0

------------------------------------------------
MBEDTLS_X509_RSASSA_PSS_SUPPORT disabled setting
------------------------------------------------
I (8814) mbedtls: IDF\components\mbedtls\mbedtls\library\ssl_tls.c:4525 client state: MBEDTLS_SSL_CLIENT_CERTIFICATE_VERIFY

I (8824) mbedtls: IDF\components\mbedtls\mbedtls\library\ssl_tls13_generic.c:1140 => write certificate verify
I (8864) mbedtls: IDF\components\mbedtls\mbedtls\library\ssl_tls13_generic.c:1100 CertificateVerify signature failed with rsa_pss_rsae_sha256

I (8874) mbedtls: IDF\components\mbedtls\mbedtls\library\ssl_tls13_generic.c:1102 mbedtls_pk_sign_ext() returned -16128 (-0x3f00)

I (8944) mbedtls: IDF\components\mbedtls\mbedtls\library\ssl_tls13_generic.c:1100 CertificateVerify signature failed with rsa_pss_rsae_sha384

I (8954) mbedtls: IDF\components\mbedtls\mbedtls\library\ssl_tls13_generic.c:1102 mbedtls_pk_sign_ext() returned -16128 (-0x3f00)

I (9024) mbedtls: IDF\components\mbedtls\mbedtls\library\ssl_tls13_generic.c:1100 CertificateVerify signature failed with rsa_pss_rsae_sha512

I (9034) mbedtls: IDF\components\mbedtls\mbedtls\library\ssl_tls13_generic.c:1102 mbedtls_pk_sign_ext() returned -16128 (-0x3f00)

W (9094) mbedtls: IDF\components\mbedtls\mbedtls\library\ssl_tls13_generic.c:1120 no suitable signature algorithm

I (9104) mbedtls: IDF\components\mbedtls\mbedtls\library\ssl_tls13_generic.c:1158 <= write certificate verify

I (9114) mbedtls: IDF\components\mbedtls\mbedtls\library\ssl_msg.c:5168 => send alert message

I (9124) mbedtls: IDF\components\mbedtls\mbedtls\library\ssl_msg.c:2943 => write record

I (9134) mbedtls: IDF\components\mbedtls\mbedtls\library\ssl_msg.c:935 => encrypt buf

I (9144) mbedtls: IDF\components\mbedtls\mbedtls\library\ssl_msg.c:1474 <= encrypt buf

I (9154) mbedtls: IDF\components\mbedtls\mbedtls\library\ssl_msg.c:2353 => flush output

I (9164) mbedtls: IDF\components\mbedtls\mbedtls\library\ssl_msg.c:2367 message length: 37, out_left: 37

I (9174) mbedtls: IDF\components\mbedtls\mbedtls\library\ssl_msg.c:2374 ssl->f_send() returned 37 (-0xffffffdb)

I (9184) mbedtls: IDF\components\mbedtls\mbedtls\library\ssl_msg.c:2401 <= flush output

I (9194) mbedtls: IDF\components\mbedtls\mbedtls\library\ssl_msg.c:3080 <= write record

I (9204) mbedtls: IDF\components\mbedtls\mbedtls\library\ssl_msg.c:5180 <= send alert message

I (9214) mbedtls: IDF\components\mbedtls\mbedtls\library\ssl_tls.c:4617 <= handshake

E (9214) esp-tls-mbedtls: mbedtls_ssl_handshake returned -0x6E00
I (9224) esp-tls-mbedtls: Certificate verified.
E (9234) esp-tls: Failed to open new connection
E (9234) transport_base: Failed to open a new connection
E (9244) HTTP_CLIENT: Connection failed, sock < 0

More Information.

No response

[msivanesancq]

We figured out the root cause of this issue.
In certificate verify stage the hash must be encoded with pkcs1 v21 standard for TLS1.3.
DS(Digital siging) peripheral is not supporting this instead it is encoding with PKCS1 v1.5 using rsa_rsassa_pkcs1_v15_encode
This issue is now fixed in these PR by introducing rsa_rsassa_pkcs1_v21_encode()
cryptoquantique#1
cryptoquantique/mbedtls#2