http client never finishes in asynchronous mode on HTTP_METHOD_HEAD (IDFGH-7444)

[MarcGeh]

Environment

• Development Kit: ESP32-DevKitC
• Kit version: v4
• Module or chip used: ESP32-WROVER-E
• IDF version: v4.4.1
• Build System: idf.py
• Compiler version: xtensa-esp32-elf-gcc (crosstool-NG esp-2021r2-patch3) 8.4.0
• Operating System: Linux
• Using an IDE?: No
• Power Supply: USB

Problem Description

When using the esp http client in asynchronous mode on a HTTPS connection with the method HTTP_METHOD_HEAD subsequent calls to esp_http_client_perform(..) will never return anything other than ESP_ERR_HTTP_EAGAIN if the connection has been established successfully and all headers have been received.

Expected Behavior

After all headers have been received esp_http_client_perform(..) should return ESP_OK just like in synchronous mode.

Actual Behavior

After all headers have been received esp_http_client_perform(..) will return ESP_ERR_HTTP_EAGAIN indefinitely.
I believe the issue is due to the behavior of esp_http_client_get_data line(..). In line 992 in esp_http_client.c this function returns 0 if the method is HTTP_METHOD_HEAD. While the client is in HTTP_STATE_RES_ON_DATA_START this function is called and since errno is not modified within it, it is most likely still set to EAGAIN from the previous transaction, which in turn causes this check in line 1181 to always be true.

Steps to reproduce

1. Setup a http client for a HTTPS connection with is_async set to true
2. Set the method to HTTP_METHOD_HEAD
3. Call esp_http_client_perform in a loop

[ESP-YJM]

@MarcGeh Thanks for your detailed description. Could you please add errno = 0 in this line

esp-idf/components/esp_http_client/esp_http_client.c

Line 991 in 179ea87

.

[MarcGeh]

@ESP-YJM Thanks, I can confirm that this fixes the issue. Will this fix be part of the next release?

[ESP-YJM]

@MarcGeh I'm not sure, but we will fix this as soon as possible.

[MarcGeh]

FYI for anyone having this issue and not wanting to modify any files inside the esp-idf to fix it, it is also sufficient to simply set errno = 0 before each call of esp_http_client_perform(...)

[AxelLin]

@MarcGeh I'm not sure, but we will fix this as soon as possible.

@ESP-YJM How is the status now? Is this fixed?

[ESP-YJM]

@AxelLin Not yet, userspace can avoid it, so I lowered the priority.

[AxelLin]

@AxelLin Not yet, userspace can avoid it, so I lowered the priority.

Then this never get fixed.

[kriegste]

Any news here?

[ESP-YJM]

@AxelLin @kriegste I have raise a MR internal.

[AxelLin]

@AxelLin @kriegste I have raise a MR internal.

How is the status now?

[AxelLin]

@MarcGeh I'm not sure, but we will fix this as soon as possible.

@ESP-YJM ASAP? I'm wondering if this will be fixed in 2023.

[ESP-YJM]

@AxelLin Sorry to response late. The MR need add some automated test and i am busy for other projects. When I have time I will add and push this MR integration. I think this problem can be avoided at the application layer first.

[AxelLin]

@ESP-YJM

Just remind this issue was reported on May 25, 2022, which is quite long time ago.
Seems you think people can easily workaround the issue.
However, for most users it's not easy to find this old issue at all.
Which means it may take quite long time to figure out why it does not work.
IMHO, you should fix it asap and backport to release branches since this is indeed a BUG.

[ESP-YJM]

@AxelLin It has beed merged into master branch and it will backport to v5.x branch.

[AxelLin]

@ESP-YJM Could you share which commit fixed this?

[ESP-YJM]

@AxelLin It has merged into the internal master branch and need wait ci passed and will sync to github.