no return from panic_abort(const char *details) (IDFGH-14221)

[ves011]

Answers checklist.

• I have read the documentation ESP-IDF Programming Guide and the issue is not addressed there.
• I have updated my IDF branch (master or release) to the latest version and checked that the issue is present there.
• I have searched the issue tracker for a similar issue and not found a similar issue.

General issue report

I have an application which ends in
*void IRAM_ATTR attribute((noreturn, no_sanitize_undefined)) panic_abort(const char details)
in panic.c line 449, and get stuck here.
Is there any reason why restart is not triggered?
The comment in user execption vector in xtensa_vectors.S line 623 is
/ never returns here - call0 is used as a jump (see note at top) /
But we need a way to recover from this situation without pressing reset button. A restart, like any other panic handlers, should be a reasonable option.
Are there other panic handlers which are not triggering restart?

[sudeep-mohanty]

Hi @ves011, Could you check if you have the the reboot behavior enabled for the panic handler?
You could do that in the menuconfig -> Component Config -> ESP System Settings -> Panic handler behavior.

Make sure you have either the Print registers and reboot or Silent reboot option selected.

[ves011]

Yes.
Its set to "Print registers and reboot".
So far all system panics ended with a restart. Its the first time i see code stuck and no message on the console.
I could capture the issue only running under debugger.

[sudeep-mohanty]

If you have a debugger attached then this behavior is expected. The system would set a breakpoint and let users probe for the faults. You can read about the configuration option CONFIG_ESP_DEBUG_OCDAWARE.

[ves011]

The issue shows up without debugger attached.
The code just get stuck: console is frozen, tcp is frozen , module is no longer accessible.
I could get to the point where it get stuck only after attaching a debugger.

[sudeep-mohanty]

Could you provide us the following information -

• Which target are you using?
• Your sdkconfig file.
• A minimal reproducible code for us to reproduce the issue and debug it at our end?

[ves011]

target is ESP32S3 WROOM1N4R2
I attached sdkconfig, generated sdkconfig.h and full sourse code of tcp_server.c
The issue pop up when running tcp_server in poor network conditions.
Reproducing is pretty difficult and will try to explain here.

• register_tcp_server() function creates in the first step tcp_server() task
• tcp_server task creates the server socket and enters accept connections loop
• if connection is accepted it creates the 2 other tasks: send_task and process_message_task
• if the 2 tasks created successfully it enters receive loop
• while in receive loop if a tcp error is detected then it breaks out the loop, deletes the 2 created tasks and return to accept loop
  Here is a meaningful extract of the code

static void send_task(void *pvParameters)
    {
    socket_message_t msg;
    int written, ret;
    int sock = ((tParams_t *)pvParameters)->socket;
    xQueueReset(tcp_send_queue);
    ESP_LOGI(TAG, "send_task started, socket: %d", sock);
    send_error = 0;
    while(1)
        {
        if(xQueueReceive(tcp_send_queue, &msg, portMAX_DELAY))
            {
            
            written = 0, ret = 0;
            while(written < sizeof(socket_message_t))
                {
                ret = send(sock, (uint8_t *)(&msg + written), sizeof(socket_message_t) - written, 0);
                if (ret < 0) 
                    {
                    send_error = 1;
                    ESP_LOGE(TAG, "Error occurred during sending message: errno %d", errno);
                    break;
                    }
                else
                    written += ret;
                }
            if(ret <= 0)
                {
                // need to handle here error while sending
                // or its enough recv() error handling in tcp_server_task ???
                }
            }
        }
    }

void tcp_server(void *pvParameters)
    {
    ...
        
    listen_sock = socket(AF_INET, SOCK_STREAM, ip_protocol);
    if (listen_sock >= 0)
        {
        ...
        ret = bind(listen_sock, (struct sockaddr *)&dest_addr, sizeof(dest_addr));
        if (ret == 0)
            {
            ESP_LOGI(TAG, "Socket bound, port %d", DEFAULT_TCP_SERVER_PORT);
            ret = listen(listen_sock, 1);
            if (ret == 0)
                {
                while (1) // accept loop
                    {
                    ...
                    server_sock = accept(listen_sock, (struct sockaddr *)&source_addr, &addr_len);
                    if(server_sock >= 0)
                        { 
                        ESP_LOGI(TAG, "server task socket %d", server_sock);
                        ...
                        while(1) //communication loop
                            {
                            len = recv(server_sock, &message, sizeof(socket_message_t), MSG_WAITALL);
                            if (len < 0)
                                {
                                ESP_LOGE(TAG, "Error occurred during receiving: errno %d", errno);
                                close(server_sock);
                                break;
                                }
                            else if (len == 0) //socket closed by client
                                {
                                ESP_LOGI(TAG, "socket closed by peer");
                                close(server_sock);
                                break;
                                }
                            else //process message based on cmd_id && commstate
                                {
                                if(commstate == IDLE)
                                    {
                                    ...

                                    }
                                else //commstate == CONNECTED
                                     //just put the message in the queue
                                    {
                                    ...
                                    //posting the message, triggers process_message task
                                    xQueueSend(tcp_receive_queue, &message, 0);
                                    }
                                }
                            }
                        if(process_message_task_handle)
                            {
                            vTaskDelete(process_message_task_handle);
                            process_message_task_handle = NULL;
                            }
                        if(send_task_handle)
                            {
                            vTaskDelete(send_task_handle);
                            send_task_handle = NULL;
                            }
                        }
                    else
                        {
                        ESP_LOGE(TAG, "Unable to accept connection: errno %d", errno);
                        break;
                        }
                    }
                }
            else
                {
                ESP_LOGE(TAG, "Error occurred during listen: errno %d", errno);
                }
            }
        else
            {
            ESP_LOGE(TAG, "Socket unable to bind: errno %d", errno);
            }
        }
    else
        {
        ESP_LOGE(TAG, "Unable to create socket: errno %d", errno);
        }
    vTaskDelete(NULL);
    }

While in receive loop tcp_server() task is blocked on recv() and only send_task() is active sending continuously messages to the connected client.
tcp socket errors like 113 or 104, pop up during recv() in tcp_server() task, and during send() in send_task().

In some rare cases, the logs show the error pop up during both recv() and send() and in some of these rare cases the issue pop-up, which makes to module to get stuck
tmp.zip

[ves011]

looking at all the above i think will be easier for you to creade a small test app with my sdkconfig and target and force it ending in *void IRAM_ATTR attribute((noreturn, no_sanitize_undefined)) panic_abort(const char details)

[sudeep-mohanty]

@ves011 I was not successful in reproducing the problem that you see. May I ask, if you see the panic banner being printed and any core-dump/register-dump on the console? It is strange that the panic handler does not reboot the chip. The fact that the problem occurs occasionally could indicate a memory corruption somewhere leading to a hang. So it is hard to say what is the cause here.