Kernel's execution state is not updated after the kernel is crashed while executing code

[skukhtichev]

Description

A Kernel's execution state is not updated when the kernel crashes while executing code in the background (e.g. due to Out Of Memory issue) with closed Notebook UI.

As a result the kernel state is reported as ´busy´ after the kernel is automatically restarted.

Reproduce

1. Create a new notebook in Jupyter Notebook UI
2. Add code cell with the following code:

while True:
   pass

5. execute a code cell
6. Close a browser window
7. Kill a kernel process from the terminal
8. Jupyter Server log will indicate that the kernel was automatically restarted: [I 2024-02-14 20:23:44.492 ServerApp] AsyncIOLoopKernelRestarter: restarting kernel (1/5), keep random ports
9. call /api/kernels endpoint
10. The kernel's execution_state will be shown as busy: [{"id": "0fadc495-97a3-4049-9c4c-adf2e37a416d", "name": "python3", "last_activity": "2024-02-14T20:22:57.083666Z", "execution_state": "busy", "connections": 0}]

The issue happens because a kernel activity monitor updates an execution state based on status messages sent via iopub channel: https://github.com/jupyter-server/jupyter_server/blob/main/jupyter_server/services/kernels/kernelmanager.py#L535-L572.
When the kernel is restarted after crashing and there is no notebook opened in the Browser, there is no kernel_info request sent to the kernel and there is no status message sent via iopub channel to kernel clients. As a results the kernel activity monitor is not able to identify that the kernel was restarted and switched to idle state.

Expected behavior

Kernel switches to idle state after it was successfully restarted

Context

• Operating System and version: MacOS and Linux
• Browser and version: Chrome 121.0.6167.160
• Jupyter Server version: 2.12.5

Troubleshoot Output

Paste the output from running `jupyter troubleshoot` from the command line here.
You may want to sanitize the paths in the output.

Command Line Output

Paste the output from your command line running `jupyter lab` here, use `--debug` if possible.

Browser Output

Paste the output from your browser Javascript console here, if applicable.

[krassowski]

FWIW JupyterLab is able to work around this condition (although it is not pretty):

• Store information about execution failure timing jupyterlab/jupyterlab#15285
• Reset execution indicator state when kernel restarts jupyterlab/jupyterlab#13832

My thinking earlier was that kernel simply cannot send a message when it dies because it is dead, but indeed a kernel manager might be able to handle it. I guess the state should not be idle though.

PS. What is kernel activity monitor? Is it related to https://github.com/jupyter-server/jupyter-resource-usage?

[skukhtichev]

There is a ´start_watching_activity´ in jupyter server's MappingKernelManager that listens on iopub channels and updates a kernel state and tracks last activity timestamp. When the kernel crashes jupyter_client restarts the kernel (https://github.com/jupyter/jupyter_client/blob/main/jupyter_client/ioloop/restarter.py#L52). So the kernel has the same id after it is restarted and has idle state.

When the kernel crashes with opened Notebook 7/JupyterLab UI the state is updated correctly on the server, but when the kernel process dies when UI is closed then the execution state is always shown as ´busy´.

Incorrect state could also impact on kernel culling logic because and execution state is monitored in https://github.com/jupyter-server/jupyter_server/blob/main/jupyter_server/services/kernels/kernelmanager.py#L664

[Wh1isper]

My idea of a so-called monitor may coincide with the idea of a server side execution like jupyterlab/jupyterlab#15448

When we can execute cell with a single command and move document changes to the backend, we can implement a process like jupyterlab/jupyterlab#12455 (Not entirely dependent on the kernel itself, but on its Manager)

[krassowski]

Maybe #990 could solve this?