Can not open existing notebooks with JupyterLab 4.2.5 and jupyter-collaboration 3

[trungleduc]

Description

When using JupyterLab 4.2.5 and jupyter-collaboration 3 together, I cannot reopen my existing notebooks

colab-bug2.mp4

Reproduce

1. Install Jupyterlab 4.2.5, create a notebook
2. Install jupyter-collaboration 3, then try to open the above notebook

Expected behavior

• I can open my old notebooks or Jupyterlab 4.2.5 and jupyter-collaboration 3 cannot be installed together

Context

• Operating System and version:
• Browser and version:
• JupyterLab version:

Troubleshoot Output

Paste the output from running `jupyter troubleshoot` from the command line here.
You may want to sanitize the paths in the output.

Command Line Output

Paste the output from your command line running `jupyter lab` here, use `--debug` if possible.

Browser Output

Paste the output from your browser Javascript console here, if applicable.

[jtpio]

For reference, installing jupyterlab=4.2.5 and jupyter-collaboration=3 together in a clean environment and then creating a new notebook seems to be working fine:

[shawnbrown]

I've been running into this bug, too.

Out of curiosity (and since I have no sense of the internals), I was curious to see if there were any format differences or added values in the ipynb JSON format. I created two small files and compared them: one before installing version 3.0.0 and one after installing.

The ipynb files are exactly the same (save for the id hashes) but one opens and the other does not.

For reference, installing jupyterlab=4.2.5 and jupyter-collaboration=3 together in a clean environment and then creating a new notebook seems to be working fine:

@jtpio, I'm not sure this is always the case. When I created some sample notebooks after installing jupyter-collaboration==3, they would sometimes give the same behavior shown in @trungleduc's video but it was not consistent.

I can't be certain, but my sense was: if I created a notebook file and had it open for a while then it seemed to be good and I never encountered the bug. But if I created a short notebook file, executed a few cells, then saved and closed it relatively quickly, then it wouldn't reopen later.

It seemed like there's something happening that takes a few seconds to complete and if I saved and close the file before that time, then I would never be able to reopen it later (and it would look just like the bug in the video).

An aside: I also tried to downgrade to jupyter-collaboration==2.1.5 but after doing this, the file-browser tab in the left sidebar was gone. However a clean install starting with version 2.1.5 seemed to work fine for me (no file-open bug and left sidebar was fine).

[abhijeetgurle-dream11]

I had also faced similar issues and I upgraded my jupyterlab to 4.3.0. After upgrading notebooks are opening but sometimes it can take a long time to open. Issue is mentioned here.

[claytonparnell]

Any update on this issue? This is a major issue for us blocking existing notebooks from opening. If we just download a notebook, rename the file, and upload the renamed notebook, we see the issue in the video where the notebook completely fails to render. My team can't currently upgrade to JL 4.3 because we maintain a Docker image following SemVer for all installed packages, and anyway the notebook package currently pins to JL<4.3. If jupyter-collaboration isn't supposed to be compatible with JL 4.2, that should be reflected in the conda-forge feedstock.

[davidbrochart]

I'm not sure what is going on. It seems that simply removing the .jupyter_ystore.db file shows this behavior. I don't have a better solution than requiring jupyterlab>=4.3.0 for jupyter-collaboration>=3.0.0.

[ellisonbg]

Is it possibly related to the reproduction that @brichet described on this issue: #390 (comment)

[ellisonbg]

I was able to reproduce this by following the exact steps in the video. One extra data point though. I did see the the following in the Terminal when the notebook failed to open:

[E 2024-12-05 19:14:27.909 ServerApp] Error saving file: Untitled.ipynb
    TypeError("'ArrayIterator' object is not iterable")
    Traceback (most recent call last):
      File "/Users/brgrange/Notebooks/bug398/.pixi/envs/default/lib/python3.13/site-packages/jupyter_server_ydoc/rooms.py", line 285, in _maybe_save_document
        "content": self._document.source,
                   ^^^^^^^^^^^^^^^^^^^^^
      File "/Users/brgrange/Notebooks/bug398/.pixi/envs/default/lib/python3.13/site-packages/jupyter_ydoc/ybasedoc.py", line 90, in source
        return self.get()
               ~~~~~~~~^^
      File "/Users/brgrange/Notebooks/bug398/.pixi/envs/default/lib/python3.13/site-packages/jupyter_ydoc/ynotebook.py", line 204, in get
        cell = self.get_cell(i)
      File "/Users/brgrange/Notebooks/bug398/.pixi/envs/default/lib/python3.13/site-packages/jupyter_ydoc/ynotebook.py", line 106, in get_cell
        cell = self._ycells[index].to_py()
      File "/Users/brgrange/Notebooks/bug398/.pixi/envs/default/lib/python3.13/site-packages/pycrdt/_map.py", line 111, in to_py
        py[key] = val.to_py()
                  ~~~~~~~~~^^
      File "/Users/brgrange/Notebooks/bug398/.pixi/envs/default/lib/python3.13/site-packages/pycrdt/_array.py", line 350, in to_py
        py = [value for value in self]
                                 ^^^^
    TypeError: 'ArrayIterator' object is not iterable

[ellisonbg]

OK, not sure if this is the root cause of this issue, but the above traceback is due to a bug in pycrdt:

jupyter-server/pycrdt#198

[ellisonbg]

FYI: I tried the fix to the pycrdt ArrayIterator locally and it doesn't seem to fix this issue, but may fix other issues.

[ellisonbg]

A few other notes:

• I am running jupyter_ydoc 3.0.2 with the streaming output fix and still seeing the issue.
• I am suspecting it may be related to the windowed rendering of cells for a couple of reasons: 1) the cell in the above screentshot are missing, just like the windowing is hiding them, 2) some of the tracebacks that I see in the browser console suggest code in the windowing logic.
• However, the problem is still there if I disable windowing in the JupyterLab settings.

[ellisonbg]

Another observation: when I delete the .jupyter_ystore file, the issue persists. This suggests this particular issue is different from some of the others where deleting this file resolves the issue.

[krassowski]

I am suspecting it may be related to the windowed rendering of cells for a couple of reasons: 1) the cell in the above screentshot are missing, just like the windowing is hiding them, 2) some of the tracebacks that I see in the browser console suggest code in the windowing logic.

@ellisonbg do you have JupyterLab 4.3.1 or 4.3.2?

[ellisonbg]

This particular bug shows up with JupyterLab 4.2.5.Sent from my iPhoneOn Dec 6, 2024, at 12:35 AM, Michał Krassowski ***@***.***> wrote: @ellisonbg do you have JupyterLab 4.3.1 or 4.3.2? —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>

[trungleduc]

Is it possibly related to the reproduction that @brichet described on this issue: #390 (comment)

it does not look like related, I do see the jupyter_server_ydoc server extension activated in the screencast above

[davidbrochart]

I found that jupyterlab v4.2.5 requires @jupyter/ydoc >=2.0.1,<3.0.0 (the JavaScript shared models), but jupyter-collaboration v3.0.0 requires jupyter-ydoc >=2.1.2,<4.0.0 (the Python shared models). Nothing prevents version 3 of the Python models to be installed in an environment with version 2 of the JavaScript models. But these models are incompatible: the cell outputs are modeled differently, which causes an infinite recursion when converting them to JSON.
I think that the root issue is that JupyterLab uses @jupyter/ydoc even when jupyter-collaboration is not installed. On the other hand, jupyter-collaboration doesn't depend on JupyterLab, at least not on the Python package. This seems to introduce a loose coupling between the JavaScript and the Python versions of jupyter-ydoc. In jupyter-collaboration v3 we wanted to support v2 or v3 of jupyter-ydoc:

jupyter-collaboration/packages/docprovider/package.json

Line 45 in 22a505b

"@jupyter/ydoc": "^2.1.3 || ^3.0.0",

jupyter-collaboration/projects/jupyter-server-ydoc/pyproject.toml

Line 32 in 22a505b

"jupyter_ydoc>=2.1.2,<4.0.0,!=3.0.0,!=3.0.1",

But this doesn't seem to enforce that if v2 is installed both the JavaScript and the Python sides should be v2, and if v3 is installed both the JavaScript and the Python sides should be v3.
Anyways, a workaround to have JupyterLab v4.2.5 and jupyter-collaboration v3 working together is to do:

pip install "jupyter-ydoc<3"