Substantial slowdown (>x10) opening large notebooks starting from version 0.19.37

[yha]

My larger notebooks have been opening much slower in more recent versions of Pluto. A large notebook that loads in 200-300 seconds on version 0.19.36 takes around 2500 seconds on 0.19.43. The slowdown seems to start at version 0.19.37 and doesn't affect all notebooks to such an extent, but I couldn't figure out so far what difference is relevant. I've been using 0.19.32 for the last months to avoid this issue.

I could reproduce some similar result with this notebook containing nothing but comments:
https://gist.github.com/yha/4ed9406d2c8b8fd7c50aadccdb5c2982

For this notebook, the initial "Running code" takes 15 seconds on 0.19.36, and 810 seconds on 0.19.43 (The effect is larger with comments than with just empty cells)

In both cases, Pluto is running on:

julia> versioninfo()
Julia Version 1.10.4
Commit 48d4fd48430 (2024-06-04 10:41 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: 48 × Intel(R) Xeon(R) Gold 6248R CPU @ 3.00GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-15.0.7 (ORCJIT, cascadelake)
Threads: 1 default, 0 interactive, 1 GC (on 48 virtual cores)

and the notebook is accessed remotely from Firefox on Windows.

[fonsp]

Thanks for the easy example notebook and for finding the version number!

It looks like this is not caused by Malt.jl, setting workspace_use_distributed_stdlib=true does not fix the issue.

A smaller notebook with the same cells is not slower, so the large number of cells is relevant.

It looks like one bottleneck is the notebook_to_js function, which takes 250ms for this notebook, most of that time is spent on cell_execution_order.

[fonsp]

Fix is ready! This will bring performance back to what it was in 0.19.36

[fonsp]

It looks like another bottleneck is deep_enough_copy, which takes 2x as long as notebook_to_js...

[fonsp]

#2974 for deep_enough_copy

[fonsp]

Thanks again for the issue report! The MWE and version numbers was really helpful, and it led to lots of PRs!

[fonsp]

The other PRs made it faster than before!

On current main

On 0.19.45

(took too long but you get the idea)

On 0.19.36