running out of memory with Julia 1.6 and Pluto.run()

[rczerminski-valo]

• MacBook Pro BigSur 11.2.3 (20D91)
• Julia 1.6 [Version 1.6.3 (2021-09-23)]
• Pluto v0.16.4

I felt brave and installed Julia 1.6 [Version 1.6.3 (2021-09-23)] on MacBook Pro BigSur 11.2.3 (20D91)
My main motivation was new Format pkg...

Now when I try Pluto.run() with my notebook julia is spawning hundreds of processes, computer is running out of memory and crashes. See below.
NOTE: the same notebook is running just fine with Julia 1.5
I am not really sure where the problem is. When I run julia 1.6 I do not see problems, but I have not really used it that much.

[...hundreds of lines removed...] 
 4957 ??         0:33.74 /Applications/Julia-1.6.app/Contents/Resources/julia/bin/julia -Cnative -J/Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/sys.dylib -O0 -g1 --startup-file=no -O0 --output-ji /Users/ryszard/.julia/compiled/v1.6/OptHarness/jl_es3HWP --output-incremental=yes --startup-file=no --history-file=no --warn-overwrite=yes --color=auto --eval eval(Meta.parse(read(stdin,String)))
 4963 ??         0:33.60 /Applications/Julia-1.6.app/Contents/Resources/julia/bin/julia -Cnative -J/Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/sys.dylib -O0 -g1 --startup-file=no -O0 --output-ji /Users/ryszard/.julia/compiled/v1.6/OptHarness/jl_x25MFB --output-incremental=yes --startup-file=no --history-file=no --warn-overwrite=yes --color=auto --eval eval(Meta.parse(read(stdin,String)))
 4967 ??         0:16.53 /Applications/Julia-1.6.app/Contents/Resources/julia/bin/julia -Cnative -J/Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/sys.dylib -O0 -g1 --startup-file=no -O0 --output-ji /Users/ryszard/.julia/compiled/v1.6/OptHarness/jl_5LsoWz --output-incremental=yes --startup-file=no --history-file=no --warn-overwrite=yes --color=auto --eval eval(Meta.parse(read(stdin,String)))
 4970 ttys001    0:00.00 grep julia
 1025 ttys004    0:01.06 julia
$ ps -A | grep julia | wc -l
     465
$

with julia 1.5 (/Applications/Julia-1.5.app/Contents/Resources/julia/bin/julia)
I am getting just few processes:

$ ps -A | grep julia | wc -l
       8
$

[fonsp]

What Julia version exactly? Which Pluto version?

[rczerminski-valo]

• MacBook Pro BigSur 11.2.3 (20D91)
• Julia 1.6 [Version 1.6.3 (2021-09-23)]
• Pluto v0.16.4

I just noticed that apparently there is higher version available (0.7.5), but when I run Pkg.add("Pluto") it installs v0.16.4.
Should I try 0.7.5? How can install it?

[fonsp]

Assuming you meant Pluto 0.17.0 instead of 0.7.5, try it again, it was only released yesterday and it can take a while for the Pkg servers to update.

Does it happen on every notebook? Can you try the Tower of Hanoi sample notebook?

[rczerminski-valo]

I cannot reproduce this behavior with Tower of Hanoi sample notebook.
I will try my notebook again with 0.17.0 when it is available.
BTW: I see 0.7.5 here -> https://juliahub.com/ui/Packages/Pluto/OJqMt/0.7.5

[fonsp]

In that case, the problem is with your notebook's code, or with one of the packages that you use. Can you try to narrow down on the problem by finding which cell causes the problem? Open the notebook in VS Code or the macos text editor and add # in front of lines before opening it in pluto.

[rczerminski-valo]

I guess I do not exactly understand what you mean by: [...] add # in front of lines [...]

If I add just single # as the very first line like this:

$ head -3 notebook.jl 
#
### A Pluto.jl notebook ###
# v0.16.4
$

I am getting an error:

The file /Users/ryszard/repos/dsi-tools/Julia/OptHarness/notebook.jl could not be loaded. Please report this error!

Go back

Error message:

File is not a Pluto.jl notebook
Stacktrace:
 [1] error(::String) at ./error.jl:33
[...]

[fonsp]

I mean that you can disable a line by turning it into a comment:

This code will run

using Plots
plot(1:10)

This code will not run anything:

# using Plots
# plot(1:10)

This means that you can modify the notebook file to "disable cells" before running it in pluto.

[rczerminski-valo]

Hmm, this would be a lot of code to comment out (disable/turn-off) and then turn-back-on and to see when this bad behavior happens...

I just found #92:
Pluto.CONFIG["PLUTO_RUN_NOTEBOOK_ON_LOAD"] = "false"; Pluto.run(1234)
I imagine, that starting Pluto this way I can enable cells one-by-one and at some point I can identify which
cell is causing the problem, etc. - would this approach work here?

One puzzle for me is why this notebook works just fine with julia 1.5.3, but not julia 1.6.3. This seems to indicate to me that maybe 1.6.3 is using different, and buggy, version of some package, rather that something in my code. Does this sound for you as a reasonable conjecture?

[fonsp]

Good ideas, try it out and see what happens! Try running the notebook using julia ~/Documents/my_notebook_file.jl to see if it also happens outside of Pluto.

I won't be able to help you through the debugging process, you need to work on this yourself, but try to divide your problem in smaller parts. Take something that works, something that doesn't work, and slowly narrow in on the problem in the middle.