Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Timeouts with MKL artifact download #4

Open
ablaom opened this issue Oct 10, 2022 · 1 comment
Open

Timeouts with MKL artifact download #4

ablaom opened this issue Oct 10, 2022 · 1 comment

Comments

@ablaom
Copy link
Contributor

ablaom commented Oct 10, 2022

Been trying to update my HelloJulia.jl Pluto course to Julia 1.8 but running into problems. I can create a sys image fine. But when I call PrecompilePlutoCourse.start() it triggers a MKL artifact download (ultimately needed by Makie) which seems to timeout and then mysteriously reports that MKL is downloaded, but pluto never launches:

 Downloading artifact: MKL
ERROR: TaskFailedException===========================>    ]  89.9 %
    Downloading [====================================>    ]  90.0 %
    nested task error:     Downloading [=====================================>  Timed out waiting to read host:port string from worker.   ]  90.4 %
    Stacktrace:
     [1] worker_from_id(pg::Distributed.ProcessGroup, i::Int64)
       @ Distributed /Applications/Julia-1.8.app/Contents/Resources/julia/share/julia/stdlib/v1.8/Distributed/src/cluster.jl:1092
     [2] worker_from_id
       @ /Applications/Julia-1.8.app/Contents/Resources/julia/share/julia/stdlib/v1.8/Distributed/src/cluster.jl:1089 [inlined]
     [3] #remote_do#170
       @ /Applications/Julia-1.8.app/Contents/Resources/julia/share/julia/stdlib/v1.8/Distributed/src/remotecall.jl:557 [inlined]
     [4] remote_do
       @ /Applications/Julia-1.8.app/Contents/Resources/julia/share/julia/stdlib/v1.8/Distributed/src/remotecall.jl:557 [inlined]
     [5] kill
       @ /Applications/Julia-1.8.app/Contents/Resources/julia/share/julia/stdlib/v1.8/Distributed/src/managers.jl:687 [inlined]
     [6] create_worker(manager::Distributed.LocalManager, wconfig::Distributed.WorkerConfig)
       @ Distributed /Applications/Julia-1.8.app/Contents/Resources/julia/share/julia/stdlib/v1.8/Distributed/src/cluster.jl:603
     [7] setup_launched_worker(manager::Distributed.LocalManager, wconfig::Distributed.WorkerConfig, launched_q::Vector{Int64})
       @ Distributed /Applications/Julia-1.8.app/Contents/Resources/julia/share/julia/stdlib/v1.8/Distributed/src/cluster.jl:544
     [8] (::Distributed.var"#45#48"{Distributed.LocalManager, Vector{Int64}, Distributed.WorkerConfig})()
       @ Distributed ./task.jl:484
    
    caused by: Timed out waiting to read host:port string from worker.
    Stacktrace:
     [1] read_worker_host_port(io::Base.PipeEndpoint)
       @ Distributed /Applications/Julia-1.8.app/Contents/Resources/julia/share/julia/stdlib/v1.8/Distributed/src/cluster.jl:344
     [2] connect(manager::Distributed.LocalManager, pid::Int64, config::Distributed.WorkerConfig)
       @ Distributed /Applications/Julia-1.8.app/Contents/Resources/julia/share/julia/stdlib/v1.8/Distributed/src/managers.jl:536
     [3] create_worker(manager::Distributed.LocalManager, wconfig::Distributed.WorkerConfig)
       @ Distributed /Applications/Julia-1.8.app/Contents/Resources/julia/share/julia/stdlib/v1.8/Distributed/src/cluster.jl:599
     [4] setup_launched_worker(manager::Distributed.LocalManager, wconfig::Distributed.WorkerConfig, launched_q::Vector{Int64})
       @ Distributed /Applications/Julia-1.8.app/Contents/Resources/julia/share/julia/stdlib/v1.8/Distributed/src/cluster.jl:544
     [5] (::Distributed.var"#45#48"{Distributed.LocalManager, Vector{Int64}, Distributed.WorkerConfig})()
       @ Distributed ./task.jl:484
Stacktrace:
 [1] sync_end(c::Channel{Any})
   @ Base ./task.jl:436
 [2] macro expansion
   @ ./task.jl:455 [inlined]
 [3] addprocs_locked(manager::Distributed.LocalManager; kwargs::    Downloading Base.Pairs{Symbol, Vector{String}, Tuple{Symbol}, NamedTuple{(:exeflags,), Tuple{Vector{String}}}})
   @ Distributed /Applications/Julia-1.8.app/Contents/Resources/julia/share/julia/stdlib/v1.8/Distributed/src/cluster.jl:490
 [4] addprocs(manager::Distributed.LocalManager; kwargs::Base.Pairs{Symbol, Vector{String}, Tuple{Symbol}, NamedTuple{(:exeflags,), Tuple{Vector{String}}}})
   @ Distributed /Applications/Julia-1.8.app/Contents/Resources/julia/share/julia/stdlib/v1.8/Distributed/src/cluster.jl:450
 [5] #addprocs#263
   @ /Applications/Julia-1.8.app/Contents/Resources/julia/share/julia/stdlib/v1.8/Distributed/src/managers.jl:455 [inlined]
 [6] start(; sysimage::String)
   @ PrecompilePlutoCourse ~/HelloJulia/temporary_julia_depot/packages/PrecompilePlutoCourse/reLu6/src/PrecompilePlutoCourse.jl:35
 [7] start()
   @ PrecompilePlutoCourse ~/HelloJulia/temporary_julia_depot/packages/PrecompilePlutoCourse/reLu6/src/PrecompilePlutoCourse.jl:30
 [8] top-level scope
   @ REPL[6]:1
  Downloaded artifact: MKL

I try calling PrecompilePlutoCourse.start() again, and no artifact download occurs, but I get a new error:

Starting a Pluto notebook in your browser.
Please use `HelloJulia.stop()` to interrupt the Pluto notebook server.

julia>       From worker 3:	[ Info: Loading...
      From worker 3:	┌ Info: 
      From worker 3:	└ Opening http://localhost:1234/open?secret=LY6KePM8&path=%2FUsers%2Fanthony%2FHelloJulia%2Fnotebooks%2Fpluto_index.jl in your default browser... ~ have fun!
      From worker 3:	┌ Info: 
      From worker 3:	│ Press Ctrl+C in this terminal to stop Pluto
      From worker 3:	└ 
      From worker 3:	
      From worker 3:	signal (11): Segmentation fault: 11
      From worker 3:	in expression starting at none:1
      From worker 3:	ijl_subtype_env at /Applications/Julia-1.8.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.8.dylib (unknown line)
      From worker 3:	YY.send_notebook_changesNOT.YY.345 at /Users/anthony/GoogleDrive/Julia/HelloJulia/precompile/hellojulia.so (unknown line)
      From worker 3:	response_update_notebook at /Users/anthony/GoogleDrive/Julia/HelloJulia/precompile/hellojulia.so (unknown line)
      From worker 3:	jfptr_response_update_notebook_154845 at /Users/anthony/GoogleDrive/Julia/HelloJulia/precompile/hellojulia.so (unknown line)
      From worker 3:	ijl_apply_generic at /Applications/Julia-1.8.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.8.dylib (unknown line)
      From worker 3:	process_ws_message at /Users/anthony/HelloJulia/temporary_julia_depot/packages/Pluto/U4wvO/src/webserver/WebServer.jl:399
      From worker 3:	ijl_apply_generic at /Applications/Julia-1.8.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.8.dylib (unknown line)
      From worker 3:	#397 at /Users/anthony/HelloJulia/temporary_julia_depot/packages/Pluto/U4wvO/src/webserver/WebServer.jl:190
      From worker 3:	#upgrade#13 at /Users/anthony/HelloJulia/temporary_julia_depot/packages/HTTP/VAOuj/src/WebSockets.jl:439
      From worker 3:	upgrade at /Users/anthony/HelloJulia/temporary_julia_depot/packages/HTTP/VAOuj/src/WebSockets.jl:419 [inlined]
      From worker 3:	#396 at /Users/anthony/HelloJulia/temporary_julia_depot/packages/Pluto/U4wvO/src/webserver/WebServer.jl:171
      From worker 3:	unknown function (ip: 0x10d4d9c02)
      From worker 3:	ijl_apply_generic at /Applications/Julia-1.8.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.8.dylib (unknown line)
      From worker 3:	jl_f__call_latest at /Applications/Julia-1.8.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.8.dylib (unknown line)
      From worker 3:	#invokelatest#2 at ./essentials.jl:729 [inlined]
      From worker 3:	invokelatest at ./essentials.jl:726 [inlined]
      From worker 3:	handle_connection at /Users/anthony/HelloJulia/temporary_julia_depot/packages/HTTP/VAOuj/src/Servers.jl:442
      From worker 3:	#16 at ./task.jl:484
      From worker 3:	unknown function (ip: 0x10d4c7dff)
      From worker 3:	ijl_apply_generic at /Applications/Julia-1.8.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.8.dylib (unknown line)
      From worker 3:	start_task at /Applications/Julia-1.8.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.8.dylib (unknown line)
      From worker 3:	Allocations: 18516703 (Pool: 18507923; Big: 8780); GC: 23
      From worker 3:	┌ Warning: temp cleanup
      From worker 3:	│   exception =
      From worker 3:	│    task switch not allowed from inside gc finalizer
      From worker 3:	│    Stacktrace:
      From worker 3:	│     [1] try_yieldto(undo::typeof(Base.ensure_rescheduled))
      From worker 3:	│       @ Base ~/GoogleDrive/Julia/HelloJulia/precompile/hellojulia.so:-1
      From worker 3:	│     [2] Channel{Tuple{String, Vector{String}, Vector{String}}}(func::Base.Filesystem.var"#28#31"{String}, size::Int64; taskref::Nothing, spawn::Bool)
      From worker 3:	│       @ Base ~/GoogleDrive/Julia/HelloJulia/precompile/hellojulia.so:-1
      From worker 3:	│     [3] prepare_for_deletion(path::String)
      From worker 3:	│       @ Base.Filesystem ~/GoogleDrive/Julia/HelloJulia/precompile/hellojulia.so:-1
      From worker 3:	│     [4] temp_cleanup_purge(; force::Bool)
      From worker 3:	│       @ Base.Filesystem ~/GoogleDrive/Julia/HelloJulia/precompile/hellojulia.so:-1
      From worker 3:	│     [5] (::Base.var"#886#887")()
      From worker 3:	│       @ Base ~/GoogleDrive/Julia/HelloJulia/precompile/hellojulia.so:-1
      From worker 3:	│     [6] _atexit()
      From worker 3:	│       @ Base ~/GoogleDrive/Julia/HelloJulia/precompile/hellojulia.so:-1
      From worker 3:	└ @ Base.Filesystem file.jl:551
      From worker 3:	task switch not allowed from inside gc finalizer
      From worker 3:	atexit hook threw an error: ErrorException("task switch not allowed from inside gc finalizer")
      From worker 3:	ijl_error at /Applications/Julia-1.8.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.8.dylib (unknown line)
      From worker 3:	ijl_switch at /Applications/Julia-1.8.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.8.dylib (unknown line)
      From worker 3:	try_yieldto at /Users/anthony/GoogleDrive/Julia/HelloJulia/precompile/hellojulia.so (unknown line)
      From worker 3:	wait at /Users/anthony/GoogleDrive/Julia/HelloJulia/precompile/hellojulia.so (unknown line)
      From worker 3:	uv_write at /Users/anthony/GoogleDrive/Julia/HelloJulia/precompile/hellojulia.so (unknown line)
      From worker 3:	unsafe_write at /Users/anthony/GoogleDrive/Julia/HelloJulia/precompile/hellojulia.so (unknown line)
      From worker 3:	print at /Users/anthony/GoogleDrive/Julia/HelloJulia/precompile/hellojulia.so (unknown line)
      From worker 3:	jfptr_print_69719 at /Users/anthony/GoogleDrive/Julia/HelloJulia/precompile/hellojulia.so (unknown line)
      From worker 3:	ijl_apply_generic at /Applications/Julia-1.8.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.8.dylib (unknown line)
      From worker 3:	showerror at ./errorshow.jl:144
      From worker 3:	unknown function (ip: 0x10d51f2ed)
      From worker 3:	ijl_apply_generic at /Applications/Julia-1.8.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.8.dylib (unknown line)
      From worker 3:	_atexit at /Users/anthony/GoogleDrive/Julia/HelloJulia/precompile/hellojulia.so (unknown line)
      From worker 3:	jfptr__atexit_73562 at /Users/anthony/GoogleDrive/Julia/HelloJulia/precompile/hellojulia.so (unknown line)
      From worker 3:	ijl_apply_generic at /Applications/Julia-1.8.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.8.dylib (unknown line)
      From worker 3:	ijl_atexit_hook at /Applications/Julia-1.8.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.8.dylib (unknown line)
      From worker 3:	ijl_exit at /Applications/Julia-1.8.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.8.dylib (unknown line)
      From worker 3:	jl_exit_thread0_cb at /Applications/Julia-1.8.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.8.dylib (unknown line)Worker 3 terminated.

Unhandled Task ERROR: EOFError: read end of file
Stacktrace:
 [1] (::Base.var"#wait_locked#680")(s::Sockets.TCPSocket, buf::IOBuffer, nb::Int64)
   @ Base ./stream.jl:945
 [2] unsafe_read(s::Sockets.TCPSocket, p::Ptr{UInt8}, nb::UInt64)
   @ Base ./stream.jl:953
 [3] unsafe_read
   @ ./io.jl:759 [inlined]
 [4] unsafe_read(s::Sockets.TCPSocket, p::Base.RefValue{NTuple{4, Int64}}, n::Int64)
   @ Base ./io.jl:758
 [5] read!
   @ ./io.jl:760 [inlined]
 [6] deserialize_hdr_raw
   @ /Applications/Julia-1.8.app/Contents/Resources/julia/share/julia/stdlib/v1.8/Distributed/src/messages.jl:167 [inlined]
 [7] message_handler_loop(r_stream::Sockets.TCPSocket, w_stream::Sockets.TCPSocket, incoming::Bool)
   @ Distributed /Applications/Julia-1.8.app/Contents/Resources/julia/share/julia/stdlib/v1.8/Distributed/src/process_messages.jl:172
 [8] process_tcp_streams(r_stream::Sockets.TCPSocket, w_stream::Sockets.TCPSocket, incoming::Bool)
   @ Distributed /Applications/Julia-1.8.app/Contents/Resources/julia/share/julia/stdlib/v1.8/Distributed/src/process_messages.jl:133
 [9] (::Distributed.var"#103#104"{Sockets.TCPSocket, Sockets.TCPSocket, Bool})()
   @ Distributed ./task.jl:484

I've tried this (from a fresh Julia depot) 3 times with the same result.

I'm guessing these errors are unrelated to my use of PrecompilePlutoCourse and am sorry for posting here, but I wondered if you had seen anything like this.
@jbrea
Copy link
Owner

jbrea commented Oct 28, 2022

I ran into a similar error message on 1.8. I didn't investigate yet, if I can fix this in this package. Currently I stick to julia 1.7.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@jbrea @ablaom