Run parallel execution paths in parallel

[fonsp]

Related and also nice: ability to execute second block if it does not depend on any running blocks.

[fonsp]

Would this make your notebook faster/more responsive? If a cell is running, then the CPU is busy anyways, and other cells will be stalled.

It would give a speedup if it would use multiple CPU cores, but the only way to do so in Julia 1.0 is with Distributed (i.e. using multiple worker processes per notebook instead of one), but this is difficult difficult.

[jtrakk]

Is this issue closed because it's resolved?

[fonsp]

We gave it this label:
one day Closed because we won't work on it soon, will be opened again later.

[oschulz]

Just wanted to ask about the possibility of parallel evaluation of non-dependent cells and found this issue. I think this would be really cool. When Julia v1.6 will (hopefully) become the next LTS version of Julia, this should become very feasible, right (I assume Pluto aims support Julia versions down to the last LTS)? One could basically spawn a Julia task (multithreaded) for each cell as soon as all the dependencies of that cell are computed, and let the Julia scheduler worry about the rest. I guess it would need to be an optional feature, not all users will write thread-safe code.

[MasonProtter]

@fonsp from #1079

The problem is that we need to assume that users do not write state-mutating code.

Yes this is true, but I think the current status quo is already dangerous for users that are writing code that mutates state between cells anyways because it's unpredictable which cell will get run first. I'm not sure that cell-parallelism would practically change that situation all that much. However, I can definitely see why it should be opt-in if it exists.

Regarding supporting julia versions older than 1.3, I think the right thing to do would just be to define

macro spawn(ex)
    @static if VERSION ≤ v"1.3"
        esc(ex)
    else
        esc(:($Threads.@spawn $ex))
    end
end

and then use this @spawn macro for launching parallel jobs. Older versions will just run single threaded, newer versions won't.

[cstjean]

Are there any notebooks out there that showcase how to use state-mutating code in a safe way? I always make sure that a cell only mutates objects that were created in that cell, and I assumed that was the only reasonable way to go about it.

EDIT: I suppose shared-global-resource access might be more problematic than plain Julia structure mutation...