Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Issue with OffsetArrays #186

Open
r-barnes opened this issue Oct 2, 2018 · 4 comments
Open

Issue with OffsetArrays #186

r-barnes opened this issue Oct 2, 2018 · 4 comments

Comments

@r-barnes
Copy link
Contributor

r-barnes commented Oct 2, 2018

I was getting nervous about the communication costs of being imprecise with communications outside of the localpart of the array, so I figured I'd build a halo exchange system by combining OffsetArrays and DistributedArrays.

I intended for that to look like the following, where each array has a 4x4 block of purely local data and a surrounding 1-cell halo to cache remote data pulled using DistributedArrays.

@everywhere using DistributedArrays
@everywhere using OffsetArrays

r1  = @spawnat 2 OffsetArray(zeros(6,6), 0:5, 0:5)
r2  = @spawnat 3 OffsetArray(zeros(6,6), 0:5, 0:5)
r3  = @spawnat 4 OffsetArray(zeros(6,6), 0:5, 0:5)
r4  = @spawnat 5 OffsetArray(zeros(6,6), 0:5, 0:5)
ras = [r1 r2; r3 r4]
D   = DArray(ras)

Unfortunately, when I try to run this, I get:

julia> D   = DArray(ras)
ERROR: MethodError: no method matching OffsetArray{Float64,2,Array{Float64,2}}(::Array{Float64,2})
Closest candidates are:
  OffsetArray{Float64,2,Array{Float64,2}}(::Any, ::Any) where {T, N, AA<:AbstractArray} at /home/rick/.julia/packages/OffsetArrays/ICRTb/src/OffsetArrays.jl:10
Stacktrace:
 [1] empty_localpart(::Type, ::Int64, ::Type) at /home/rick/.julia/packages/DistributedArrays/XV7NS/src/darray.jl:66
 [2] macro expansion at ./task.jl:264 [inlined]
 [3] macro expansion at /home/rick/.julia/packages/DistributedArrays/XV7NS/src/darray.jl:84 [inlined]
 [4] macro expansion at ./task.jl:244 [inlined]
 [5] DArray(::Tuple{Int64,Int64}, ::Array{Future,2}, ::Tuple{Int64,Int64}, ::Array{Int64,2}, ::Array{Tuple{UnitRange{Int64},UnitRange{Int64}},2}, ::Array{Array{Int64,1},1}) at /home/rick/.julia/packages/DistributedArrays/XV7NS/src/darray.jl:82
 [6] macro expansion at ./task.jl:266 [inlined]
 [7] macro expansion at /home/rick/.julia/packages/DistributedArrays/XV7NS/src/darray.jl:194 [inlined]
 [8] macro expansion at ./task.jl:244 [inlined]
 [9] DArray(::Array{Future,2}) at /home/rick/.julia/packages/DistributedArrays/XV7NS/src/darray.jl:192
 [10] top-level scope at none:0

which doesn't seem like an issue with OffsetArrays because using SparseArrays in an analogous way produces no problems:

@everywhere using DistributedArrays
@everywhere using SparseArrays

r1  = @spawnat 2 sparse(zeros(6,6))
r2  = @spawnat 3 sparse(zeros(6,6))
r3  = @spawnat 4 sparse(zeros(6,6))
r4  = @spawnat 5 sparse(zeros(6,6))
ras = [r1 r2; r3 r4]
D   = DArray(ras)

Sadly, I'm not familiar enough with Julia to know why the interaction of OffsetArrays and DistributedArrays is problematic.

I've got:

  [aaf54ef3] DistributedArrays v0.5.1
  [6fe1bfb0] OffsetArrays v0.8.1
@vchuravy
Copy link
Member

vchuravy commented Oct 2, 2018

Should hopefully be fixed by #175, but what is the end goal? Would be lovely to have a larger test-case, I suspect the OffsetArrays will shake out a couple of assumption we have about axes vs indices

@r-barnes
Copy link
Contributor Author

I have a problem that I'd usually use C++ and MPI for which involves many nodes loading data and holding that data in memory for the duration of the program. The data collectively represents a large domain. Local computation is performed and halos (the edge cells of the local matrices) are exchanged to keep things synchronized.

Julia doesn't seem well-suited to this paradigm from a library perspective, though the language itself seems fine. So I've been experimenting with ways to achieve this behaviour.

@vchuravy
Copy link
Member

Yes that sounds like a usecase I would love to eventually support. We have a project right now that will require doing something similar.

@jishnub
Copy link

jishnub commented May 2, 2021

Note that with the conversion method now being defined in OffsetArrays.jl, the construction of the DArray works:

julia> @everywhere using DistributedArrays

julia> @everywhere using OffsetArrays

julia> r1  = @spawnat 2 OffsetArray(zeros(6,6), 0:5, 0:5)
Future(2, 1, 93, nothing)

julia> r2  = @spawnat 3 OffsetArray(zeros(6,6), 0:5, 0:5)
Future(3, 1, 94, nothing)

julia> r3  = @spawnat 4 OffsetArray(zeros(6,6), 0:5, 0:5)
Future(4, 1, 95, nothing)

julia> r4  = @spawnat 5 OffsetArray(zeros(6,6), 0:5, 0:5)
Future(5, 1, 96, nothing)

julia> ras = [r1 r2; r3 r4]
2×2 Matrix{Future}:
 Future(2, 1, 93, nothing)  Future(3, 1, 94, nothing)
 Future(4, 1, 95, nothing)  Future(5, 1, 96, nothing)

julia> D   = DArray(ras);

julia> D |> typeof
DArray{Float64, 2, OffsetMatrix{Float64, Matrix{Float64}}}

However the indices are not correct as DArray explicitly assumes 1-based indexing.

julia> D |>axes
(Base.OneTo(12), Base.OneTo(12))

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants