Utilities for managing deeply nested data?

[kapple19]

TL; DR

I'm looking to simplify code for managing nested DimArrays and DimStacks.
E.g. feeding nested data to AlgebraOfGraphics.
Maybe an add_dims function? I've prototyped one that I'm happy to PR:

function add_dims(dimarray, newdims...)
    alldims = (dims(dimarray)..., newdims...)
    return DimArray(
        reshape(
            dimarray |> parent,
            alldims .|> length
        ),
        alldims;
        name = dimarray.name,
        metadata = dimarray.metadata,
        refdims = dimarray.refdims
    )
end

Motivating Example

My work typically involves exploring nested data structures that are best not laid out as a long-format data table. A (non-obvious) example is visualised in the following plot:

where the titles are bearings [deg], the horizontal axis is sound speed [m/s], the vertical axis is depth [m] and the colour is range [m].

And this is still a simplified version of the type of data I manage probably half my days at work.

The data plotted above is generated by the following code:

##
using DimensionalData
using CairoMakie, AlgebraOfGraphics

using DimensionalData: dims

##
function munk_profile(z::Real; ϵ = 7.37e-3)
    z̃ = 2(z/1300 - 1)
    return 1500(
        1 + ϵ * (
            z̃ - 1 + exp(-z̃)
        )
    )
end

rand_munk_profile(z::Real) = munk_profile(z) + 5rand()

##
dimprofile(z, c) = DimArray(c, Dim{:z}(z); name = :c)
dimslice(r, zc) = DimArray(zc, Dim{:r}(r); name = :slice)
dimenvironment(θ, rzc) = DimArray(rzc, Dim{:θ}(θ); name = :environment)

rand_depths() = [0 : 10 : 50; 5e3rand(rand(20:30))] |> sort! |> unique!
rand_ranges() = [0; 10e3rand(rand(2:5))] |> sort! |> unique!

rand_dimprofile() = rand_depths() |> zs -> dimprofile(zs, rand_munk_profile.(zs))
rand_dimslice() = rand_ranges() |> rs -> dimslice(rs, [rand_dimprofile() for _ in rs])
rand_environment() = (45 |> step -> 0 : step : 360-step) |> θs -> dimenvironment(θs, [rand_dimslice() for _ in θs])

env = rand_environment()

I can show the nested structures of env by:

julia> dims(env)
(↓ θ Sampled{Int64} 0:45:315 ForwardOrdered Regular Points)

julia> dims(env[1])
(↓ r Sampled{Float64} [0.0, 3597.4471629977843, 5534.263563590665, 7028.512437622645] ForwardOrdered Irregular Points)

julia> dims(env[1][1])
(↓ z Sampled{Float64} [0.0, 10.0, …, 4977.663346971497, 4990.064674258991] ForwardOrdered Irregular Points)

Each bearing Dim{:θ} has a collection of ranges Dim{:r} whose values aren't shared between each θ.
Likewise each Dim{:r} has a collection of depth-sound speed Dim{:z}-Dim{:c} pairs whose z values aren't shared between each r.

(In other words, vector data I guess 😅 but with structures resembling raster data, DimensionalData is the best I've found so far for managing such data.)

I've played around with ways to efficiently and flexibly visualise the above in different ways.
My current preference involves the auxiliary function add_dims I've defined above, which behaves like reshape but for DimArrays.

The idea for add_dims is that when you access a DimArray inside another DimArray, that inner DimArray doesn't hold information on the Dim(s) you used to call it.

I'll define the compass bearing relationships:

lat_cols = ["N"; ""; "S"]
lon_rows = ["W" "" "E"]
compass_dirs = lat_cols .* lon_rows
compass_bearings = 0 : 45 : 360-45
compass_grid = Dict(
    θ => 2 .+ (
        -round(Int, cosd(θ), RoundFromZero),
        round(Int, sind(θ), RoundFromZero),
    )
    for θ in compass_bearings
)
compass_directions = Dict(
    θ => compass_dirs[compass_grid[θ]...]
    for θ in compass_bearings
)

and then plot like so:

draw(
    sum(
        add_dims(
            env[θ = At(θ)][r = At(r)],
            Dim{:θ}([θ]),
            Dim{:r}([r])
        ) |> data
        for θ in 45 |> step -> 0 : step : 360-step
        for r in dims(env[θ = At(θ)], :r) |> reverse
    )
    # * mapping(:c, :z, color = :r, layout = :θ => (θ -> compass_directions[θ]) => presorted)
    * mapping(:c, :z, color = :r, layout = :θ => nonnumeric)
    * visual(Lines),
    scales(;
        Color = (; colormap = Reverse(:blues)),
        Layout = (; palette = [compass_grid[θ] for θ in compass_bearings])
    );
    axis = (; yreversed = true)
)

Okay so, what's your question?

My question is, does anyone know any better ways to manage/handle this data, including interaction with AlgebraOfGraphics?
The above is still only a simplified example of the type of data I handle about half by work days.
Re AoG I'll be looking into pagination next as well, excited to see what it can do.

If you guys like add_dims I'm happy to PR it with docs and example of course.

I'm partially open to alternatives. I'd like to avoid pregrouped, it makes my soul hurt and in my opinion goes against the whole declarative syntax idea AoG was made for (but I still understand why pregrouped exists).

Are there other packages of storage structures that work better on this type of vector data but allows dimensional names and indexing, including grammar-of-graphics interoperability like what AoG provides?

[rafaqz]

This makes me think refdims should also be table columns (just the same value repeated), so you could be setting refdims instead of reshaping.

It's pretty much what they are meant for - tracking where in a larger array a slice comes from.

We could even add a DimNestedArray object that holds other DimArray/DimStack that updates refdims automatically on indexing?

[kapple19]

Oh I see. I'll check out refdims then.

This makes me think refdims should also be table columns

Are you saying it doesn't implement the Tables.jl interface so it won't work out-of-the-box with AoG? I'm about to check it out now and see what it can do.

We could even add a DimNestedArray object that holds other DimArray/DimStack that updates refdims automatically on indexing?

I see. From my superficial understanding, DimArray already has the features for nesting multiple layers, but you have an idea for a new object that would work better?

The automatic refdims update upon indexing would indeed simplify the syntax for passing to AoG.data.

I also found eachslice.

[tiemvanderdeure]

+1 for treating refdims as table columns. I have a completely unrelated usecase where this would also make my life a lot easier

[kapple19]

We could even add a DimNestedArray object that holds other DimArray/DimStack that updates refdims automatically on indexing?

One thing DimNestedArray would do differently/on top of normal DimArray/DimStack is that each nested layer doesn't have conflicting Dims?

This realisation of mine negates my above thought that:

From my superficial understanding, DimArray already has the features for nesting multiple layers

On top of your suggestion that accessing layers updates refdims.

[rafaqz]

Ok I've made a new issue to add refdims as tabel columns #884