Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support Oceananigans v0.95 and bump v0.13.4 #233

Merged
merged 27 commits into from
Jan 2, 2025
Merged

Conversation

ali-ramadhan
Copy link
Collaborator

🤞 that everything works out of the box!

@ali-ramadhan ali-ramadhan requested a review from jagoosw November 16, 2024 01:46
@ali-ramadhan
Copy link
Collaborator Author

ali-ramadhan commented Nov 16, 2024

I think I'm seeing a couple of different errors pointing to this function

function intercept_tracer_tendencies!(model, intercepted_tendencies)
    for (name, field) in enumerate(intercepted_tendencies)
        field .= Array(interior(model.timestepper.Gⁿ[name + 3]))
    end
end

in test_sediments.jl. Something about name + 3 being out of bounds, and something about the array dimensions not matching depending on the test.

I'm skipping the sediment tests for now just to see if the rest of the tests pass on CI (they did on my machine).

@jagoosw I hope you don't mind that I re-organized test_sediments.jl a bit while investigating.

@jagoosw
Copy link
Collaborator

jagoosw commented Nov 16, 2024

That is quite strange, the changes since 0.93 shouldn't have effected any of this right?

@ali-ramadhan
Copy link
Collaborator Author

Yeah I agree and I couldn't find any hints in the diffs or the changelog... I'll try to take a closer look later today.

@ali-ramadhan
Copy link
Collaborator Author

Because of CliMA/Oceananigans.jl#3930 I had to rename ab2_step_free_surface! to step_free_surface! .

I think there's a wider discussion to be had around supporting hydrostatic RK3 time stepping with OceanBioME.jl but I'll open a new issue for it.

@ali-ramadhan
Copy link
Collaborator Author

Hmmm, new error seems to be related to store_flat_tendencies! going out of bounds for sediment models.

@ali-ramadhan
Copy link
Collaborator Author

Let's try this again.

Right now we're depending on the main branch of Oceananigans.jl until a new version is tagged (hopefully v0.95.0 soon).

Docs are failing because they're pulling in v0.92.x based on the [compat] entries and there's no way to enforce a specific branch via [compat] so I added a Manifest.toml to docs/ to get the tests to pass. The aim is to remove this Manifest.toml once we can upgrade to v0.95.0.

@ali-ramadhan ali-ramadhan changed the title Support Oceananigans v0.94 and bump v0.13.3 Support Oceananigans v0.94 and bump v0.13.4 Dec 4, 2024
@ali-ramadhan
Copy link
Collaborator Author

I think we're also encountering CliMA/Oceananigans.jl#3964 here.

@ali-ramadhan ali-ramadhan changed the title Support Oceananigans v0.94 and bump v0.13.4 Support Oceananigans v0.95 and bump v0.13.4 Dec 20, 2024
@ali-ramadhan
Copy link
Collaborator Author

Updating to Oceananigans v0.95 now!

@ali-ramadhan
Copy link
Collaborator Author

Updating to Oceananigans v0.95.6 now! Need to dig more but PR CliMA/Oceananigans.jl#4008 might be related to the failing tests?

@jagoosw
Copy link
Collaborator

jagoosw commented Dec 31, 2024

Hi @ali-ramadhan,

Sorry, I've not been keeping on top of this.

I've had a go at refactoring the sediments, do you think it would be best to merge this PR with sediments broken and put that in a different PR, or do it all here?

@ali-ramadhan
Copy link
Collaborator Author

Hey @jagoosw! It is just test_flat_sediment that's broken right now so I've used @test_skip there for now.

Maybe the test will just pass with your refactor? Happy to merge this PR and find out. If the test still fails I'm happy to help debug.

I just left a docs/Manifest.toml which isn't there in the main branch so I'll remove it and see if the docs build okay.

@ali-ramadhan
Copy link
Collaborator Author

ali-ramadhan commented Dec 31, 2024

Just for reference here's the error I'm getting with julia -g2. Not too informative and I couldn't immediately find where it might go out of bounds. Stack trace is trimmed to fit within the GitHub comment character limit.

ERROR: a BoundsError was thrown during kernel execution on thread (33, 1, 1) in block (1, 1, 1).
Out-of-bounds array access
Stacktrace:
 [1] throw_boundserror at /home/alir/.julia/packages/CUDA/2kjXI/src/device/quirks.jl:15
 [2] multiple call sites at unknown:0

GPU(), QuasiAdamsBashforth2, Instant remineralisation, NPZD, Rectilinear grid: Error During Test at /home/alir/atdepth/OceanBioME.jl/test/test_sediments.jl:207
  Test threw exception
  Expression: test_flat_sediment(grid, biogeochemistry, model; timestepper)
  KernelException: exception thrown during kernel execution on device NVIDIA GeForce RTX 4090
  Stacktrace:
    [1] check_exceptions()
      @ CUDA ~/.julia/packages/CUDA/2kjXI/src/compiler/exceptions.jl:39
    [2] device_synchronize(; blocking::Bool, spin::Bool)
      @ CUDA ~/.julia/packages/CUDA/2kjXI/lib/cudadrv/synchronization.jl:191
    [3] device_synchronize
      @ ~/.julia/packages/CUDA/2kjXI/lib/cudadrv/synchronization.jl:178 [inlined]
    [4] checked_cuModuleLoadDataEx(_module::Base.RefValue{Ptr{CUDA.CUmod_st}}, image::Ptr{UInt8}, numOptions::Int64, options::Vector{CUDA.CUjit_option_enum}, optionValues::Vector{Ptr{Nothing}})
      @ CUDA ~/.julia/packages/CUDA/2kjXI/lib/cudadrv/module.jl:18
    [5] CuModule(data::Vector{UInt8}, options::Dict{CUDA.CUjit_option_enum, Any})
      @ CUDA ~/.julia/packages/CUDA/2kjXI/lib/cudadrv/module.jl:60
    [6] CuModule
      @ ~/.julia/packages/CUDA/2kjXI/lib/cudadrv/module.jl:49 [inlined]
    [7] link(job::GPUCompiler.CompilerJob, compiled::@NamedTuple{image::Vector{UInt8}, entry::String})
      @ CUDA ~/.julia/packages/CUDA/2kjXI/src/compiler/compilation.jl:409
    [8] actual_compilation(cache::Dict{Any, CuFunction}, src::Core.MethodInstance, world::UInt64, cfg::GPUCompiler.CompilerConfig{GPUCompiler.PTXCompilerTarget, CUDA.CUDACompilerParams}, compiler::typeof(CUDA.compile), linker::typeof(CUDA.link))
      @ GPUCompiler ~/.julia/packages/GPUCompiler/2CW9L/src/execution.jl:262
    [9] cached_compilation(cache::Dict{Any, CuFunction}, src::Core.MethodInstance, cfg::GPUCompiler.CompilerConfig{GPUCompiler.PTXCompilerTarget, CUDA.CUDACompilerParams}, compiler::Function, linker::Function)
      @ GPUCompiler ~/.julia/packages/GPUCompiler/2CW9L/src/execution.jl:151
   [10] macro expansion
      @ ~/.julia/packages/CUDA/2kjXI/src/compiler/execution.jl:380 [inlined]
   [11] macro expansion
      @ ./lock.jl:267 [inlined]
   [12] cufunction(f::typeof(OceanBioME.Models.Sediments.gpu_calculate_sediment_tendencies!), tt::Type{Tuple{KernelAbstractions.CompilerMetadata{KernelAbstractions.NDIteration.StaticSize{(3, 3)}, KernelAbstractions.NDIteration.DynamicCheck, Nothing, Nothing, KernelAbstractions.NDIteration.NDRange{2, KernelAbstractions.NDIteration.StaticSize{(1, 1)}, KernelAbstractions.NDIteration.StaticSize{(16, 16)}, Nothing, Nothing}}, InstantRemineralisation{Float64, @NamedTuple{N_storage::OffsetArrays.OffsetArray{Float64, 3, CuDeviceArray{Float64, 3, 1}}}, Nothing, CuDeviceMatrix{Int64, 1}}, NutrientPhytoplanktonZooplanktonDetritus{Float64, @NamedTuple{P::OffsetArrays.OffsetArray{Float64, 3, CuDeviceArray{Float64, 3, 1}}, D::OffsetArrays.OffsetArray{Float64, 3, CuDeviceArray{Float64, 3, 1}}}}, RectilinearGrid{Float64, Periodic, Periodic, Bounded, Oceananigans.Grids.StaticVerticalCoordinate{OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Float64}, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Nothing}, Centered{1, Float64, Nothing, Nothing, Nothing, Nothing}, @NamedTuple{P::OffsetArrays.OffsetArray{Float64, 3, CuDeviceArray{Float64, 3, 1}}, D::OffsetArrays.OffsetArray{Float64, 3, CuDeviceArray{Float64, 3, 1}}, N::OffsetArrays.OffsetArray{Float64, 3, CuDeviceArray{Float64, 3, 1}}}, OffsetArrays.OffsetArray{Float64, 3, CuDeviceArray{Float64, 3, 1}}, @NamedTuple{N_storage::OffsetArrays.OffsetArray{Float64, 3, CuDeviceArray{Float64, 3, 1}}}, Float64}}; kwargs::@Kwargs{always_inline::Bool, maxthreads::Int64})
      @ CUDA ~/.julia/packages/CUDA/2kjXI/src/compiler/execution.jl:375
   [13] macro expansion
      @ ~/.julia/packages/CUDA/2kjXI/src/compiler/execution.jl:112 [inlined]
   [14] (::KernelAbstractions.Kernel{CUDABackend, KernelAbstractions.NDIteration.StaticSize{(16, 16)}, KernelAbstractions.NDIteration.StaticSize{(3, 3)}, typeof(OceanBioME.Models.Sediments.gpu_calculate_sediment_tendencies!)})(::InstantRemineralisation{Float64, @NamedTuple{N_storage::Field{Center, Center, Center, Nothing, RectilinearGrid{Float64, Periodic, Periodic, Bounded, Oceananigans.Grids.StaticVerticalCoordinate{OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Float64}, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, GPU}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float64, 3, CuArray{Float64, 3, CUDA.DeviceMemory}}, Float64, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.Fields.FieldBoundaryBuffers{Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}}, @NamedTuple{Gⁿ::@NamedTuple{N_storage::Field{Center, Center, Center, Nothing, RectilinearGrid{Float64, Periodic, Periodic, Bounded, Oceananigans.Grids.StaticVerticalCoordinate{OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Float64}, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, GPU}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float64, 3, CuArray{Float64, 3, CUDA.DeviceMemory}}, Float64, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.Fields.FieldBoundaryBuffers{Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}}, G⁻::@NamedTuple{N_storage::Field{Center, Center, Center, Nothing, RectilinearGrid{Float64, Periodic, Periodic, Bounded, Oceananigans.Grids.StaticVerticalCoordinate{OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Float64}, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, GPU}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float64, 3, CuArray{Float64, 3, CUDA.DeviceMemory}}, Float64, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.Fields.FieldBoundaryBuffers{Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}}}, CuArray{Int64, 2, CUDA.DeviceMemory}}, ::Vararg{Any}; ndrange::Nothing, workgroupsize::Nothing)
      @ CUDA.CUDAKernels ~/.julia/packages/CUDA/2kjXI/src/CUDAKernels.jl:103
   [15] (::KernelAbstractions.Kernel{CUDABackend, KernelAbstractions.NDIteration.StaticSize{(16, 16)}, KernelAbstractions.NDIteration.StaticSize{(3, 3)}, typeof(OceanBioME.Models.Sediments.gpu_calculate_sediment_tendencies!)})(::InstantRemineralisation{Float64, @NamedTuple{N_storage::Field{Center, Center, Center, Nothing, RectilinearGrid{Float64, Periodic, Periodic, Bounded, Oceananigans.Grids.StaticVerticalCoordinate{OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Float64}, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, GPU}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float64, 3, CuArray{Float64, 3, CUDA.DeviceMemory}}, Float64, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.Fields.FieldBoundaryBuffers{Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}}, @NamedTuple{Gⁿ::@NamedTuple{N_storage::Field{Center, Center, Center, Nothing, RectilinearGrid{Float64, Periodic, Periodic, Bounded, Oceananigans.Grids.StaticVerticalCoordinate{OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Float64}, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, GPU}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float64, 3, CuArray{Float64, 3, CUDA.DeviceMemory}}, Float64, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.Fields.FieldBoundaryBuffers{Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}}, G⁻::@NamedTuple{N_storage::Field{Center, Center, Center, Nothing, RectilinearGrid{Float64, Periodic, Periodic, Bounded, Oceananigans.Grids.StaticVerticalCoordinate{OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Float64}, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, GPU}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float64, 3, CuArray{Float64, 3, CUDA.DeviceMemory}}, Float64, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.Fields.FieldBoundaryBuffers{Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}}}, CuArray{Int64, 2, CUDA.DeviceMemory}}, ::Vararg{Any})
      @ CUDA.CUDAKernels ~/.julia/packages/CUDA/2kjXI/src/CUDAKernels.jl:89
   [16] #_launch!#12
      @ ~/.julia/packages/Oceananigans/GXAdX/src/Utils/kernel_launching.jl:295 [inlined]
   [17] _launch!
      @ ~/.julia/packages/Oceananigans/GXAdX/src/Utils/kernel_launching.jl:272 [inlined]
   [18] launch!
      @ ~/.julia/packages/Oceananigans/GXAdX/src/Utils/kernel_launching.jl:255 [inlined]
   [19] update_tendencies!
...
   [32] macro expansion
      @ ~/atdepth/OceanBioME.jl/test/test_sediments.jl:207 [inlined]
   [33] macro expansion
      @ ~/.julia/juliaup/julia-1.10.6+0.x64.linux.gnu/share/julia/stdlib/v1.10/Test/src/Test.jl:669 [inlined]
   [34] macro expansion
      @ ~/atdepth/OceanBioME.jl/test/test_sediments.jl:207 [inlined]
   [35] macro expansion
      @ ~/.julia/juliaup/julia-1.10.6+0.x64.linux.gnu/share/julia/stdlib/v1.10/Test/src/Test.jl:1577 [inlined]
   [36] macro expansion
      @ ~/atdepth/OceanBioME.jl/test/test_sediments.jl:207 [inlined]
   [37] macro expansion
      @ ~/.julia/juliaup/julia-1.10.6+0.x64.linux.gnu/share/julia/stdlib/v1.10/Test/src/Test.jl:1577 [inlined]
   [38] top-level scope
      @ ~/atdepth/OceanBioME.jl/test/test_sediments.jl:132

@ali-ramadhan
Copy link
Collaborator Author

Docs are fine without docs/Manifest.toml now so this PR is ready for review!

@jagoosw
Copy link
Collaborator

jagoosw commented Jan 1, 2025

Okay since its going to be a pretty major refactor so its less of a mess I'll do it in a different PR, should we add a warning to the sediment model constructors that they're broken?

@ali-ramadhan
Copy link
Collaborator Author

Good idea. I just added extra warnings for InstantRemineralisation and SimpleMultiG.

Copy link
Collaborator

@jagoosw jagoosw left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Thank you for sorting this out, I'll fix the sediment imminently

@jagoosw jagoosw mentioned this pull request Jan 2, 2025
@ali-ramadhan ali-ramadhan merged commit db05016 into main Jan 2, 2025
5 checks passed
@ali-ramadhan ali-ramadhan deleted the ali/oceananigans-0.94 branch January 2, 2025 18:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants