Run Examples on GPU for Documentation (#104)

* add single column to docs * changed filename * this should work, now start with documenting * updating Oceananigans * small bugfix * new compact * bugfix * might be right now * start without near global omip * fix ecco test * another bugfix * Update src/OceanSeaIceModels/CrossRealmFluxes/roughness_lengths.jl Co-authored-by: Gregory L. Wagner <[email protected]> * Update src/OceanSeaIceModels/CrossRealmFluxes/tabulated_albedo.jl Co-authored-by: Gregory L. Wagner <[email protected]> * Update src/OceanSeaIceModels/CrossRealmFluxes/tabulated_albedo.jl Co-authored-by: Gregory L. Wagner <[email protected]> * correct typo * Update src/OceanSeaIceModels/ocean_only_model.jl Co-authored-by: Gregory L. Wagner <[email protected]> * TemperatureDependentAirViscosity and ReynoldsScalingFunction * update ScalarRoughnessLength * correct FT in albedo * bugfix * bugfix * FT for tabulated albedo * using zstar * small bugfix * bugfix * another try * better default for ECCOMetadata * some initial comments * exposing al parameters * better stability functions * bugfixes * bugfixxed * some refactoring + adding flux examples * some grammar improvements * bugfix in docs * remove useless file * bugfix * functions for time quantities * bottom drag coefficient * back to reference height * some improvements * bugfix * revert to chloride * changed pipeline * use github documenter for the moment * bugfix * change measurement to reference * conversion depending on k * change to the * change name * bugfix * namechange * add surface flux tests * change name to "gusty" velocity difference * typo * bugfix * correctly include runoff in forcing * bugfix * speed up generate fluxes * error in project * ok found the issue! * correct the functions * change to z_data * improve example * closure * extrinsic-intrinsic * remove interpolate * z_data instead of h_data * add warning for longitude range * small change * change the CFL * comments * remove zstar prototype * bugfix * small bugfix * bugfix * allow precompilation * small bugfix * change default to ecco2daily * correct JRA55 * remove one division * some optimizations * let's see if this speeds up the computation * updating catke to new syntax * also new manifest * add on_architecture * the conversion is done _inside_ fts * another try * small verification * fix a typo * typo * last try * another bugfix * another bugfix * fix back * back to square one. Still very slow * switching to an array * some changes to the docstring * bug with reciprocal of rho * more fixes * switch reference frames * try fixing tests with secret tokens * take ecco 4 monthly * maybe here it will work? * userame and password in docs * ecco secrets in ci * remove all ECCO2 from examples * more verbose error * test new buildkite env variables * fix tests * fix tests * fix tests * some fixes * the simulation that is running * name changes * some changes to the simulation * try it out * test documentation on GPU * correct docs key * timeout the docs after 24 hours instead of the default 2 * med example on the GPU * update Oceananigans * namechange to pipeline * rename the examples build * remove size threshold * GLMakie -> CairoMakie * add GPUs * yet another bugfix * yet another bugfix * test on Oceananigans#main * test on Oceananigans#main * instantiate first * Oceananigans#main * smooth out a couple of bugs * remove Oceananigans from docs Project.toml * fix gpu problem * try only the near global at the moment * remove reaive humidity * remove reative humidity cause not used * bugfix * just check the near global configuration * try with everything together * typo * try without the giba simulation * simple cpu example * adding a slurm time * try with SSL_no_verify * try with wget * typo * full fledged model on GPU * Update .buildkite/examples_build.yml Co-authored-by: Gregory L. Wagner <[email protected]> * add literate debug * back to previous project / manifest * add visualization * check if it deploys * test the deploy keys * test reading the ssh key * typo * the docs deploy? * check deployment again * test docs deployment * remove `winthenv` * near global simulation in docs * better comments * another test * try new keys * progress it stuck on single_column_grid * test only with snapshot * gif -> mp4 gifs are very big in size since there is no compression * remove also "inspect_ecco_data.jl" * try with a video? * print every 500 iterations * max abs * some changes to simulation details * smaller titles * lift correct variables * reduce frames by a factor of two * bugfix * another bugfix * adding generate bathymetry * some bugfixes * better colormaps * more massaging * try the full global ocean * global ocean simulation * Parameter space problem * unfortunately we cannot use active cells map * few tweaks * code cleanup * code cleanup * code cleanup * clearer explanation * clearer explanation * try the global ocean * better visualizations * try again * fix bug * change figure size * one movie per quantity * bugfix * fix typo * some more bugfixes * first small change * lakes -> connected regions * warm up -> spin up * add colorbars * code alignment * bugfix * some formatting changes * bugfix * typo * bugfix * back to how it was * fix precompilation issue * Update examples/near_global_ocean_simulation.jl Co-authored-by: Gregory L. Wagner <[email protected]> * fix examples then merge * different unit of measure --------- Co-authored-by: Gregory L. Wagner <[email protected]> Co-authored-by: Navid C. Constantinou <[email protected]>
CliMA · Aug 13, 2024 · 41aa1e4 · 41aa1e4
1 parent e2b8280
commit 41aa1e4
Show file tree

Hide file tree

Showing 28 changed files with 555 additions and 753 deletions.
diff --git a/.buildkite/examples_build.yml b/.buildkite/examples_build.yml
@@ -0,0 +1,57 @@
+agents:
+  queue: new-central
+  slurm_mem: 8G
+  modules: climacommon/2024_05_27
+  slurm_time: 48:00:00
+
+timeout_in_minutes: 1440
+
+env:
+  JULIA_LOAD_PATH: "${JULIA_LOAD_PATH}:${BUILDKITE_BUILD_CHECKOUT_PATH}/.buildkite"
+  OPENBLAS_NUM_THREADS: 1
+  OMPI_MCA_opal_warn_on_missing_libcuda: 0
+
+steps:
+  - label: "initialize"
+    key: "init"
+    command:
+      - "echo '--- Instantiate project'"
+      - "julia --project -e 'using Pkg; Pkg.instantiate(; verbose=true); Pkg.precompile(; strict=true)'"
+      # force the initialization of the CUDA runtime as it is lazily loaded by default
+      - "julia --project -e 'using CUDA; CUDA.precompile_runtime()'"
+
+    agents:
+      slurm_mem: 120G
+      slurm_gpus: 1
+      slurm_cpus_per_task: 8
+    env:
+      JULIA_NUM_PRECOMPILE_TASKS: 8
+
+  - wait
+
+  - label: "Run documentation"
+    key: "build_documentation"
+    commands:
+      - "julia --color=yes --project=docs/ -e 'using Pkg; Pkg.instantiate(); Pkg.develop(PackageSpec(path=pwd()))'"
+      - "julia --color=yes --project=docs/ docs/make.jl"
+
+    agents:
+      slurm_mem: 120G
+      slurm_gpus: 1
+      slurm_cpus_per_task: 8
+      slurm_ntasks: 1
+      slurm_gpus_per_task: 1
+      slurm_time: 48:00:00
+
+    env:
+      JULIA_NUM_PRECOMPILE_TASKS: 8
+      JULIA_DEBUG: "Documenter"
+      # This environment variable is needed to avoid SSL verification errors when Downloads.jl 
+      # tries to download the bathymetry data. It should not be required so we need to fix our certificates.
+      # and remove this environment variable. ref: https://github.com/JuliaLang/Downloads.jl/issues/97
+      JULIA_SSL_NO_VERIFY: "**" 
+
+    timeout_in_minutes: 1440
+
+  - wait: ~
+    continue_on_failure: true
diff --git a/.buildkite/pipeline.yml b/.buildkite/pipeline.yml
@@ -20,6 +20,7 @@ steps:
 
     agents:
       slurm_mem: 32G
+      slurm_gpus: 1
       slurm_cpus_per_task: 8
       slurm_gpus: 1
       slurm_ntasks: 1

diff --git a/Manifest.toml b/Manifest.toml
@@ -885,7 +885,9 @@ version = "1.2.0"
 
 [[deps.Oceananigans]]
 deps = ["Adapt", "CUDA", "Crayons", "CubedSphere", "Dates", "Distances", "DocStringExtensions", "FFTW", "Glob", "IncompleteLU", "InteractiveUtils", "IterativeSolvers", "JLD2", "KernelAbstractions", "LinearAlgebra", "Logging", "MPI", "NCDatasets", "OffsetArrays", "OrderedCollections", "PencilArrays", "PencilFFTs", "Pkg", "Printf", "Random", "Rotations", "SeawaterPolynomials", "SparseArrays", "Statistics", "StructArrays"]
-git-tree-sha1 = "942c78ad3c95e47bcbab3eeacb0b8a846bcfb33b"
+git-tree-sha1 = "937a5ba25152cbcd0b74e66d21ce931a620a6cc4"
+repo-rev = "main"
+repo-url = "https://github.com/CliMA/Oceananigans.jl.git"
 uuid = "9e8cae18-63c1-5223-a75c-80ca9d6e9a09"
 version = "0.91.4"
 

diff --git a/docs/Project.toml b/docs/Project.toml
@@ -3,10 +3,9 @@ CairoMakie = "13f3f980-e62b-5c42-98c6-ff1f3baf88f0"
 DataDeps = "124859b0-ceae-595e-8997-d05f6a7a8dfe"
 Documenter = "e30172f5-a6a5-5a46-863b-614d45cd2de4"
 Literate = "98b081ad-f1c9-55d3-8b20-4c87d4299306"
-Oceananigans = "9e8cae18-63c1-5223-a75c-80ca9d6e9a09"
+OrthogonalSphericalShellGrids = "c2be9673-fb75-4747-82dc-aa2bb9f4aed0"
 
 [compat]
 CairoMakie = "0.10.12"
 DataDeps = "0.7"
 Documenter = "1"
-Oceananigans = "0.91.4"
diff --git a/docs/make.jl b/docs/make.jl
@@ -15,26 +15,30 @@ const EXAMPLES_DIR = joinpath(@__DIR__, "..", "examples")
 const OUTPUT_DIR   = joinpath(@__DIR__, "src/literated")
 
 to_be_literated = [
-    "inspect_ecco_data.jl",
+    # "inspect_ecco_data.jl",
+    "generate_bathymetry.jl",
     "generate_surface_fluxes.jl",
-    "single_column_simulation.jl",
-    # "near_global_omip_simulation.jl"
+    # "single_column_simulation.jl",
+    # "mediterranean_simulation_with_ecco_restoring.jl",
+    "near_global_ocean_simulation.jl"
 ]
 
 for file in to_be_literated
     filepath = joinpath(EXAMPLES_DIR, file)
-    Literate.markdown(filepath, OUTPUT_DIR; flavor = Literate.DocumenterFlavor())
+    withenv("JULIA_DEBUG" => "Literate") do
+        Literate.markdown(filepath, OUTPUT_DIR; flavor = Literate.DocumenterFlavor(), execute = true)
+    end
 end
 
 #####
 ##### Build and deploy docs
 #####
 
 format = Documenter.HTML(
-       collapselevel = 2,
-      size_threshold = nothing,
-          prettyurls = get(ENV, "CI", nothing) == "true",
-           canonical = "https://clima.github.io/ClimaOceanDocumentation/dev/",
+   collapselevel = 2,
+  size_threshold = nothing,
+      prettyurls = get(ENV, "CI", nothing) == "true",
+       canonical = "https://clima.github.io/ClimaOceanDocumentation/dev/",
 )
 
 pages = [
@@ -48,9 +52,12 @@ pages = [
         ],
 
     "Examples" => [
-        "Inspect ECCO2 data" => "literated/inspect_ecco_data.md",
+        # "Inspect ECCO2 data" => "literated/inspect_ecco_data.md",
+        "Generate bathymetry" => "literated/generate_bathymetry.md",
         "Surface fluxes" => "literated/generate_surface_fluxes.md",
-        "Single column simulation" => "literated/single_column_simulation.md",
+        # "Single column simulation" => "literated/single_column_simulation.md",
+        # "Mediterranean simulation with ECCO restoring" => "literated/mediterranean_simulation_with_ecco_restoring.md",
+        "Near-global Ocean simulation" => "literated/near_global_ocean_simulation.md",
         ]
 ]
 
@@ -86,10 +93,8 @@ for file in files
     rm(file)
 end
 
-withenv("GITHUB_REPOSITORY" => "CliMA/ClimaOceanDocumentation") do
-    deploydocs(        repo = "github.com/CliMA/ClimaOceanDocumentation.git",
-                   versions = ["stable" => "v^", "v#.#.#", "dev" => "dev"],
-                  forcepush = true,
-                  devbranch = "main",
-               push_preview = true)
-end
+deploydocs(repo = "github.com/CliMA/ClimaOceanDocumentation.git",
+       versions = ["stable" => "v^", "v#.#.#", "dev" => "dev"],
+      forcepush = true,
+      devbranch = "main",
+   push_preview = true)
diff --git a/examples/freely_decaying_mediterranean.jl b/examples/freely_decaying_mediterranean.jl