Marine ens. var. #1353

guillaumevernieres · 2024-10-30T13:15:02Z

Companion PR to NOAA-EMC/global-workflow#3041

This PR needs an update to jcb-gdas (NOAA-EMC/jcb-gdas#46)

test/gw-ci/CMakeLists.txt

Companion PR to NOAA-EMC/GDASApp#1353 --------- Co-authored-by: Dan Holdaway <[email protected]>

guillaumevernieres · 2024-12-02T21:29:41Z

Automated GW GDASApp Testing Results: Machine: hera

Start: Mon Dec  2 20:56:32 UTC 2024 on hfe11
---------------------------------------------------
Build:                                  *FAILED*
Build: Failed at Mon Dec  2 21:14:36 UTC 2024
Build: see output at /scratch1/NCEPDEV/da/role.jedipara/CI/GDASApp/workflow/PR/1353/global-workflow/sorc/log.build

Not quite sure why it returned "Build Failed", it's still building.

guillaumevernieres · 2024-12-02T21:55:51Z

ctests started:

guillaumevernieres · 2024-12-03T01:06:59Z

The new batch of ctests is just not practical. I'm not even sure this is going to run overnight.
1 - ~~we need to add dependencies to the ctests, if a test fail, the subsequent tests shouldn't run~~
2 - we should add labels specific to aero/atm/snow and marine. Our work is usually perpendicular to each-others, that pr for example should only run the marine DA stuff
3 - We shouldn't run the forecast after the DA, this will be tested again in the g-w tests ... debatable ...
4 - is it time to stage the model fcst from the 1/2 cycle on hpc's so we don't have to always run the fcst?

I'll create an issue tomorrow with what's above.

RussTreadon-NOAA · 2024-12-03T01:07:34Z

Multiple ctests are failing / will fail because the expanded ctests suite now requires the GSI package (gsi.x and enkf.x) to be compiled. For example,

 31/134 Test #1964: test_gdasapp_C96C48_hybatmDA_gdas_anal_202112210000 ......................***Failed  210.87 sec

failed because gsi.x does not exist

+ exglobal_atmos_analysis.sh[934]: /bin/cp -p /scratch1/NCEPDEV/da/role.jedipara/CI/GDASApp/workflow/PR/1353/global-workflow/exec/gsi.x /scratch1/NCEPDEV/stmp2/role.jedipara/RUNDIRS/C96C48_hybatmDA/gdas.2021122100/anal.4045548
/bin/cp: cannot stat '/scratch1/NCEPDEV/da/role.jedipara/CI/GDASApp/workflow/PR/1353/global-workflow/exec/gsi.x': No such file or directory
+ exglobal_atmos_analysis.sh[1]: postamble exglobal_atmos_analysis.sh 1733177865 1
+ preamble.sh[70]: set +x
End exglobal_atmos_analysis.sh at 22:18:30 with error code 1 (time elapsed: 00:00:45)
+ JGLOBAL_ATMOS_ANALYSIS[1]: postamble JGLOBAL_ATMOS_ANALYSIS 1733177846 1

Script /scratch1/NCEPDEV/da/role.jedipara/CI/GDASApp/repo/ci/run_gw_ci.sh builds g-w using

rm -rf log.build
./build_all.sh -u &>> log.build
build_status=$?

-u builds GDASApp. This was fine when our g-w ctests did not include GSI-based DA. The expanded set of tests now includes GSI-based DA. We need to add -g to build GSI.

I changed /scratch1/NCEPDEV/da/role.jedipara/CI/GDASApp/repo/ci/run_gw_ci.sh to read

rm -rf log.build
./build_all.sh -ugk &>> log.build
build_status=$?

I added -k to kill all builds build in case any build fails.

guillaumevernieres · 2024-12-03T01:40:54Z

Damn ... Never ending story. Of course the one liner change I decide not to test just failed.
To the reviewers: I'll run the marine DA related "ci" manually on msu tomorrow.

guillaumevernieres · 2024-12-03T01:42:46Z

Multiple ctests are failing / will fail because the expanded ctests suite now requires the GSI package (gsi.x and enkf.x) to be compiled. For example,
 31/134 Test #1964: test_gdasapp_C96C48_hybatmDA_gdas_anal_202112210000 ......................***Failed  210.87 sec
failed because gsi.x does not exist
+ exglobal_atmos_analysis.sh[934]: /bin/cp -p /scratch1/NCEPDEV/da/role.jedipara/CI/GDASApp/workflow/PR/1353/global-workflow/exec/gsi.x /scratch1/NCEPDEV/stmp2/role.jedipara/RUNDIRS/C96C48_hybatmDA/gdas.2021122100/anal.4045548
/bin/cp: cannot stat '/scratch1/NCEPDEV/da/role.jedipara/CI/GDASApp/workflow/PR/1353/global-workflow/exec/gsi.x': No such file or directory
+ exglobal_atmos_analysis.sh[1]: postamble exglobal_atmos_analysis.sh 1733177865 1
+ preamble.sh[70]: set +x
End exglobal_atmos_analysis.sh at 22:18:30 with error code 1 (time elapsed: 00:00:45)
+ JGLOBAL_ATMOS_ANALYSIS[1]: postamble JGLOBAL_ATMOS_ANALYSIS 1733177846 1
Script /scratch1/NCEPDEV/da/role.jedipara/CI/GDASApp/repo/ci/run_gw_ci.sh builds g-w using
rm -rf log.build
./build_all.sh -u &>> log.build
build_status=$?
-u builds GDASApp. This was fine when our g-w ctests did not include GSI-based DA. The expanded set of tests now includes GSI-based DA. We need to add -g to build GSI.

I changed /scratch1/NCEPDEV/da/role.jedipara/CI/GDASApp/repo/ci/run_gw_ci.sh to read
rm -rf log.build
./build_all.sh -ugk &>> log.build
build_status=$?
I added -k to kill all builds build in case any build fails.

Thanks for digging into this @RussTreadon-NOAA .

…SApp into feature/marineenvar

guillaumevernieres · 2024-12-03T13:33:41Z

@DavidNew-NOAA , @RussTreadon-NOAA , @CoryMartin-NOAA , @danholdaway , I'm encapsulating the new ctests inside of cmake options and I'm turning them off by default. For the time being, developers will have to manually turn on/off the tests that are relevant to the code changes. Not great but a simple way to move forward until we have a very low-res atmosphere.

DavidNew-NOAA · 2024-12-03T13:45:23Z

@guillaumevernieres What is the problem with having these tests on by default? If we do encapsulate them in CMake options, then we need a way to turn those options on at build time for nightly testing and PR testing

guillaumevernieres · 2024-12-03T14:14:12Z

@guillaumevernieres What is the problem with having these tests on by default? If we do encapsulate them in CMake options, then we need a way to turn those options on at build time for nightly testing and PR testing

@DavidNew-NOAA , the tests are too slow, to the point that they are completely useless right now. It wasn't even 1/2 way through this am when I checked.
You can add the options to your cmake build to turn them on for "over-night" testing, but we shouldn't run all these for pr testing.

guillaumevernieres · 2024-12-03T14:57:49Z

testing on my side:

something fishy with the ens. fcst. rocoto job ... but the ens. forecast was successful.
Here's the rest:

shlyaeva

I am happy to reapprove all the previous changes (with the caveat of non expert review), but I have no opinion on the latest changes (encapsulating tests in the cmake options). Since it seems that it needs discussion and is not fully related to this PR one option would be to separate that change in a different PR (unless it can be discussed and decided in this PR in a reasonable time).

guillaumevernieres · 2024-12-03T16:10:35Z

I am happy to reapprove all the previous changes (with the caveat of non expert review), but I have no opinion on the latest changes (encapsulating tests in the cmake options). Since it seems that it needs discussion and is not fully related to this PR one option would be to separate that change in a different PR (unless it can be discussed and decided in this PR in a reasonable time).

Good point. I'll add @DavidNew-NOAA to the PR review, see what he says first. As it is right now, we can't run the ci that we trigger through the github labels.

DavidNew-NOAA

Most is fine by me. I'll just comment on the testing stuff:

After some thought, I think it will be easier to only run a subset of tests at the command line rather than with CMake flags. For the run_gw_ci.sh script in GDASApp here, we run:

ctest -j${NTASKS_TESTS} -R gdasapp --output-on-failure &>> log.ctest

We can just run ctest -R C96C48_ufs_hybatmDA or ctest -R C48mx500_3DVarAOWCDA depending on which label one adds to a PR on GitHub.

It will be much easier for me to implement that than turning on those CMake flags at build time, because build.sh in GDASApp is called by build_gdas.sh in GW which is called by build_all.sh, so I'll have to had several different options tobuild_all.sh.

I was meaning to add a test reference for the C384mx025 marinevar ctest, but I can do it in a separate PR. Up to you.

AndrewEichmann-NOAA

No objections from inspection

guillaumevernieres · 2024-12-03T16:47:58Z

Most is fine by me. I'll just comment on the testing stuff:

After some thought, I think it will be easier to only run a subset of tests at the command line rather than with CMake flags. For the run_gw_ci.sh script in GDASApp here, we run:

ctest -j${NTASKS_TESTS} -R gdasapp --output-on-failure &>> log.ctest

We can just run ctest -R C96C48_ufs_hybatmDA or ctest -R C48mx500_3DVarAOWCDA depending on which label one adds to a PR on GitHub.

It will be much easier for me to implement that than turning on those CMake flags at build time, because build.sh in GDASApp is called by build_gdas.sh in GW which is called by build_all.sh, so I'll have to had several different options tobuild_all.sh.

I was meaning to add a test reference for the C384mx025 marinevar ctest, but I can do it in a separate PR. Up to you.

We're not going to run these test on hera for this pr, so I can go back to running the test by default if that helps your nightly testing @DavidNew-NOAA . No need yet for a reference file for the c384.

KatherineLukens-NOAA

I second Andy's comment. No objections here.

DavidNew-NOAA

Thanks @guillaumevernieres . I will work on specific CI labels for GitHub testing when Hera is back up

guillaumevernieres and others added 15 commits September 24, 2024 01:41

wip, weird h values in ens iau incr

293459f

early exit recenter

348e8d7

wip

8dd6217

Merge branch 'develop' into feature/marineenvar

8b8c5b8

iso date and tidy

3f95af2

added hyb test

870eb32

Merge branch 'develop' into feature/marineenvar

4af3902

...

d454e8b

pynorm

ef2b7c7

pynorm

d31e8ed

cpp norm

b872bc3

Merge branch 'develop' into feature/marineenvar

1d7496e

Merge branch 'develop' into feature/marineenvar

cf48626

hyb test, speed up diffusion

f923841

removed ens recenter

71e9e5f

guillaumevernieres mentioned this pull request Oct 30, 2024

Marine ens. var. NOAA-EMC/jcb-gdas#37

Merged

AndrewEichmann-NOAA reviewed Oct 30, 2024

View reviewed changes

test/gw-ci/CMakeLists.txt Outdated Show resolved Hide resolved

danholdaway added a commit to NOAA-EMC/jcb-gdas that referenced this pull request Oct 30, 2024

Marine ens. var. (#37)

1e9d844

Companion PR to NOAA-EMC/GDASApp#1353 --------- Co-authored-by: Dan Holdaway <[email protected]>

guillaumevernieres mentioned this pull request Nov 13, 2024

Skip the ice specific diffusion #1359

Closed

guillaumevernieres added 11 commits November 19, 2024 10:42

Merge branch 'develop' into feature/marineenvar

56dd9ba

e-folding of bkgerr

a35966e

wip

1ae7042

update submodules

7128edf

update submodules

2bf5c77

updated submodules

d506930

fixed bugs, added e-folding for bkgerr

6275dfa

live and don't learn

c1a7e3e

Merge branch 'develop' into feature/marineenvar

416a558

var name issue

931142c

Merge branch 'develop' into feature/marineenvar

50bf828

guillaumevernieres removed the hera-GW-RT-Failed Automated testing with global-workflow failed on Hera label Dec 3, 2024

guillaumevernieres added 2 commits December 3, 2024 07:32

optional gw ctests

17d6db8

Merge branch 'feature/marineenvar' of https://github.com/NOAA-EMC/GDA…

133a854

…SApp into feature/marineenvar

guillaumevernieres dismissed shlyaeva’s stale review via 133a854 December 3, 2024 13:33

guillaumevernieres requested a review from shlyaeva December 3, 2024 15:00

shlyaeva reviewed Dec 3, 2024

View reviewed changes

typo

6e23615

guillaumevernieres requested a review from DavidNew-NOAA December 3, 2024 16:10

DavidNew-NOAA reviewed Dec 3, 2024

View reviewed changes

AndrewEichmann-NOAA previously approved these changes Dec 3, 2024

View reviewed changes

KatherineLukens-NOAA previously approved these changes Dec 3, 2024

View reviewed changes

default gw test to on

5d47396

guillaumevernieres dismissed stale reviews from KatherineLukens-NOAA and AndrewEichmann-NOAA via 5d47396 December 3, 2024 16:52

DavidNew-NOAA approved these changes Dec 3, 2024

View reviewed changes

shlyaeva approved these changes Dec 3, 2024

View reviewed changes

guillaumevernieres merged commit d91663b into develop Dec 3, 2024
5 checks passed

guillaumevernieres deleted the feature/marineenvar branch December 3, 2024 18:25

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Marine ens. var. #1353

Marine ens. var. #1353

guillaumevernieres commented Oct 30, 2024 •

edited

Loading

guillaumevernieres commented Dec 2, 2024

guillaumevernieres commented Dec 2, 2024

guillaumevernieres commented Dec 3, 2024 •

edited

Loading

RussTreadon-NOAA commented Dec 3, 2024

guillaumevernieres commented Dec 3, 2024

guillaumevernieres commented Dec 3, 2024

guillaumevernieres commented Dec 3, 2024

DavidNew-NOAA commented Dec 3, 2024 •

edited

Loading

guillaumevernieres commented Dec 3, 2024

guillaumevernieres commented Dec 3, 2024

shlyaeva left a comment

guillaumevernieres commented Dec 3, 2024

DavidNew-NOAA left a comment •

edited

Loading

AndrewEichmann-NOAA left a comment

guillaumevernieres commented Dec 3, 2024

KatherineLukens-NOAA left a comment

DavidNew-NOAA left a comment

Marine ens. var. #1353

Marine ens. var. #1353

Conversation

guillaumevernieres commented Oct 30, 2024 • edited Loading

guillaumevernieres commented Dec 2, 2024

guillaumevernieres commented Dec 2, 2024

guillaumevernieres commented Dec 3, 2024 • edited Loading

RussTreadon-NOAA commented Dec 3, 2024

guillaumevernieres commented Dec 3, 2024

guillaumevernieres commented Dec 3, 2024

guillaumevernieres commented Dec 3, 2024

DavidNew-NOAA commented Dec 3, 2024 • edited Loading

guillaumevernieres commented Dec 3, 2024

guillaumevernieres commented Dec 3, 2024

shlyaeva left a comment

Choose a reason for hiding this comment

guillaumevernieres commented Dec 3, 2024

DavidNew-NOAA left a comment • edited Loading

Choose a reason for hiding this comment

AndrewEichmann-NOAA left a comment

Choose a reason for hiding this comment

guillaumevernieres commented Dec 3, 2024

KatherineLukens-NOAA left a comment

Choose a reason for hiding this comment

DavidNew-NOAA left a comment

Choose a reason for hiding this comment

guillaumevernieres commented Oct 30, 2024 •

edited

Loading

guillaumevernieres commented Dec 3, 2024 •

edited

Loading

DavidNew-NOAA commented Dec 3, 2024 •

edited

Loading

DavidNew-NOAA left a comment •

edited

Loading