Merge Radmon scripts for monitoring the satellite radiance DA #162

xyzemc · 2023-12-12T16:56:07Z

This PR is for run the Radiance Monitor (Radmon) Package in RRFS workflow for generating the real-time the satellite radiance DA statistics information, like O-B/O-A, bias correction, data number for each instrument and each channel. The code has been merged to rrfs_utl as described in: NOAA-GSL/rrfs_utl#43, and NOAA-GSL/rrfs_utl#46.

This PR includes:
Added the rrfs_radmon directory under ./ush;
Added the required fix files for running radmon;
Modifiedexrrfs_run_gsidiag.sh scripts to run radmon, and setup some local variables that radmon scripts required;
Added the 'gzip' in exrrfs_run_gsidiag.sh to compress the diag file for saving the space, and fit the requirement of radmon scripts.

ISSUE:

Fixes the issue(s) mentioned in Add radiance monitoring package to workflow #62

…diance; Added the required fix files for running radmon; Added the gzip in gsi scripts to comporess the diag file for saving disk space.

fix/gsi/gdas_radmon_base.tar

ush/rrfs_radmon/clean_tankdir.sh

jobs/JRRFS_RUN_GSIDIAG

scripts/exrrfs_run_gsidiag.sh

ush/rrfs_radmon/exnam_verfrad.sh

chan-hoo · 2023-12-13T12:46:02Z

@xyzemc, could you open an issue for this PR?

chan-hoo · 2023-12-13T12:58:07Z

@xyzemc, I found that an issue for this PR had been already issued. I'll add it to the above description.

xyzemc · 2023-12-13T13:43:08Z

@chan-hoo Thanks for linking the issue to this PR.

Confirmed with Ed. this file is not needed for radmon in RRFS workflow.

moved DO_RADMON="FALSE" from exrrfs_run_gsidiag.sh to config_defaults.sh

MatthewPyle-NOAA · 2023-12-18T13:59:30Z

@xyzemc I'd like to get this one merged today or tomorrow if possible. A big merge that went in today has created some script conflicts that need to be resolved.

xyzemc · 2023-12-18T19:43:34Z

@xyzemc I'd like to get this one merged today or tomorrow if possible. A big merge that went in today has created some script conflicts that need to be resolved.

resolved all the conflicts.

MatthewPyle-NOAA

Congrats on clearing out the conflicts, @xyzemc. Just one small request before approving.

MatthewPyle-NOAA · 2023-12-18T19:48:16Z

ush/rrfs_radmon/exnam_verfrad.sh

Could this be renamed to either exrrfs_verfrad.sh or radmon_verfrad.sh? Mostly just would like to get rid of the "nam" aspect.

Could this be renamed to either exrrfs_verfrad.sh or radmon_verfrad.sh? Mostly just would like to get rid of the "nam" aspect.

Yes. I was thinking that too.

Thanks @xyzemc I'd like to run a quick test to make sure nothing breaks with these changes, but looks good to me.

MatthewPyle-NOAA

Giving conditional approval, but should a DA-engineering test be run? What do you think, @chan-hoo?

chan-hoo · 2023-12-20T12:04:10Z

@MatthewPyle-NOAA, sorry for my late response. Yes, I think so. We should check at least two tasks "run_analysis" and "run_gsidiag" in a DA eng test.

MatthewPyle-NOAA · 2023-12-21T16:40:18Z

@chan-hoo is a DA engineering test best done on WCOSS2? Real time or retro? I've never done much with it. Thanks!

chan-hoo · 2023-12-21T19:32:26Z

@MatthewPyle-NOAA, I agree. I also feel it is not good as an engineering test. I think we'll need a retro or ensemble case for wcoss2 (not real-time).

chan-hoo · 2024-01-02T13:08:56Z

@xyzemc, could you let us know which sample configuration we can use to test this PR?

…default.

… radmon

chan-hoo · 2024-01-03T11:57:53Z

@xyzemc, currently the DA engineering test configuration does not work on Hera (Issue #180). Once it is available, I'll test your PR.

xyzemc · 2024-01-03T14:21:06Z

@xyzemc, currently the DA engineering test configuration does not work on Hera (Issue #180). Once it is available, I'll test your PR.

Thank you!

scripts/exrrfs_run_gsidiag.sh

chan-hoo · 2024-01-06T02:53:46Z

ush/rrfs_radmon/exnam_verfrad.sh

+
+#################################################################################
+
+export USHradmon=${USHradmon:-${USHDIR}/rrfs_radmon}


${USHDIR} -> ${USHdir}

It is not changed yet.

ush/rrfs_radmon/exrrfs_verfrad.sh

chan-hoo · 2024-01-09T11:59:56Z

@xyzemc, the test run failed. Please resolve the issues. You can use the engineering test script ush/sample_configs/DA_eng/config.DA.retro.hera.sh.

xyzemc · 2024-01-09T14:05:35Z

@xyzemc, the test run failed. Please resolve the issues. You can use the engineering test script ush/sample_configs/DA_eng/config.DA.retro.hera.sh.

Will do it when Hera is back from maintenance.

MatthewPyle-NOAA · 2024-01-17T21:42:49Z

@xyzemc Where does this PR stand now?

xyzemc · 2024-01-18T04:23:42Z

@xyzemc Where does this PR stand now?

Still working on the engineering test

MatthewPyle-NOAA · 2024-01-18T12:05:58Z

@xyzemc Where does this PR stand now?

Still working on the engineering test

Okay, thanks. No hurry - just wanted to make sure I wasn't holding up the process.

… radmon

xyzemc · 2024-01-18T16:37:08Z

Added an option in exrrfs_run_gsidiag.sh to skip the radmon job when the 'radstat' file is not existing.
Engendering test has been pasted from my end.

chan-hoo · 2024-01-18T16:39:52Z

Thanks for the update. I'm testing it again now.

chan-hoo · 2024-01-18T19:17:15Z

It looks working well. Approving.

MatthewPyle-NOAA

Reapproving as well, and will merge. Thanks for seeing this one through, @chan-hoo and @xyzemc

Merged the scripts for running the radmon to monitor the satellite ra…

2b4f9b5

…diance; Added the required fix files for running radmon; Added the gzip in gsi scripts to comporess the diag file for saving disk space.

xyzemc requested review from MatthewPyle-NOAA, JacobCarley-NOAA and chan-hoo as code owners December 12, 2023 16:56

guoqing-noaa reviewed Dec 13, 2023

View reviewed changes

fix/gsi/gdas_radmon_base.tar Outdated Show resolved Hide resolved

ush/rrfs_radmon/clean_tankdir.sh Show resolved Hide resolved

chan-hoo reviewed Dec 13, 2023

View reviewed changes

chan-hoo linked an issue Dec 13, 2023 that may be closed by this pull request

Add radiance monitoring package to workflow #62

Closed

xyzemc and others added 4 commits December 13, 2023 16:06

Address the reviewer's coments.

0a7af77

Delete fix/gsi/gdas_radmon_base.tar

d407efc

Confirmed with Ed. this file is not needed for radmon in RRFS workflow.

modified USHDIR to USHdir

8a28ed2

moved DO_RADMON="FALSE" from exrrfs_run_gsidiag.sh to config_defaults.sh

Change USHnam to USHradmon

7c6bf37

xyzemc and others added 3 commits December 18, 2023 15:01

resove the conflict with dev-sci

c35e456

resove the merge conflict

0d24581

Merge branch 'dev-sci' into radmon

a19d0f8

MatthewPyle-NOAA reviewed Dec 18, 2023

View reviewed changes

modified the exnam_verfrad.sh to exrrfs_verfrad.sh

30775d2

MatthewPyle-NOAA requested review from chan-hoo and MatthewPyle-NOAA December 19, 2023 13:51

MatthewPyle-NOAA previously approved these changes Dec 19, 2023

View reviewed changes

Merge branch 'dev-sci' into radmon

fdef8b8

Removed the local defination of DO_RADMON which is defined in config_…

2c69010

…default.

Merge branch 'radmon' of https://github.com/xyzemc/rrfs-workflow into…

ed5f609

… radmon

xyzemc dismissed MatthewPyle-NOAA’s stale review via ed5f609 January 2, 2024 19:38

Merge branch 'dev-sci' into radmon

e82911a

chan-hoo reviewed Jan 6, 2024

View reviewed changes

address reviewer's comments

121e915

chan-hoo and others added 3 commits January 16, 2024 15:22

Merge branch 'dev-sci' into radmon

c560a3b

Merge branch 'dev-sci' into radmon

bb0ac27

Merge branch 'dev-sci' into radmon

b8a12ef

xyzemc added 2 commits January 18, 2024 16:17

Add the option to skip the radmon job when radstat file is not existing.

61bf322

Merge branch 'radmon' of https://github.com/xyzemc/rrfs-workflow into…

676cd29

… radmon

chan-hoo approved these changes Jan 18, 2024

View reviewed changes

MatthewPyle-NOAA self-requested a review January 18, 2024 19:19

MatthewPyle-NOAA approved these changes Jan 18, 2024

View reviewed changes

MatthewPyle-NOAA merged commit acbeb01 into NOAA-EMC:dev-sci Jan 18, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Merge Radmon scripts for monitoring the satellite radiance DA #162

Merge Radmon scripts for monitoring the satellite radiance DA #162

xyzemc commented Dec 12, 2023 •

edited by chan-hoo

Loading

chan-hoo commented Dec 13, 2023

chan-hoo commented Dec 13, 2023

xyzemc commented Dec 13, 2023

MatthewPyle-NOAA commented Dec 18, 2023

xyzemc commented Dec 18, 2023

MatthewPyle-NOAA left a comment

MatthewPyle-NOAA Dec 18, 2023

xyzemc Dec 18, 2023

MatthewPyle-NOAA Dec 18, 2023

MatthewPyle-NOAA left a comment

chan-hoo commented Dec 20, 2023

MatthewPyle-NOAA commented Dec 21, 2023

chan-hoo commented Dec 21, 2023

chan-hoo commented Jan 2, 2024

chan-hoo commented Jan 3, 2024

xyzemc commented Jan 3, 2024

chan-hoo Jan 6, 2024

xyzemc Jan 6, 2024

chan-hoo Jan 7, 2024

chan-hoo commented Jan 9, 2024

xyzemc commented Jan 9, 2024

MatthewPyle-NOAA commented Jan 17, 2024

xyzemc commented Jan 18, 2024

MatthewPyle-NOAA commented Jan 18, 2024

xyzemc commented Jan 18, 2024 •

edited

Loading

chan-hoo commented Jan 18, 2024

chan-hoo commented Jan 18, 2024

MatthewPyle-NOAA left a comment


		#################################################################################

		export USHradmon=${USHradmon:-${USHDIR}/rrfs_radmon}

Merge Radmon scripts for monitoring the satellite radiance DA #162

Merge Radmon scripts for monitoring the satellite radiance DA #162

Conversation

xyzemc commented Dec 12, 2023 • edited by chan-hoo Loading

ISSUE:

chan-hoo commented Dec 13, 2023

chan-hoo commented Dec 13, 2023

xyzemc commented Dec 13, 2023

MatthewPyle-NOAA commented Dec 18, 2023

xyzemc commented Dec 18, 2023

MatthewPyle-NOAA left a comment

Choose a reason for hiding this comment

MatthewPyle-NOAA Dec 18, 2023

Choose a reason for hiding this comment

xyzemc Dec 18, 2023

Choose a reason for hiding this comment

MatthewPyle-NOAA Dec 18, 2023

Choose a reason for hiding this comment

MatthewPyle-NOAA left a comment

Choose a reason for hiding this comment

chan-hoo commented Dec 20, 2023

MatthewPyle-NOAA commented Dec 21, 2023

chan-hoo commented Dec 21, 2023

chan-hoo commented Jan 2, 2024

chan-hoo commented Jan 3, 2024

xyzemc commented Jan 3, 2024

chan-hoo Jan 6, 2024

Choose a reason for hiding this comment

xyzemc Jan 6, 2024

Choose a reason for hiding this comment

chan-hoo Jan 7, 2024

Choose a reason for hiding this comment

chan-hoo commented Jan 9, 2024

xyzemc commented Jan 9, 2024

MatthewPyle-NOAA commented Jan 17, 2024

xyzemc commented Jan 18, 2024

MatthewPyle-NOAA commented Jan 18, 2024

xyzemc commented Jan 18, 2024 • edited Loading

chan-hoo commented Jan 18, 2024

chan-hoo commented Jan 18, 2024

MatthewPyle-NOAA left a comment

Choose a reason for hiding this comment

xyzemc commented Dec 12, 2023 •

edited by chan-hoo

Loading

xyzemc commented Jan 18, 2024 •

edited

Loading