Replace gfs_cyc with an interval (NOAA-EMC#2928)

# Description To facilitate longer and more flexible GFS cadences, the `gfs_cyc` variable is replaced with a specified interval. Up front, this is reflected in a change in the arguments for setup_exp to: ``` --interval <n_hours> ``` Where `n_hours` is the interval (in hours) between gfs forecasts. `n_hours` must be a multiple of 6. If 0, no gfs will be run (only gdas; only valid for cycled mode). The default value is 6 (every cycle). (This is a change from current behavior of 24.) In cycled mode, there is an additional argument to control which cycle will be the first gfs cycle: ``` --sdate_gfs <YYYYMMDDHH> ``` The default if not provided is `--idate` + 6h (first full cycle). This is the same as current behavior when `gfs_cyc` is 6, but may vary from current behavior for other cadences. As part of this change, some of the validation of the dates has been added. `--edate` has also been made optional and defaults to `--idate` if not provided. During `config.base` template-filling, `INTERVAL_GFS` (renamed from `STEP_GFS`) is defined as `--interval` and `SDATE_GFS as `--sdate_gfs`. Some changes were necessary to the gfs verification (metp) job, as `gfs_cyc` was being used downstream by verif-global. That has been removed, and instead workflow will be responsible for only running metp on the correct cycles. This also removes "do nothing" metp tasks that exit immediately, because only the last GFS cycle in a day would actually process verification. Now, metp has its own cycledef and will (a) always runs at 18z, regardless of whether gfs is running at 18z or not, if the interval is less than 24h; (b) use the same cycledef as gfs if the interval is 24h or greater. This is simpler than trying to determine the last gfs cycle of a day when it could change from day to day. To facilitate this change, support for the undocumented rocoto dependency tag `taskvalid` is added, as the metp task needs to know whether the cycle has a gfsarch task or not. metp will trigger on gfsarch completing (as before), or look backwards for the last gfsarch to exist. Additionally, a couple EE2 issues with the metp job are resolved (even though it is not run in ops): - verif-global update replaced `$CDUMP` with `$RUN` - `$DATAROOT` is no longer redefined in the metp job Also corrects some dependency issues with the extractvars job for replay and the replay CI test. Depends on NOAA-EMC/EMC_verif-global#137 Resolves NOAA-EMC#260 Refs NOAA-EMC#1299 --------- Co-authored-by: David Huber <[email protected]>
KateFriedman-NOAA · Oct 22, 2024 · 720eb4c · 720eb4c
1 parent 76befe1
commit 720eb4c
Show file tree

Hide file tree

Showing 34 changed files with 227 additions and 183 deletions.
diff --git a/ci/cases/gfsv17/C384mx025_3DVarAOWCDA.yaml b/ci/cases/gfsv17/C384mx025_3DVarAOWCDA.yaml
@@ -8,7 +8,7 @@ arguments:
   resdetatmos: 384
   resdetocean: 0.25
   nens: 0
-  gfs_cyc: 4
+  interval: 6
   start: cold
   comroot: {{ 'RUNTESTS' | getenv }}/COMROOT
   expdir: {{ 'RUNTESTS' | getenv }}/EXPDIR

diff --git a/ci/cases/pr/C48_S2SWA_gefs.yaml b/ci/cases/pr/C48_S2SWA_gefs.yaml
@@ -9,7 +9,7 @@ arguments:
   resdetocean: 5.0
   resensatmos: 48
   nens: 2
-  gfs_cyc: 1
+  interval: 24
   start: cold
   comroot: {{ 'RUNTESTS' | getenv }}/COMROOT
   expdir: {{ 'RUNTESTS' | getenv }}/EXPDIR

diff --git a/ci/cases/pr/C48mx500_3DVarAOWCDA.yaml b/ci/cases/pr/C48mx500_3DVarAOWCDA.yaml
@@ -13,7 +13,7 @@ arguments:
   idate: 2021032412
   edate: 2021032418
   nens: 0
-  gfs_cyc: 0
+  interval: 0
   start: warm
   yaml: {{ HOMEgfs }}/ci/cases/yamls/soca_gfs_defaults_ci.yaml
 

diff --git a/ci/cases/pr/C96C48_hybatmDA.yaml b/ci/cases/pr/C96C48_hybatmDA.yaml
@@ -14,6 +14,6 @@ arguments:
   idate: 2021122018
   edate: 2021122106
   nens: 2
-  gfs_cyc: 1
+  interval: 24
   start: cold
   yaml: {{ HOMEgfs }}/ci/cases/yamls/gfs_defaults_ci.yaml
diff --git a/ci/cases/pr/C96C48_hybatmaerosnowDA.yaml b/ci/cases/pr/C96C48_hybatmaerosnowDA.yaml
@@ -13,7 +13,7 @@ arguments:
   idate: 2021122012
   edate: 2021122100
   nens: 2
-  gfs_cyc: 1
+  interval: 24
   start: cold
   yaml: {{ HOMEgfs }}/ci/cases/yamls/atmaerosnowDA_defaults_ci.yaml
 

diff --git a/ci/cases/pr/C96C48_ufs_hybatmDA.yaml b/ci/cases/pr/C96C48_ufs_hybatmDA.yaml
@@ -13,7 +13,7 @@ arguments:
   idate: 2024022318
   edate: 2024022406
   nens: 2
-  gfs_cyc: 1
+  interval: 24
   start: warm
   yaml: {{ HOMEgfs }}/ci/cases/yamls/ufs_hybatmDA_defaults.ci.yaml
 

diff --git a/ci/cases/pr/C96_S2SWA_gefs_replay_ics.yaml b/ci/cases/pr/C96_S2SWA_gefs_replay_ics.yaml
@@ -9,11 +9,14 @@ arguments:
   resdetocean: 1.0
   resensatmos: 96
   nens: 2
-  gfs_cyc: 1
+  interval: 6
   start: warm
   comroot: {{ 'RUNTESTS' | getenv }}/COMROOT
   expdir: {{ 'RUNTESTS' | getenv }}/EXPDIR
   idate: 2020110100
   edate: 2020110100
   yaml: {{ HOMEgfs }}/ci/cases/yamls/gefs_replay_ci.yaml
   icsdir: {{ 'ICSDIR_ROOT' | getenv }}/C96mx100/20240610
+
+skip_ci_on_hosts:
+  - wcoss2
diff --git a/ci/cases/pr/C96_atm3DVar.yaml b/ci/cases/pr/C96_atm3DVar.yaml
@@ -12,7 +12,7 @@ arguments:
   idate: 2021122018
   edate: 2021122106
   nens: 0
-  gfs_cyc: 1
+  interval: 24
   start: cold
   yaml: {{ HOMEgfs }}/ci/cases/yamls/gfs_defaults_ci.yaml
 

diff --git a/ci/cases/pr/C96_atm3DVar_extended.yaml b/ci/cases/pr/C96_atm3DVar_extended.yaml
@@ -12,7 +12,7 @@ arguments:
   idate: 2021122018
   edate: 2021122118
   nens: 0
-  gfs_cyc: 4
+  interval: 6
   start: cold
   yaml: {{ HOMEgfs }}/ci/cases/yamls/gfs_extended_ci.yaml
 

diff --git a/ci/cases/weekly/C384C192_hybatmda.yaml b/ci/cases/weekly/C384C192_hybatmda.yaml
@@ -14,6 +14,6 @@ arguments:
   idate: 2023040118
   edate: 2023040200
   nens: 2
-  gfs_cyc: 1
+  interval: 24
   start: cold
   yaml: {{ HOMEgfs }}/ci/cases/yamls/gfs_defaults_ci.yaml
diff --git a/ci/cases/weekly/C384_atm3DVar.yaml b/ci/cases/weekly/C384_atm3DVar.yaml
@@ -14,6 +14,6 @@ arguments:
   idate: 2023040118
   edate: 2023040200
   nens: 0
-  gfs_cyc: 1
+  interval: 24
   start: cold
   yaml: {{ HOMEgfs }}/ci/cases/yamls/gfs_defaults_ci.yaml
diff --git a/docs/source/jobs.rst b/docs/source/jobs.rst
@@ -8,7 +8,7 @@ GFS Configuration
 
 The sequence of jobs that are run for an end-to-end (analysis+forecast+post processing+verification) GFS configuration using the Global Workflow is shown above. The system utilizes a collection of scripts that perform the tasks for each step.
 
-For any cycle the system consists of two suites -- the "gdas" suite which provides the initial guess fields, and the "gfs" suite which creates the initial conditions and forecast of the system. As with the operational system, the gdas runs for each cycle (00, 06, 12, and 18 UTC), however, to save time and space in experiments, the gfs (right side of the diagram) is initially setup to run for only the 00 UTC cycle (See the "run GFS this cycle?" portion of the diagram). The option to run the GFS for all four cycles is available (see the ``gfs_cyc`` variable in configuration file).
+For any cycle the system consists of two suites -- the "gdas" suite which provides the initial guess fields, and the "gfs" suite which creates the initial conditions and forecast of the system.
 
 An experimental run is different from operations in the following ways:
 

diff --git a/docs/source/setup.rst b/docs/source/setup.rst
@@ -32,7 +32,7 @@ The following command examples include variables for reference but users should
 ::
 
    cd workflow
-   ./setup_expt.py gfs forecast-only --idate $IDATE --edate $EDATE [--app $APP] [--start $START] [--gfs_cyc $GFS_CYC] [--resdetatmos $RESDETATMOS] [--resdetocean $RESDETOCEAN]
+   ./setup_expt.py gfs forecast-only --idate $IDATE --edate $EDATE [--app $APP] [--start $START] [--interval $INTERVAL_GFS] [--resdetatmos $RESDETATMOS] [--resdetocean $RESDETOCEAN]
      [--pslot $PSLOT] [--configdir $CONFIGDIR] [--comroot $COMROOT] [--expdir $EXPDIR]
 
 where:
@@ -51,12 +51,12 @@ where:
 
    * ``$START`` is the start type (warm or cold [default])
    * ``$IDATE`` is the initial start date of your run (first cycle CDATE, YYYYMMDDCC)
-   * ``$EDATE`` is the ending date of your run (YYYYMMDDCC) and is the last cycle that will complete
+   * ``$EDATE`` is the ending date of your run (YYYYMMDDCC) and is the last cycle that will complete [default: $IDATE]
    * ``$PSLOT`` is the name of your experiment [default: test]
    * ``$CONFIGDIR`` is the path to the ``/config`` folder under the copy of the system you're using [default: $TOP_OF_CLONE/parm/config/]
    * ``$RESDETATMOS`` is the resolution of the atmosphere component of the system (i.e. 768 for C768) [default: 384]
    * ``$RESDETOCEAN`` is the resolution of the ocean component of the system (i.e. 0.25 for 1/4 degree) [default: 0.; determined based on atmosphere resolution]
-   * ``$GFS_CYC`` is the forecast frequency (0 = none, 1 = 00z only [default], 2 = 00z & 12z, 4 = all cycles)
+   * ``$INTERVAL_GFS`` is the forecast interval in hours [default: 6]
    * ``$COMROOT`` is the path to your experiment output directory. Your ``ROTDIR`` (rotating com directory) will be created using ``COMROOT`` and ``PSLOT``. [default: $HOME (but do not use default due to limited space in home directories normally, provide a path to a larger scratch space)]
    * ``$EXPDIR`` is the path to your experiment directory where your configs will be placed and where you will find your workflow monitoring files (i.e. rocoto database and xml file). DO NOT include PSLOT folder at end of path, it will be built for you. [default: $HOME]
 
@@ -67,7 +67,7 @@ Atm-only:
 ::
 
    cd workflow
-   ./setup_expt.py gfs forecast-only --pslot test --idate 2020010100 --edate 2020010118 --resdetatmos 384 --gfs_cyc 4 --comroot /some_large_disk_area/Joe.Schmo/comroot --expdir /some_safe_disk_area/Joe.Schmo/expdir
+   ./setup_expt.py gfs forecast-only --pslot test --idate 2020010100 --edate 2020010118 --resdetatmos 384 --interval 6 --comroot /some_large_disk_area/Joe.Schmo/comroot --expdir /some_safe_disk_area/Joe.Schmo/expdir
 
 Coupled:
 
@@ -144,7 +144,8 @@ The following command examples include variables for reference but users should
 ::
 
    cd workflow
-   ./setup_expt.py gfs cycled --idate $IDATE --edate $EDATE [--app $APP] [--start $START] [--gfs_cyc $GFS_CYC]
+   ./setup_expt.py gfs cycled --idate $IDATE --edate $EDATE [--app $APP] [--start $START]
+     [--interval $INTERVAL_GFS] [--sdate_gfs $SDATE_GFS]
      [--resdetatmos $RESDETATMOS] [--resdetocean $RESDETOCEAN] [--resensatmos $RESENSATMOS] [--nens $NENS] [--run $RUN]
      [--pslot $PSLOT] [--configdir $CONFIGDIR] [--comroot $COMROOT] [--expdir $EXPDIR] [--icsdir $ICSDIR]
 
@@ -163,9 +164,10 @@ where:
      - S2SWA: atm-ocean-ice-wave-aerosols
 
    * ``$IDATE`` is the initial start date of your run (first cycle CDATE, YYYYMMDDCC)
-   * ``$EDATE`` is the ending date of your run (YYYYMMDDCC) and is the last cycle that will complete
+   * ``$EDATE`` is the ending date of your run (YYYYMMDDCC) and is the last cycle that will complete [default: $IDATE]
    * ``$START`` is the start type (warm or cold [default])
-   * ``$GFS_CYC`` is the forecast frequency (0 = none, 1 = 00z only [default], 2 = 00z & 12z, 4 = all cycles)
+   * ``$INTERVAL_GFS`` is the forecast interval in hours [default: 6]
+   * ``$SDATE_GFS`` cycle to begin GFS forecast [default: $IDATE + 6]
    * ``$RESDETATMOS`` is the resolution of the atmosphere component of the deterministic forecast [default: 384]
    * ``$RESDETOCEAN`` is the resolution of the ocean component of the deterministic forecast [default: 0.; determined based on atmosphere resolution]
    * ``$RESENSATMOS`` is the resolution of the atmosphere component of the ensemble forecast [default: 192]
@@ -184,7 +186,7 @@ Example:
 ::
 
    cd workflow
-   ./setup_expt.py gfs cycled --pslot test --configdir /home/Joe.Schmo/git/global-workflow/parm/config --idate 2020010100 --edate 2020010118 --comroot /some_large_disk_area/Joe.Schmo/comroot --expdir /some_safe_disk_area/Joe.Schmo/expdir --resdetatmos 384 --resensatmos 192 --nens 80 --gfs_cyc 4
+   ./setup_expt.py gfs cycled --pslot test --configdir /home/Joe.Schmo/git/global-workflow/parm/config --idate 2020010100 --edate 2020010118 --comroot /some_large_disk_area/Joe.Schmo/comroot --expdir /some_safe_disk_area/Joe.Schmo/expdir --resdetatmos 384 --resensatmos 192 --nens 80 --interval 6
 
 Example ``setup_expt.py`` on Orion:
 

diff --git a/gempak/ush/gfs_meta_comp.sh b/gempak/ush/gfs_meta_comp.sh
@@ -24,7 +24,7 @@ device="nc | ${metaname}"
 
 export COMIN="gfs.multi"
 mkdir "${COMIN}"
-for cycle in $(seq -f "%02g" -s ' ' 0  "${STEP_GFS}" "${cyc}"); do
+for cycle in $(seq -f "%02g" -s ' ' 0  "${INTERVAL_GFS}" "${cyc}"); do
     YMD=${PDY} HH=${cycle} GRID="1p00" declare_from_tmpl gempak_dir:COM_ATMOS_GEMPAK_TMPL
     for file_in in "${gempak_dir}/gfs_1p00_${PDY}${cycle}f"*; do
         file_out="${COMIN}/$(basename "${file_in}")"

diff --git a/gempak/ush/gfs_meta_mar_comp.sh b/gempak/ush/gfs_meta_mar_comp.sh
@@ -15,7 +15,7 @@ cp "${HOMEgfs}/gempak/fix/datatype.tbl" datatype.tbl
 
 export COMIN="gfs.multi"
 mkdir -p "${COMIN}"
-for cycle in $(seq -f "%02g" -s ' ' 0 "${STEP_GFS}" "${cyc}"); do
+for cycle in $(seq -f "%02g" -s ' ' 0 "${INTERVAL_GFS}" "${cyc}"); do
     YMD=${PDY} HH=${cycle} GRID="1p00" declare_from_tmpl gempak_dir:COM_ATMOS_GEMPAK_TMPL
     for file_in in "${gempak_dir}/gfs_1p00_${PDY}${cycle}f"*; do
         file_out="${COMIN}/$(basename "${file_in}")"

diff --git a/jobs/JGFS_ATMOS_VERIFICATION b/jobs/JGFS_ATMOS_VERIFICATION
@@ -16,12 +16,6 @@ source "${HOMEgfs}/ush/jjob_header.sh" -e "metp" -c "base metp"
 ## METPCASE : METplus verification use case (g2g1 | g2o1 | pcp1)
 ###############################################################
 
-# TODO: This should not be permitted as DATAROOT is set at the job-card level.
-# TODO: DATAROOT is being used as DATA in metp jobs.  This should be rectified in metp.
-# TODO: The temporary directory is DATA and is created at the top of the J-Job.
-# TODO: remove this line
-export DATAROOT=${DATA}
-
 VDATE=$(date --utc +%Y%m%d%H -d "${PDY} ${cyc} - ${VRFYBACK_HRS} hours")
 export VDATE=${VDATE:0:8}
 

diff --git a/parm/config/gefs/config.base b/parm/config/gefs/config.base
@@ -227,7 +227,8 @@ export FHOUT_OCN=3
 export FHOUT_ICE=3
 
 # GFS cycle info
-export gfs_cyc=@gfs_cyc@ # 0: no GFS cycle, 1: 00Z only, 2: 00Z and 12Z only, 4: all 4 cycles.
+export INTERVAL_GFS=@INTERVAL_GFS@ # Frequency of GFS forecast
+export SDATE_GFS=@SDATE_GFS@
 
 # set variables needed for use with REPLAY ICs
 export REPLAY_ICS=@REPLAY_ICS@
@@ -255,11 +256,6 @@ export FHOUT_WAV=3
 export FHMAX_HF_WAV=120
 export FHOUT_HF_WAV=1
 export FHMAX_WAV=${FHMAX_GFS}
-if (( gfs_cyc != 0 )); then
-    export STEP_GFS=$(( 24 / gfs_cyc ))
-else
-    export STEP_GFS="0"
-fi
 export ILPOST=1           # gempak output frequency up to F120
 
 export FHMIN_ENKF=${FHMIN_GFS}

diff --git a/parm/config/gfs/config.base b/parm/config/gfs/config.base
@@ -283,7 +283,8 @@ export FHOUT_ICE=3
 export EUPD_CYC="@EUPD_CYC@"
 
 # GFS cycle info
-export gfs_cyc=@gfs_cyc@ # 0: no GFS cycle, 1: 00Z only, 2: 00Z and 12Z only, 4: all 4 cycles.
+export INTERVAL_GFS=@INTERVAL_GFS@ # Frequency of GFS forecast
+export SDATE_GFS=@SDATE_GFS@
 
 # GFS output and frequency
 export FHMIN_GFS=0
@@ -302,11 +303,7 @@ export FHMAX_HF_WAV=120
 export FHOUT_HF_WAV=1
 export FHMAX_WAV=${FHMAX:-9}
 export FHMAX_WAV_GFS=${FHMAX_GFS}
-if (( gfs_cyc != 0 )); then
-    export STEP_GFS=$(( 24 / gfs_cyc ))
-else
-    export STEP_GFS="0"
-fi
+
 # TODO: Change gempak to use standard out variables (#2348)
 export ILPOST=${FHOUT_HF_GFS}           # gempak output frequency up to F120
 if (( FHMAX_HF_GFS < 120 )); then

diff --git a/parm/config/gfs/config.resources b/parm/config/gfs/config.resources
@@ -988,8 +988,8 @@ case ${step} in
     threads_per_task=1
     walltime_gdas="03:00:00"
     walltime_gfs="06:00:00"
-    ntasks=4
-    tasks_per_node=4
+    ntasks=1
+    tasks_per_node=1
     export memory="80G"
     ;;
 

diff --git a/parm/config/gfs/config.wave b/parm/config/gfs/config.wave
@@ -117,7 +117,7 @@ if [[ "${RUN}" == "gdas" ]]; then
   export WAVNCYC=4
   export WAVHCYC=${assim_freq:-6}
   export FHMAX_WAV_CUR=48 # RTOFS forecasts only out to 8 days
-elif [[ ${gfs_cyc} -ne 0 ]]; then
+elif (( INTERVAL_GFS > 0 )); then
   export WAVHCYC=${assim_freq:-6}
   export FHMAX_WAV_CUR=192 # RTOFS forecasts only out to 8 days
 else

diff --git a/scripts/exgfs_aero_init_aerosol.py b/scripts/exgfs_aero_init_aerosol.py
@@ -11,13 +11,13 @@
 ---------
 This script requires the following environment variables be set beforehand:
 
-CDATE: 		Initial time in YYYYMMDDHH format
-STEP_GFS: 	Forecast cadence (frequency) in hours
-FHMAX_GFS: 	Forecast length in hours
-RUN: 		Forecast phase (gfs or gdas). Currently always expected to be gfs.
-ROTDIR: 	Rotating (COM) directory
-USHgfs: 	Path to global-workflow `ush` directory
-PARMgfs: 	Path to global-workflow `parm` directory
+CDATE:        Initial time in YYYYMMDDHH format
+INTERVAL_GFS: Forecast cadence (frequency) in hours
+FHMAX_GFS:    Forecast length in hours
+RUN:          Forecast phase (gfs or gdas). Currently always expected to be gfs.
+ROTDIR:       Rotating (COM) directory
+USHgfs:       Path to global-workflow `ush` directory
+PARMgfs:      Path to global-workflow `parm` directory
 
 Additionally, the following data files are used:
 
@@ -66,7 +66,7 @@
 def main() -> None:
     # Read in environment variables and make sure they exist
     cdate = get_env_var("CDATE")
-    incr = int(get_env_var('STEP_GFS'))
+    incr = int(get_env_var('INTERVAL_GFS'))
     fcst_length = int(get_env_var('FHMAX_GFS'))
     run = get_env_var("RUN")
     rot_dir = get_env_var("ROTDIR")

diff --git a/sorc/verif-global.fd b/sorc/verif-global.fd
diff --git a/workflow/applications/applications.py b/workflow/applications/applications.py
@@ -68,6 +68,7 @@ def __init__(self, conf: Configuration) -> None:
 
         self.nens = base.get('NMEM_ENS', 0)
         self.fcst_segments = base.get('FCST_SEGMENTS', None)
+        self.interval_gfs = to_timedelta(f"{base.get('INTERVAL_GFS')}H")
 
         if not AppConfig.is_monotonic(self.fcst_segments):
             raise ValueError(f'Forecast segments do not increase monotonically: {",".join(self.fcst_segments)}')
@@ -109,9 +110,6 @@ def _init_finalize(self, conf: Configuration):
         # Save base in the internal state since it is often needed
         base = self.configs['_no_run']['base']
 
-        # Get more configuration options into the class attributes
-        self.gfs_cyc = base.get('gfs_cyc')
-
         # Get task names for the application
         self.task_names = self.get_task_names()
 
@@ -199,19 +197,6 @@ def get_task_names(self, run="_no_run") -> Dict[str, List[str]]:
         '''
         pass
 
-    @staticmethod
-    def get_gfs_interval(gfs_cyc: int) -> timedelta:
-        """
-        return interval in hours based on gfs_cyc
-        """
-
-        gfs_internal_map = {'1': '24H', '2': '12H', '4': '6H'}
-
-        try:
-            return to_timedelta(gfs_internal_map[str(gfs_cyc)])
-        except KeyError:
-            raise KeyError(f'Invalid gfs_cyc = {gfs_cyc}')
-
     @staticmethod
     def is_monotonic(test_list: List, check_decreasing: bool = False) -> bool:
         """

diff --git a/workflow/applications/gefs.py b/workflow/applications/gefs.py
@@ -42,7 +42,6 @@ def _get_app_configs(self):
     def _update_base(base_in):
 
         base_out = base_in.copy()
-        base_out['INTERVAL_GFS'] = AppConfig.get_gfs_interval(base_in['gfs_cyc'])
         base_out['RUN'] = 'gefs'
 
         return base_out