PPT showing VPOT and STRM at 200, 500, 1000 mb as output by UPP with CFS data and SFS data.
VPOT and STRM Plots.pptx

PPT comparing CFS operational products against GFSv16 model output data processed with UPP.
VPOT and STRM Plots (2).pptx

On 08/06/2024, Wen organized a meeting to discuss generating stream function and velocity potential as SFS products(here). Huiya, Jesse, George and Karina attended the meeting. Based on the runtime issue in Karina’s UPP PR #951, Two approaches were considered for moving forward. Option 1), Karina will continue to optimize code in her UPP PR #951 to reduce runtime; Option 2), Karina will upgrade the program “cfs_genpsiandchi.fd” in the CFS v2.3 package with GRIB2 I/O and transition to the global-workflow repository. The generation of stream function and velocity potential will be configured as part of the atmospheric product generation process in the SFS workflow via the global-workflow. The new process will ingest pgb datasets in 0.5/1.0 degree resolution, generate stream function and velocity fields on isobaric levels, and then append to them the pgb datasets. Krina thought the Option 2 would be feasible for her. George and Jesse are willing to provide resource to support Karina.

Quorating from Huiya's email on 08/16/2024:

Karina, recalled Wen proposed option 2 to compute velocity potential and stream function is to run a stand-alone utility after UPP to generate these two products?
However, EIB has recently rejected this idea, citing maintaining this additional utility
in their workflow is outside of their scope.

Wen just met with Jason and me to discuss this recent development. We would
like to propose option 3, which is to compute these two variables using discretized
numerical analysis. This will allow you to take full advantage of MPI in the UPP
and improve efficiency and reduce memory footprints. The algorithm you had put
into UPP right now does full spectral transform which is more accurate but is
time consuming.

Feel free to schedule a meeting if you want to discuss option 3. Please include George and Jesse who will be the one approving and supporting this new task.

The utility process was rejected by EIB, so we will work on option 3: computing STRM and VPOT using discrete numerical analysis.

@KarinaAsmar-NOAA Thanks for the heads-up.

@WenMeng-NOAA We are still working on option 3, but wanted to check with you regarding runtimes. The numerical analysis requires integration and the respective boundary conditions, so we still need to collect the delta values at all processes, compute the integration numerically at process 0, and then scatter to all processes (similar to how we were doing option 1). I am still working on setting up the numerical integration properly, but checking runtimes with the collect/scatter process:

SFS data with 4 threads: 64 sec
GFS data with 4 threads: ~4 minutes
GFS data with 8 threads: ~5 minutes

Are these runtimes acceptable for implementing into UPP?

@KarinaAsmar-NOAA How many MPI tasks were assigned for processing GFS? Currently the runtime of processing GFS C768 with 96 tasks is within 2 minutes.

@KarinaAsmar-NOAA How many MPI tasks were assigned for processing GFS? Currently the runtime of processing GFS C768 with 96 tasks is within 2 minutes.

@WenMeng-NOAA Do you mean the -n flag for mpiexec? If so, the runtimes above used 48 tasks. Running GFS with 96 tasks and 4 threads takes ~6 minutes

@KarinaAsmar-NOAA I don't think the 6-minute runtime of gfs C768 processing with 96 tasks is acceptable. That means your new algorithm for calculating velocity potential will cost 4 minutes? Currently the MDL2P is 7 seconds. Please work on optimizing MPI code in your new subroutine.

@KarinaAsmar-NOAA For your reference, here is GFS C768 testing information on WCOSS2:

GFS C768: 3 nodes, 120 tasks
total runtime: 147 seconds
runtime of subroutine  MDL2P: 5.5 seconds

You may find runtime information from runtime log of the UPP standalone test.

@KarinaAsmar-NOAA For your reference, here is GFS C768 testing information on WCOSS2:

GFS C768: 3 nodes, 120 tasks
total runtime: 147 seconds
runtime of subroutine  MDL2P: 5.5 seconds

You may find runtime information from runtime log of the UPP standalone test.

Thank you @WenMeng-NOAA . To get the runtime, I am submitting a job to run the UPP standalone test, and then I get the start time and end times of the job from the log. Is this the proper way to get the runtime you need?

@KarinaAsmar-NOAA The sample runtime log for gfs can be found at /u/wen.meng/ptmp/karina/outpost_master_2019083006.

@KarinaAsmar-NOAA The sample runtime log for gfs can be found at /u/wen.meng/ptmp/karina/outpost_master_2019083006.

Thanks! Using the outpost* log of the latest standalone test (more debugging is still needed) the current runtime with 48 tasks is 3 minutes and 50 seconds.

@WenMeng-NOAA We have reduced the runtimes (still using the spectral calculation from CFS) to the following:

GFS 120 tasks

• 3:15 minutes with no STRM/VPOT in control files
• 3:37 minutes with STRM/VPOT at 200mb added to control files
• 14:02 minutes with STRM/VPOT at 26 isobaric levels 
   
  SFS 48 tasks
• 38 sec with no STRM/VPOT in control files
• 35 sec with STRM/VPOT at 200mb added to control files
• 37 sec with STRM/VPOT at 26 isobaric levels

CPC had requested STRM/VPOT only at 200hPa for SFS. Are these 200mb-only runtimes acceptable for implementing into UPP?

@KarinaAsmar-NOAA @JesseMeng-NOAA Can you submit a PR with the new algorithms as a draft so we can review the changes?

@KarinaAsmar-NOAA @JesseMeng-NOAA Can you submit a PR with the new algorithms as a draft so we can review the changes?

@WenMeng-NOAA Yes, we'll submit a PR.

STRM_VPOT_Poisson_Results_v3.pptx
Results with Poisson solver compared to legacy spectral method. Optimization of the process is in progress.

STRM_VPOT_Poisson_Results_v4.pptx
Updated Poisson Solver results with added convergence condition for improved runtimes.