Update the ConMon to use GSI-monitor shared resources

[EdwardSafford-NOAA]

Update the ConMon to use the shared DA monitor resources. Remove support for deprecated systems, add support for orion.

Note that, unlike the other DA monitors, the ConMon is not in the global workflow. Data extraction jobs can be started by the ConMon/data_xtrct/ush/ConMon_DE.sh script any time after the cnvstat file is generated.

Also note that the Transfer.sh script only works on wcoss2.

[DavidHuber-NOAA]

The code changes here look fine. However, on a test run, I am seeing a large number of errors being reported from src/Conventional_Monitor/nwprod/conmon_shared/ush/diag2grad_uv_case.sh at lines 116 and 122. I'm guessing the intent is that these lines should be, e.g.

grads_list=`ls *grads*uv*.${run}`

In order to capture the just created files.

There are similar errors at
/data/users/dhuber/gsi-monitor/src/Conventional_Monitor/nwprod/conmon_shared/ush/diag2grad_q_case.sh: 98, 104
and
/data/users/dhuber/gsi-monitor/src/Conventional_Monitor/nwprod/conmon_shared/ush/diag2grad_q_case.sh: 96, 102

If these aren't appropriate for this issue and/or are non-critical, I can open another issue to track them.

[EdwardSafford-NOAA]

Hmmm. That's interesting and a bit distressing. I'll re-run and see if I can reproduce those errors.

[EdwardSafford-NOAA]

I think I have it figured out. In my test I was able to reproduce the failures to build the grads_list and scater_list variables from the ls command for specific data types/subtypes. For type q for example, I got errors when trying to find types q130, q135, and q182. When I examined the NetCDF diag file with ncdump I found no Observation_Type values for 130, 135, and 182. So the issue appears to be no data for those types. I checked the uv diag file and found a similar match between reported errors and no data.

So it looks like this isn't really a problem, per se, but warrants additional investigation. The ConMon still has embedded lists of types and subtypes in the scripts. It's possible those lists are out of date, and anyway should be replaced with an external file for better clarity and ease of maintenance.

If you agree I'll open two new issues. One will be for the scripts to handle this situation more elegantly, report the situation more clearly, and abort processing the missing type/subtypes sooner in the process flow. The second issue will be to investigate what data types/subtypes are actually expected and which are no longer available, and then create an external type file like the other monitors use.

[DavidHuber-NOAA]

@EdwardSafford-NOAA Glad to know everything still works. I think opening a couple of new issues is a good approach. I'll run a test on the plotting routines then wrap up my review.

[DavidHuber-NOAA]

I received errors from GrADS that Open Error: Can't open Station/Index map file tupair.map when plotting plothorz. I'm not sure if this is a GrADS version issue or a file that should be linked into the run directory. Similarly, during plothorz_uv execution, it is looking for uvsfc11.map.

I also received several sets of the following during plothorz.sh

SET DFILE error:  file 4 not open
Data Request Error:  File number out of range
  Variable = obg.4
Operation Error:  Error from maskout function
Operation Error:  Error from maskout function
  Error ocurred at column 1
DISPLAY error:  Invalid expression

And very similar errors from plothorz_uv.sh.

[EdwardSafford-NOAA]

The plot issues appear to be connected to missing data. I've worked through plotting for data class q, and when I add a check on for the existence of the necessary data files before firing the GrADS plot scripts, the maskout and missing map file errors are no more. The map files are created when the required data is available.

All data types that use GrADS plot script plot_uvsfc_horz.gz generate a raft of errors probably all related to

Processing time step 3
Low Level I/O Error: Read error on data file
Error reading 4 bytes at location 88392
Possible cause: Premature EOF
Possible cause: Invalid sequential data

The affected data types/subtypes are uv229_00, uv280_00, uv281_00, uv282_00, uv284_00, uv287_00. I believe the quickest fix for this is to temporarily disable plots for these data types and open an issue to figure out what's up with plot_uvsfc_horz.gz.

There is one other problem related to the analysis and guess files format change from wgrib to wgrib2. This will require change(s) to one or more GrADS plot scripts. I'll work through the remaining data types individually, confirm these changes eliminate the plot errors, and describe the changes fully when I add them to this PR.

[EdwardSafford-NOAA]

New issues have been opened in connection with problems encountered in this PR. They are:

#60 to investigate missing data and move the expected class/type/subtype information out of the plot scripts and into a fix file.
#61 to fix a warning of insufficient room within certain plots for the colorbar
#62 fix the plot_uvsfc_horz.gs script, which fails for all data cases
#63 investigate warnings conditionally generated by plotstas_time_count_ps.gs script.
#64 investigate warnings for uv235 in time series plots.

[EdwardSafford-NOAA]

Note that plotting still produces warnings but no longer (in my tests) produces any fatal errors. Expected GrADS warnings include:

• gaaxis internal logic check 24 -- to be addressed in Investigate ConMon plot_time_ps.sh and plot_time.sh warnings #63
• gaaxis internal logic check 25 -- to be addressed in Investigate ConMon plot_time_ps.sh and plot_time.sh warnings #63
• Cannot plot data - all undefined values -- to be addressed in Investigate ConMon plot issue with type uv235 #64

[DavidHuber-NOAA]

Sorry this fell off my radar with AMS. I've run a test case and it all looks good to me. Thanks!

[EdwardSafford-NOAA]

@DavidHuber-NOAA no worries! I figured the AMS week was going to be a distraction and set my expectations accordingly. Sure glad to have this one done -- it's been quite the slog. And thanks again for your help.