Additional conventional data plots

[EdwardSafford-NOAA]

The time series and vertical plots for conventional data sources have been added to obs-monitor. The legacy ConMon produces two additional plots which are not (yet) created by Obs-Monitor: histograms and horizontal plots. Examples of each are below:

Are these plots routinely of use? Could they be made a sort of second tier plots available for in-depth investigation rather than being regularly generated with each cycle? Or could either/both just be scrapped?

From an implementation standpoint it should be possible to generate the histograms, but it will take a bit of work. The legacy ConMon produces these plots by running executables to bin the extracted data, then the output is fed to GrADS to generate the plot. Within Obs-Monitor it should be possible either transform the data within eva or pre-process it like the ConMon does.

The horizontal plots are more of a challenge. The input for the lower right figure is a trimmed down grid file that the ConMon saves along with the extracted obs data. GrADS can read that file directly (using a generated control file) but eva doesn't have such capabilities, and I'm not eager to make that happen unless that plot is deemed to be particularly valuable.

So I don't intend to start work on any of this but I would like to solicit input from @CoryMartin-NOAA, @kevindougherty-noaa , and particularly @ADCollard on the question of 'Do we need the histogram and horizontal obs plots?'

[CoryMartin-NOAA]

My $0.02 is I think there is benefit in having automated histograms for O-B checks of ob types, but for the maps, I think thinking of them for only "in-depth investigation" is the way to go. And for that, we can manually plot the diag files as a whole for specific cycles, which is already supported by EVA.

[EdwardSafford-NOAA]

I've studied the data handling that goes on in preparation to make legacy ConMon plots and it's, unfortunately, less than straightforward. Using type/subtype q120_00 as an example the key steps are:

• Data file q120_00.scater.ges.yyyymmddhh is extracted and saved with each cycle. It is not used directly to plot though.
• The plot_hist.sh script calls a Fortran executable, conmon_read_q_IG.x in this case (there is one for each conventional data type), which reads the scater file and generates a much smaller binary file named out_q120_00_ges.$PDATE as well as a text file with meta-data named grads_info_q120_00_ges.$PDATE.
• The extraction is repeated for the corresponding anl file.

I'm not concerned about the grads_info files. We can probably generate that meta-data from eva. The big issue in my view is the generation of the out_q120_00_[ges|anl].$PDATE files. I can see 3 options to enable Obs-monitor to make the histogram plots:

1. Incorporate the Fortran programs that create the out_{dtype}_[ges|anl].$PDATE files into Obs-monitor's process flow for as long as we need to support legacy ConMon data. The programs would be run prior to invoking eva.
2. Modify the legacy ConMon to generate the out_{dtype}_[ges|anl].$PDATE files and save those instead of the $dtype_$subtype.scater[ges|anl].yyyymmddhh files.
3. Skip it for now and produce histogram plots only from JEDI data using the future extractors.

Thoughts?

[CoryMartin-NOAA]

Given the timelines of things as they stand, I'm inclined to go with option 3. We keep legacy monitor running for conventional and can switch to the new for radiance when ready, and switch all to new with JEDI.