harp-verif

A set of R scripts to carry out point verification using the harp package. These scripts are generally just wrappers in R to harp functions and are designed to carry out the full RRJVSP (Read forecasts, Read obs, Join, Verify, Save, and Plot) workflow.

The scripts require vfld and vobs data (e.g. from a HARMONIE-AROME experiment), or the equivalent forecast and observation sqlite tables, as input, and can perform the following tasks:

• Conversion of vfld/vobs data to sqlite format.
• Point verification of deterministic or ensemble forecasts over multiple groups, producing verification rds files which can be visualised using harpVis::shiny_plot_point_verif.
• Plotting a set of standard verification scores, producing png files which can be visualised using the local shiny app provided in this repo.
• Scorecard generation.
• Extraction of point data from grib/ICM files.

Please see the documentation for details on how to run the point verification workflow. Note that the scripts largely follow the methodology outlined in the harp 2022 and 2024 training courses, to which users are referred to for background information.

A description of the contents of this repo is given below. Directories which are not relevant to the point verification workflow are marked in italics.

config_files

The majority of verification options (e.g. leadtimes, models, paths etc.) are set via a configuration file. Several examples are included:

• config_det_example.yml: A sample config file for when dealing with determinstic models.
• config_eps_example.yml: A sample config file for using ensemble data.
• config_eps_ctrl_example.yml: A sample config file for comparing individual members from multiple ensembles (e.g. the control member of each).

docs

Documentation on how to use the point verification scripts is available in this tutorial.

pre_processing

Scripts to pre-process the data:

• vobs2sql.R: Convert vobs files to sqlite.
• vfld2sql.R: Convert vfld files to sqlite.
• extract_point_param.R: Extract point data from grib or ICM and save to rds.
• pull_dates_sql.R: Query what dates are available in a given sqlite file.

spatial_verif

Spatial verificaiton against radar data. Currently under development.

verification

The main verification script:

• point_verif.R: Perform point verification for deterministic or ensemble models over various groups (e.g. lead_time, valid_dttm, SID, etc.). By default harp rds files are produced, while a set of png files for standard verification metrics will also be generated if indicated in the config file. Scorecard generation is also available via a flag in the config file.
• set_params.R: Defines scalings, thresholds, and quality control settings for parameters appearing in the vfld/vobs files or sqlite tables (e.g. T2m, Q). See harpIO::show_param_definitions() and harpIO::harp_params for more details, e.g.:

unlist(harp_params)[grep("harp_param",names(unlist(harp_params)))]

• fn_verif_helpers.R: Contains various common functions for pre-processing and verification.
• fn_station_selection.R: Used to define and generate SID lists. These SID lists can either be explicitly defined by users, or can be generated by looking for all stations within a given geographic regions (either a lat/lon bounding box or defined by a polygon file). New regions can also be defined by users. Static SID lists generated from geographic regions are contained in sid_lists.rds, however the use of this file is now deprecated.
• source_options.R: A file specifying the options required by point_verif.R when sourcing the verification script within R/RStudio. Running the scripts in interactive mode is useful for querying data, for example.
• allsynop_sids.list: All SIDs from HARMONIE-AROME's allsynop.list.
• poly_files: A directory to store polygon files used for station selection.

visualization

Plotting and visualization scripts:

• fn_plot_point_verif.R: The main wrapper script for generating pngs during point verification.
• fn_plot_aux_scores.R: Used to generate auxiliary scores which are/were not in harp by default (e.g. forecast timeseries).
• fn_plot_helpers.R: Contains various plotting functions.
• fn_scorecard_signif.R: A wrapper for plotting scorecards and model differences.
• fn_plot_signif_diff.R: Plots the difference between two models and the confidence of that difference (from Andrew Singleton).
• fn_plot_tile_scorecard.R: An alternative plotting tool for scorecard data.
• visapp/app.R: A simple shiny app to display static png images. See the documentation for further instructions.
• visapp/sample_images: Sample images to use for testing app functionality.
• visapp/all_experiment_names.R: A tool for changing the display name of experiments in the app.
• visapp/sample_style.css: Style settings for the app.

Package dependencies

Instructions for harp installation can be found here. Installation can take some time due to the larger number of dependencies. The latest release of the point verification scripts is compatible with harp version 0.2.2. To install a specific release of harp, use:

remotes::install_github("harphub/[email protected]")

Quite often a hotfix may be applied to the develop branch of a specific harp repository (e.g. harpPoint) which is not yet in an official release. In order to get this update, you can use:

remotes::install_github("harphub/harpPoint@develop")

The following packages are used by the point verifiation scripts:

• harp
• here
• argparse
• yaml
• dplyr
• tidyr
• purr
• forcats
• stringr
• RColorBrewer
• gridExtra
• grid
• pracma
• RSQlite
• scales
• pals
• shiny
• shinyWidgets
• lubridate
• cowplot
• scico
• sf (optional, only required for polygon filtering)