Skip to content

Meeting Notes

Jump to bottom
Katie Dagon edited this page Oct 24, 2024 · 104 revisions

UPDATE 10/24/24: PPE meeting notes are now here

Older notes are still preserved below

CLM5 PPE meeting notes

October 3, 2024

Agenda:

  • Linnia will present the latest on the upcoming CLM6 tuning experiments
  • Timeline and CLM6 status updates
  • Parameter selection
  • Retraining the sparsegrid

CLM6-PPE parameter list: https://docs.google.com/spreadsheets/d/1R0AybNR0YAmMDjRqp9oyUffDhKeAWv1QF4yWTHqiXXM/edit?usp=sharing

Notes:

  • Timeline and updates

    • On track getting CLM6 PPE running, working with SEs
    • Adrianna is helping with mesh files and datm subsetting, performance is much slower using full grid datm (vs. sparse grid)
    • Thanks to Sean, now have a script that subsets datm file for GSWP3, working on CRU data now
    • Ideally develop a tool that inputs the sparse gridcells you want and outputs the mesh file, since you would need to repeat this step every time you want to use a different sparse grid
    • Using list of constraints from previous meeting, with OAAT to identify parameters
    • Planning on a new mini-OAAT with CLM6, working towards a new LHC (~1500 members) with CLM6 and a new parameter list using several rounds of history matching for calibration

  • Parameter selection:

    • Thinking beyond LAI; goals include tuning and UQ
    • Easier to start with a larger set of parameters than it is to add later
    • Running with crop model off, Sam will help with crop model tuning separately
    • tpu25ratio: the fact that it has an effect (irrespective of range) is interesting (Rosie), some evidence for taking TPU out of land models (Charlie), only has a parameter effect at elevated CO2 (Daniel), can also ask Danica
    • Acclimation: might not want to treat all params as independent, can we span the overall range and sensitivity of acclimation by designing a specific sampling routine; but there is a question on why we pre-select these relationships - is it actually needed (Ben), Rosie has a spreadsheet that might allow us to play with these params independently, is it safer to treat them independently first vs. trying to understand those relationships later - with some computational tradeoffs (Daniel), benefit to reducing the dimensionality (Linnia)
    • Phenology: need metrics to assess importance of these params, other params we didn't include in CLM5 OAAT?; soilpsi is not linear so be mindful of +/- percentages (Peter), be mindful of sparse grid limitations for these params (Will/Rosie), could try slope of LAI (Adrianna); maybe do a mini OAAT to determine whether phenology should be included in CLM6 calibration
    • FUN parameters are already perturbed together as KCN and could use that as a code example for combining acclimation parameters, be mindful of FUN bug in CLM5.1 - will include KCN in mini OAAT for that reason
    • Hydrology: multiple ways to perturb pedotransfer function parameters, makes sense to perturb them independently; what are the tuning goals for these parameters (Daniel), relates to parameter range definition and spatial dependence of parameter values (Guoqiang), may want to explore how to sample the parameter distributions (Daniel)
    • maximum_leaf_wetted_fraction was found to be important in land-atmosphere interactions (Meg), it is the maximum value fwet can take (Sean)
    • Snow: n_melt_coef is useful and range could be expanded (Guoqiang), SNICAR stuff was useful for Arctic simulations (Will), zsno probably only useful for coupled simulations and was in Claire's ensemble
    • Decomposition: likely some groupings here like respiration fraction litter to SOM and SOM to SOM params and check on ranges (Will), check q10_mr variable name in code (Charlie), watch for making fractions too low (Charlie), check on new CLM6 coarse woody debris parameter rf_cwd (Will), minpsi_hr and maxpsi_hr could be influential (Charlie), think about running bundled params separately for OAAT, consider independence of respiration fraction params (Katie R.)

August 8, 2024

Linnia & Daniel

Discussion of experimental design for tuning CLM6

Objective 1: Tuning LAI in CLM6 for CESM3

Objective 2: Carbon cycle uncertainty

May 16, 2024

Joint with CLM Meeting [10 people in room, 19 on call]

Announcements

Agenda

  • Kachinga Silwimba (PhD student at Boise State Univ., ASP GVP visitor through end of May) presents his work on "History Matching with Gaussian Processes and Evidential Deep Neural Networks to Improve Total Water Storage Simulations in the Community Land Model"

Notes

CLM-PPE analysis for hydrological applications

  • Tuning CLM Total Water Storage (TWS) to GRACE satellite anomalies

First objectives:

  • Develop methodology for emulating timeseries and seasonality
  • Test and compare multiple ML methods for emulating CLM
  • Perform sensitivity tests (Fourier amplitude & Sobol sensitivity tests)
    • Using the emulator to perform the sensitivity tests.

Longer term objective: Once an emulator is built and evaluated, the next step will be to perform history matching (in progress)

  • GRACE TWS will be used as observational target data set.

Step 1: Augment the dataset to emulate timeseries (specifically seasonality) of TWS

  • PPE dataset was augmented to incorporate time
  • Training data:
    • X: input array (500 parameters * 100 years * 12 month = 600,000 )
      • include month and year as predictors
      • use cyclic encoding of month (to ensure December and January are "close together" rather than 12 and 1)
    • y: monthly TWS (1901-2000)
  • Validation data: (2001-2014 montly TWS)

Step 2: Compare three different tools for emulation, Gaussian process, Depp Neural Network, and Evidential Neural Network

  • Gaussian Processes: -Gaussian process emulators do not scale well with larger datasets (impossible to train on 600k training data) -Trained GP to emulate the annual mean TWS. -Fourier Amplitude Sensitivity test (performed with GP) showed the most influential parameters were related to ET and soil moisture (fff, d_max) as expected.

DNN

  • Performs well and shows high skill for predicting seasonality (2001-2014)
  • Performed Fourier amplitude sensitivity test with DNN (same parameters are important as with GP)

Evidential DNN : "best of both GP and DNN" scales well with large training dataset and provides an estimate of uncertainty in predictions.

  • validation looks good
  • partitions aleatoric and epistemic uncertainty
  • Epistemic is larger (as expected)
  • Sensitivity tests with EDNN show similar results to DNN and GP

Summary: Sobol and FAST sensitivity analysis show similar results (regardless of emulator used)

  • indicates robustness of sensitivity tests

Emulation of timeseries (seasonality) is working well with DNN and Evidential DNN

Evidential Deep Neural Network is a promising tool for PPE emulation

  • scales well with large datasets
  • provides uncertainty estimates in predictions (critical for history matching)
  • partitions uncertainty into aleatoric and epistemic

Next steps:

  • Emulate the CLM-PPE TWS anomalies and evaluate them with GRACE
  • Incorporate space (in addition to time) to emulation.
  • Perform history matching.

Discussion:

  • Can you apply the SOBOL test to EDNN uncertainty?
    • maybe, something to try
  • Why doesn't GP work for emulating timeseries?
    • dataset is too large (compute scales cubically with the input data size)
  • Compare the uncertainty estimates from GP to EDNN
    • GP uncertainty is sensitive to choice of kernel but a comparison would be useful.
  • Evidential DNN captures expected epistemic uncertainty relative to magnitude of aleatoric
    • Epistemic is a combination of uncertainty in hyperparameter tuning and sparse PPE sample
    • Epistemic could be reduced by adding more PPE ensemble members
  • Quantile transform may be useful when incorporating space.

April 11, 2024

[4 people in room, 16 on call]

Announcements

Agenda

Notes

  • Motivation: is the sparse grid still serving us?
  • Also motivated by Adrianna's FATES work
  • Large computational savings from sparse grid (14x faster than 2 degree)
  • Why 400? Based on variable representativeness (relative to full grid) and what we could afford
    • Use ILAMB plots to look at bias and tradeoffs with # gridcells
  • Limitations
  • Current sparsegrid is fine for gridcell output, but now we are interested in PFT-level output
    • Can we construct a similarity matrix at the patch level? Each PFT has its own similarity matrix?
    • Would potentially increase # gridcells to run at each simulation
  • Consider validation data as well, consider matching with obs
    • Building an emulator could use idealized setup, then switch to more realistic setup for validation
  • Alternatives
    1. Use current sparse grid
    2. Re-run clustering for PFT information
    3. Select gridcells for dominant PFTs
    4. Start with dominant and add co-dominant cells
    5. Full grid
    6. Idealized surface datasets
  • Where do we put our uncertainty?
    • Error in sparse grid propagates to calibration
    • Error in PFT representation
  • PFT interactions within gridcells for cluster analysis with PFT information - how do we account for this?
    • Relationship between PFTs within a column
    • Subset by common combinations of PFTs? Use those to seed the clustering
    • Linnia: 1 PFT tends to co-exist with 1 or 2 other PFTs at most, so it seems like this is reasonable
    • Daniel: run clustering on h1 data (patches), then you are selecting patches to run vs. selecting gridcells
  • Adrianna: proportional area vs. dominance thresholds plots
    • How can we pick different grids that map up to different PFTs?
    • Defining dominance ecologically
    • Defining co-dominance, secondary grids after running some initial simulations
  • PFT interactions - might need additional simulations to diagnose these
  • Iterative process of running clustering algorithm

March 14, 2024

[28 people on call]

Announcements

Agenda

  • Linnia presents updates on the CLM-PPE and discusses some of the implications of our experimental design.

Notes

  • Wave 1 of history matching: focused on PFT x biome spatial aggregation, vary PFT parameters independently
    • Some PFTs are harder to emulate / have greater emulator uncertainty
  • Wave 0 (original LHC PPE) scaled PFT parameters uniformly
    • Implausibility score rules out parts of the parameter space
    • Ignoring structural uncertainty (for now)
  • Sample from plausible sets: how best to do this?
    • Goal is to reduce emulator uncertainty
    • Use close to latin hypercube to select 100 members to re-run in CLM (Wave 1)
  • Wave 1 results
    • Observational uncertainty varies across biomes
    • Emulator uncertainty is dominating
    • Wave 1 from CLM falls generally within uncertainty band (obs + emulator uncertainty)
    • Vary PFT parameters independently works ok
  • Issues to address
    • Uncertainty: observational, forcing, emulator
    • Sparse grid: PFT interactions
    • Metrics: trend, IAV, seasonality
  • Emulator uncertainty
    • Adding 100 ensemble members is not reducing emulator uncertainty - why?
    • Wave 1 allowed PFT parameters to vary independently, which matters when PFTs share gridcells
    • Needleleaf dominated gridcell: adding wave0 and wave1 ensemble members decreases variance
    • Needleleaf/broadleaf split gridcell: adding wave1 ensembles members does NOT decrease variance
    • Test adding NL LAI as a predictor: adding wave1 ensemble members decreases variance (Ben: how sensitive is the performance to adding LAI from which wave?)
    • Rosie: should we just focus on dominated gridcells?
  • Rethinking sparse grid
    • Option 1: Select gridcells with one PFT dominating (FATES approach)
    • Option 2: Select new sparse grid with PFT as a factor
      • Sparse grid redesign likely important to sampling enough PFT gridcells
      • Adding targets that may not be aggregated by PFT (e.g., GPP) - do the data exist somewhere? Gordon: GPP product has an underlying landcover classification, could calculate FLUXCOM GPP using CLM land surface dataset. Ben: useful variance information
  • Andy: Try parameter selection sequentially based on PFTs in the gridcell? Conditional optimization in a sequence. Hydrology analogy would be calibrating snow first, fixing it, then calibrating soil parameters. (Adrianna is taking this approach. Learn more in dominant gridcells and then introduce a new grid partway through.) Linnia: Small parts of grass everywhere is challenging, keep it fixed?
  • Adrianna: biome land area vs. "dominance threshold" for diagnosing spatial importance of a particular PFT. Each PFT would get its own set of gridcells (think about how many), consider observational uncertainty. [Great future discussion topic for this group!]
  • Ben: spatial element to the calibration cascade, bringing in information in the data as well
  • Aleya: what is a PFT interaction? All below ground interactions via moisture/nutrients because all PFTs in a gridcell share the same soil column
  • Linnia: started looking at trend in LAI, Daniel has shown that LAI trend is an important constraint
  • Andy: resampling from emulator results, different approaches to interpreting implausibility score
    • Early waves may not want to focus the resampling too narrow (to reduce emulator uncertainty), but later waves could consider this. Also consider how observational uncertainty maps onto interpreting implausibility. Draw on hydrology community experience.
  • How do wave0 training members impact emulators? Add more to wave1? Throw out the worst of wave0?

February 15, 2024

Announcements

  • We have a google group / email list! Feel free to add yourself and share with others.
  • CESM Land Model Working Group meeting is February 27-29, register here.
    • Lots of PPE talks!
  • Linnia is running a PPE tag on Derecho! Thanks to Erik, Keith, Sam L., Daniel for making this happen.
  • Thinking about LAI calibration at the PFT-biome level, working towards calibration for CLM6.
  • Adrianna is working on FATES calibration cascade.

Agenda

  • Khachik Sargsyan (DOE Sandia) and Daniel Ricciuto (DOE Oak Ridge) present on "Reduced-Dimensional Neural Network Surrogate Construction and Calibration of the E3SM Land Model." Slides here.

Notes

  • Forward UQ: global sensitivity analysis & inverse UQ: model calibration
  • Reduced dimensional surrogate construction
  • Using satellite phenology version of ELM, will work with BGC model next
  • 2 degree resolution, 275 members, 10 parameters
  • Karhunen-Loeve expansion: SVD type dimensionality reduction but centralized, continuous form
    • linear encoder of the output
    • solving for eigen-features
    • uncertain parameters and "certain" conditions
    • working with latent space
    • they have tried autoencoders but not worth the effort, linear encoding is acceptable here
  • neural network based surrogate, they do compare with GP but prefer combining outputs
  • Evaluating at 96 FLUXNET sites, 180 months
    • using different time averaging: monthly, monthly climo, seasonal climo, annual
    • issues with memory and high dimensionality
  • Retaining 8 eigenvalues (FLUXNET sites) and 11 (global) retaining most of the variance
  • Recommend residual NN (ResNet)
  • fLNR most sensitive parameter, mbbopt second most (Ball-Berry stomatal conductance slope)
  • Reference data is FLUXCOM GPP
  • Posterior sampling via MCMC
  • Likelihood in reduced space: project observed data to KL eigenspace to calibrate
  • Also looked at local (site-specific) parameter posterior PDFs - how to interpret?
    • Correlate PFT fractions with best fLNR values

Questions

  • Linear approximation w/ PFT-output? Using aggregated output for now as a "compression".
  • Issues with GPP calibration making another variable worse? Could add latent heat flux to the methodology via vectorization.
  • Informing structural issues? Embedded model error approach - augment Bayesian likelihood for model inadequacy. Internal approaches to add statistical representations - some preliminary work.
  • Performance at different sites? Surrogate performance does vary based on site, haven't looked at calibration yet.
  • Losing information from PCA - is KL method different? PCA does not centralize the data. Formally this is not PCA, it is more like SVD.
  • Choice of eigenvectors based on default? A different eigenbasis would impact this workflow.
  • Model is forced with reanalysis but also looking at FLUXNET sites? FLUXCOM is the target.

Additional Questions (follow-up)

  • Comparing NN vs. GP - with CLM we have found separate emulators work well.
    • We have compared with PC (polynomial chaos - see some of the additional slides)
    • GP like PC would have to be built separately for each latent feature. Putting all outputs together might be a bit awkward, particularly if each GP is tuned to its own best hyperparameter setting (and tuning I feel might be required for high-d input space, e.g. 10 parameters).
    • advantage of GP or PC is that the have a natural way of quantifying the emulator error, which NN does not have readily.
  • Details on ResNet?
  • Quantifying the NN emulator error?
    • not really, but there is a whole industry on quantifying NN emulator errors with many methods like Laplace approximation, MC-dropout, variational inference etc... None of them are ideal or easy to train/interpret though.
  • Sensitivity metric?
    • Sobol sensitivity indices that capture variance fraction due to a given parameter.
  • How is the model prior defined?
    • Nothing tricky so far, just uniform prior for each parameter in a range driven by literature/expert knowledge.
  • In all this, the elephant in the room is the model error - there is no ideal way to tackle it of course, but the embedded approach (https://www.dl.begellhouse.com/journals/52034eb04b657aea,5a3895a14afb242f,1d2810e66490c327.html) is what we have been trying with some success.

October 26, 2023

Announcements

Agenda

  • Hossein Kaviani (UVA) presents "Quantifying parameter uncertainty in environmental models" --PhD Student at UVA doing an internship with Katie Dagon

Notes

Bayesian inference: incorporate prior information into our predictions

  • Can be used for estimating posterior and constraining range of parameter values MCMC: Markov chain Monte Carlo
  • Markov chain is an explorer, following a gradient
  • Needs a 'burn-in' period (adjustable parameter)
  • Many methods use multiple chains

Developed new diagnostic assessment of MCMC algorithms

  • MCMC needs to be effective (converge to true posterior)
  • efficient (in time)
  • reliable (consistent across random seeds)
  • controllable (insensitive to hyper-parameters)

Goal: to come up with diagnostics that ensure an algorithm is insensitive to "unimportant" hyper parameters

  • Comparing three existing MCMC methods: -- DREAM, metropolis-hastings (MH) and Adaptive Metropolis (AM)

  • Test problems: 10D bimodal mixed gaussian distribution

  • Hyper parameters of interest: number of chains & number of function evaluations

  • KL divergence tells us how close the estimated posterior is to our true posterior (true posterior is known)

  • Diagnostics developed can a priori tell a researcher what algorithms are appropriate for different problems (due to their insensitivity to choice of hyper parameters).

Research question 2: Integrating multi-objective, Real-Time control into storm water management

  • implement BORG algorithm in SWM moswl (storm water management)
  • Objectives: minimizing floods (overflows) & minimizing sum of squared errors between simulated flows and pre-development flows
  • Many of the target objectives, in practice have conditional constraints, BORG can capture the objective goals with constraints.

Internship at NCAR:

  • Coupling Bayesian inference with emulators of CLM
  • Tuning 32 CLM parameters
  • Develop likelihood function to capture a probabilistic space between observations and model simulations
  • Using DREAM algorithm

September 27, 2023

Announcements

  • The Parameter Estimation Interest Group will be having its first meeting next week, Wednesday October 4 at 9am MT. You can join the email list and slack channel to stay up to date on group activities.

Agenda

  • Nina Raoult (Univ. Exeter) presents: "Parameter perturbation experiments in land surface modelling"

Notes

  • Working with JULES and ORCHIDEE
  • Parameter sensitivity, history matching (mouse w/ cheese!), and emergent constraints
  • Parameter sensitivity: Morris and Sobol
    • Morris to find most sensitive
    • Sobol to capture interactions; needs many more simulations which can be computationally challenging
    • sensitivity at different timesteps
  • Bayesian framework: optimized parameters
  • History matching: ruling out unlikely parameters using implausibility function
    • Sampling for history matching framework? Latin hypercube for initial sampling and emulators (LHC for dense sampling too)
    • Advantage to tune against multiple metrics (vs. traditional Bayesian)
    • Can discover parameters with double minimums -- don't change cut off too soon (i.e., how to define implausibility)
    • Comparing to gradient based descent: can get stuck (many local mins)
    • Generate ensembles from posterior distributions to test the model
  • Parameter spread in future projections (e.g., Booth et al. 2012)
    • Use emergent constraints to combine parameter constraint (via calibration) with future relationship

Discussion

  • Emergent constraint method: using posterior distribution of one parameter (Topt), combine with linear regression from future projection experiments
  • Ecological plausibility of optimized parameters: constraining the prior distributions; parameters are making up for compensating errors, how many parameter do you want/need?
  • Improving structural biases: when parameter optimization can't get close to obs, likely pointing to a structural process (e.g., missing process)
  • History matching vs. behavioral/non-behavioral terminology like GLUE (Beven) in hydrology
  • Standard "cutoff" for history matching is 3 based on standard deviations, can think about how to adjust this during process
  • Assessing emulator performance during waves
  • Emergent constraints: need to understand the physical mechanisms behind them, otherwise can be noise
  • Computational constraints of Sobol: approach it as a chain of methods with Morris first, limit the number of parameters; use ensembles, emulators with or without history matching; in reality really only using Sobol on site examples due to computational limitations
  • What are the limitations of gradient descent? Not really giving us uncertainty

August 31, 2023

Announcements

  • Introduce NCAR visitor Hossein Kaviani (UVA)
  • The ILMF is organizing a series of webinars including one on parameter estimation on October 10, 9-11am MT. Organized by Rosie, Daniel, and Nina Raoult. Linnia will be speaking about the CLM PPE work! Register here.
  • Next CLM PPE meeting will be at a different day/time: Wednesday September 27th at 10am MT, Nina Raoult (Univ. Exeter) will be presenting.

Agenda

  • Daniel: OAAT manuscript update and SSP extension runs
  • Linnia: Updates on LAI calibration

Link to PFT parameter sheet: https://docs.google.com/spreadsheets/d/1ZIM5ZT6DLdWm9s2uLXqd9YUbYME4Qw6p65S92gG6r4o/edit?usp=drive_link

Notes

  • Will/Rosie: there is a FUN parameter name switch, we can test out how that impacts simulations with the sparse grid. It is baked into all the PPE simulations.
  • Daniel: OAAT manuscript updates
    • CTSM/CLM naming convention for publications, a bit open ended at this point. How to best track the literature / be consistent?
    • Data available on request
    • Hosting the CLM PPE output on /glade/campaign/cgd/tss (~3 TB), plan to create a DOI/link e.g., through Climate Data Gateway; backup smaller dataset ready for OAAT publication and analysis.
    • Comments welcome on the OAAT manuscript, email Daniel.
  • Daniel: SSP3-7 extensions update
    • Runs to 2100 are complete
    • Huge spread in land carbon sink
    • Can compare to CESM LE, TRENDY, ISIMIP, CMIP
    • Rosie: Color-code the lines in the spread plot with dominant mode of variability in carbon loss?
    • Will/Charlie: ILAMB scores to weight likelihood of various parameter sets
    • Andy: Behavioral vs. non-behavioral outcomes, matching physically realistic metrics
    • Sanjiv: What is the largest hammer in the system? Should we have a poll to guess? Likely leafcn, slatop, jmax (continent on the LAI-selected parameters which is a subset of all CLM parameters)
  • Linnia: LAI calibration slides
    • Lessons learned: compound cost functions are hard to emulate, design emulation for physical relationships, use ensemble of emulators
    • Calibrating PFT-specific parameters independently without running a huge number of simulations
    • Universal vs. PFT-varying parameters
    • Need help defining relationships for these parameters
    • Gaussian process emulators for each PFT, PFT-level calibration target is CLM-SP, spatial PFT mean of annual max LAI
    • Deciduous emulators aren't as good, overall performance is very good
      • Charlie: Deciduous phenology parameters may be partially represented in PPE (e.g., crit_dayl)
    • Hierarchy of sampling: 10^6 samples per PFT, univariate sampling - could we do this better?
    • Plausibility criteria: universal sets with at least one PFT set within observational tolerance of CLM-SP target, very narrow to start (+/- 1%), 15% of original sample is plausible, then passing sets for PFT (still hundreds), then randomly sample 25 for running CLM
    • Within testing, can improve many PFT biases in LAI; broadleaf deciduous temperate tree is struggling due to emulator uncertainty
    • Still neglecting interactions between PFT parameters
    • GP emulator uncertainty seems robust
    • Take-away: need to take plant trait relationships into account
    • Guoqiang: they have used an optimization workflow that updates the emulator (iterative)
    • Sanjiv: emulating with meteorological information, will work on this for next steps (see Baker et al. 2022)
    • Charlie: to tackle within PFT relationships could use something in the cost function, or take into account trait relationships in the initial ensemble design (i.e., don't vary the parameters independently). We can try both.
    • Ben: what's the lowest dimensionality to represent forcing data?
    • Followup: set up a call to brainstorm trait relationships, see spreadsheet

July 6, 2023

Announcements

  • CESM Workshop and Parameter Estimation cross-working group session was a success!
    • Presentation slides are here
    • Follow-up parameter estimation interest group in development - look out for an email survey
  • PPE session at AGU: abstract deadline is Aug 2

Agenda

  • Guoqiang Tang (NCAR) presents "Improving the hydrological performance of CTSM through parameter optimization and large-sample watershed modeling”
  • Slides here

NOTES

  • Hydrology calibration toolboxes (e.g., OSTRICH) have been implemented for CTSM
  • Multiple optimization algorithms can be implemented (e.g., DDS, MO-SMO)
  • CAMELS dataset used as calibration target (expanding to CARAVAN).
  • Configuration for NCAR HPC system has required infrastructure development.

MO-ASMO example:

  • initially need 15-20X parameters simulations,
  • trains a surrogate (random forest)
  • finds new parameter sets (pareto front, NSGA-II multiobjective optimization)
  • Runs the forward model with sample
  • repeat until criteria is met (14 iterations used in example)

May 11, 2023

Announcements

  • CESM Workshop and Parameter Estimation cross-working group session
  • Analysis of PPEs in Atmospheric Research (APPEAR) Virtual Seminar Series
  • Linnia ran a new mini-ensemble - nice work!
  • Katie and Daniel received funding from NCAR to work on PPE this summer
  • Volunteers/suggestions for June 8 CLM PPE meeting?
    • Guoqiang can't make June 8 but would be interested in presenting at a later date
  • Ben: interest in CMIP activity regarding PPEs? Email Ben for more info / to get involved

Agenda

  • discuss the CLM5 OAAT paper
    • figures are coming along really well
  • preview the OAAT diagnostics page
    • https://webext.cgd.ucar.edu/I2000/PPEn11_OAAT/
    • interested in feedback!
      • list by parameter?
      • links to parameter spreadsheet and CLM variable names
    • should follow-up with CISL on cloud deployment
    • also some potential for UCSB data science students to contribute here
  • discuss next steps for CLM5-PPE
    • software and datasets
      • tutorial once we have a calibration example?
      • storing parameter information in parameter files (e.g., ranges)
    • follow-on experiments
      • SSP extensions: 2 scenarios?
      • DART CAM reanalysis product
      • analyzing CLM PPE SP simulations

March 16, 2023

Agenda

  • Short status update on the various CLM-PPE projects
    • one-at-a-time paper 'in prep'
      • can share figures, or can wait and share full draft
      • will be seeking input on diagnostics suite to put online
    • conversation started with CISL re: interactive visualization hosting
    • two follow-on experiments in the pipeline
      • extending current LHC to 2100
      • running default params for 80-member DART ensemble (Raeder et al. 2021)
    • proposing a parameter estimation cross-working group session for CESM workshop
  • Exploring parametric dependence of climate feedbacks using a Perturbed Parameter Ensemble (PPE)
    • Saloua Peatier from CERFACS
    • Results from an atmospheric PPE, examining equilibrium climate sensitivity (ECS)

NOTES

Exploring the influence of parameters on equilibrium climate sensitivity in the CNRM-CM5-1 model.

Experimental Design:

  • Examine parameters related to cloud micropohysics, convection, and cloud radiation (ice cloud microphysics were most influential)
  • LHC sampling - 30 parameters - 102 ensemble members
  • 2 experiments (control and future) - 3 year simulations

Performance metrics:

  • LW,SW, T, P annual mean combined into an error metric
    • Used CFMIP as uncertainty range.
  • Used EOF analysis to reduce spatial dimensionality and reconstruct RMSE
  • Climate feedback parameter
  • CNRM-CM6-1 LHC ensemble spread in climate feedback parameter was larger than multimodal ensemble (CFMIP), but ECS was shifted higher.

Emulation & Calibration:

  • Multi-linear regression used as emulator to predict error and climate feedback parameter.
  • Used emulator and gradient reduction optimization to identify “candidate” parameterizations of the model
    • Minimize the Total Error metric, for evenly spaced values of the climate feedback parameter.
  • Generated ~15 “candidate” parameterizations
  • Ran the atmospheric model with “candidate” parameterizations

Results:

  • Most candidate parameterizations were within the error range for error but had a wide spread in the climate feedback parameter lambda.
  • None of the parameter sets in the Latin-hypercube ensemble, or the calibrated candidates, were able to beat the default hand tunded model.
  • Most “candidates” improved temperature but did not improve precipitation.
  • Candidates with high ECS were often way out of top of atmosphere energy balance.

Takeaways:

  • Model parameterizations that were within the plausible uncertainty range of observational error simulated a wide range of equilibrium climate sensitivities.
  • Model developers are remarkably skilled at hand tuning!

February 16, 2023

Agenda

  • (Re)Introduction to this group and the purpose of these meetings
    • Share updates on this project
    • Hear from others on PPE-related projects
  • LMWG recordings are online
  • Land DA Town Hall on Machine Learning for Land Data Assimilation on Feb 22
  • Brief simulation/analysis update
    • OAAT (one-at-a-time) paper in prep
    • LHC (latin hypercube) analysis underway
  • Linnia Hawkins presents on LAI calibration approaches for the LHC ensemble. Slides are available here.

Notes

  • Establish an ML-methodology for LAI calibration
  • Hope to improve fluxes through vegetation structure
  • ML-based emulators to interpolate beyond PPE simulations
  • Thinking about what metrics to use (e.g., global mean, multiple objectives)
    • How to incorporate spatial variability
  • GP emulators have training considerations (e.g., covariance structure, hyperparameter tuning), but you also get measure of uncertainty
  • Choosing metrics and tools for hyperparameter tuning
    • How sensitive is emulator response to hyperparameters?
    • Using covariance structure in hyperparameter to sample PFT-dependent parameters
  • ILAMB has a new scoring methodology that tries to keep weighting away from the tropics
  • Accounting for structural limitations
  • Combining different metrics for multi-objective calibration
    • Equifinality challenges
  • How to define uncertainty in observations?
    • Use obs uncertainty to weigh targets
  • Stability of MCMC sampler chain is something to look at
  • Pareto sampling for multi-objective optimization
  • How to define parameter ranges or re-define based on optimization results
    • Coordination among traits in plants, can we use that to define covariance
  • Biases in atmospheric forcing data
    • Use precip and temp as predictors in emulation
  • This workflow has a lot of variation and possibility!

October 27, 2022

Agenda

  • Yifan Cheng presents on Arctic hydrology parameter optimization

Notes

  • Funded via NSF Navigating the New Arctic Project
    • in collaboration with indigenous communities
    • trying to move CESM towards more actionable science
  • Developed an optimization workflow for CTSM
    • utilizing ASMO
    • medlynintercept
      • high values yield cold/misty summer in Alaska
      • constrained to [1,2e4] instead of [1,2e5]
    • improved general flow simulation in 13 of 15 basins
      • better high-flow (flood) predictions
      • degraded low-flow predictions
    • snow performance is similar
      • met forcing more important than params
  • Regional Arctic Systems Model
    • running the optimized parameter set with coupled CTSM-WRF (mizuRoute)
    • manual WRF namelist perturbations
    • comprehensive terrestrial hydrology and hydrometeorology assessment
    • streamflow not degraded, actually improved in >half the basins
    • importantly, utilized a previous WRF simulation for downscaling
      • minimized the shock of offline-> coupled

September 29, 2022

Agenda

  • Land surface modeling summit report back
  • CLM5 PPE status update (OAAT/LHC)
  • Oct 27th meeting topic
  • CESM coupled PPE update
    • Claire Zarakas and Abby Swann

Notes

  • Land surface modeling summit
    • Sept 12-15th, Oxford, UK
    • presentations available online
    • Join the AIMES land data assimilation working group
    • From Dave:
      • Dave and Eleanor Blythe will be organizing an international land modeling forum, which will be open / of interest to many involved here
      • Likely to contain working group on parameter estimation
    • From Rosie:
      • Tristan Quaife will be presenting at the FATES modeling call on Nov 10th
      • Kristoffer Aalstad gave a great talk recently on snow data assimilation
        • good candidate for an external presenter, here or at CLM meeting
  • CLM5 PPE status update (OAAT/LHC)
    • Writing commences for OAAT, target: JAMES
    • Rerunning LHC
      • backup hiccup
      • fix GSWP3 bug
      • add krmax
    • Will save all restarts and spinup
  • CESM coupled PPE update
    • Claire Zarakas ([email protected])
    • 18 parameters varied one-at-a-time
    • concentration driven, CAM6-CLM5-slab ocean
    • many interesting results!
      • large effects
      • zeta_max_stable not very important in coupled model
      • medlynslope often confusing
      • coupling affects carbon / water cycle means differently
      • coupling can erode and/or scramble IAV signals from the offline model
    • would have liked double the time to discuss
      • Claire will hopefully present again in the future
Clone this wiki locally