Skip to content

Commit

Permalink
added results to GSA docs
Browse files Browse the repository at this point in the history
  • Loading branch information
odunbar committed Jul 31, 2024
1 parent 228c98f commit aa02c0c
Show file tree
Hide file tree
Showing 7 changed files with 46 additions and 6 deletions.
2 changes: 1 addition & 1 deletion docs/make.jl
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ examples = [
"Emulator testing" => [
"examples/emulators/regression_2d_2d.md",
"examples/emulators/lorenz_integrator_3d_3d.md",
"examples/emulators/ishigami_3d_1d.md",
"examples/emulators/global_sens_analysis.md",
],
]

Expand Down
Binary file added docs/src/assets/GFunction_sens_RF-scalar_10.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/src/assets/GFunction_sens_RF-scalar_3.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/src/assets/GFunction_sens_RF-scalar_6.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/src/assets/GFunction_slices_RF-scalar_6.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Original file line number Diff line number Diff line change
@@ -1,13 +1,25 @@
# Global Sensitiviy Analysis for an emulated Ishigami function
# Global Sensitiviy Analysis (GSA) test functions

!!! info "How do I run this code?"
The full code is found in the [`examples/Emulator/`](https://github.com/CliMA/CalibrateEmulateSample.jl/tree/main/examples/Emulator) directory of the github repository

In this example, we assess directly the performance of our machine learning emulators. The task is to learn a function for use in a [global sensitivity analysis](https://en.wikipedia.org/wiki/Variance-based_sensitivity_analysis). In particular, we learn the Ishigami function
In this example, we assess directly the performance of our machine learning emulators. The task is to learn a function for use in a [global sensitivity analysis](https://en.wikipedia.org/wiki/Variance-based_sensitivity_analysis). In particular, we have two cases:

- The Ishigami function
```math
f(x; a, b) = (1 + bx_3^4)\sin(x_1) + a \sin(x_2), \forall x\in [-\pi,\pi]^3
```
with ``a=7, b=0.1``. In this example, global sensitivity analysis refers to calculation of two Sobol indices. The first index collects proportions ``V_i`` (a.k.a `firstorder`) of the variance of ``f`` attributable to the input ``x_i``, and the second index collects proportions ``TV_i`` (a.k.a `totalorder`) of the residual variance having removed contributions attributable to inputs ``x_j`` ``\forall j\neq i``. The Ishigami function has an analytic formula for these Sobol indices, it is also known that one can obtain numerical approximation through quasi-Monto-Carlo methods by evaluating the Ishigami function on a special quasi-random Sobol sequence.
with ``a=7, b=0.1``.
- The Sobol G-function
```math
f(x; a) = \prod_{i=1}^d \frac{|4x_i - 2|+a_i}{1+x_i},
```
with the user-chosen input dimension ``d``, and coefficients ``a_i = \frac{i-1}{2} \geq 0`` for ``i=1,\dots,d``, where small $i$ (and thus small $a_i$) imply larger first-order effects, and interactions are primarily present between these variables.


In this example, global sensitivity analysis refers to calculation of two Sobol indices. The first index collects proportions ``V_i`` (a.k.a `firstorder`) of the variance of ``f`` attributable to the input ``x_i``, and the second index collects proportions ``TV_i`` (a.k.a `totalorder`) of the residual variance having removed contributions attributable to inputs ``x_j`` ``\forall j\neq i``. The Ishigami function has an analytic formula for these Sobol indices, it is also known that one can obtain numerical approximation through quasi-Monto-Carlo methods by evaluating the Ishigami function on a special quasi-random Sobol sequence.

## Ishigami

To emulate the Ishigami function, the data consists of 300 pairs ``\{x,f(x)+\eta\}`` where ``\eta \sim N(0,\Sigma)`` is additive noise, and the x are sampled from the Sobol sequence. The emulators are validated by evaluating the posterior mean function on the full 16000 points of the Sobol sequence and the Sobol indices are estimated. We rerun the experiment for many repeats of the random feature hyperparameter optimization and present the statistics of these indices, as well as plotting a realization of the emulator.

Expand Down Expand Up @@ -123,7 +135,7 @@ y_pred, y_var = predict(emulator, samples', transform_to_real = true)
result_pred = analyze(data, y_pred')
```

## Gaussian Process Emulator (sci-kit learn `GP`)
### Gaussian Process Emulator (sci-kit learn `GP`)

Here is the plot for one emulation
```@raw html
Expand All @@ -147,7 +159,7 @@ Sampled Emulated Sobol Indices (# obs 300, noise var 0.01)
totalorder: [0.5502469909342245, 0.4587734278791574, 0.23542404141319245]
```

## Random feature emulator (Separable Low-rank kernel `RF-scalar`)
### Random feature emulator (Separable Low-rank kernel `RF-scalar`)

Here is the plot for one emulation
```@raw html
Expand Down Expand Up @@ -175,3 +187,31 @@ Sampled Emulated Sobol Indices (# obs 300, noise var 0.01)
```

## Sobol G-function results

To emulate the Sobol function, a similar setup is created to that of Ishigami, and the code follows similarly
Rather than a table though, plots are produced of the Sobol function values.

For example, repeating the scalar random feature emulation task 30 times produces (1.) A slice of the function in the (up-to) three most sensitive dimensions from one realization, and (2.) The sensitivities' variability to each dimension averaged over the realizations.

For three input dimensions one obtains

```@raw html
<img src="../../../assets/GFunction_sens_RF-scalar_3.png" width="600">
```

```@raw html
<img src="../../../assets/GFunction_slices_RF-scalar_3.png" width="600">
```

For ten input dimensions one obtains

```@raw html
<img src="../../../assets/GFunction_sens_RF-scalar_10.png" width="600">
```

```@raw html
<img src="../../../assets/GFunction_slices_RF-scalar_10.png" width="600">
```


0 comments on commit aa02c0c

Please sign in to comment.