From 1db9895ee24d522ae90fd449b47ac2cbf806b369 Mon Sep 17 00:00:00 2001 From: rswamina Date: Tue, 27 Jun 2023 15:32:50 +0100 Subject: [PATCH 01/12] version changed --- _episodes/06-preprocessor.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_episodes/06-preprocessor.md b/_episodes/06-preprocessor.md index dff4f078..7813674d 100644 --- a/_episodes/06-preprocessor.md +++ b/_episodes/06-preprocessor.md @@ -2,7 +2,7 @@ title: "Writing your own recipe" teaching: 15 exercises: 30 -compatibility: ESMValTool v2.6.0 +compatibility: ESMValTool v2.9.0 questions: - "How do I create a new recipe?" From 9116eeacf374550f8c426da323edb6eec1c217e3 Mon Sep 17 00:00:00 2001 From: rswamina Date: Wed, 28 Jun 2023 13:49:58 +0100 Subject: [PATCH 02/12] changed data set for CMIP5 --- _episodes/06-preprocessor.md | 45 ++++++++++++++++++++++++------------ 1 file changed, 30 insertions(+), 15 deletions(-) diff --git a/_episodes/06-preprocessor.md b/_episodes/06-preprocessor.md index 7813674d..ee373724 100644 --- a/_episodes/06-preprocessor.md +++ b/_episodes/06-preprocessor.md @@ -22,7 +22,7 @@ keypoints: ## Introduction -One of the key strenghts of ESMValTool is in making complex analyses reusable +One of the key strengths of ESMValTool is in making complex analyses reusable and reproducible. But that doesn't mean everything in ESMValTool needs to be complex. Sometimes, the biggest challenge is in keeping things simple. You probably know the 'warming stripes' visualization by Professor Ed Hawkins. On @@ -38,14 +38,14 @@ ESMValTool. We have prepared a small Python script that takes a NetCDF file with timeseries data, and visualizes it in the form of our desired warming stripes figure. -You can find the diagnostic script that we will use -[here (`warming_stripes.py`)](../files/warming_stripes.py). +The diagnostic script that we will use is called `warming_stripes.py` and +can be downloaded [here](../files/warming_stripes.py). Download the file and store it in your working directory. If you want, you may also have a look at the contents, but it is not necessary to follow along. We will write an ESMValTool recipe that takes some data, performs the necessary -preprocessing, and then runs our Python script. +preprocessing, and then runs this Python script. > ## Drawing up a plan > @@ -55,7 +55,7 @@ preprocessing, and then runs our Python script. > > > ## Answer > > -> > In this episode, we will need to do 2 tasks: +> > In this episode, we will need to do the following two tasks: > > > > - A preprocessing task that converts the gridded temperature data to a timeseries > > of global temperature anomalies @@ -73,17 +73,18 @@ start from scratch. This forces us to think about all the steps. We will deal with common errors as they occur throughout the development. Remember the basic structure of a recipe, and notice that each of them is -extensively described in the documentation under the header ["The recipe -format"](https://docs.esmvaltool.org/projects/esmvalcore/en/latest/recipe/overview.html): +extensively described in the documentation under the header + ["The recipe +format"](https://docs.esmvaltool.org/projects/esmvalcore/en/latest/recipe/overview.html){:target="_blank"}: - [documentation](https://docs.esmvaltool.org/projects/esmvalcore/en/latest/ -recipe/overview.html#recipe-section-documentation) +recipe/overview.html#recipe-section-documentation){:target="_blank"} - [datasets](https://docs.esmvaltool.org/projects/esmvalcore/en/latest/recipe/ -overview.html#recipe-section-datasets) +overview.html#recipe-section-datasets){:target="_blank"} - [preprocessors](https://docs.esmvaltool.org/projects/esmvalcore/en/latest/ -recipe/overview.html#recipe-section-preprocessors) +recipe/overview.html#recipe-section-preprocessors){:target="_blank"} - [diagnostics](https://docs.esmvaltool.org/projects/esmvalcore/en/latest/ -recipe/overview.html#recipe-section-diagnostics) +recipe/overview.html#recipe-section-diagnostics){:target="_blank"} This is the first place to look for help if you get stuck. @@ -201,8 +202,8 @@ Although there is no actual error in the recipe, ESMValTool assumes you mistaken left out a variable name to process and alerts you with this error message. ## Adding a dataset entry -Let's add a datasets section. We will reuse the same datasets that we used in -previous episodes. +Let's add a datasets section. + > ## Filling in the dataset keys > @@ -210,7 +211,7 @@ previous episodes. > and look at the explanation of the dataset entry > in the [ESMValTool > documentation](https://docs.esmvaltool.org/projects/esmvalcore/en/latest/ -recipe/overview.html#recipe-section-documentation). +recipe/overview.html#recipe-section-documentation){:target="_blank"}. > For both the datasets, write down the following properties: > > - project @@ -229,7 +230,7 @@ recipe/overview.html#recipe-section-documentation). > > | project | CMIP6 | CMIP5 | > > | short name | tas | tas | > > | CMIP table | Amon | Amon | -> > | dataset | BCC-ESM1 | CanESM2 | +> > | dataset | BCC-ESM1 | bcc-csm1-1| > > | experiment | historical | historical | > > | ensemble | r1i1p1f1 | r1i1p1 | > > | grid | gn (native grid) | N/A | @@ -249,9 +250,23 @@ of the `diagnostic` section in the recipe example seen in [Running your first recipe]({{ page.root }}{% link _episodes/04-recipe.md %}). ```yaml +# ESMValTool +# recipe_warming_stripes.yml +--- +documentation: + description: Reproducing Ed Hawkins' warming stripes visualization + title: Reproducing Ed Hawkins' warming stripes visualization. + + authors: + - doe_john datasets: - {dataset: BCC-ESM1, project: CMIP6, mip: Amon, exp: historical, ensemble: r1i1p1f1, grid: gn, start_year: 1850, end_year: 2014} + +diagnostics: + dummy_diagnostic_1: + scripts: null + ``` The recipe should run but produce the same message as in the previous case since we From ea5821305e8f0c9b77833b4603a995dcd6a572d4 Mon Sep 17 00:00:00 2001 From: rswamina Date: Tue, 14 May 2024 12:03:59 +0100 Subject: [PATCH 03/12] _episodes/06-preprocessor.md --- _includes/links.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/_includes/links.md b/_includes/links.md index fc512f82..fb27db67 100644 --- a/_includes/links.md +++ b/_includes/links.md @@ -14,6 +14,7 @@ [cran-knitr]: https://cran.r-project.org/package=knitr [cran-stringr]: https://cran.r-project.org/package=stringr [dc-lessons]: http://www.datacarpentry.org/lessons/ +[datasets-overview]: https://docs.esmvaltool.org/projects/ESMValCore/en/latest/recipe/overview.html#datasets [diagnostic]: https://github.com/ESMValGroup/ESMValTool/blob/main/esmvaltool/diag_scripts/examples/diagnostic.py [ds]: https://github.com/ESMValGroup/ESMValTool_Tutorial/blob/main/data/dataset.urls [email]: mailto:team@carpentries.org @@ -52,6 +53,7 @@ [python-gapminder]: https://swcarpentry.github.io/python-novice-gapminder/ [pyyaml]: https://pypi.python.org/pypi/PyYAML [recipe]: https://github.com/ESMValGroup/ESMValTool/blob/main/esmvaltool/recipes/examples/recipe_python.yml +[recipe-overview]: https://docs.esmvaltool.org/projects/esmvalcore/en/latest/recipe/overview.html [r-markdown]: https://rmarkdown.rstudio.com/ [rstudio]: https://www.rstudio.com/ [ruby-install-guide]: https://www.ruby-lang.org/en/downloads/ From e4b88f35f87e18916a6738fdb0a23d85c3f71f78 Mon Sep 17 00:00:00 2001 From: rswamina Date: Tue, 14 May 2024 12:13:53 +0100 Subject: [PATCH 04/12] _episodes/06-preprocessor.md --- _episodes/06-preprocessor.md | 83 ++++++++++++++++--- ...ming_stripes_multiple_ensemble_members.yml | 37 +++++++++ 2 files changed, 107 insertions(+), 13 deletions(-) create mode 100644 files/recipe_warming_stripes_multiple_ensemble_members.yml diff --git a/_episodes/06-preprocessor.md b/_episodes/06-preprocessor.md index ee373724..555c268c 100644 --- a/_episodes/06-preprocessor.md +++ b/_episodes/06-preprocessor.md @@ -75,7 +75,7 @@ with common errors as they occur throughout the development. Remember the basic structure of a recipe, and notice that each of them is extensively described in the documentation under the header ["The recipe -format"](https://docs.esmvaltool.org/projects/esmvalcore/en/latest/recipe/overview.html){:target="_blank"}: +format"][recipe-overview]{:target="_blank"}: - [documentation](https://docs.esmvaltool.org/projects/esmvalcore/en/latest/ recipe/overview.html#recipe-section-documentation){:target="_blank"} @@ -276,6 +276,26 @@ us to reuse this dataset entry with different variable names later on. This is not really necessary for our simple use case, but it is common practice in ESMValTool. + +> ## Pro-tip: Automatically populating a recipe with all available datasets +> +> You can select all available models for processing using +> `glob` patterns or wildcards. An example `datasets` section that uses all +> available CMIP6 models and ensemble members for the `historical` experiment +> will look like this: +> ```yaml``` +>datasets: +> - project: CMIP6 +> exp: historical +> dataset: '*' +> institute: '*' +> ensemble: '*' +> grid: '*' +> Note that you will have to set the `search_esgf` option in the `config_file to +> `always` so that you can download data from ESGF nodes as needed. +{: .callout} + + ## Adding the preprocessor section Above, we already described the preprocessing task that needs to convert the @@ -311,7 +331,8 @@ esmvalcore.preprocessor.html#preprocessor-functions): > {: .solution} {: .challenge} -Add the following block to your recipe file: +Add the following block to your recipe file between the `datasets` and `diagnostics` +block: ```yaml preprocessors: @@ -332,8 +353,8 @@ preprocessors: ## Completing the diagnostics section -Now we are ready to finish our diagnostics section. Remember that we want to -make two tasks: a preprocessor task, and a diagnostic task. To illustrate that +We are now ready to finish our diagnostics section. Remember that we want to +create two tasks: a preprocessor task, and a diagnostic task. To illustrate that we can also pass settings to the diagnostic script, we add the option to specify a custom colormap. @@ -374,30 +395,64 @@ a custom colormap. > {: .solution} {: .challenge} -Now you should be able to run the recipe to get your own warming stripes. +You should now be able to run the recipe to get your own warming stripes. Note: for the purpose of simplicity in this episode, we have not added logging or provenance tracking in the diagnostic script. Once you start to develop your own diagnostic scripts and want to add them to the ESMValTool repositories, this -will be required. However, writing your own diagnostic script is beyond the -scope of the basic tutorial. +will be required. Writing your own diagnostic script is discussed in a +[later episode]({{ page.root }}{% link _episodes/08-diagnostics.md %}). ## Bonus exercises -Below are a couple of exercise to practice modifying the recipe. For your +Below are a few exercises to practice modifying an ESMValTool recipe. For your reference, here's a copy of the [recipe at this point](../files/recipe_warming_stripes.yml). This will be the point of departure for each of the modifications we'll make below. -> ## Specific location +> ## Multiple ensemble members +> +> You can choose data from multiple ensemble members for a model in a single line +> +>> ## Solution +>> +>> The `dataset` section allows you to choose more than one ensemble member +>> Here's a copy of the changed +>> [recipe](../files/recipe_warming_stripes_multiple_ensemble_members.yml) +>>to do that. +>> Changes made are shown in the diff output below: +>> +>> --- recipe_warming_stripes.yml +>> +++ recipe_warming_stripes_multiple_ensemble_members.yml +>> @@ -8,7 +8,7 @@ +>> - righi_mattia +>> +>> datasets: +>> - - {dataset: BCC-ESM1, project: CMIP6, mip: Amon, exp: historical, ensemble: r1i1p1f1, grid: gn, start_year: 1850, end_year: 2014} +>> + - {dataset: BCC-ESM1, project: CMIP6, mip: Amon, exp: historical, ensemble: "r(1:2)i1p1f1", grid: gn, start_year: 1850, end_year: 2014} +>> +>> preprocessors: +>> global_anomalies: +>> +>> +>> Pro-tip: Check out the section on a different way to use multiple ensemble +>> members or even multiple experiments at +>> [Concatenating data corresponding to multiple facets][datasets-overview]. +>> +> {: .solution} +{:.challenge} + + +> ## Specific location selection > -> On showyourstripes.org, you can download stripes for specific locations. We -> will reproduce this possibility. Look at the available preprocessors in the -> documentation, and replace the global mean with a suitable alternative. +> On showyourstripes.org, you can download stripes for specific locations. Here we +> show how this can be done with ESMValTool. +> Instead of the global mean, we can pick a location to plot the stripes for. +> Can you find a suitable preprocessor to do this? > > > ## Solution > > -> > You could have used `extract_point` or `extract_region`. We used +> > You can use `extract_point` or `extract_region` to select a location. We used > > `extract_point`. Here's a copy of the [recipe at this > > point](../files/recipe_warming_stripes_local.yml) and this is the difference > > from the previous recipe: @@ -576,3 +631,5 @@ for each of the modifications we'll make below. > > > {: .solution} {:.challenge} + +{% include links.md %} diff --git a/files/recipe_warming_stripes_multiple_ensemble_members.yml b/files/recipe_warming_stripes_multiple_ensemble_members.yml new file mode 100644 index 00000000..a2fce156 --- /dev/null +++ b/files/recipe_warming_stripes_multiple_ensemble_members.yml @@ -0,0 +1,37 @@ +# ESMValTool +# recipe_warming_stripes.yml +--- +documentation: + description: Reproducing Ed Hawkins' warming stripes visualization + title: Reproducing Ed Hawkins' warming stripes visualization. + authors: + - righi_mattia + +datasets: + - {dataset: BCC-ESM1, project: CMIP6, mip: Amon, exp: historical, ensemble: "r(1:2)i1p1f1", grid: gn, start_year: 1850, end_year: 2014} + +preprocessors: + global_anomalies: + area_statistics: + operator: mean + anomalies: + period: month + reference: + start_year: 1981 + start_month: 1 + start_day: 1 + end_year: 2010 + end_month: 12 + end_day: 31 + standardize: false + +diagnostics: + diagnostic_warming_stripes: + variables: + global_temperature_anomalies: + short_name: tas + preprocessor: global_anomalies + scripts: + warming_stripes_script: + script: ~/esmvaltool_tutorial/warming_stripes.py + colormap: 'bwr' From 6eb834853e566d04658cc55f55ba611dc643e40f Mon Sep 17 00:00:00 2001 From: rswamina Date: Mon, 27 May 2024 23:33:55 +0100 Subject: [PATCH 05/12] updates for v2.10 --- _episodes/06-preprocessor.md | 165 +++++++++++++++++------------------ _includes/links.md | 9 ++ 2 files changed, 89 insertions(+), 85 deletions(-) diff --git a/_episodes/06-preprocessor.md b/_episodes/06-preprocessor.md index 555c268c..8983fe40 100644 --- a/_episodes/06-preprocessor.md +++ b/_episodes/06-preprocessor.md @@ -2,7 +2,7 @@ title: "Writing your own recipe" teaching: 15 exercises: 30 -compatibility: ESMValTool v2.9.0 +compatibility: ESMValTool v2.10.0 questions: - "How do I create a new recipe?" @@ -73,18 +73,13 @@ start from scratch. This forces us to think about all the steps. We will deal with common errors as they occur throughout the development. Remember the basic structure of a recipe, and notice that each of them is -extensively described in the documentation under the header - ["The recipe -format"][recipe-overview]{:target="_blank"}: - -- [documentation](https://docs.esmvaltool.org/projects/esmvalcore/en/latest/ -recipe/overview.html#recipe-section-documentation){:target="_blank"} -- [datasets](https://docs.esmvaltool.org/projects/esmvalcore/en/latest/recipe/ -overview.html#recipe-section-datasets){:target="_blank"} -- [preprocessors](https://docs.esmvaltool.org/projects/esmvalcore/en/latest/ -recipe/overview.html#recipe-section-preprocessors){:target="_blank"} -- [diagnostics](https://docs.esmvaltool.org/projects/esmvalcore/en/latest/ -recipe/overview.html#recipe-section-diagnostics){:target="_blank"} +extensively described in the documentation following the header + ["Overview"][recipe-overview]{:target="_blank"}: + +- [documentation][recipe-section-documentation]{:target="_blank"} +- [datasets][recipe-section-datasets]{:target="_blank"} +- [preprocessors][recipe-section-preprocessors]{:target="_blank"} +- [diagnostics][recipe-section-diagnostics]{:target="_blank"} This is the first place to look for help if you get stuck. @@ -120,17 +115,18 @@ esmvaltool run recipe_warming_stripes.yml In this case, it gives an error. Below you see the last few lines of the error message. ``` ... -Error validating data /home/user/esmvaltool_tutorial/recipe_warming_stripes.yml - with schema /home/user/mambaforge/envs/esmvaltool_tutorial/lib/python3.10 - /site-packages/esmvalcore/recipe_schema.yml - documentation.authors: Required field missing -YYYY-MM-DD HH:mm:SS,NNN UTC [19451] INFO If you have a question or need help, - please start a new discussion on https://github.com/ESMValGroup/ - ESMValTool/discussions -If you suspect this is a bug, please open an issue on - https://github.com/ESMValGroup/ESMValTool/issues -To make it easier to find out what the problem is, please consider attaching - the files run/recipe_*.yml and run/main_log_debug.txt from the output directory. +yamale.yamale_error.YamaleError: +Error validating data '/home/users/username/esmvaltool_tutorial/recipe_warming_stripes.yml' +with schema '/apps/jasmin/community/esmvaltool/miniconda3_py311_23.11.0-2/envs/esmvaltool/lib/python3.11/ +site-packages/esmvalcore/_recipe/recipe_schema.yml' + documentation.authors: Required field missing +2024-05-27 13:21:23,805 UTC [41924] INFO +If you have a question or need help, please start a new discussion on +https://github.com/ESMValGroup/ESMValTool/discussions +If you suspect this is a bug, please open an issue on https://github.com/ESMValGroup/ESMValTool/issues +To make it easier to find out what the problem is, please consider attaching the +files run/recipe_*.yml and run/main_log_debug.txt from the output directory. + ``` {: .error} @@ -169,23 +165,22 @@ This is the minimal recipe layout that is required by ESMValTool. If we now run the recipe again, you will probably see the following error: ``` -ValueError: Tag 'doe_john' does not exist in section 'authors' of - /home/user/mambaforge/envs/esmvaltool_tutorial/python3.10/ - site-packages/esmvaltool/config-references.yml +ValueError: Tag 'doe_john' does not exist in section +'authors' of /apps/jasmin/community/esmvaltool/ESMValTool_2.10.0/esmvaltool/config-references.yml + ``` {: .error} > ## Pro tip: config-references.yml > > The error message above points to a file named -> [config-references.yml](https://github.com/ESMValGroup/ESMValTool/blob/main -/esmvaltool/config-references.yml). +> [config-references.yml][config-references] + > This is where ESMValTool stores all its citation information. To add yourself > as an author, add your name in the form `lastname_firstname` in alphabetical > order following the existing entries, under the `# Development team` comment. > See the -> [List of authors](https://docs.esmvaltool.org/en/latest/community/ -code_documentation.html#list-of-authors) +> [List of authors][list-of-authors]{:target="_blank"} > section in the ESMValTool documentation for more information. {: .callout} @@ -210,8 +205,7 @@ Let's add a datasets section. > Use the paths specified in the configuration file to explore the data directory, > and look at the explanation of the dataset entry > in the [ESMValTool -> documentation](https://docs.esmvaltool.org/projects/esmvalcore/en/latest/ -recipe/overview.html#recipe-section-documentation){:target="_blank"}. +> documentation][recipe-section-datasets]{:target="_blank"}. > For both the datasets, write down the following properties: > > - project @@ -283,15 +277,17 @@ in ESMValTool. > `glob` patterns or wildcards. An example `datasets` section that uses all > available CMIP6 models and ensemble members for the `historical` experiment > will look like this: -> ```yaml``` ->datasets: -> - project: CMIP6 -> exp: historical -> dataset: '*' -> institute: '*' -> ensemble: '*' -> grid: '*' -> Note that you will have to set the `search_esgf` option in the `config_file to +> +>>```yaml +>>datasets: +>> - project: CMIP6 +>> exp: historical +>> dataset: '*' +>> institute: '*' +>> ensemble: '*' +>> grid: '*' +>> ``` +> Note that you will have to set the `search_esgf` option in the `config_file` to > `always` so that you can download data from ESGF nodes as needed. {: .callout} @@ -304,8 +300,7 @@ standard, gridded temperature data to a timeseries of temperature anomalies. > ## Defining the preprocessor > > Have a look at the available preprocessors in the -> [documentation](https://docs.esmvaltool.org/projects/esmvalcore/en/latest -/recipe/preprocessor.html). +> [documentation][preprocessor]{:target="_blank"}. > Write down > > - Which preprocessor functions do you think we should use? @@ -322,8 +317,7 @@ standard, gridded temperature data to a timeseries of temperature anomalies. > > case we want to compute the `mean`). > > > > The default order in which these preprocessors are applied can be seen -> > [here](https://docs.esmvaltool.org/projects/esmvalcore/en/latest/api/ -esmvalcore.preprocessor.html#preprocessor-functions): +> > [here][preprocessor-functions]{:target="_blank"}: > > `area_statistics` comes before `anomalies`. If you want to change this, you > > can use the `custom_order` preprocessor. We will keep it like this. > > @@ -410,39 +404,6 @@ reference, here's a copy of the [recipe at this point](../files/recipe_warming_stripes.yml). This will be the point of departure for each of the modifications we'll make below. -> ## Multiple ensemble members -> -> You can choose data from multiple ensemble members for a model in a single line -> ->> ## Solution ->> ->> The `dataset` section allows you to choose more than one ensemble member ->> Here's a copy of the changed ->> [recipe](../files/recipe_warming_stripes_multiple_ensemble_members.yml) ->>to do that. ->> Changes made are shown in the diff output below: ->> ->> --- recipe_warming_stripes.yml ->> +++ recipe_warming_stripes_multiple_ensemble_members.yml ->> @@ -8,7 +8,7 @@ ->> - righi_mattia ->> ->> datasets: ->> - - {dataset: BCC-ESM1, project: CMIP6, mip: Amon, exp: historical, ensemble: r1i1p1f1, grid: gn, start_year: 1850, end_year: 2014} ->> + - {dataset: BCC-ESM1, project: CMIP6, mip: Amon, exp: historical, ensemble: "r(1:2)i1p1f1", grid: gn, start_year: 1850, end_year: 2014} ->> ->> preprocessors: ->> global_anomalies: ->> ->> ->> Pro-tip: Check out the section on a different way to use multiple ensemble ->> members or even multiple experiments at ->> [Concatenating data corresponding to multiple facets][datasets-overview]. ->> -> {: .solution} -{:.challenge} - - > ## Specific location selection > > On showyourstripes.org, you can download stripes for specific locations. Here we @@ -493,11 +454,11 @@ for each of the modifications we'll make below. > {: .solution} {:.challenge} -> ## Different periods +> ## Different time periods > -> Split the diagnostic in 2: the second one should use a different period. -> You're free to choose the periods yourself. For example, 1 could be 'recent', -> the other '20th_century'. For this, you'll have to add a new variable group. +> Split the diagnostic in two with two different time periods for the same variable. +> You can choose the time periods yourself. For example, the recent past and +> the 20th century. You can do this by using variable groups. > > > ## Solution > > @@ -508,7 +469,6 @@ for each of the modifications we'll make below. > > --- recipe_warming_stripes_local.yml > > +++ recipe_warming_stripes_periods.yml > > @@ -7,7 +7,7 @@ -> > - righi_mattia > > > > datasets: > > - - {dataset: BCC-ESM1, project: CMIP6, mip: Amon, exp: historical, @@ -547,7 +507,7 @@ for each of the modifications we'll make below. > preprocessors. Add a second preprocessor to add another location of your > choosing. > -> Pro-tip: if you want to avoid repetition, you can use YAML anchors. +> > > > ## Solution > > @@ -601,6 +561,11 @@ for each of the modifications we'll make below. > {:.challenge} +> +> ## Pro-tip: YAML anchors +> If you want to avoid repetition, you can use YAML anchors as seen in the recipe above. +{:.callout} + > ## Additional datasets > > So far we have defined the datasets in the datasets section of the recipe. @@ -632,4 +597,34 @@ for each of the modifications we'll make below. > {: .solution} {:.challenge} +> ## Multiple ensemble members +> +> You can choose data from multiple ensemble members for a model in a single line +> +>> ## Solution +>> +>> The `dataset` section allows you to choose more than one ensemble member +>> Here's a copy of the changed +>> [recipe](../files/recipe_warming_stripes_multiple_ensemble_members.yml) +>>to do that. +>> Changes made are shown in the diff output below: +>>```diff +>>--- recipe_warming_stripes.yml 2024-05-27 15:37:52.340358967 +0100 +>>+++ recipe_warming_stripes_multiens.yml 2024-05-27 22:18:42.035558837 +0100 +>>@@ -10,7 +10,7 @@ +>>- ensemble: r1i1p1f1, grid: gn, start_year: 1850, end_year: 2014} +>>+ ensemble: "r(1:2)i1p1f1", grid: gn, start_year: 1850, end_year: 2014} +>>``` +>> +> {: .solution} +{:.challenge} + +> ## Pro-tip: Concatenating datasets +> Check out the section on a different way to use multiple ensemble +> members or even multiple experiments at [Concatenating data corresponding to multiple facets] +>[concatenating-datasets]{:target="_blank"}. +> +{: .callout} + + {% include links.md %} diff --git a/_includes/links.md b/_includes/links.md index 61c30915..18a08aec 100644 --- a/_includes/links.md +++ b/_includes/links.md @@ -8,8 +8,10 @@ [coc-reporting]: https://docs.carpentries.org/topic_folders/policies/incident-reporting.html [coc]: https://docs.carpentries.org/topic_folders/policies/code-of-conduct.html [code-documentation]: https://docs.esmvaltool.org/en/latest/community/code_documentation.html +[concatenating-datasets]: https://docs.esmvaltool.org/projects/ESMValCore/en/latest/recipe/overview.html#concatenating-data-corresponding-to-multiple-facets [concept-maps]: https://carpentries.github.io/instructor-training/05-memory/ [config-file]: https://docs.esmvaltool.org/projects/ESMValCore/en/latest/quickstart/configure.html +[config-references]: https://github.com/ESMValGroup/ESMValTool/blob/main/esmvaltool/config-references.yml [contrib-covenant]: https://contributor-covenant.org/ [contributing]: {{ repo_url }}/blob/{{ source_branch }}/CONTRIBUTING.md [cran-checkpoint]: https://cran.r-project.org/package=checkpoint @@ -43,6 +45,7 @@ [lesson-recipe]: {{ relative_root_path }}{% link _episodes/04-recipe.md %} [lesson-reference]: {{ relative_root_path }}{% link reference.md %} [lesson-setup]: {{ relative_root_path }}{% link setup.md %} +[list-of-authors]: https://docs.esmvaltool.org/en/latest/community/code_documentation.html#list-of-authors [manage-environments]: https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html [mit-license]: https://opensource.org/licenses/mit-license.html [morea]: https://morea-framework.github.io/ @@ -52,11 +55,17 @@ [pandoc]: https://pandoc.org/ [paper-now]: https://github.com/PeerJ/paper-now [preproc-area-manipulation]: https://docs.esmvaltool.org/projects/ESMValCore/en/latest/recipe/preprocessor.html#area-operations +[preprocessor]: https://docs.esmvaltool.org/projects/esmvalcore/en/latest/recipe/preprocessor.html +[preprocessor-functions]: https://docs.esmvaltool.org/projects/esmvalcore/en/latest/api/esmvalcore.preprocessor.html#preprocessor-functions [provenance]: https://docs.esmvaltool.org/en/latest/community/diagnostic.html?highlight=provenance#recording-provenance [python-gapminder]: https://swcarpentry.github.io/python-novice-gapminder/ [pyyaml]: https://pypi.python.org/pypi/PyYAML [recipe]: https://github.com/ESMValGroup/ESMValTool/blob/main/esmvaltool/recipes/examples/recipe_python.yml [recipe-overview]: https://docs.esmvaltool.org/projects/esmvalcore/en/latest/recipe/overview.html +[recipe-section-datasets]: https://docs.esmvaltool.org/projects/esmvalcore/en/latest/recipe/overview.html#recipe-section-datasets +[recipe-section-diagnostics]: https://docs.esmvaltool.org/projects/esmvalcore/en/latest/recipe/overview.html#recipe-section-diagnostics +[recipe-section-documentation]: https://docs.esmvaltool.org/projects/esmvalcore/en/latest/recipe/overview.html#recipe-section-documentation +[recipe-section-preprocessors]: https://docs.esmvaltool.org/projects/esmvalcore/en/latest/recipe/overview.html#recipe-section-preprocessors [r-markdown]: https://rmarkdown.rstudio.com/ [rstudio]: https://www.rstudio.com/ [ruby-install-guide]: https://www.ruby-lang.org/en/downloads/ From e76475449f7f9318f1b789395162857069de6881 Mon Sep 17 00:00:00 2001 From: rswamina Date: Tue, 28 May 2024 09:05:11 +0100 Subject: [PATCH 06/12] fixed errors in call outs --- _episodes/06-preprocessor.md | 21 ++++++--------------- _includes/links.md | 1 + 2 files changed, 7 insertions(+), 15 deletions(-) diff --git a/_episodes/06-preprocessor.md b/_episodes/06-preprocessor.md index 8983fe40..6a6006f8 100644 --- a/_episodes/06-preprocessor.md +++ b/_episodes/06-preprocessor.md @@ -117,13 +117,15 @@ In this case, it gives an error. Below you see the last few lines of the error m ... yamale.yamale_error.YamaleError: Error validating data '/home/users/username/esmvaltool_tutorial/recipe_warming_stripes.yml' -with schema '/apps/jasmin/community/esmvaltool/miniconda3_py311_23.11.0-2/envs/esmvaltool/lib/python3.11/ +with schema +'/apps/jasmin/community/esmvaltool/miniconda3_py311_23.11.0-2/envs/esmvaltool/lib/python3.11/ site-packages/esmvalcore/_recipe/recipe_schema.yml' documentation.authors: Required field missing 2024-05-27 13:21:23,805 UTC [41924] INFO If you have a question or need help, please start a new discussion on https://github.com/ESMValGroup/ESMValTool/discussions -If you suspect this is a bug, please open an issue on https://github.com/ESMValGroup/ESMValTool/issues +If you suspect this is a bug, please open an issue on +https://github.com/ESMValGroup/ESMValTool/issues To make it easier to find out what the problem is, please consider attaching the files run/recipe_*.yml and run/main_log_debug.txt from the output directory. @@ -175,10 +177,9 @@ ValueError: Tag 'doe_john' does not exist in section > > The error message above points to a file named > [config-references.yml][config-references] - > This is where ESMValTool stores all its citation information. To add yourself > as an author, add your name in the form `lastname_firstname` in alphabetical -> order following the existing entries, under the `# Development team` comment. +> order following the existing entries, under the `# Development team` section. > See the > [List of authors][list-of-authors]{:target="_blank"} > section in the ESMValTool documentation for more information. @@ -276,17 +277,7 @@ in ESMValTool. > You can select all available models for processing using > `glob` patterns or wildcards. An example `datasets` section that uses all > available CMIP6 models and ensemble members for the `historical` experiment -> will look like this: -> ->>```yaml ->>datasets: ->> - project: CMIP6 ->> exp: historical ->> dataset: '*' ->> institute: '*' ->> ensemble: '*' ->> grid: '*' ->> ``` +> will look is available [here] [include-all-datasets]{:target="_blank"}. > Note that you will have to set the `search_esgf` option in the `config_file` to > `always` so that you can download data from ESGF nodes as needed. {: .callout} diff --git a/_includes/links.md b/_includes/links.md index 18a08aec..b368bb70 100644 --- a/_includes/links.md +++ b/_includes/links.md @@ -25,6 +25,7 @@ [generate-ssh-key]: https://docs.github.com/en/authentication/connecting-to-github-with-ssh/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent [github-importer]: https://import.github.com/ [importer]: https://github.com/new/import +[include-all-datasets]: https://docs.esmvaltool.org/projects/ESMValCore/en/latest/recipe/overview.html#automatically-populating-a-recipe-with-all-available-datasets [install-from-source]: https://docs.esmvaltool.org/en/latest/quickstart/installation.html#install-from-source [interface]: https://docs.esmvaltool.org/projects/esmvalcore/en/latest/interfaces.html [jekyll-collection]: https://jekyllrb.com/docs/collections/ From 60cc98cd60c59fcdc394d5ea58cd0a4563bcf1a7 Mon Sep 17 00:00:00 2001 From: rswamina Date: Tue, 28 May 2024 13:25:27 +0100 Subject: [PATCH 07/12] small modifications to improve readability --- _episodes/06-preprocessor.md | 59 +++++++++++++++++++------------- _includes/links.md | 1 + files/recipe_warming_stripes.yml | 1 + 3 files changed, 37 insertions(+), 24 deletions(-) diff --git a/_episodes/06-preprocessor.md b/_episodes/06-preprocessor.md index 6a6006f8..01a57b2c 100644 --- a/_episodes/06-preprocessor.md +++ b/_episodes/06-preprocessor.md @@ -9,6 +9,7 @@ questions: - "Can I use different preprocessors for different variables?" - "Can I use different datasets for different variables?" - "How can I combine different preprocessor functions?" +- "Can I run the same recipe for multiple ensemble members?" objectives: - "Create a recipe with multiple preprocessors" - "Use different preprocessors for different variables" @@ -18,6 +19,7 @@ keypoints: - "The setting `additional_datasets` can be used to add a different dataset." - "Variable groups are useful for defining different settings for different variables." +- "Multiple ensemble members and experiments can be analysed in a single recipe through concatenation." --- ## Introduction @@ -26,7 +28,7 @@ One of the key strengths of ESMValTool is in making complex analyses reusable and reproducible. But that doesn't mean everything in ESMValTool needs to be complex. Sometimes, the biggest challenge is in keeping things simple. You probably know the 'warming stripes' visualization by Professor Ed Hawkins. On -the site you can find the same visualization for +the site {:target="_blank"} you can find the same visualization for many regions in the world. ![Warming stripes](../fig/warming_stripes.png) *Shared by Ed Hawkins under a @@ -42,7 +44,7 @@ The diagnostic script that we will use is called `warming_stripes.py` and can be downloaded [here](../files/warming_stripes.py). Download the file and store it in your working directory. If you want, you may -also have a look at the contents, but it is not necessary to follow along. +also have a look at the contents, but it is not necessary to do so for this lesson. We will write an ESMValTool recipe that takes some data, performs the necessary preprocessing, and then runs this Python script. @@ -50,8 +52,8 @@ preprocessing, and then runs this Python script. > ## Drawing up a plan > > Previously, we saw that running ESMValTool executes a number of -> tasks. Write down what tasks we will need to execute in this episode and what -> each of these tasks does? +> tasks. What tasks do you think we will need to execute and what +> should each of these tasks do to generate the warming stripes? > > > ## Answer > > @@ -69,11 +71,11 @@ preprocessing, and then runs this Python script. The easiest way to make a new recipe is to start from an existing one, and modify it until it does exactly what you need. However, in this episode we will -start from scratch. This forces us to think about all the steps. We will deal -with common errors as they occur throughout the development. +start from scratch. This forces us to think about all the steps involved in processing the data. +We will also deal with commonly occurring errors through the development of the recipe. -Remember the basic structure of a recipe, and notice that each of them is -extensively described in the documentation following the header +Remember the basic structure of a recipe, and notice that each component is +extensively described in the documentation under the section, ["Overview"][recipe-overview]{:target="_blank"}: - [documentation][recipe-section-documentation]{:target="_blank"} @@ -102,11 +104,11 @@ documentation: ``` -Notice that `yaml` always requires 2 spaces indentation between the different +Notice that `yaml` always requires `two spaces` indentation between the different levels. Pressing `ctrl+o` will save the file. Verify the filename at the bottom and press enter. Then use `ctrl+x` to exit the editor. -We will try to run the recipe after every modification we make, to see if it (still) works. +We will try to run the recipe after every modification we make, to see if it (still) works! ```bash esmvaltool run recipe_warming_stripes.yml @@ -132,9 +134,13 @@ files run/recipe_*.yml and run/main_log_debug.txt from the output directory. ``` {: .error} -Here, ESMValTool is telling us that it is missing a required field, namely the -authors. We see that ESMValTool always tries to validate the recipe -at an early stage. +We can use the the log message above, to understand why ESMValTool failed. Here, this is because +we missed a required field with author names. The text `documentation.authors: Required field missing` +tells us that. We see that ESMValTool always tries to validate the recipe +at an early stage. Note also the suggestion to open a GitHub issue if +you need help debugging the error message. This is something most +users do when they cannot understand the error or are not able to fix it +on their own. Let's add some additional information to the recipe. Open the recipe file again, and add an authors section below the description. ESMValTool expects the authors @@ -238,8 +244,8 @@ Let's add a datasets section. > {: .solution} {: .challenge} -We start with the BCC-ESM1 dataset and add a datasets section to the recipe, -listing a single dataset, as shown below. Note that key fields such +Let us start with the BCC-ESM1 dataset and add a datasets section to the recipe, +listing this single dataset, as shown below. Note that key fields such as `mip` or `start_year` are included in the `datasets` section here but are part of the `diagnostic` section in the recipe example seen in [Running your first recipe]({{ page.root }}{% link _episodes/04-recipe.md %}). @@ -277,7 +283,7 @@ in ESMValTool. > You can select all available models for processing using > `glob` patterns or wildcards. An example `datasets` section that uses all > available CMIP6 models and ensemble members for the `historical` experiment -> will look is available [here] [include-all-datasets]{:target="_blank"}. +> is available [here] [include-all-datasets]{:target="_blank"}. > Note that you will have to set the `search_esgf` option in the `config_file` to > `always` so that you can download data from ESGF nodes as needed. {: .callout} @@ -302,7 +308,7 @@ standard, gridded temperature data to a timeseries of temperature anomalies. > > ## Solution > > > > We need to calculate anomalies and global means. There is an `anomalies` -> > preprocessor which needs a granularity, a reference period, and whether or +> > preprocessor which takes in as arguments, a time period, a reference period, and whether or > > not to standardize the data. The global means can be calculated with the > > `area_statistics` preprocessor, which takes an operator as argument (in our > > case we want to compute the `mean`). @@ -310,7 +316,8 @@ standard, gridded temperature data to a timeseries of temperature anomalies. > > The default order in which these preprocessors are applied can be seen > > [here][preprocessor-functions]{:target="_blank"}: > > `area_statistics` comes before `anomalies`. If you want to change this, you -> > can use the `custom_order` preprocessor. We will keep it like this. +> > can use the `custom_order` preprocessor as described [here][recipe-section-preprocessors]{:target="_blank"}. +>> For this example, we will keep the default order.. > > > > Let's name our preprocessor `global_anomalies`. > {: .solution} @@ -448,8 +455,9 @@ for each of the modifications we'll make below. > ## Different time periods > > Split the diagnostic in two with two different time periods for the same variable. -> You can choose the time periods yourself. For example, the recent past and -> the 20th century. You can do this by using variable groups. +> You can choose the time periods yourself. In the example below, we have +> chosen the recent past and +> the 20th century and have used variable grouping. > > > ## Solution > > @@ -552,17 +560,20 @@ for each of the modifications we'll make below. > {:.challenge} -> + + + > ## Pro-tip: YAML anchors -> If you want to avoid repetition, you can use YAML anchors as seen in the recipe above. +> If you want to avoid retyping the arguments used in your preprocessor, you can use YAML anchors as +seen in the `anomalies` preprocessor specifications in the recipe above. {:.callout} > ## Additional datasets > > So far we have defined the datasets in the datasets section of the recipe. > However, it's also possible to add specific datasets only for specific -> variable groups. Look at the documentation to learn about the -> 'additional_datasets' keyword, and add a second dataset only for one of the +> variables or variable groups. Take a look at the documentation to learn about the +> `additional_datasets` keyword [here][additional-datasets]{:target="_blank"}, and add a second dataset only for one of the > variable groups. > > > ## Solution diff --git a/_includes/links.md b/_includes/links.md index b368bb70..9d788081 100644 --- a/_includes/links.md +++ b/_includes/links.md @@ -1,6 +1,7 @@ {% include base_path.html %} [activate-environment]: https://docs.esmvaltool.org/en/latest/quickstart/installation.html#pre-installed-versions-on-hpc-clusters-other-servers [add-ssh-key]: https://docs.github.com/en/authentication/connecting-to-github-with-ssh/adding-a-new-ssh-key-to-your-github-account +[additional-datasets]: https://docs.esmvaltool.org/projects/ESMValCore/en/latest/recipe/overview.html#diagnostic-and-variable-specific-datasets [cc-by-human]: https://creativecommons.org/licenses/by/4.0/ [cc-by-legal]: https://creativecommons.org/licenses/by/4.0/legalcode [ci]: http://communityin.org/ diff --git a/files/recipe_warming_stripes.yml b/files/recipe_warming_stripes.yml index 7e72be2f..6eb48e6c 100644 --- a/files/recipe_warming_stripes.yml +++ b/files/recipe_warming_stripes.yml @@ -29,6 +29,7 @@ preprocessors: diagnostics: diagnostic_warming_stripes: + description: visualize global temperature anomalies as warming stripes variables: global_temperature_anomalies: short_name: tas From 70c5cf1d999998c19c80b7e89a8145c2b7cacf62 Mon Sep 17 00:00:00 2001 From: rswamina Date: Tue, 28 May 2024 13:29:40 +0100 Subject: [PATCH 08/12] small modifications to improve readability --- _episodes/06-preprocessor.md | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-) diff --git a/_episodes/06-preprocessor.md b/_episodes/06-preprocessor.md index 01a57b2c..67b2a14a 100644 --- a/_episodes/06-preprocessor.md +++ b/_episodes/06-preprocessor.md @@ -135,7 +135,8 @@ files run/recipe_*.yml and run/main_log_debug.txt from the output directory. {: .error} We can use the the log message above, to understand why ESMValTool failed. Here, this is because -we missed a required field with author names. The text `documentation.authors: Required field missing` +we missed a required field with author names. +The text `documentation.authors: Required field missing` tells us that. We see that ESMValTool always tries to validate the recipe at an early stage. Note also the suggestion to open a GitHub issue if you need help debugging the error message. This is something most @@ -316,7 +317,8 @@ standard, gridded temperature data to a timeseries of temperature anomalies. > > The default order in which these preprocessors are applied can be seen > > [here][preprocessor-functions]{:target="_blank"}: > > `area_statistics` comes before `anomalies`. If you want to change this, you -> > can use the `custom_order` preprocessor as described [here][recipe-section-preprocessors]{:target="_blank"}. +> > can use the `custom_order` preprocessor as +>> described [here][recipe-section-preprocessors]{:target="_blank"}. >> For this example, we will keep the default order.. > > > > Let's name our preprocessor `global_anomalies`. @@ -564,8 +566,9 @@ for each of the modifications we'll make below. > ## Pro-tip: YAML anchors -> If you want to avoid retyping the arguments used in your preprocessor, you can use YAML anchors as -seen in the `anomalies` preprocessor specifications in the recipe above. +> If you want to avoid retyping the arguments used in your preprocessor, +> you can use YAML anchors as seen in the `anomalies` preprocessor +> specifications in the recipe above. {:.callout} > ## Additional datasets @@ -573,8 +576,8 @@ seen in the `anomalies` preprocessor specifications in the recipe above. > So far we have defined the datasets in the datasets section of the recipe. > However, it's also possible to add specific datasets only for specific > variables or variable groups. Take a look at the documentation to learn about the -> `additional_datasets` keyword [here][additional-datasets]{:target="_blank"}, and add a second dataset only for one of the -> variable groups. +> `additional_datasets` keyword [here][additional-datasets]{:target="_blank"}, and +> add a second dataset only for one of the variable groups. > > > ## Solution > > From b659d8623d33e333cc8bff89e7f4bc2de0ee7ecb Mon Sep 17 00:00:00 2001 From: rswamina <42945525+rswamina@users.noreply.github.com> Date: Tue, 28 May 2024 16:16:21 +0100 Subject: [PATCH 09/12] Update 06-preprocessor.md Added full stop to end of line. --- _episodes/06-preprocessor.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_episodes/06-preprocessor.md b/_episodes/06-preprocessor.md index 67b2a14a..0eed72d9 100644 --- a/_episodes/06-preprocessor.md +++ b/_episodes/06-preprocessor.md @@ -604,7 +604,7 @@ for each of the modifications we'll make below. > ## Multiple ensemble members > -> You can choose data from multiple ensemble members for a model in a single line +> You can choose data from multiple ensemble members for a model in a single line. > >> ## Solution >> From 10ec4f57a4c456bcee071fd7e76a5ab977e65c5f Mon Sep 17 00:00:00 2001 From: rswamina <42945525+rswamina@users.noreply.github.com> Date: Wed, 29 May 2024 14:31:27 +0100 Subject: [PATCH 10/12] Create dependabot.yml --- .github/dependabot.yml | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) create mode 100644 .github/dependabot.yml diff --git a/.github/dependabot.yml b/.github/dependabot.yml new file mode 100644 index 00000000..2e2ea667 --- /dev/null +++ b/.github/dependabot.yml @@ -0,0 +1,19 @@ +# To get started with Dependabot version updates, you'll need to specify which +# package ecosystems to update and where the package manifests are located. +# Please see the documentation for all configuration options: +# https://docs.github.com/code-security/dependabot/dependabot-version-updates/configuration-options-for-the-dependabot.yml-file + +version: 2 +updates: + - package-ecosystem: "" # See documentation for possible values + directory: "/" # Location of package manifests + schedule: + interval: "weekly" + + # Enable version updates for Docker + - package-ecosystem: "docker" + # Look for a `Dockerfile` in the `root` directory + directory: "/" + # Check for updates once a week + schedule: + interval: "weekly" From 7d6fea929d42990101bc575aa06cd4c687de8d78 Mon Sep 17 00:00:00 2001 From: rswamina <42945525+rswamina@users.noreply.github.com> Date: Wed, 29 May 2024 15:04:34 +0100 Subject: [PATCH 11/12] Update dependabot.yml Added bundler package --- .github/dependabot.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/dependabot.yml b/.github/dependabot.yml index 2e2ea667..4d088d9d 100644 --- a/.github/dependabot.yml +++ b/.github/dependabot.yml @@ -5,7 +5,7 @@ version: 2 updates: - - package-ecosystem: "" # See documentation for possible values + - package-ecosystem: "bundler" # See documentation for possible values directory: "/" # Location of package manifests schedule: interval: "weekly" From c0aac1a576b158bbf608f198159a43bf4ec299c0 Mon Sep 17 00:00:00 2001 From: rswamina <42945525+rswamina@users.noreply.github.com> Date: Wed, 29 May 2024 15:54:22 +0100 Subject: [PATCH 12/12] Update dependabot.yml for package npm --- .github/dependabot.yml | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/.github/dependabot.yml b/.github/dependabot.yml index 4d088d9d..aadaec36 100644 --- a/.github/dependabot.yml +++ b/.github/dependabot.yml @@ -9,6 +9,11 @@ updates: directory: "/" # Location of package manifests schedule: interval: "weekly" + # Enable updates for package np which has jquery + - package-ecosystem: "npm" # See documentation for possible values + directory: "/" # Location of package manifests + schedule: + interval: "weekly" # Enable version updates for Docker - package-ecosystem: "docker"