Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove NCAR branding and source images more consistently #13

Merged
merged 3 commits into from
Oct 4, 2023
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/nightly-build.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ jobs:
if: ${{ github.repository_owner == 'ProjectPythia' }}
uses: ProjectPythia/cookbook-actions/.github/workflows/build-book.yaml@main
with:
environment_name: dask-cookbook-dev
environment_name: dask-cookbook

link-check:
if: ${{ github.repository_owner == 'ProjectPythia' }}
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/publish-book.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ jobs:
build:
uses: ProjectPythia/cookbook-actions/.github/workflows/build-book.yaml@main
with:
environment_name: dask-cookbook-dev
environment_name: dask-cookbook

deploy:
needs: build
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/trigger-book-build.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,6 @@ jobs:
build:
uses: ProjectPythia/cookbook-actions/.github/workflows/build-book.yaml@main
with:
environment_name: dask-cookbook-dev
environment_name: dask-cookbook
artifact_name: book-zip-${{ github.event.number }}
# Other input options are possible, see ProjectPythia/cookbook-actions/.github/workflows/build-book.yaml
14 changes: 8 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,4 @@
<img src="notebooks/images/NCAR_CISL_NSF_banner.jpeg" alt="NCAR CISL logo" />

# NCAR Dask Tutorial
# Dask Cookbook
negin513 marked this conversation as resolved.
Show resolved Hide resolved

[![nightly-build](https://github.com/ProjectPythia/cookbook-template/actions/workflows/nightly-build.yaml/badge.svg)](https://github.com/ProjectPythia/cookbook-template/actions/workflows/nightly-build.yaml)
[![Binder](https://mybinder.org/badge_logo.svg)](http://binder.projectpythia.org/v2/gh/ProjectPythia/dask-cookbook/main?labpath=notebooks)
Expand All @@ -25,17 +23,21 @@ The motivation behind this repository is to provide a clear and concise resource
<a href="https://github.com/benkirk/demo_containers/graphs/contributors">
<img src="https://contrib.rocks/image?repo=benkirk/demo_containers" />
</a>
<a href="https://github.com/ProjectPythia/dask-cookbook/graphs/contributors">
<img src="https://contrib.rocks/image?repo=ProjectPythia/dask-cookbook" />
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure what would be the best way to handle contributors' list for the Pythia cookbooks for the cookbooks that are adapted from previous materials. This might include my profile twice (but I am okay with that and no need to change) ;-)

@jsignell An alternative will be to add your profiles individually or point to a repo that you are the only contributor too. Please see above for how I included Ben...

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah good point. I kind of like the idea of just relying on the contributors information on github for this repo. I'm not too worried about people showing up twice but eventually if everyone who contributed to the NCAR version has also contributed over here, then we can just have the one contributor list.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @jsignell for your response. In that case, I will leave this as is.
Overall, this is a bigger issue for the project Pythia maintainers on how to handle the list of contributors for many projects like this, where the cookbooks are adapted from other projects and are collected here as a resource for the community....

Maybe a good topic for future Pythia hackathons...

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just remembered that there is an authors section as well where there are links to specific individuals, so maybe it already is ok to just rely on github for the list of contributors?

</a>

## Note on Content Origin

This cookbook is part of the extensive material used in our NCAR tutorial, ["Using Dask on HPC systems"](https://github.com/NCAR/dask-tutorial.git), which was held in February 2023. The complete tutorial series also includes an in-depth exploration and practical use cases of Dask on HPC systems and best practices for Dask on HPC . For the complete set of materials, including these additional insights on Dask on HPC, please refer to the main tutorial content available [here](https://ncar.github.io/dask-tutorial/README.html).
This cookbook is derived from the extensive material used in the NCAR tutorial, ["Using Dask on HPC systems"](https://github.com/NCAR/dask-tutorial.git), which was held in February 2023. The NCAR tutorial series also includes an in-depth exploration and practical use cases of Dask on HPC systems and best practices for Dask on HPC. For the complete set of NCAR tutorial materials, including these additional insights
on Dask on HPC, please refer to the main NCAR tutorial content available [here](https://ncar.github.io/dask-tutorial/README.html).

## Structure

In the first chapter of this cookbook, we provide step-by-step tutorials on the basic concepts of Dask, including Dask arrays and Dask dataframes, which are powerful tools for parallel computing and distributed data processing. We explain the key differences between these Dask data structures and their counterparts in NumPy and Pandas.

In the second chapter of the repository, we move on to more advanced topics, such as distributed computing and Dask+Xarray integration. We provide examples of how to use Dask+Xarray to efficiently work with large, labelled multi-dimensional datasets.
Finally, we will discuss some best practices regarding Dask + Xarray.
Finally, we will discuss some best practices regarding Dask+Xarray.

## Running the Notebooks

Expand Down Expand Up @@ -78,7 +80,7 @@ If you are interested in running this material locally on your computer, you wil

```bash
conda env create -f environment.yml
conda activate dask-cookbook-example
conda activate dask-cookbook
```

1. Move into the `notebooks` directory and start up Jupyterlab
Expand Down
4 changes: 2 additions & 2 deletions _config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ execute:

# Add a few extensions to help with parsing content
parse:
myst_enable_extensions: # default extensions to enable in the myst parser. See https://myst-parser.readthedocs.io/en/latest/using/syntax-optional.html
myst_enable_extensions: # default extensions to enable in the myst parser. See https://myst-parser.readthedocs.io/en/stable/using/syntax-optional.html
- amsmath
- colon_fence
- deflist
Expand All @@ -33,7 +33,7 @@ sphinx:
html_permalinks_icon: '<i class="fas fa-link"></i>'
html_theme_options:
home_page_in_toc: true
repository_url: https://github.com/negin513/dask-cookbook.git # Online location of your book
repository_url: https://github.com/ProjectPythia/dask-cookbook.git # Online location of your book
repository_branch: main # Which branch of the repository should be used when creating links (optional)
use_issues_button: true
use_repository_button: true
Expand Down
2 changes: 1 addition & 1 deletion environment.yml
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
name: dask-cookbook-dev
name: dask-cookbook
channels:
- conda-forge
- nodefaults
Expand Down
44 changes: 12 additions & 32 deletions notebooks/00-dask-overview.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -2,33 +2,18 @@
"cells": [
{
"cell_type": "markdown",
"id": "4db5d14d-ee57-4791-9ab7-58cb2ff2cc3b",
"id": "a1ccac1c-5d4e-47dc-9d5e-4f417126df94",
"metadata": {},
"source": [
"<img src=\"https://raw.githubusercontent.com/NCAR/dask-tutorial/main/images/NCAR-contemp-logo-blue.png\"\n",
" width=\"750px\"\n",
" alt=\"NCAR logo\"\n",
" style=\"vertical-align:middle;margin:30px 0px\"/>\n",
"\n",
"<img src=\"https://docs.dask.org/en/stable/_images/dask_horizontal.svg\"\n",
" width=\"30%\"\n",
" alt=\"Dask logo\"\n",
" align=\"right\"\n",
"/>\n",
"\n",
"# Dask Overview\n",
"\n",
"**ESDS Dask Tutorial | 06 February, 2023** \n",
"\n",
"Negin Sobhani, Brian Vanderwende, Deepak Cherian, Ben Kirk \n",
"Computational & Information Systems Lab (CISL) \n",
"[[email protected]](mailto:[email protected]), [[email protected]](mailto:[email protected])\n",
"\n",
"\n",
"---------"
]
},
{
"cell_type": "markdown",
"id": "5b6211a0-3762-41a2-8a45-6b19ce32f658",
"metadata": {},
"source": [
"**In this tutorial, you learn:**\n",
"### In this tutorial, you learn:\n",
"\n",
"* What is Dask?\n",
"* Why Dask in Geosciences?\n",
Expand All @@ -54,11 +39,6 @@
"\n",
"## What is Dask?\n",
"\n",
"<img src=\"https://raw.githubusercontent.com/NCAR/dask-tutorial/main/images/dask_horizontal.svg\"\n",
" width=\"500px\"\n",
" alt=\"NCAR logo\"\n",
" style=\"vertical-align:middle;margin:30px 0px\"/>\n",
"\n",
"* Dask is an open-source Python library for parallel and distributed computing that scales the existing Python ecosystem.\n",
"\n",
"* Dask was developed to scale Python packages such as Numpy, Pandas, and Xarray to multi-core machines and distributed clusters when datasets exceed memory.\n",
Expand Down Expand Up @@ -109,7 +89,7 @@
"\n",
"</ul>\n",
"And keep in mind - all of the above steps improve your code whether you end up using Dask or not!</br></br>\n",
"<img src=\"https://raw.githubusercontent.com/NCAR/dask-tutorial/main/images/dask_twitter.png\"\n",
"<img src=\"https://raw.githubusercontent.com/ProjectPythia/dask-cookbook/main/notebooks/images/dask_twitter.png\"\n",
" width=\"500px\"/>\n",
"\n",
"</div>\n",
Expand Down Expand Up @@ -166,11 +146,11 @@
"\n",
"These are very powerfull tools, but it is easy to write something using a delayed function that could be executed faster and more simply using a high-level collection \n",
"\n",
"<img src=\"https://raw.githubusercontent.com/NCAR/dask-tutorial/main/images/high_vs_low_level_coll_analogy.png\"\n",
"<img src=\"https://raw.githubusercontent.com/ProjectPythia/dask-cookbook/main/notebooks/images/high_vs_low_level_coll_analogy.png\"\n",
" width=\"83%\"\n",
" alt=\"Dask Collections\"/>\n",
" \n",
"*Image credit: Anaconda, Inc. and contributors*\n",
"*Image credit: Dask Contributors*\n",
"\n",
"\n",
"### 2. Dynamic Task Scheduling\n",
Expand All @@ -190,7 +170,7 @@
" width=\"75%\"\n",
" alt=\"Dask Distributed Cluster\"/>\n",
" \n",
"*Image credit: Anaconda, Inc. and contributors*\n",
"*Image credit: Dask Contributors*\n",
"\n",
"\n",
"We will learn more about Dask Collections and Dynamic Task Scheduling in the next tutorials."
Expand Down Expand Up @@ -231,7 +211,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.16"
"version": "3.9.18"
},
"widgets": {
"application/vnd.jupyter.widget-state+json": {
Expand Down
28 changes: 10 additions & 18 deletions notebooks/01-dask-array.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -7,21 +7,13 @@
"tags": []
},
"source": [
"<img src=\"https://raw.githubusercontent.com/NCAR/dask-tutorial/main/images/NCAR-contemp-logo-blue.png\"\n",
" width=\"750px\"\n",
" alt=\"NCAR logo\"\n",
" style=\"vertical-align:middle;margin:30px 0px\"/>\n",
"<img src=\"https://docs.dask.org/en/stable/_images/dask_horizontal.svg\"\n",
" width=\"30%\"\n",
" alt=\"Dask logo\"\n",
" align=\"right\"\n",
"/>\n",
"\n",
"# Dask Arrays\n",
"\n",
"**ESDS Dask Tutorial | 06 February, 2023** \n",
"\n",
"Negin Sobhani, Brian Vanderwende, Deepak Cherian, Ben Kirk \n",
"Computational & Information Systems Lab (CISL) \n",
"[[email protected]](mailto:[email protected]), [[email protected]](mailto:[email protected])\n",
"\n",
"\n",
"---------\n",
"# Dask Array\n",
"\n",
"### In this tutorial, you learn:\n",
"\n",
Expand All @@ -31,8 +23,8 @@
"\n",
"**Related Dask Array Documentation**\n",
"\n",
"* [Dask Array documentation](https://docs.dask.org/en/latest/array.html)\n",
"* [Dask Array API](https://docs.dask.org/en/latest/array-api.html)\n",
"* [Dask Array documentation](https://docs.dask.org/en/stable/array.html)\n",
"* [Dask Array API](https://docs.dask.org/en/stable/array-api.html)\n",
"* [Dask Array examples](https://examples.dask.org/array.html)\n",
"\n",
"\n",
Expand All @@ -41,7 +33,7 @@
"\n",
"<img src=\"https://docs.dask.org/en/stable/_images/dask-array.svg\" width=\"500px\" style=\"horizontal-align:middle\"/>\n",
"\n",
"*Image credit: Anaconda, Inc. and contributors*\n",
"*Image credit: Dask Contributors*\n",
"\n",
"Dask Array can be used as a drop-in replacement for NumPy arrays, with a similar API and support for a subset of NumPy functions. \n",
"\n",
Expand Down Expand Up @@ -1114,7 +1106,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.12"
"version": "3.9.18"
},
"widgets": {
"application/vnd.jupyter.widget-state+json": {
Expand Down
39 changes: 13 additions & 26 deletions notebooks/02-dask-dataframe.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -5,22 +5,14 @@
"id": "990cfa4c-2117-4435-9806-ff9048890398",
"metadata": {},
"source": [
"<img src=\"https://raw.githubusercontent.com/NCAR/dask-tutorial/main/images/NCAR-contemp-logo-blue.png\"\n",
" width=\"750px\"\n",
" alt=\"NCAR logo\"\n",
" style=\"vertical-align:middle;margin:30px 0px\"/>\n",
"<img src=\"https://docs.dask.org/en/stable/_images/dask_horizontal.svg\"\n",
" width=\"30%\"\n",
" alt=\"Dask logo\"\n",
" align=\"right\"\n",
"/>\n",
"\n",
"# Dask DataFrame\n",
"\n",
"\n",
"**ESDS dask tutorial | 06 February, 2023** \n",
"\n",
"Negin Sobhani, Brian Vanderwende, Deepak Cherian, Ben Kirk \n",
"Computational & Information Systems Lab (CISL) \n",
"[[email protected]](mailto:[email protected]), [[email protected]](mailto:[email protected])\n",
"\n",
"---------\n",
"## Overview\n",
"### In this tutorial, you learn:\n",
"\n",
"* Basic concepts and features of Dask DataFrames\n",
Expand All @@ -31,8 +23,8 @@
"\n",
"### Related Documentation\n",
"\n",
"* [Dask DataFrame documentation](https://docs.dask.org/en/latest/dataframe.html)\n",
"* [Dask DataFrame API](https://docs.dask.org/en/latest/dataframe-api.html)\n",
"* [Dask DataFrame documentation](https://docs.dask.org/en/stable/dataframe.html)\n",
"* [Dask DataFrame API](https://docs.dask.org/en/stable/dataframe-api.html)\n",
"* [Dask DataFrame examples](https://examples.dask.org/dataframe.html)\n",
"* [pandas documentation](https://pandas.pydata.org/pandas-docs/stable/)\n",
"\n",
Expand All @@ -47,13 +39,12 @@
"\n",
"## Introduction\n",
"\n",
"\n",
"<img src=\"https://raw.githubusercontent.com/NCAR/dask-tutorial/main/images/dask_dataframe.png\"\n",
"<img src=\"https://docs.dask.org/en/stable/_images/dask-dataframe.svg\"\n",
" align=\"right\"\n",
" width=\"530px\"\n",
" alt=\"Dask DataFrame is composed of pandas DataFrames\"/>\n",
" \n",
" \n",
"\n",
"*Image credit: Dask Contributors*\n",
"\n",
"pandas is a very popular tool for working with **tabular datasets**, but the dataset needs to **fit into the memory**. \n",
"\n",
Expand Down Expand Up @@ -303,12 +294,8 @@
"\n",
"For example, we have heard that this could be Denver's first January in 13 years with no 60-degree days. \n",
"\n",
"\n",
"\n",
"\n",
"\n",
"<img src=\"https://raw.githubusercontent.com/NCAR/dask-tutorial/main/images/denver2.png\"\n",
" alt=\"Dask DataFrame is composed of pandas DataFrames\"/>\n",
"<img src=\"https://raw.githubusercontent.com/ProjectPythia/dask-cookbook/main/notebooks/denver.png\"\n",
" alt=\"News article showing that this January is abnormally warm\"/>\n",
"\n",
"Below, we show all days with high temperature above 60°F (155.5°C/10) since 2010:"
]
Expand Down Expand Up @@ -970,7 +957,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.12"
"version": "3.9.18"
},
"widgets": {
"application/vnd.jupyter.widget-state+json": {
Expand Down
Loading