Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ppai/weather: add weather forecasting sample #8345

Merged
Merged
Show file tree
Hide file tree
Changes from 25 commits
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
1b4749b
ppai/weather-foreacasting: add weather sample
davidcavazos Aug 26, 2022
6937f26
remove temporary test model
davidcavazos Jan 9, 2023
a80d3b4
remove test scripts
davidcavazos Jan 9, 2023
6336bbc
install local packages before testing
davidcavazos Jan 9, 2023
b39bb84
update datasets
davidcavazos Jan 9, 2023
82d3de3
add local packages to test requirements
davidcavazos Jan 10, 2023
05b8426
Merge branch 'main' of github.com:GoogleCloudPlatform/python-docs-sam…
davidcavazos Jan 10, 2023
dcd488b
split notebook and tests
davidcavazos Jan 11, 2023
b39d1d0
Merge branch 'main' of github.com:GoogleCloudPlatform/python-docs-sam…
davidcavazos Jan 11, 2023
66b6c79
add predictions notebook and tests
davidcavazos Jan 11, 2023
b616fee
add missing requirements
davidcavazos Jan 11, 2023
c6dbbc3
update notebooks
davidcavazos Jan 11, 2023
ee3a83b
update test names
davidcavazos Jan 11, 2023
126a252
fix tests
davidcavazos Jan 12, 2023
8293dfa
enable vertex test
davidcavazos Jan 12, 2023
66e7d7e
Merge branch 'main' of github.com:GoogleCloudPlatform/python-docs-sam…
davidcavazos Jan 12, 2023
6120fa4
copy file to directory
davidcavazos Jan 12, 2023
d18fd7b
Merge branch 'main' of github.com:GoogleCloudPlatform/python-docs-sam…
davidcavazos Jan 13, 2023
5a5c294
update dependencies
davidcavazos Jan 13, 2023
c38e7f3
Merge branch 'main' of github.com:GoogleCloudPlatform/python-docs-sam…
davidcavazos Jan 17, 2023
4cad59d
Merge branch 'main' into ppai-weather-forecasting
gcf-merge-on-green[bot] Jan 17, 2023
20450e3
Merge branch 'main' of github.com:GoogleCloudPlatform/python-docs-sam…
davidcavazos Jan 18, 2023
8b2d13a
Merge branch 'ppai-weather-forecasting' of github.com:davidcavazos/py…
davidcavazos Jan 18, 2023
e125c50
ignore checkpoints directory
davidcavazos Jan 18, 2023
6b046f1
Merge branch 'main' into ppai-weather-forecasting
davidcavazos Jan 18, 2023
5152e7f
Simplify boolean test
davidcavazos Jan 18, 2023
b867eb1
Improve wording
davidcavazos Jan 18, 2023
227f40c
Improve wording
davidcavazos Jan 18, 2023
d3ff301
rename go to graph_objects
davidcavazos Jan 18, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
41 changes: 33 additions & 8 deletions people-and-planet-ai/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,8 @@

## 🦏 [Wildlife Insights -- _image-classification_](image-classification)

[![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/GoogleCloudPlatform/python-docs-samples/blob/main/people-and-planet-ai/image-classification/README.ipynb)

> [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/GoogleCloudPlatform/python-docs-samples/blob/main/people-and-planet-ai/image-classification/README.ipynb)
>
> [Watch the video in YouTube<br> ![thumbnail](http://img.youtube.com/vi/hUzODH3uGg0/0.jpg)](https://youtu.be/hUzODH3uGg0)

This model is trained to recognize animal species from
Expand All @@ -20,10 +20,12 @@ pictures.
[AutoML]: https://cloud.google.com/vertex-ai/docs/beginner/beginners-guide
[Vertex AI]: https://cloud.google.com/vertex-ai

## 🗺 [Global Fishing Watch -- _timeseries-classification_](timeseries-classification)
---

[![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/GoogleCloudPlatform/python-docs-samples/blob/main/people-and-planet-ai/timeseries-classification/README.ipynb)
## 🗺 [Global Fishing Watch -- _timeseries-classification_](timeseries-classification)

> [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/GoogleCloudPlatform/python-docs-samples/blob/main/people-and-planet-ai/timeseries-classification/README.ipynb)
>
> [Watch the video in YouTube<br> ![thumbnail](http://img.youtube.com/vi/LnEhSVEJUuY/0.jpg)](https://youtu.be/LnEhSVEJUuY)

This model is trained to classify if a ship is fishing or not every hour from their
Expand All @@ -41,10 +43,12 @@ location data.
[Keras]: https://keras.io
[Vertex AI]: https://cloud.google.com/vertex-ai

## 🏭 [Coal Plant Predictions -- _geospatial-classification_](geospatial-classification)
---

[![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/GoogleCloudPlatform/python-docs-samples/blob/main/people-and-planet-ai/geospatial-classification/README.ipynb)
## 🏭 [Coal Plant Predictions -- _geospatial-classification_](geospatial-classification)

> [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/GoogleCloudPlatform/python-docs-samples/blob/main/people-and-planet-ai/geospatial-classification/README.ipynb)
>
> [Watch the video in YouTube<br> ![thumbnail](http://img.youtube.com/vi/8amFK7T_n30/0.jpg)](https://youtu.be/8amFK7T_n30)

This model uses satellite data to predict if a coal plant is turned on and producing carbon emissions. The satellite data comes from [Google Earth Engine.](https://earthengine.google.com/)
Expand All @@ -60,10 +64,12 @@ This model uses satellite data to predict if a coal plant is turned on and produ
[TensorFlow]: https://www.tensorflow.org/
[Vertex AI]: https://cloud.google.com/vertex-ai

## 🌍 [Land cover classification -- _image segmentation_](land-cover-classification)
---

[![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/GoogleCloudPlatform/python-docs-samples/blob/main/people-and-planet-ai/land-cover-classification/README.ipynb)
## 🌍 [Land cover classification -- _image segmentation_](land-cover-classification)

> [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/GoogleCloudPlatform/python-docs-samples/blob/main/people-and-planet-ai/land-cover-classification/README.ipynb)
>
> [Watch the video in YouTube<br> ![thumbnail](http://img.youtube.com/vi/zImQf91ffFo/0.jpg)](https://youtu.be/zImQf91ffFo)

This model uses satellite data to classify what is on Earth. The satellite data comes from [Google Earth Engine.](https://earthengine.google.com/)
Expand All @@ -81,3 +87,22 @@ This model uses satellite data to classify what is on Earth. The satellite data
[Earth Engine]: https://earthengine.google.com/
[TensorFlow]: https://www.tensorflow.org/
[Vertex AI]: https://cloud.google.com/vertex-ai

---

## 🌦 Weather forecasting -- _timeseries regression_

> [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/GoogleCloudPlatform/python-docs-samples/blob/main/people-and-planet-ai/weather-forecasting/README.ipynb)
<!-- > [Watch the video in YouTube<br> ![thumbnail](http://img.youtube.com/vi/TODO/0.jpg)](https://youtu.be/TODO) -->

This model uses satellite data to forecast precipitation for the next 2 and 6 hours. The satellite data comes from [Google Earth Engine.](https://earthengine.google.com/)

* **Model**: 2D Fully Convolutional Network in [PyTorch]
* **Creating datasets**: [Sentinel-2] satellite data and [ESA WorldCover] from [Earth Engine] with [Dataflow]
* **Training the model**: [PyTorch] in [Vertex AI]
* **Getting predictions**: [PyTorch] locally

[Dataflow]: https://cloud.google.com/dataflow
[Earth Engine]: https://earthengine.google.com/
[PyTorch]: https://pytorch.org/
[Vertex AI]: https://cloud.google.com/vertex-ai
64 changes: 50 additions & 14 deletions people-and-planet-ai/conftest.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,18 +14,19 @@

from __future__ import annotations

from datetime import datetime
import multiprocessing
import os
import platform
import re
import subprocess
from unittest import mock
import sys
import textwrap
import uuid
from collections.abc import Callable, Iterable
from datetime import datetime
from unittest import mock

from google.cloud import storage
import pytest
from google.cloud import storage


@pytest.fixture(scope="session")
Expand Down Expand Up @@ -202,9 +203,9 @@ def run_notebook(
skip_shell_commands: bool = False,
until_end: bool = False,
) -> None:
import nbformat
from nbclient.client import NotebookClient
from nbclient.exceptions import CellExecutionError
import nbformat

def notebook_filter_section(
start: str,
Expand Down Expand Up @@ -262,13 +263,7 @@ def notebook_filter_section(
cmd = "pass"
cell["source"] = shell_command_re.sub(cmd, cell["source"])
else:
cmd = [
"import subprocess",
"_cmd = f'''\\1'''",
"print(f'>> {_cmd}')",
"subprocess.run(_cmd, shell=True, check=True)",
]
cell["source"] = shell_command_re.sub("\n".join(cmd), cell["source"])
cell["source"] = shell_command_re.sub(r"_run(f'''\1''')", cell["source"])

# Apply variable substitutions.
for regex, new_value in compiled_substitutions:
Expand All @@ -278,8 +273,38 @@ def notebook_filter_section(
for old, new in replace.items():
cell["source"] = cell["source"].replace(old, new)

# Clear outputs.
cell["outputs"] = []

# Prepend the prelude cell.
nb.cells = [nbformat.v4.new_code_cell(prelude)] + nb.cells
prelude_src = textwrap.dedent(
"""\
def _run(cmd):
import subprocess as _sp
import sys as _sys
_p = _sp.run(cmd, shell=True, stdout=_sp.PIPE, stderr=_sp.PIPE)
_stdout = _p.stdout.decode('utf-8').strip()
_stderr = _p.stderr.decode('utf-8').strip()
if _stdout:
print(f'➜ !{cmd}')
print(_stdout)
if _stderr:
print(f'➜ !{cmd}', file=_sys.stderr)
print(_stderr, file=_sys.stderr)
if _p.returncode:
raise RuntimeError('\\n'.join([
f"Command returned non-zero exit status {_p.returncode}.",
f"-------- command --------",
f"{cmd}",
f"-------- stderr --------",
f"{_stderr}",
f"-------- stdout --------",
f"{_stdout}",
]))
"""
+ prelude
)
nb.cells = [nbformat.v4.new_code_cell(prelude_src)] + nb.cells

# Run the notebook.
error = ""
Expand All @@ -289,7 +314,16 @@ def notebook_filter_section(
except CellExecutionError as e:
# Remove colors and other escape characters to make it easier to read in the logs.
# https://stackoverflow.com/a/33925425
error = re.sub(r"(\x9B|\x1B\[)[0-?]*[ -\/]*[@-~]", "", str(e))
color_chars = re.compile(r"(\x9B|\x1B\[)[0-?]*[ -\/]*[@-~]")
error = color_chars.sub("", str(e))
for cell in nb.cells:
if cell["cell_type"] != "code":
continue
for output in cell["outputs"]:
if output.get("name") == "stdout":
print(color_chars.sub("", output["text"]))
elif output.get("name") == "stderr":
print(color_chars.sub("", output["text"]), file=sys.stderr)

if error:
raise RuntimeError(
Expand All @@ -305,6 +339,8 @@ def run_notebook_parallel(
replace: dict[str, str] = {},
skip_shell_commands: bool = False,
) -> None:
import multiprocessing

args = [
{
"ipynb_file": ipynb_file,
Expand Down
7 changes: 7 additions & 0 deletions people-and-planet-ai/weather-forecasting/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
**/*.egg-info
**/build
**/dist
checkpoints/
data/
data-training/
model/
16 changes: 16 additions & 0 deletions people-and-planet-ai/weather-forecasting/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
## 🌦 Weather forecasting -- _timeseries regression_

> [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/GoogleCloudPlatform/python-docs-samples/blob/main/people-and-planet-ai/weather-forecasting/README.ipynb)
davidcavazos marked this conversation as resolved.
Show resolved Hide resolved
<!-- > [Watch the video in YouTube<br> ![thumbnail](http://img.youtube.com/vi/TODO/0.jpg)](https://youtu.be/TODO) -->

This model uses satellite data to forecast precipitation for the next 2 and 6 hours. The satellite data comes from [Google Earth Engine.](https://earthengine.google.com/)

* **Model**: 2D Fully Convolutional Network in [PyTorch]
* **Creating datasets**: [Sentinel-2] satellite data and [ESA WorldCover] from [Earth Engine] with [Dataflow]
* **Training the model**: [PyTorch] in [Vertex AI]
* **Getting predictions**: [PyTorch] locally

[Dataflow]: https://cloud.google.com/dataflow
[Earth Engine]: https://earthengine.google.com/
[PyTorch]: https://pytorch.org/
[Vertex AI]: https://cloud.google.com/vertex-ai
Loading