Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add dot-env-secrets recipe #265

Merged
merged 3 commits into from
Mar 17, 2024
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
63 changes: 63 additions & 0 deletions flows-advanced/dot-env-secrets/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
# Dot-env Secrets

Recipe to quickly load your `.env` secrets into
Prefect's [Secret block](https://docs.prefect.io/latest/api-ref/prefect/blocks/system/#prefect.blocks.system.Secret).

## Reasoning

Often times, I would find myself adapting existing Python script into Prefect.
lucianosrp marked this conversation as resolved.
Show resolved Hide resolved
Storing secret environment variables in a `.env` file is a common solution and
the [dotenv](https://pypi.org/project/python-dotenv/) package can seamlessly load those
variables into the runtime environment.

I needed a tool to load all the variables present in this file into my Prefect workplace
so that my flows no longer need to rely on this file and can just fetch the variable
from Prefect.

## Dependencies

In order to run this script you would
need [python-dot-env](https://pypi.org/project/python-dotenv/).
A *prefect-only* solution is also possible but not (yet) presented in this recipe.
Make sure to be authenticated before running the script.

## Solution

Running this script will:

- Load the variables present in the .env file (specified in the `fp` parameter)
- Convert the variable names to `kebab-case`
- Upload Secret block to your Prefect workspace

## Example

In this example `sample-dot-env` contains a variable `LOAD_DOT_ENV` which is a string.
In the `load_secrets` function we specify the path to the `sample-dot-env` file:

```shell
LOAD_DOT_ENV=it_works
```

```python
if __name__ == "__main__":
env_file = "flows-advanced/dot-env-secrets/sample-dot-env"
load_secrets(env_file)
```

Executing it will print each variable name and its block ID

```shell
load-dot-env ffb9096a-efd3-4fcf-a37e-f79b8549f27f
```

You can now use this Secret block in your Prefect code!

![screenshot](static/block-loaded-example.png)

## More

More could be done to this recipe! For example:

- Provide a solution to 'load' all Prefect's secrets onto a file
- Remove the `dotenv` dependency
- Load all variables concurrently
41 changes: 41 additions & 0 deletions flows-advanced/dot-env-secrets/main.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
from typing import Any

from dotenv import dotenv_values
from prefect import flow
from prefect.blocks.system import Secret


def rename_keys(d: dict[str, Any]) -> dict[str, Any]:
"""
Variable names need to be renamed in order to be valid Block names.

Examples:
--------
>>> original = {'TEST_KEY':"test_value", "test-key-valid":"test_value"}
>>> rename_keys(original)
{'test-key': 'test_value', 'test-key-valid': 'test_value'}

"""
n = {}
for k, v in d.items():
n[k.replace("_", "-").casefold()] = v
return n


@flow(log_prints=True)
def load_secrets(fp: str, overwrite: bool = False):
original_values = dotenv_values(fp, verbose=True)

if original_values == {}:
raise IOError("File not found or empty")

d = rename_keys(original_values)
for k, v in d.items():
secret = Secret(value=v)
uuid = secret.save(name=k, overwrite=overwrite)
print(k, uuid)


if __name__ == "__main__":
env_file = "flows-advanced/dot-env-secrets/sample-dot-env"
load_secrets(env_file)
2 changes: 2 additions & 0 deletions flows-advanced/dot-env-secrets/requirements.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
dotenv
prefect
1 change: 1 addition & 0 deletions flows-advanced/dot-env-secrets/sample-dot-env
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
LOAD_DOT_ENV=it_works
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.