diff --git a/.github/workflows/yaml-lint.yaml b/.github/workflows/yaml-lint.yaml index b555d6c..b54fb51 100644 --- a/.github/workflows/yaml-lint.yaml +++ b/.github/workflows/yaml-lint.yaml @@ -13,3 +13,8 @@ jobs: uses: actions/checkout@v3 - name: yaml-lint uses: ibiqlik/action-yamllint@v3 + with: + config_data: | + extends: default + rules: + line-length: disable diff --git a/README.md b/README.md index 3a81471..4602788 100644 --- a/README.md +++ b/README.md @@ -2,23 +2,22 @@ ## Details -A Github Action to deploy a Prefect flow via [Prefect Projects](https://docs.prefect.io/latest/concepts/projects/#projects). Note - all configuration defined in your `deployment.yaml` will be infered at run time; this means you do **not** need to duplicate cli arguments that are already defined. For example, if your `deployment.yaml` looks like: +A Github Action to deploy one or more Prefect deployments via [Prefect Projects](https://docs.prefect.io/latest/concepts/projects/#projects). Note - all configuration must be defined in your `deployment.yaml`, which will be infered at run time; this means you **cannot** pass any additional CLI arguments. For example, your `deployment.yaml` should have the following configuration in place: ```yaml -description: null -entrypoint: examples/simple/flow.py:call_api -flow_name: null -name: Simple -parameters: {} -schedule: null -tags: [] -version: null -work_pool: - job_variables: - image: prefecthq/prefect:2-latest - name: simple-pool - work_queue_name: null +deployments: + - name: Simple + description: null + entrypoint: examples/simple/flow.py:call_api + flow_name: null + parameters: {} + schedule: null + tags: [] + version: null + work_pool: + name: simple-pool ``` -You will not need to pass your work-pool name or the deployment name to this action. + +Additionally, the `prefect deploy` command needs to load your flow in order to gather some information about it. This results in the module that the flow is in being loaded, which can result in errors if not all the dependencies are present (issue [9512](https://github.com/PrefectHQ/prefect/issues/9512)). As a result, this action takes in a comma-seperated list of requirments to pre-load these ahead of running `prefect deploy`. This will **not** result in a generic image being created, but rather used to satisfy a pre-flight check required by the Prefect CLI. If building one or many custom docker images, those will still be isolated and only install the relevant dependencies defined as a part of your Dockerfile. ## Requirements @@ -30,11 +29,10 @@ You will not need to pass your work-pool name or the deployment name to this act ## Inputs -| Input | Desription | Required | Default | -|-------|------------|----------|---------| -| additional-args | Any additional arguments to pass to the Prefect Deploy command. Available additional arguments are listed below. | false | | -| entrypoint | The path to a flow entrypoint within a project, in format: `./path/to/file.py:flow_func_name`. | true | | -| requirements-file-path | Path to requirements files to correctly install dependencies for your Prefect flow. | false | `./requirements.txt` | +| Input | Desription | Required | +|-------|------------|----------| +| deployment-names | Comma separated list of deployment names defined in the deployment.yaml file. | true | +| requirements-file-paths | Comma sepearated list of paths to requirements files to correctly install dependencies for your Prefect flow(s). | false | ## Examples @@ -64,12 +62,34 @@ jobs: prefect-workspace: ${{ secrets.PREFECT_WORKSPACE }} - name: Run Prefect Deploy - uses: PrefectHQ/actions-prefect-deploy@v2 + uses: PrefectHQ/actions-prefect-deploy@v3 with: - requirements-file-path: ./examples/simple/requirements.txt - entrypoint: ./examples/simple/flow.py:call_api - additional-args: --cron '30 19 * * 0' + deployment-names: Simple + requirements-file-paths: ./examples/simple/requirements.txt ``` + +### Multi-Deployment Prefect Deploy + +Deploy multiple Prefect deployments that doesn't have a `push` step defined in the `prefect.yaml` +```yaml +name: Deploy multiple Prefect deployments +on: + push: + branches: + - main +jobs: + deploy_flow: + runs-on: ubuntu-latest + steps: + - ... + + - name: Run Prefect Deploy + uses: PrefectHQ/actions-prefect-deploy@v3 + with: + deployment-names: Simple_Deployment_1,Simple_Deployment_2 + requirements-file-paths: ./examples/multi-deployment/deployment-1/requirements.txt,./examples/multi-deployment/deployment-2/requirements.txt +``` + ### Basic Docker Auth w/ Prefect Deploy Deploy a Prefect flow and also build a Docker artifact that pushes to a defined repository in the `prefect.yaml` file. @@ -101,11 +121,10 @@ jobs: prefect-workspace: ${{ secrets.PREFECT_WORKSPACE }} - name: Run Prefect Deploy - uses: PrefectHQ/actions-prefect-deploy@v2 + uses: PrefectHQ/actions-prefect-deploy@v3 with: - requirements-file-path: ./examples/docker/requirements.txt - entrypoint: ./examples/docker/flow.py:call_api - additional-args: --cron '30 19 * * 0' --pool docker-pool + deployment-names: Docker + requirements-file-paths: ./examples/docker/requirements.txt ``` ### GCP Workload Identity w/ Prefect Deploy @@ -144,30 +163,11 @@ jobs: prefect-workspace: ${{ secrets.PREFECT_WORKSPACE }} - name: Run Prefect Deploy - uses: PrefectHQ/actions-prefect-deploy@v2 + uses: PrefectHQ/actions-prefect-deploy@v3 with: - requirements-file-path: ./examples/docker/requirements.txt - entrypoint: ./examples/docker/flow.py:call_api - additional-args: --cron '30 19 * * 0' --pool docker-pool + deployment-names: Docker + requirements-file-paths: ./examples/docker/requirements.txt ``` -## Additional Arguments - -| Arg Name | Description | Example | -|----------|-------------|---------| -| --anchor-date | The anchor date for an interval schedule. | | -| --cron | A cron string that will be used to set a CronSchedule on the deployment. | `--cron '30 19 * * 0' `| -| --description | The description to give the deployment. If not provided, the description will be populated from the flow's description. | | -| --interval | An integer specifying an interval (in seconds) that will be used to set an IntervalSchedule on the deployment. | `--interval 60` | -| --name | The name to give the deployment. | `--name 'Test Flow'` | -| --param | An optional parameter override, values are parsed as JSON strings | `--param question=ultimate --param answer=42` | -| --params | An optional parameter override in a JSON string format. | `--params='{"question": "ultimate", "answer": 42}'` | -| --pool | The work pool that will handle this deployment's runs. | `--pool docker-pool` | -| --rrule | An RRule that will be used to set an RRuleSchedule on the deployment. | | -| --tag | One or more optional tags to apply to the deployment - Note: tags are used only for organizational purposes. | | -| --timezone | Deployment schedule timezone string. | `--timezone 'America/New_York'` | -| --variable | One or more job variable overrides for the work pool. | `--variable foo=bar` | -| --version | A version to give the deployment. | | -| --work-queue | The work queue that will handle this deployment's runs. It will be created if it doesn't already exist. | `--work-queue test` | ## Terms & Conditions See here for the Prefect's [Terms and Conditions](https://www.prefect.io/legal/terms/). diff --git a/action.yaml b/action.yaml index d59f52e..5fd6bfe 100644 --- a/action.yaml +++ b/action.yaml @@ -7,36 +7,46 @@ branding: color: blue inputs: - additional-args: + deployment-names: description: - Any additinal arguments that need to be passed to - the prefect deployment run in a space separated format. - i.e - `--variable foo=bar -t 1.0.0 --cron "* * * * *"`. + Comma separated list of names of your + Prefect deployments. + example 'deployment1,deployment2' required: false + default: deployment - entrypoint: + requirements-file-paths: description: - The path to a flow entrypoint within a project, - in format `./path/to/file.py:flow_func_name`. - required: true - - requirements-file-path: - description: - Path to requirements files to correctly install + Comma separated list of paths to + requirements files to correctly install dependencies for your Prefect flow(s). + example './flow1/requirements.txt,./flow2/requirements.txt' required: false - default: ./requirements.txt - + default: '' runs: using: composite steps: - id: install-local-requirements - run: pip install -r ${{ inputs.requirements-file-path }} + if: ${{ inputs.requirements-file-paths != '' }} + run: | + IFS=',' read -r -a requirements_paths <<< ${{ inputs.requirements-file-paths }} + for req in ${requirements_paths}; do + pip install -r $req + done shell: bash - id: prefect-deploy run: | - prefect deploy ${{ inputs.entrypoint }} \ - ${{ inputs.additional-args }} + PREFECT_VERSION=$(prefect --version) + PREFECT_SHORT_VERSION="$(cut -d "." -f 2 <<< "$PREFECT_VERSION")"."$(cut -d "." -f 3 <<< "$PREFECT_VERSION")" + if [ $(echo $PREFECT_SHORT_VERSION'<'10.6 | bc -l) -eq 1 ]; + then + echo "The currently installed Prefect version (2.$PREFECT_SHORT_VERSION) is not compatible. Updating to the latest Prefect version..." + pip install prefect -U "prefect>=2.10.8" + fi + IFS=',' read -r -a deployment_names <<< ${{ inputs.deployment-names }} + for name in ${deployment_names}; do + prefect deploy --name $name + done shell: bash diff --git a/examples/docker/deployment.yaml b/examples/docker/deployment.yaml index 65b1798..70747d7 100644 --- a/examples/docker/deployment.yaml +++ b/examples/docker/deployment.yaml @@ -1,13 +1,14 @@ -description: null -entrypoint: examples/docker/flow.py:call_api -flow_name: null -name: null -parameters: {} -schedule: null -tags: [] -version: null -work_pool: - job_variables: - image: '{{ image_name }}' - name: null - work_queue_name: null +deployments: + - name: Docker + description: null + entrypoint: examples/docker/flow.py:call_api + flow_name: null + parameters: {} + schedule: null + tags: [] + version: null + work_pool: + job_variables: + image: '{{ image_name }}' + name: docker-pool + work_queue_name: null diff --git a/examples/docker/requirements.txt b/examples/docker/requirements.txt index 6133339..879c961 100644 --- a/examples/docker/requirements.txt +++ b/examples/docker/requirements.txt @@ -1 +1 @@ -prefect==2.10.4 +prefect==2.10.8 diff --git a/examples/multi-deployment/deployment-1/.prefectignore b/examples/multi-deployment/deployment-1/.prefectignore new file mode 100644 index 0000000..7ce5ed8 --- /dev/null +++ b/examples/multi-deployment/deployment-1/.prefectignore @@ -0,0 +1,41 @@ +# prefect artifacts +.prefectignore + +# python artifacts +__pycache__/ +*.py[cod] +*$py.class +*.egg-info/ +*.egg + +# Type checking artifacts +.mypy_cache/ +.dmypy.json +dmypy.json +.pyre/ + +# IPython +profile_default/ +ipython_config.py +*.ipynb_checkpoints/* + +# Environments +.python-version +.env +.venv +env/ +venv/ + +# MacOS +.DS_Store + +# Dask +dask-worker-space/ + +# Editors +.idea/ +.vscode/ + +# VCS +.git/ +.hg/ diff --git a/examples/multi-deployment/deployment-1/flow.py b/examples/multi-deployment/deployment-1/flow.py new file mode 100755 index 0000000..c140d36 --- /dev/null +++ b/examples/multi-deployment/deployment-1/flow.py @@ -0,0 +1,11 @@ +import json + +import requests +from prefect import flow + +@flow(name="Simple Flow", log_prints=True) +def call_api(url: str = "http://time.jsontest.com/"): + """Sends a GET request to the provided URL and returns the JSON response""" + resp = requests.get(url).json() + print(resp) + return resp diff --git a/examples/multi-deployment/deployment-1/prefect.yaml b/examples/multi-deployment/deployment-1/prefect.yaml new file mode 100644 index 0000000..fac5059 --- /dev/null +++ b/examples/multi-deployment/deployment-1/prefect.yaml @@ -0,0 +1,20 @@ +# File for configuring project / deployment build, push and pull steps + +# Generic metadata about this project +name: simple +prefect-version: 2.10.8 + +# build section allows you to manage and build docker images +build: null + +# push section allows you to manage if +# and how this project is uploaded to remote locations +push: null + +# pull section allows you to provide instructions +# for cloning this project in remote locations +pull: + - prefect.projects.steps.git_clone_project: + repository: git@github.com:PrefectHQ/actions-prefect-deploy.git + branch: main + access_token: "{{ prefect.blocks.secret.simple-github-pat }}" diff --git a/examples/multi-deployment/deployment-1/requirements.txt b/examples/multi-deployment/deployment-1/requirements.txt new file mode 100644 index 0000000..879c961 --- /dev/null +++ b/examples/multi-deployment/deployment-1/requirements.txt @@ -0,0 +1 @@ +prefect==2.10.8 diff --git a/examples/multi-deployment/deployment-2/.prefectignore b/examples/multi-deployment/deployment-2/.prefectignore new file mode 100644 index 0000000..7ce5ed8 --- /dev/null +++ b/examples/multi-deployment/deployment-2/.prefectignore @@ -0,0 +1,41 @@ +# prefect artifacts +.prefectignore + +# python artifacts +__pycache__/ +*.py[cod] +*$py.class +*.egg-info/ +*.egg + +# Type checking artifacts +.mypy_cache/ +.dmypy.json +dmypy.json +.pyre/ + +# IPython +profile_default/ +ipython_config.py +*.ipynb_checkpoints/* + +# Environments +.python-version +.env +.venv +env/ +venv/ + +# MacOS +.DS_Store + +# Dask +dask-worker-space/ + +# Editors +.idea/ +.vscode/ + +# VCS +.git/ +.hg/ diff --git a/examples/multi-deployment/deployment-2/flow.py b/examples/multi-deployment/deployment-2/flow.py new file mode 100755 index 0000000..c140d36 --- /dev/null +++ b/examples/multi-deployment/deployment-2/flow.py @@ -0,0 +1,11 @@ +import json + +import requests +from prefect import flow + +@flow(name="Simple Flow", log_prints=True) +def call_api(url: str = "http://time.jsontest.com/"): + """Sends a GET request to the provided URL and returns the JSON response""" + resp = requests.get(url).json() + print(resp) + return resp diff --git a/examples/multi-deployment/deployment-2/prefect.yaml b/examples/multi-deployment/deployment-2/prefect.yaml new file mode 100644 index 0000000..fac5059 --- /dev/null +++ b/examples/multi-deployment/deployment-2/prefect.yaml @@ -0,0 +1,20 @@ +# File for configuring project / deployment build, push and pull steps + +# Generic metadata about this project +name: simple +prefect-version: 2.10.8 + +# build section allows you to manage and build docker images +build: null + +# push section allows you to manage if +# and how this project is uploaded to remote locations +push: null + +# pull section allows you to provide instructions +# for cloning this project in remote locations +pull: + - prefect.projects.steps.git_clone_project: + repository: git@github.com:PrefectHQ/actions-prefect-deploy.git + branch: main + access_token: "{{ prefect.blocks.secret.simple-github-pat }}" diff --git a/examples/multi-deployment/deployment-2/requirements.txt b/examples/multi-deployment/deployment-2/requirements.txt new file mode 100644 index 0000000..879c961 --- /dev/null +++ b/examples/multi-deployment/deployment-2/requirements.txt @@ -0,0 +1 @@ +prefect==2.10.8 diff --git a/examples/multi-deployment/deployment.yaml b/examples/multi-deployment/deployment.yaml new file mode 100644 index 0000000..70ed7cc --- /dev/null +++ b/examples/multi-deployment/deployment.yaml @@ -0,0 +1,27 @@ +deployments: + - name: Simple_Deployment_1 + description: null + entrypoint: examples/multi-deployment/deployment-1/flow.py:call_api + flow_name: null + parameters: {} + schedule: null + tags: [] + version: null + work_pool: + job_variables: + image: prefecthq/prefect:2-latest + name: simple-pool + work_queue_name: null + - name: Simple_Deployment_2 + description: null + entrypoint: examples/multi-deployment/deployment-2/flow.py:call_api + flow_name: null + parameters: {} + schedule: null + tags: [] + version: null + work_pool: + job_variables: + image: prefecthq/prefect:2-latest + name: simple-pool + work_queue_name: null diff --git a/examples/simple/deployment.yaml b/examples/simple/deployment.yaml index 59e6158..1377af0 100644 --- a/examples/simple/deployment.yaml +++ b/examples/simple/deployment.yaml @@ -1,13 +1,14 @@ -description: null -entrypoint: examples/simple/flow.py:call_api -flow_name: null -name: Simple -parameters: {} -schedule: null -tags: [] -version: null -work_pool: - job_variables: - image: prefecthq/prefect:2-latest - name: simple-pool - work_queue_name: null +deployments: + - name: Simple + description: null + entrypoint: examples/simple/flow.py:call_api + flow_name: null + parameters: {} + schedule: null + tags: [] + version: null + work_pool: + job_variables: + image: prefecthq/prefect:2-latest + name: simple-pool + work_queue_name: null diff --git a/examples/simple/requirements.txt b/examples/simple/requirements.txt index 6133339..879c961 100644 --- a/examples/simple/requirements.txt +++ b/examples/simple/requirements.txt @@ -1 +1 @@ -prefect==2.10.4 +prefect==2.10.8