feat: support deployment of >1 prefect deployment (#13)

* Initial commit * ID with underscore * Try different for loop * Test * Verify array is generated * Use GH env var * start here for deploy * Update default * Remove unneeded steps * Try forcing array * Reintroduce deploy names * Readme * Examples * if statement for req test * a note about reqs * force env * typos * update inputs * Try env * deplyoment name * docker example * Set no default for req txt * Try it this way * Remove create lists step as not needed * remove list step completely * Pass as var * Readme v3 * Readme remove addtional args * Try to test requirements * Add finalizer * Use correct option * test * Try this * Ok start from the top * Try this * Maybe try reading it as a special char * Maybe this but likely a special char issue * Purposefully set IFS * Fix spacing * Comparing prefect versions * Update if statement * Cleanup * Add information to step * Verify again * Forgot IFS * Force prefect upgrade * Cleanup verbiage * Cleanup verbiage * Correct install * Cleanup * lint * Sets line length as warning * Disable line length * Update action.yaml Co-authored-by: Edward Park <[email protected]> * Update action.yaml Co-authored-by: Edward Park <[email protected]> * Remove unneeded duplicate steps * Example prefect version update --------- Co-authored-by: Jimi Dodd-o <[email protected]> Co-authored-by: Edward Park <[email protected]>
PrefectHQ · May 11, 2023 · b1a5ede · b1a5ede
1 parent 48f6386
commit b1a5ede
Show file tree

Hide file tree

Showing 16 changed files with 285 additions and 95 deletions.
diff --git a/.github/workflows/yaml-lint.yaml b/.github/workflows/yaml-lint.yaml
@@ -13,3 +13,8 @@ jobs:
         uses: actions/checkout@v3
       - name: yaml-lint
         uses: ibiqlik/action-yamllint@v3
+        with:
+          config_data: |
+            extends: default
+            rules:
+              line-length: disable
diff --git a/README.md b/README.md
@@ -2,23 +2,22 @@
 
 ## Details
 
-A Github Action to deploy a Prefect flow via [Prefect Projects](https://docs.prefect.io/latest/concepts/projects/#projects). Note - all configuration defined in your `deployment.yaml` will be infered at run time; this means you do **not** need to duplicate cli arguments that are already defined. For example, if your `deployment.yaml` looks like: 
+A Github Action to deploy one or more Prefect deployments via [Prefect Projects](https://docs.prefect.io/latest/concepts/projects/#projects). Note - all configuration must be defined in your `deployment.yaml`, which will be infered at run time; this means you **cannot** pass any additional CLI arguments. For example, your `deployment.yaml` should have the following configuration in place: 
 ```yaml
-description: null
-entrypoint: examples/simple/flow.py:call_api
-flow_name: null
-name: Simple
-parameters: {}
-schedule: null
-tags: []
-version: null
-work_pool:
-  job_variables:
-    image: prefecthq/prefect:2-latest
-  name: simple-pool
-  work_queue_name: null
+deployments:
+  - name: Simple
+    description: null
+    entrypoint: examples/simple/flow.py:call_api
+    flow_name: null
+    parameters: {}
+    schedule: null
+    tags: []
+    version: null
+    work_pool:
+      name: simple-pool
 ```
-You will not need to pass your work-pool name or the deployment name to this action.
+
+Additionally, the `prefect deploy` command needs to load your flow in order to gather some information about it. This results in the module that the flow is in being loaded, which can result in errors if not all the dependencies are present (issue [9512](https://github.com/PrefectHQ/prefect/issues/9512)). As a result, this action takes in a comma-seperated list of requirments to pre-load these ahead of running `prefect deploy`. This will **not** result in a generic image being created, but rather used to satisfy a pre-flight check required by the Prefect CLI. If building one or many custom docker images, those will still be isolated and only install the relevant dependencies defined as a part of your Dockerfile.
 
 ## Requirements
 
@@ -30,11 +29,10 @@ You will not need to pass your work-pool name or the deployment name to this act
 
 ## Inputs
 
-| Input | Desription | Required | Default |
-|-------|------------|----------|---------|
-| additional-args | Any additional arguments to pass to the Prefect Deploy command. Available additional arguments are listed below. | false | |
-| entrypoint | The path to a flow entrypoint within a project, in format: `./path/to/file.py:flow_func_name`. | true | |
-| requirements-file-path | Path to requirements files to correctly install dependencies for your Prefect flow. | false | `./requirements.txt` |
+| Input | Desription | Required |
+|-------|------------|----------|
+| deployment-names | Comma separated list of deployment names defined in the deployment.yaml file. | true |
+| requirements-file-paths | Comma sepearated list of paths to requirements files to correctly install dependencies for your Prefect flow(s). | false |
 
 ## Examples
 
@@ -64,12 +62,34 @@ jobs:
           prefect-workspace: ${{ secrets.PREFECT_WORKSPACE }}
 
       - name: Run Prefect Deploy
-        uses: PrefectHQ/actions-prefect-deploy@v2
+        uses: PrefectHQ/actions-prefect-deploy@v3
         with:
-          requirements-file-path: ./examples/simple/requirements.txt
-          entrypoint: ./examples/simple/flow.py:call_api
-          additional-args: --cron '30 19 * * 0'
+          deployment-names: Simple
+          requirements-file-paths: ./examples/simple/requirements.txt
 ```
+
+### Multi-Deployment Prefect Deploy
+
+Deploy multiple Prefect deployments that doesn't have a `push` step defined in the `prefect.yaml`
+```yaml
+name: Deploy multiple Prefect deployments
+on:
+  push:
+    branches:
+      - main
+jobs:
+  deploy_flow:
+    runs-on: ubuntu-latest
+    steps:
+      - ...
+
+      - name: Run Prefect Deploy
+        uses: PrefectHQ/actions-prefect-deploy@v3
+        with:
+          deployment-names: Simple_Deployment_1,Simple_Deployment_2
+          requirements-file-paths: ./examples/multi-deployment/deployment-1/requirements.txt,./examples/multi-deployment/deployment-2/requirements.txt
+```
+
 ### Basic Docker Auth w/ Prefect Deploy
 
 Deploy a Prefect flow and also build a Docker artifact that pushes to a defined repository in the `prefect.yaml` file.
@@ -101,11 +121,10 @@ jobs:
           prefect-workspace: ${{ secrets.PREFECT_WORKSPACE }}
 
       - name: Run Prefect Deploy
-        uses: PrefectHQ/actions-prefect-deploy@v2
+        uses: PrefectHQ/actions-prefect-deploy@v3
         with:
-          requirements-file-path: ./examples/docker/requirements.txt
-          entrypoint: ./examples/docker/flow.py:call_api
-          additional-args: --cron '30 19 * * 0' --pool docker-pool
+          deployment-names: Docker
+          requirements-file-paths: ./examples/docker/requirements.txt
 ```
 ### GCP Workload Identity w/ Prefect Deploy
 
@@ -144,30 +163,11 @@ jobs:
           prefect-workspace: ${{ secrets.PREFECT_WORKSPACE }}
 
       - name: Run Prefect Deploy
-        uses: PrefectHQ/actions-prefect-deploy@v2
+        uses: PrefectHQ/actions-prefect-deploy@v3
         with:
-          requirements-file-path: ./examples/docker/requirements.txt
-          entrypoint: ./examples/docker/flow.py:call_api
-          additional-args: --cron '30 19 * * 0' --pool docker-pool
+          deployment-names: Docker
+          requirements-file-paths: ./examples/docker/requirements.txt
 ```
-## Additional Arguments
-
-| Arg Name | Description | Example |
-|----------|-------------|---------|
-| --anchor-date | The anchor date for an interval schedule. | |
-| --cron | A cron string that will be used to set a CronSchedule on the deployment. | `--cron '30 19 * * 0' `|
-| --description | The description to give the deployment. If not provided, the description will be populated from the flow's description. | |
-| --interval | An integer specifying an interval (in seconds) that will be used to set an IntervalSchedule on the deployment. | `--interval 60` |
-| --name | The name to give the deployment. | `--name 'Test Flow'` |
-| --param | An optional parameter override, values are parsed as JSON strings | `--param question=ultimate --param answer=42` |
-| --params | An optional parameter override in a JSON string format. | `--params='{"question": "ultimate", "answer": 42}'` |
-| --pool | The work pool that will handle this deployment's runs. | `--pool docker-pool` |
-| --rrule | An RRule that will be used to set an RRuleSchedule on the deployment. | |
-| --tag | One or more optional tags to apply to the deployment - Note: tags are used only for organizational purposes. | |
-| --timezone | Deployment schedule timezone string. | `--timezone 'America/New_York'` |
-| --variable | One or more job variable overrides for the work pool. | `--variable foo=bar` |
-| --version | A version to give the deployment. | |
-| --work-queue | The work queue that will handle this deployment's runs. It will be created if it doesn't already exist. | `--work-queue test` |
 
 ## Terms & Conditions
 See here for the Prefect's [Terms and Conditions](https://www.prefect.io/legal/terms/).
diff --git a/action.yaml b/action.yaml
@@ -7,36 +7,46 @@ branding:
   color: blue
 
 inputs:
-  additional-args:
+  deployment-names:
     description:
-      Any additinal arguments that need to be passed to
-      the prefect deployment run in a space separated format.
-      i.e - `--variable foo=bar -t 1.0.0 --cron "* * * * *"`.
+      Comma separated list of names of your
+      Prefect deployments.
+      example 'deployment1,deployment2'
     required: false
+    default: deployment
 
-  entrypoint:
+  requirements-file-paths:
     description:
-      The path to a flow entrypoint within a project,
-      in format `./path/to/file.py:flow_func_name`.
-    required: true
-
-  requirements-file-path:
-    description:
-      Path to requirements files to correctly install
+      Comma separated list of paths to
+      requirements files to correctly install
       dependencies for your Prefect flow(s).
+      example './flow1/requirements.txt,./flow2/requirements.txt'
     required: false
-    default: ./requirements.txt
-
+    default: ''
 
 runs:
   using: composite
   steps:
     - id: install-local-requirements
-      run: pip install -r ${{ inputs.requirements-file-path }}
+      if: ${{ inputs.requirements-file-paths != '' }}
+      run: |
+        IFS=',' read -r -a requirements_paths <<< ${{ inputs.requirements-file-paths }}
+        for req in ${requirements_paths}; do
+          pip install -r $req
+        done
       shell: bash
 
     - id: prefect-deploy
       run: |
-        prefect deploy ${{ inputs.entrypoint }} \
-         ${{ inputs.additional-args }}
+        PREFECT_VERSION=$(prefect --version)
+        PREFECT_SHORT_VERSION="$(cut -d "." -f 2 <<< "$PREFECT_VERSION")"."$(cut -d "." -f 3 <<< "$PREFECT_VERSION")"
+        if [ $(echo $PREFECT_SHORT_VERSION'<'10.6 | bc -l) -eq 1 ];
+        then
+          echo "The currently installed Prefect version (2.$PREFECT_SHORT_VERSION) is not compatible. Updating to the latest Prefect version..."
+          pip install prefect -U "prefect>=2.10.8"
+        fi
+        IFS=',' read -r -a deployment_names <<< ${{ inputs.deployment-names }}
+        for name in ${deployment_names}; do
+          prefect deploy --name $name
+        done
       shell: bash
diff --git a/examples/docker/deployment.yaml b/examples/docker/deployment.yaml
@@ -1,13 +1,14 @@
-description: null
-entrypoint: examples/docker/flow.py:call_api
-flow_name: null
-name: null
-parameters: {}
-schedule: null
-tags: []
-version: null
-work_pool:
-  job_variables:
-    image: '{{ image_name }}'
-  name: null
-  work_queue_name: null
+deployments:
+  - name: Docker
+    description: null
+    entrypoint: examples/docker/flow.py:call_api
+    flow_name: null
+    parameters: {}
+    schedule: null
+    tags: []
+    version: null
+    work_pool:
+      job_variables:
+        image: '{{ image_name }}'
+      name: docker-pool
+      work_queue_name: null
diff --git a/examples/docker/requirements.txt b/examples/docker/requirements.txt
@@ -1 +1 @@
-prefect==2.10.4
+prefect==2.10.8
diff --git a/examples/multi-deployment/deployment-1/.prefectignore b/examples/multi-deployment/deployment-1/.prefectignore
@@ -0,0 +1,41 @@
+# prefect artifacts
+.prefectignore
+
+# python artifacts
+__pycache__/
+*.py[cod]
+*$py.class
+*.egg-info/
+*.egg
+
+# Type checking artifacts
+.mypy_cache/
+.dmypy.json
+dmypy.json
+.pyre/
+
+# IPython
+profile_default/
+ipython_config.py
+*.ipynb_checkpoints/*
+
+# Environments
+.python-version
+.env
+.venv
+env/
+venv/
+
+# MacOS
+.DS_Store
+
+# Dask
+dask-worker-space/
+
+# Editors
+.idea/
+.vscode/
+
+# VCS
+.git/
+.hg/
diff --git a/examples/multi-deployment/deployment-1/flow.py b/examples/multi-deployment/deployment-1/flow.py
@@ -0,0 +1,11 @@
+import json
+
+import requests
+from prefect import flow
+
+@flow(name="Simple Flow", log_prints=True)
+def call_api(url: str = "http://time.jsontest.com/"):
+    """Sends a GET request to the provided URL and returns the JSON response"""
+    resp = requests.get(url).json()
+    print(resp)
+    return resp
diff --git a/examples/multi-deployment/deployment-1/prefect.yaml b/examples/multi-deployment/deployment-1/prefect.yaml
@@ -0,0 +1,20 @@
+# File for configuring project / deployment build, push and pull steps
+
+# Generic metadata about this project
+name: simple
+prefect-version: 2.10.8
+
+# build section allows you to manage and build docker images
+build: null
+
+# push section allows you to manage if
+# and how this project is uploaded to remote locations
+push: null
+
+# pull section allows you to provide instructions
+# for cloning this project in remote locations
+pull:
+  - prefect.projects.steps.git_clone_project:
+      repository: [email protected]:PrefectHQ/actions-prefect-deploy.git
+      branch: main
+      access_token: "{{ prefect.blocks.secret.simple-github-pat }}"
diff --git a/examples/multi-deployment/deployment-1/requirements.txt b/examples/multi-deployment/deployment-1/requirements.txt
@@ -0,0 +1 @@
+prefect==2.10.8
diff --git a/examples/multi-deployment/deployment-2/.prefectignore b/examples/multi-deployment/deployment-2/.prefectignore
@@ -0,0 +1,41 @@
+# prefect artifacts
+.prefectignore
+
+# python artifacts
+__pycache__/
+*.py[cod]
+*$py.class
+*.egg-info/
+*.egg
+
+# Type checking artifacts
+.mypy_cache/
+.dmypy.json
+dmypy.json
+.pyre/
+
+# IPython
+profile_default/
+ipython_config.py
+*.ipynb_checkpoints/*
+
+# Environments
+.python-version
+.env
+.venv
+env/
+venv/
+
+# MacOS
+.DS_Store
+
+# Dask
+dask-worker-space/
+
+# Editors
+.idea/
+.vscode/
+
+# VCS
+.git/
+.hg/
diff --git a/examples/multi-deployment/deployment-2/flow.py b/examples/multi-deployment/deployment-2/flow.py
@@ -0,0 +1,11 @@
+import json
+
+import requests
+from prefect import flow
+
+@flow(name="Simple Flow", log_prints=True)
+def call_api(url: str = "http://time.jsontest.com/"):
+    """Sends a GET request to the provided URL and returns the JSON response"""
+    resp = requests.get(url).json()
+    print(resp)
+    return resp
diff --git a/examples/multi-deployment/deployment-2/prefect.yaml b/examples/multi-deployment/deployment-2/prefect.yaml
@@ -0,0 +1,20 @@
+# File for configuring project / deployment build, push and pull steps
+
+# Generic metadata about this project
+name: simple
+prefect-version: 2.10.8
+
+# build section allows you to manage and build docker images
+build: null
+
+# push section allows you to manage if
+# and how this project is uploaded to remote locations
+push: null
+
+# pull section allows you to provide instructions
+# for cloning this project in remote locations
+pull:
+  - prefect.projects.steps.git_clone_project:
+      repository: [email protected]:PrefectHQ/actions-prefect-deploy.git
+      branch: main
+      access_token: "{{ prefect.blocks.secret.simple-github-pat }}"
diff --git a/examples/multi-deployment/deployment-2/requirements.txt b/examples/multi-deployment/deployment-2/requirements.txt
@@ -0,0 +1 @@
+prefect==2.10.8