feat: Terraform & GitHub Actions (#351)

* More backend module setup

* Move RDS

* Serverless DB config

* Scaling config

* Setup lambda

* Lambda setup

* More setup

* Build setup

* Move S3

* Lambda terraform

* Lambda stuff

* Variables for lambda deploy

* Unquote vars

* Add iam

* TF cleanup, path fixes

* TF vars

* Fix lambda build

* Fix build script

* More lambda setup

* version missmatch: apollo-server-fastify doesn't support fastify v4

* Move to express from fastify

* Express serverless initial setup

* fixed destroy script

* Express fixes

* Set up listener

* Fix express deploy

* Lambda URL setup

* Fix endpoint record for serverless

* Start API gateway setup

* API Gateway setup

* Add comments

* Regional module setup

* Separate VPC for lambda

* Add separate cert module

* API Gateway contents setup

* Fix API Gateway

* Working link from API gateway to domain

* Improve serverless API deploy

* Initial CORS setup for API gateway

* Fix dependencies, still issues on POST

* CORS stuff

* Docs

* Fix weird dependencies

* Resource changes & logging for API gateway

* Fix cyclic dependency

* Fix API gateway logging

* JWTStrategy fix for serverless

* Logging for testing

* Switch to serverless RDS v1

* Add optional serverless v2

* Fix v2

* Fix v2

* JWT testing setup

* Current state

* Make lambda private

* Move options cors to proxy

* Remove cors

* Serverless update

* Fix zip path

* Add hash output

* Fix update script

* More update script fixes

* Fix lambda redeploy

* Cleanup

* Cleanup

* Add webpack for lambda, not done yet

* Basic webpack config

* Webpack cleanup

* Change lambda memory

* Lambda optimization

* Set up per-mode configuration of infrastructure

* Fix db config

* Prepare setup

* Major script cleanup

* Fix type

* Remove todo

* Fix update, fix caching

* Add invalidation to cloudfront distribution

* Add dev update github action

* Fix script path

* Add commands

* switch invalidation to null resource

* step 1 is now also applied for dev

* added workflow images

* added better images

* Fix cache invalidation

* Fix cache invalidation

* Fix SPA frontend redeploy

* Fix template

* Config

* Fix destroy script

* Fix path in action

* Add login

* RM login

* Add inits

* rm id

* Remove init

* Fix zsh

* use bash

* Add ts-node dep, use sudo for bash

* Sudo node-prune

* ts-node

* Add typescript

* More sudo

* Test init

* Validate

* Login

* Use setup v2

* use setup

* test init

* test config

* Try explicit shell

* Manual login

* LS test

* cat file

* Test file path

* Log test

* logging

* Move export

* add token

* test log

* Add more test content

* more test

* Folder test

* Add ls

* multi ls

* fix

* Add sudo

* config file location echo

* config file location echo

* Add tf cli config file

* Remove quotes

* Try with no sudo

* Cleanup

* remove wrapper

* Add existence check in script

* Package

* fix force

* shorten desc

* Fix online status check

* Add https check

* add error codes

* Fix config

* Per-mode backup manager name

* Destroy script

* more logs

* condition fix

* Fix ordering

* Fix destroy script for action

* Add init

* fix output path

* Fix paths

* Fix final snapshot skip

* Fix order of build in destroy script

* Switch redirect-record config

* Make build sudo in destroy

* Better error message

* Fix depends-on

* Fix log exporter naming

* Modularize frontend subnet, chmod node_modules

* change package.json

* Change webpack config

* move lambda entrypoint

* Fix lambda path

* change webpack config

* Try other config

* Test don't use webpack

* main path

* Revert "main path"

This reverts commit 2f13df2.

* Revert "Test don't use webpack"

This reverts commit 10edbb6.

* Revert "Try other config"

This reverts commit faf67aa.

* Revert "change webpack config"

This reverts commit 6fbddaf.

* Revert "Fix lambda path"

This reverts commit da22c0f.

* Revert "move lambda entrypoint"

This reverts commit dd0e6ff.

* Revert "Change webpack config"

This reverts commit 155a8f1.

* Revert "change package.json"

This reverts commit d995a28.

* Revert "Modularize frontend subnet, chmod node_modules"

This reverts commit 415bcc4.

* cleanup

* Remove webpack

* Add more confirmation for force deployment

* Confirm not required

* Fix path in live mode

* Fix typo in path

* Change live path for testing

* Fix security groups

* add echo

* Fix text entry

* fix input more

* Update node

* Correct subnets for API EBS

* Enable destruction of live system

* Change test config

* Fix destroy action input

* Remove online check for customer-facing systems

* live/test check on initial deploy as well

* Release scripts setup

* Update deployment actions

* Test update

* Fix path

* Fix shell

* Update setup terraform

* remove sudo in actions

* sudo

* Test update

* Try modifying cert

* Docs, fix cert

* Add log

* remove automation on branch

* Fix jq path

* use actual cert

* feat: Change from vuex to pinia (#353)

* Placeholder workflow

* Placeholder workflow contents

* Add files via upload (#352)

* Add workflow

* reÃplaced vuex with pinia

* fixed quote issue

* Removed vue app email

Co-authored-by: David Wyss <[email protected]>

* PR fixes

* Update frontend/src/stores/index.ts

Co-authored-by: davwys <[email protected]>

* Update backend/src/flox/modules/auth/auth.guard.ts

Co-authored-by: Marinolino <[email protected]>

* Update frontend/src/data/readme.md

Co-authored-by: Marinolino <[email protected]>

* Update scripts/aws-update/aws-update.sh

Co-authored-by: Marinolino <[email protected]>

* Update scripts/aws-update/aws-update.sh

Co-authored-by: Marinolino <[email protected]>

* Update scripts/aws-initial-setup/2_main-setup/api-ebs/variables.tf

Co-authored-by: Marinolino <[email protected]>

* Update scripts/aws-initial-setup/2_main-setup/api-ebs/variables.tf

Co-authored-by: Marinolino <[email protected]>

* Update scripts/support/destroy-system.sh

Co-authored-by: Marinolino <[email protected]>

* Update scripts/support/destroy-system.sh

Co-authored-by: Marinolino <[email protected]>

* Update scripts/support/destroy-system.sh

Co-authored-by: Marinolino <[email protected]>

* Update scripts/aws-initial-setup/2_main-setup/api-serverless/variables.tf

Co-authored-by: Marinolino <[email protected]>

* Update scripts/aws-initial-setup/2_main-setup/api-serverless/variables.tf

Co-authored-by: Marinolino <[email protected]>

* Update scripts/aws-initial-setup/initial-aws-setup.sh

Co-authored-by: Marinolino <[email protected]>

* Update scripts/aws-initial-setup/initial-aws-setup.sh

Co-authored-by: Marinolino <[email protected]>

* PR changes

* Fix tests for user module

* Centralize test utils

* Remove required keys, add TODO

* feat: GitHub Actions for Staging Workflow (#354)

* staging workflows

* Update .github/workflows/remove-stage.yml

Co-authored-by: David Wyss <[email protected]>

* PR fixes

* PR final fixes

* Update .github/workflows/initial-setup.yml

Co-authored-by: David Wyss <[email protected]>

* Fix type

* Fix order

* Remove fastify

* Fix import

* shorten bucket-prefix

* changed stage branch name generation to fewer chars

* Fix regex

* Fix script modes

* Fix destroy scripts

* comment

* remove destroy config

Co-authored-by: Johann Schwabe <[email protected]>
Co-authored-by: Marinolino <[email protected]>
Co-authored-by: Marinolino <[email protected]>

.env

@@ -1,3 +1,4 @@
+PROJECT_NAME=flox
 SERVER_PORT=3000
 FRONTEND_PORT=3001
 DB_DATABASE=app_db # TODO: Application specific

.github/workflows/destroy-system.yml

@@ -1,7 +1,8 @@
-name: Destroy AWS
+name: Destroy AWS infrastructure
 
+# Should only be run manually from the Actions tab
+# ATTENTION: this system may destroy infrastructure seen by customers!
 on:
-  # Testing only
   workflow_dispatch:
     inputs:
       mode:
@@ -11,9 +12,44 @@ on:
         options:
           - test
           - dev
+          - live
+          - stage
+      force:
+        required: false
+        type: boolean
+        description: Force destruction
+        default: false
+      confirm:
+        required: false
+        type: string
+        description: Enter "confirm" to confirm you are aware that DATA LOSS, INCLUDING ACCOUNTS, will occur when forcing destruction!
+      stage-branch-name:
+        required: false
+        type: string
+        description: Name of the staging branch to destroy in 'stage' mode
+
 jobs:
-  setup-aws:
+  destroy-aws:
     runs-on: ubuntu-latest
     steps:
-      - name: Test
-        run: echo "Testing!"
+      - name: Checkout 🛎
+        uses: actions/checkout@v3
+
+      - name: Make scripts executable
+        run: |
+          sudo chmod +x ./scripts/support/build.sh
+          sudo chmod +x ./scripts/support/destroy-system.sh
+
+      - name: Set up Terraform
+        uses: hashicorp/setup-terraform@v2
+        with:
+          cli_config_credentials_token: ${{ secrets.TF_API_TOKEN }}
+          terraform_wrapper: false
+
+      - name: Validate Terraform
+        working-directory: ./scripts/support
+        run: terraform validate
+
+      - name: Destroy AWS infrastructure
+        working-directory: ./scripts/support
+        run: bash destroy-system.sh ${{ github.event.inputs.mode }} false ${{ github.event.inputs.force }} ${{ github.event.inputs.confirm }} ${{ github.event.inputs.stage-branch-name }}

.github/workflows/initial-setup.yml

@@ -1,7 +1,7 @@
-name: Initial AWS setup
+name: Set up AWS infrastructure
 
+# Should only be run manually from the Actions tab (only once!)
 on:
-  # Testing only
   workflow_dispatch:
     inputs:
       mode:
@@ -10,11 +10,40 @@ on:
         description: System deployment mode
         options:
           - test
-          - live
           - dev
+          - live
+          - stage
+      force:
+        required: false
+        type: boolean
+        description: Force deployment
+        default: false
+      confirm:
+        required: false
+        type: string
+        description: Enter "confirm" to confirm you are aware that DATA LOSS, INCLUDING ACCOUNTS, may occur when forcing deployment!
 jobs:
   setup-aws:
     runs-on: ubuntu-latest
     steps:
-      - name: Test
-        run: echo "Testing!"
+      - name: Checkout 🛎
+        uses: actions/checkout@v3
+
+      - name: Make scripts executable
+        run: |
+          sudo chmod +x ./scripts/support/build.sh
+          sudo chmod +x ./scripts/aws-initial-setup/initial-aws-setup.sh
+
+      - name: Set up Terraform
+        uses: hashicorp/setup-terraform@v2
+        with:
+          cli_config_credentials_token: ${{ secrets.TF_API_TOKEN }}
+          terraform_wrapper: false
+
+      - name: Validate Terraform
+        working-directory: ./scripts/aws-initial-setup
+        run: terraform validate
+
+      - name: Build & set up AWS infrastructure
+        working-directory: ./scripts/aws-initial-setup
+        run: bash initial-aws-setup.sh ${{ github.event.inputs.mode }} false ${{ github.event.inputs.force }} ${{ github.event.inputs.confirm }}

.github/workflows/manually-tag-branch.yml

@@ -1,11 +1,31 @@
-name: Create draft release
+name: "Stage 3: Manually Tag a branch as release, which will result in an automated draft Release"
 
 on:
-  # Testing only
   workflow_dispatch:
+    inputs:
+      logLevel:
+        description: 'Log level'
+        required: true
+        default: 'warning'
+      version:
+        description: 'Version for release draft. Example: "1.2203.1" -> v1.2203.1'
+        required: true
+      postfix:
+        description: 'Optional postfix. Example: "-beta-1" -> v1.2203.1-beta-1'
+        required: false
+
 jobs:
-  setup-aws:
+  add-tag:
+    name: "Add tag to specified branch, triggering a release draft"
     runs-on: ubuntu-latest
     steps:
-      - name: Test
-        run: echo "Testing!"
+      - name: Checkout
+        uses: actions/checkout@v3
+        with:
+          token: ${{ secrets.TAG_TOKEN }}
+
+      - name: Create and push new Tag
+        run: |
+          echo "Log level: v${{ github.event.inputs.version }}${{ github.event.inputs.postfix }}"
+          git tag v${{ github.event.inputs.version }}${{ github.event.inputs.postfix }}
+          git push origin v${{ github.event.inputs.version }}${{ github.event.inputs.postfix }}

.github/workflows/merge-master-in-dev.yml

@@ -0,0 +1,45 @@
+name: "Merges Master into Dev on update, will open a PR if a merge conflict exists"
+
+on:
+  push:
+    branches:
+      - master
+
+jobs:
+  merge-master-to-dev:
+    runs-on: ubuntu-latest
+
+    steps:
+      - name: Checkout 🛎
+      - uses: actions/checkout@v3
+
+      - name: Make scripts executable
+        run:  sudo chmod +x ./scripts/support/merge-master-into-dev.sh
+
+      - name: Set Git config
+        id: setup
+        run: |
+          git config --local user.email "[email protected]"
+          git config --local user.name "GitHub Action"
+
+      - name: Merge master back to dev
+        id: merge
+        continue-on-error: true
+        run: bash merge-master-into-dev.sh
+
+      - name: Merge successful
+        id: push
+        if: steps.merge.conclusion == success
+        run: git push
+
+      - name: Merge conflict
+        id: solve_conflict
+        if: steps.merge.conclusion != success
+        run: 'gh pr create -B "dev" -t "stage: Auto-update dev from master" -b "Resolve merge conflict" -r polygon-software/team/developers'
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+
+      - name: PR already exists
+        id: error
+        if: steps.solve_conflict.conclusion != success
+        run: echo A PR from master to dev already exists!

.github/workflows/monday-dev-pr-review.yml

@@ -0,0 +1,50 @@
+name: "PR: Update Label on monday.com items based on PR Status"
+
+on:
+  pull_request_review:
+    types: [submitted, edited]
+
+jobs:
+  set-status:
+    name: "Update monday.com Labels to 'PR: Commented' / 'PR: Approved' / 'PR: Rejected'"
+    runs-on: ubuntu-latest
+    steps:
+      # If PR was 'COMMENTED'
+      - name: "Update state of monday.com items to 'PR: Commented'"
+        uses: polygon-software/action-monday-state@main
+        if: ${{ github.event.review.state == 'commented' && github.base_ref != 'master' }}
+        with:
+          monday-token: ${{ secrets.MONDAY_TOKEN }}
+          allow-no-item-id: true
+          multiple: true
+          text: ${{ github.event.pull_request.title }}
+          prefix: '#'
+          status-column-title: 'Item Status!'
+          set-status: 'PR: Commented'
+
+      # If PR was 'APPROVED'
+      - name: "Update state of monday.com items to 'PR: Approved'"
+        uses: polygon-software/action-monday-state@main
+        if: ${{ github.event.review.state == 'approved' && github.base_ref != 'master' }}
+        with:
+          monday-token: ${{ secrets.MONDAY_TOKEN }}
+          multiple: true
+          allow-no-item-id: true
+          text: ${{ github.event.pull_request.title }}
+          prefix: '#'
+          status-column-title: 'Item Status!'
+          set-status: 'PR: Approved'
+
+      # If PR was 'REJECTED'
+      - name: "Update state of monday.com items to 'PR: Rejected'"
+        uses: polygon-software/action-monday-state@main
+        if: ${{ github.event.review.state == 'rejected' && github.base_ref != 'master' }}
+        with:
+          monday-token: ${{ secrets.MONDAY_TOKEN }}
+          multiple: true
+          allow-no-item-id: true
+          text: ${{ github.event.pull_request.title }}
+          prefix: '#'
+          status-column-title: 'Item Status!'
+          set-status: 'PR: Rejected'
+

.github/workflows/monday-dev-pr.yml

@@ -0,0 +1,31 @@
+name: "PR: Label monday.com items in a dev PR as 'PR: In Review'"
+
+on:
+  pull_request:
+    branches: [dev]
+
+jobs:
+  set-status:
+    runs-on: ubuntu-latest
+    steps:
+      # Set status to 'PR: In Review' if base branch is not 'master' branch
+      - name: "Update state of monday.com items to 'PR: In Review'"
+        id: monday-state
+        uses: polygon-software/action-monday-state@main
+        if: ${{ github.base_ref != 'master' }}
+        with:
+          monday-token: ${{ secrets.MONDAY_TOKEN }}
+          multiple: true
+          allow-no-item-id: true
+          monday-organization: polygonsoftware
+          text: ${{ github.event.pull_request.title }}
+          prefix: '#'
+          status-column-title: 'Item Status!'
+          set-status: 'PR: In Review'
+
+      - name: Comment PR
+        uses: polygon-software/actions-comment-pull-request@v1
+        if: ${{ github.base_ref != 'master' }}
+        with:
+          message: ${{ steps.monday-state.outputs.message }}
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

.github/workflows/monday-master-pr.yml

@@ -0,0 +1,29 @@
+name: "Stage 1: Label monday.com items in a master PR as 'Stage 1: Staged'"
+
+on:
+  pull_request:
+    branches:
+      - master
+jobs:
+  set-status:
+    runs-on: ubuntu-latest
+    steps:
+      # The monday.com item IDs must appear in the Pull-Requests Body / Description. Take the commit messages
+      - name: "Update state of monday.com items to 'Stage 1: Staged'"
+        id: monday-state
+        uses: polygon-software/action-monday-state@main
+        with:
+          monday-token: ${{ secrets.MONDAY_TOKEN }}
+          allow-no-item-id: true
+          multiple: true
+          monday-organization: polygonsoftware
+          text: ${{ github.event.pull_request.body }}
+          prefix: '#'
+          status-column-title: 'Item Status!'
+          set-status: 'Stage 1: Staged'
+
+      - name: Comment PR
+        uses: polygon-software/actions-comment-pull-request@v1
+        with:
+          message: ${{ steps.monday-state.outputs.message }}
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

.github/workflows/monday-push-dev.yml

@@ -0,0 +1,24 @@
+name: "Stage 0: Label monday.com items arriving in DEV as 'Stage 0: Dev'"
+
+on:
+  push:
+    branches:
+      - dev
+
+jobs:
+  set-status:
+    name: Set monday.com Item Status
+    runs-on: ubuntu-latest
+    steps:
+      # Set status on-dev for items that appear in a commit message in DEV
+      - name: "Update state of monday.com items to 'State 0: Dev'"
+        uses: polygon-software/action-monday-state@main
+        with:
+          monday-token: ${{ secrets.MONDAY_TOKEN }}
+          allow-no-item-id: true
+          multiple: true
+          text: ${{ join(github.event.commits.*.message, ', ') }}
+          prefix: '#'
+          status-column-title: 'Item Status!'
+          set-status: 'State 0: Dev'
+

.github/workflows/monday-push-master.yml

@@ -0,0 +1,24 @@
+name: "Stage 2: Label monday.com items arriving in DEV as 'Stage 2: Master'"
+
+on:
+  push:
+    branches:
+      - master
+
+jobs:
+  set-status:
+    name: Set monday.com Item Status
+    runs-on: ubuntu-latest
+    steps:
+      # Set status on-dev for items that appear in a commit message in DEV
+      - name: "Update state of monday.com items to 'Stage 2: Master'"
+        uses: polygon-software/action-monday-state@main
+        with:
+          monday-token: ${{ secrets.MONDAY_TOKEN }}
+          allow-no-item-id: true
+          multiple: true
+          text: ${{ join(github.event.commits.*.message, ', ') }}
+          prefix: '#'
+          status-column-title: 'Item Status!'
+          set-status: 'Stage 2: Master'
+

.github/workflows/monday-push-stage.yml

@@ -0,0 +1,24 @@
+name: "Stage 1: Label monday.com items arriving in stage as 'Stage 1: Staged'"
+
+on:
+  push:
+    branches:
+      - stage
+
+jobs:
+  set-status:
+    name: Set monday.com Item Status
+    runs-on: ubuntu-latest
+    steps:
+      # Set status on-stage for items that appear in a commit message in stage
+      - name: "Update state of monday.com items to 'State 1: Staged'"
+        uses: polygon-software/action-monday-state@main
+        with:
+          monday-token: ${{ secrets.MONDAY_TOKEN }}
+          allow-no-item-id: true
+          multiple: true
+          text: ${{ join(github.event.commits.*.message, ', ') }}
+          prefix: '#'
+          status-column-title: 'Item Status!'
+          set-status: 'State 1: Staged'
+