Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

3223 - Terraform Infrastructure #3387

Draft
wants to merge 46 commits into
base: develop
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
46 commits
Select commit Hold shift + click to select a range
2eb6de0
Moving workflows around into env blocks
andrew-jameson Dec 19, 2024
6256949
Stacking terraform after deploy jobs
andrew-jameson Dec 19, 2024
f87f7a0
minor whitespace update
andrew-jameson Dec 19, 2024
747a609
Nulling cf cmds in deploy scripts
andrew-jameson Dec 19, 2024
cf3d81f
altering github webhooks for streamlined circleci behavior
andrew-jameson Dec 19, 2024
d279d52
requires e-v
andrew-jameson Dec 19, 2024
425e7bc
streamlining circleci didn't take, trying again
andrew-jameson Dec 19, 2024
86b30bc
deploy on push to this branch
andrew-jameson Dec 19, 2024
9793ce3
whitespace to trigger deploy
andrew-jameson Dec 19, 2024
37e01c1
reverting to label
andrew-jameson Dec 19, 2024
090aa19
apk doesn't need a 'y'
andrew-jameson Dec 19, 2024
1244be4
instead of converting everything to alpine, maybe just use docker
andrew-jameson Dec 19, 2024
767ea55
install tf on ubuntu executor
andrew-jameson Dec 19, 2024
b8db872
bash syntax placeholder
andrew-jameson Dec 19, 2024
be6d225
retrying deploy on push
andrew-jameson Dec 19, 2024
9d0acc5
deploy-on-push
andrew-jameson Dec 19, 2024
9ad1f7e
getting ref/branch
andrew-jameson Dec 19, 2024
d8a7066
giving up on push
andrew-jameson Dec 19, 2024
0673f81
running init manually
andrew-jameson Dec 30, 2024
5ae7985
typo
andrew-jameson Dec 30, 2024
f1eeb9f
checking plan works
andrew-jameson Dec 30, 2024
1cb7d7c
our terraform is out of date, downgrading installed ver for compatibi…
andrew-jameson Dec 30, 2024
4a9ca94
missing > syntax
andrew-jameson Dec 30, 2024
3dc4310
another syntax
andrew-jameson Dec 30, 2024
509e67e
env
andrew-jameson Dec 30, 2024
a164899
debugging pathing
andrew-jameson Dec 31, 2024
7afd842
fixed, removing debugging
andrew-jameson Dec 31, 2024
f2be7bb
a
andrew-jameson Dec 31, 2024
1c4dd4b
dir change
andrew-jameson Dec 31, 2024
465fbee
with successful pipeline, let's bring in app changes from old branch
andrew-jameson Dec 31, 2024
852d6ce
strat
andrew-jameson Dec 31, 2024
e756a43
And finally, apply
andrew-jameson Dec 31, 2024
cfaa234
its missing login info?
andrew-jameson Dec 31, 2024
19a7783
frontend path
andrew-jameson Jan 1, 2025
60ed96d
deploy script was removing directory it built
andrew-jameson Jan 1, 2025
97d8db1
pinning version, our nginx cfg is out of date
andrew-jameson Jan 1, 2025
2db79dd
need environment variables or nginx crashes
andrew-jameson Jan 1, 2025
2e51a4c
env syntax
andrew-jameson Jan 1, 2025
7e0bd0f
variables within variables
andrew-jameson Jan 1, 2025
4914a2a
hardcoding, self-referencing isn't allowed
andrew-jameson Jan 1, 2025
0f6136d
now lets try adding backend
andrew-jameson Jan 1, 2025
edc3a35
missed quotes on backend
andrew-jameson Jan 1, 2025
24d819f
only double quotes
andrew-jameson Jan 1, 2025
37301b2
adding command
andrew-jameson Jan 1, 2025
cdb0d54
buildpacks keyword wasnt found. can i just use a list on non-s one?
andrew-jameson Jan 1, 2025
dc9c049
switching to only one
andrew-jameson Jan 1, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
56 changes: 30 additions & 26 deletions .circleci/deployment/commands.yml
Original file line number Diff line number Diff line change
Expand Up @@ -117,18 +117,12 @@
steps:
- get-app-deploy-strategy:
appname: <<parameters.backend-appname>>
- run:
name: Install dependencies
command: |
sudo apt update
sudo apt-get install -y wget
sudo wget -qO /usr/local/bin/yq https://github.com/mikefarah/yq/releases/download/v4.42.1/yq_linux_amd64 && chmod +x /usr/local/bin/yq
- run:
name: Apply database migrations
command: |
bash ./scripts/apply-remote-migrations.sh <<parameters.backend-appname>>
- run:
name: Deploy backend application
name: Prepare backend application
command: |
bash ./scripts/deploy-backend.sh \
$DEPLOY_STRATEGY \
Expand Down Expand Up @@ -177,7 +171,7 @@
- get-app-deploy-strategy:
appname: <<parameters.frontend-appname>>
- run:
name: Deploy frontend application
name: Prepare frontend application
command: |
bash ./scripts/deploy-frontend.sh \
$DEPLOY_STRATEGY \
Expand Down Expand Up @@ -235,36 +229,46 @@
- run:
name: Export S3 Credentials for TFState
command: |
strip() {
# Usage: strip "string" "pattern"
printf '%s\n' "${1##$2}"
}
# The cloud.gov space defined via environment variable (e.g., "tanf-dev", "tanf-staging")
env=$(strip <<parameters.cf-space>> "tanf-")

S3_CREDENTIALS=$(cf service-key tdp-tf-states tdp-tf-key | tail -n +2)
{
echo "access_key = \"$(echo "${S3_CREDENTIALS}" | jq -r .access_key_id)\""
echo "secret_key = \"$(echo "${S3_CREDENTIALS}" | jq -r .secret_access_key)\""
echo "region = \"$(echo "${S3_CREDENTIALS}" | jq -r '.region')\""
echo "bucket = \"$(echo "${S3_CREDENTIALS}" | jq -r '.bucket')\""
} >> ./backend_config.tfvars
- run:
name: Prepare Terraform Variables
command: |
} >> ./terraform/$env/backend_config.tfvars

S3_CREDENTIALS=$(cf service-key tdp-tf-states tdp-tf-key | tail -n +2)
{
echo "cf_password = \"$<<parameters.cf-password>>\""
echo "cf_user = \"$<<parameters.cf-username>>\""
echo "cf_space_name = \"<<parameters.cf-space>>\""
echo "cf_app_name = \"<<parameters.cf-app>>\""
} >> ./variables.tfvars
- terraform/init:
path: <<parameters.tf-path>>
backend_config_file: ./backend_config.tfvars
- terraform/validate:
path: <<parameters.tf-path>>
- terraform/fmt:
path: <<parameters.tf-path>>
- terraform/plan:
path: <<parameters.tf-path>>
var_file: ./variables.tfvars
- terraform/apply:
path: <<parameters.tf-path>>
var_file: ./variables.tfvars
} >> ./terraform/$env/variables.tfvars
- run:
name: Terraform Check
command: |
bash ./scripts/terraform-check.sh
- run:
name: Terraform Init
command: |
strip() {
# Usage: strip "string" "pattern"
printf '%s\n' "${1##$2}"
}
# The cloud.gov space defined via environment variable (e.g., "tanf-dev", "tanf-staging")
env=$(strip <<parameters.cf-space>> "tanf-")
cd ./terraform/$env
terraform init -backend-config backend_config.tfvars -var-file variables.tfvars
terraform validate
terraform plan -out tfapply -var-file ./variables.tfvars
terraform apply tfapply

enable-s3-versioning:
parameters:
Expand Down
13 changes: 4 additions & 9 deletions .circleci/deployment/jobs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,8 @@
- deploy-cloud-dot-gov:
backend-appname: tdp-backend-<< parameters.target_env >>
frontend-appname: tdp-frontend-<< parameters.target_env >>
- deploy-infrastructure:
cf-app: << parameters.target_env >>

deploy-staging:
executor: docker-executor
Expand All @@ -64,15 +66,8 @@
cf-space: tanf-staging
cf-username: CF_USERNAME_STAGING

deploy-infrastructure-dev:
parameters:
target_env:
type: string
executor: terraform/default
working_directory: ~/tdp-deploy
steps:
- deploy-infrastructure:
cf-app: << parameters.target_env >>



deploy-infrastructure-staging:
executor: terraform/default
Expand Down
106 changes: 52 additions & 54 deletions .circleci/deployment/workflows.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,39 +10,38 @@
- not: << pipeline.parameters.run_nightly_owasp_scan >>

jobs:
- deploy-project-updates-site:
# Dev environments
- deploy-dev:
target_env: << pipeline.parameters.target_env >>
filters:
branches:
only:
ignore:
- develop
- deploy-infrastructure-dev:
target_env: << pipeline.parameters.target_env >>
- main
- master

- enable-versioning:
requires:
- deploy-dev
filters:
branches:
ignore:
- develop
- main
- master

# Develop
- deploy-infrastructure-staging:
filters:
branches:
only:
- develop
- main
- deploy-infrastructure-production:
- deploy-project-updates-site:
filters:
branches:
only:
- master
- enable-versioning:
requires:
- deploy-infrastructure-dev
filters:
branches:
ignore:
- develop
- main
- master
- enable-versioning:
requires:
- deploy-infrastructure-staging
Expand All @@ -54,94 +53,93 @@
branches:
only:
- develop
- enable-versioning:
- build-and-tag-develop:
requires:
- deploy-infrastructure-staging
target_env: staging
cf-password: CF_PASSWORD_STAGING
cf-username: CF_USERNAME_STAGING
cf-space: tanf-staging
filters:
branches:
only:
- main
- enable-versioning:
- develop
- deploy-develop:
requires:
- deploy-infrastructure-production
target_env: prod
cf-password: CF_PASSWORD_PROD
cf-username: CF_USERNAME_PROD
cf-space: tanf-prod
- build-and-tag-develop
filters:
branches:
only:
- master
- prod-deploy-clamav:
- develop
- test-deployment-e2e:
requires:
- deploy-infrastructure-production
- deploy-develop
filters:
branches:
only:
- master
- deploy-dev:
target_env: << pipeline.parameters.target_env >>
- develop

# Staging
- build-and-tag-staging:
requires:
- deploy-infrastructure-dev
- deploy-infrastructure-staging
filters:
branches:
ignore:
- develop
only:
- main
- master
- build-and-tag-develop:
- enable-versioning:
requires:
- deploy-infrastructure-staging
target_env: staging
cf-password: CF_PASSWORD_STAGING
cf-username: CF_USERNAME_STAGING
cf-space: tanf-staging
filters:
branches:
only:
- develop
- deploy-develop:
- main
- deploy-staging:
requires:
- build-and-tag-develop
- build-and-tag-staging
filters:
branches:
only:
- develop
- build-and-tag-staging:
requires:
- deploy-infrastructure-staging
- main

# Production
- deploy-infrastructure-production:
filters:
branches:
only:
- main
- deploy-staging:
- master
- enable-versioning:
requires:
- build-and-tag-staging
- deploy-infrastructure-production
target_env: prod
cf-password: CF_PASSWORD_PROD
cf-username: CF_USERNAME_PROD
cf-space: tanf-prod
filters:
branches:
only:
- main
- build-and-tag-production:
- master
- prod-deploy-clamav:
requires:
- deploy-infrastructure-production
filters:
branches:
only:
- master
- deploy-production:
- build-and-tag-production:
requires:
- build-and-tag-production
- deploy-infrastructure-production
filters:
branches:
only:
- master
- test-deployment-e2e:
- deploy-production:
requires:
- deploy-develop
- build-and-tag-production
filters:
branches:
only:
- develop
- master
- make_erd: # from ../util folder
filters:
branches:
Expand Down
3 changes: 2 additions & 1 deletion .github/workflows/build-all.yml
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@ on:
- main
- master
- 'release/**'
- 'feat/3223-terraform-cf'
paths:
- 'scripts/**'
- '.circleci/**'
Expand All @@ -53,6 +54,6 @@ jobs:
project-slug: ${{ github.repository }}
branch: ${{ (github.event_name == 'pull_request') && github.head_ref || github.ref_name }}
payload: '{
"build_and_test_all": true,
"build_and_test_all": false,
"triggered": true
}'
3 changes: 2 additions & 1 deletion .github/workflows/build-pr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ on:
- main
- master
- 'release/**'
- 'feat/3223-terraform-cf'
types: [review_requested, ready_for_review, synchronize]
jobs:
build_and_test_pr:
Expand All @@ -42,6 +43,6 @@ jobs:
project-slug: ${{ github.repository }}
branch: ${{ (github.event_name == 'pull_request') && github.head_ref || github.ref_name }}
payload: '{
"build_and_test_all": true,
"build_and_test_all": false,
"triggered": true
}'
2 changes: 1 addition & 1 deletion .github/workflows/deploy-on-label.yml
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,7 @@ jobs:

- name: Circle CI Deployment Trigger
id: curl-circle-ci
if: steps.get-pr-state.outputs.STATE == 'success'
#if: steps.get-pr-state.outputs.STATE == 'success'
uses: promiseofcake/circleci-trigger-action@v1
with:
user-token: ${{ secrets.CIRCLE_CI_V2_TOKEN }}
Expand Down
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,7 @@ npm-debug.log
/output/
/cache/
boilerplate.zip
.circleci/generated_config.yml

# Spritesmith
spritesmith-generated/
Expand Down Expand Up @@ -117,3 +118,4 @@ cypress.env.json
# DB seeds
tdrs-backend/*.pg
tdrs-backend/django.log

6 changes: 4 additions & 2 deletions scripts/apply-remote-migrations.sh
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,10 @@ app=${1}
cd ./tdrs-backend

echo "Install dependencies..."
sudo apt-get install -y gcc && sudo apt-get install -y graphviz && sudo apt-get install -y graphviz-dev
sudo apt install -y libpq-dev python3-dev
sudo apt-get install -y gcc wget graphviz graphviz-dev libpq-dev python3-dev
sudo apt update
sudo wget -qO /usr/local/bin/yq https://github.com/mikefarah/yq/releases/download/v4.42.1/yq_linux_amd64 && chmod +x /usr/local/bin/yq


python -m venv ./env
source ./env/bin/activate
Expand Down
Loading
Loading