Use secrets in bqetl_marketing_suppression_list DAG #6247

scholtzan · 2024-09-23T21:14:29Z

Description

This rewrites some of the DAG configurations to use secrets instead of variables.

bqetl_marketing_suppression_list__campaign_monitor_api_key needs to be added to GSM. This should be set to the same value as var.value.campaign_monitor_api_key. I already added the client_id to GSM

Reviewer, please follow this checklist

┆Issue is synchronized with this Jira Task

dataops-ci-bot · 2024-09-23T21:21:27Z

Integration report for "Use secrets in bqetl_marketing_suppression_list DAG"

`sql.diff`

Click to expand!

diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/dags/bqetl_marketing_suppression_list.py /tmp/workspace/generated-sql/dags/bqetl_marketing_suppression_list.py
--- /tmp/workspace/main-generated-sql/dags/bqetl_marketing_suppression_list.py	2024-09-23 21:18:51.000000000 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_marketing_suppression_list.py	2024-09-23 21:20:48.000000000 +0000
@@ -4,6 +4,7 @@
 from airflow.sensors.external_task import ExternalTaskMarker
 from airflow.sensors.external_task import ExternalTaskSensor
 from airflow.utils.task_group import TaskGroup
+from airflow.providers.cncf.kubernetes.secret import Secret
 import datetime
 from operators.gcp_container_operator import GKEPodOperator
 from utils.constants import ALLOWED_STATES, FAILED_STATES
@@ -28,6 +29,31 @@
 * repo/bigquery-etl
 """
 
+marketing_suppression_list_external__campaign_monitor_suppression_list__v1_bqetl_marketing_suppression_list__campaign_monitor_api_key = Secret(
+    deploy_type="env",
+    deploy_target="CAMPAIGN_MONITOR_API_KEY",
+    secret="airflow-gke-secrets",
+    key="bqetl_marketing_suppression_list__campaign_monitor_api_key",
+)
+marketing_suppression_list_external__campaign_monitor_suppression_list__v1_bqetl_marketing_suppression_list__campaign_monitor_client_id = Secret(
+    deploy_type="env",
+    deploy_target="CAMPAIGN_MONITOR_CLIENT_ID",
+    secret="airflow-gke-secrets",
+    key="bqetl_marketing_suppression_list__campaign_monitor_client_id",
+)
+marketing_suppression_list_external__send_suppression_list_update_to_campaign_monitor__v1_bqetl_marketing_suppression_list__campaign_monitor_api_key = Secret(
+    deploy_type="env",
+    deploy_target="CAMPAIGN_MONITOR_API_KEY",
+    secret="airflow-gke-secrets",
+    key="bqetl_marketing_suppression_list__campaign_monitor_api_key",
+)
+marketing_suppression_list_external__send_suppression_list_update_to_campaign_monitor__v1_bqetl_marketing_suppression_list__campaign_monitor_client_id = Secret(
+    deploy_type="env",
+    deploy_target="CAMPAIGN_MONITOR_CLIENT_ID",
+    secret="airflow-gke-secrets",
+    key="bqetl_marketing_suppression_list__campaign_monitor_client_id",
+)
+
 
 default_args = {
     "owner": "[email protected]",
@@ -149,13 +175,14 @@
             "python",
             "sql/moz-fx-data-shared-prod/marketing_suppression_list_external/campaign_monitor_suppression_list_v1/query.py",
         ]
-        + [
-            "--api_key={{ var.value.campaign_monitor_api_key }}",
-            "--client_id={{ var.value.campaign_monitor_client_id }}",
-        ],
+        + [],
         image="gcr.io/moz-fx-data-airflow-prod-88e0/bigquery-etl:latest",
         owner="[email protected]",
         email=["[email protected]"],
+        secrets=[
+            marketing_suppression_list_external__campaign_monitor_suppression_list__v1_bqetl_marketing_suppression_list__campaign_monitor_api_key,
+            marketing_suppression_list_external__campaign_monitor_suppression_list__v1_bqetl_marketing_suppression_list__campaign_monitor_client_id,
+        ],
     )
 
     marketing_suppression_list_external__send_suppression_list_update_to_campaign_monitor__v1 = GKEPodOperator(
@@ -164,13 +191,14 @@
             "python",
             "sql/moz-fx-data-shared-prod/marketing_suppression_list_external/send_suppression_list_update_to_campaign_monitor_v1/query.py",
         ]
-        + [
-            "--api_key={{ var.value.campaign_monitor_api_key }}",
-            "--client_id={{ var.value.campaign_monitor_client_id }}",
-        ],
+        + [],
         image="gcr.io/moz-fx-data-airflow-prod-88e0/bigquery-etl:latest",
         owner="[email protected]",
         email=["[email protected]"],
+        secrets=[
+            marketing_suppression_list_external__send_suppression_list_update_to_campaign_monitor__v1_bqetl_marketing_suppression_list__campaign_monitor_api_key,
+            marketing_suppression_list_external__send_suppression_list_update_to_campaign_monitor__v1_bqetl_marketing_suppression_list__campaign_monitor_client_id,
+        ],
     )
 
     checks__fail_marketing_suppression_list_derived__main_suppression_list__v1.set_upstream(
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/marketing_suppression_list_external/campaign_monitor_suppression_list_v1/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/marketing_suppression_list_external/campaign_monitor_suppression_list_v1/metadata.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/marketing_suppression_list_external/campaign_monitor_suppression_list_v1/metadata.yaml	2024-09-23 21:15:33.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/marketing_suppression_list_external/campaign_monitor_suppression_list_v1/metadata.yaml	2024-09-23 21:16:08.000000000 +0000
@@ -9,9 +9,11 @@
   dag: bqetl_marketing_suppression_list
 scheduling:
   dag_name: bqetl_marketing_suppression_list
-  arguments:
-  - --api_key={{ var.value.campaign_monitor_api_key }}
-  - --client_id={{ var.value.campaign_monitor_client_id }}
+  secrets:
+  - deploy_target: CAMPAIGN_MONITOR_API_KEY
+    key: bqetl_marketing_suppression_list__campaign_monitor_api_key
+  - deploy_target: CAMPAIGN_MONITOR_CLIENT_ID
+    key: bqetl_marketing_suppression_list__campaign_monitor_client_id
 bigquery: null
 workgroup_access:
 - role: roles/bigquery.dataViewer
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/marketing_suppression_list_external/campaign_monitor_suppression_list_v1/query.py /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/marketing_suppression_list_external/campaign_monitor_suppression_list_v1/query.py
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/marketing_suppression_list_external/campaign_monitor_suppression_list_v1/query.py	2024-09-23 21:13:48.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/marketing_suppression_list_external/campaign_monitor_suppression_list_v1/query.py	2024-09-23 21:13:52.000000000 +0000
@@ -88,6 +88,7 @@
 @click.command
 @click.option(
     "--api_key",
+    envvar='CAMPAIGN_MONITOR_API_KEY',
     required=True,
     help="Campaign Monitor API key to use for authentication.",
 )
@@ -111,6 +112,7 @@
 )
 @click.option(
     "--client_id",
+    envvar='CAMPAIGN_MONITOR_CLIENT_ID',
     required=True,
     help="Client Id for Campaign Monitor.",
 )
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/marketing_suppression_list_external/send_suppression_list_update_to_campaign_monitor_v1/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/marketing_suppression_list_external/send_suppression_list_update_to_campaign_monitor_v1/metadata.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/marketing_suppression_list_external/send_suppression_list_update_to_campaign_monitor_v1/metadata.yaml	2024-09-23 21:15:33.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/marketing_suppression_list_external/send_suppression_list_update_to_campaign_monitor_v1/metadata.yaml	2024-09-23 21:16:08.000000000 +0000
@@ -10,9 +10,11 @@
   dag: bqetl_marketing_suppression_list
 scheduling:
   dag_name: bqetl_marketing_suppression_list
-  arguments:
-  - --api_key={{ var.value.campaign_monitor_api_key }}
-  - --client_id={{ var.value.campaign_monitor_client_id }}
+  secrets:
+  - deploy_target: CAMPAIGN_MONITOR_API_KEY
+    key: bqetl_marketing_suppression_list__campaign_monitor_api_key
+  - deploy_target: CAMPAIGN_MONITOR_CLIENT_ID
+    key: bqetl_marketing_suppression_list__campaign_monitor_client_id
   depends_on:
   - task_id: marketing_suppression_list_external__campaign_monitor_suppression_list__v1
     dag_name: bqetl_marketing_suppression_list
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/marketing_suppression_list_external/send_suppression_list_update_to_campaign_monitor_v1/query.py /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/marketing_suppression_list_external/send_suppression_list_update_to_campaign_monitor_v1/query.py
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/marketing_suppression_list_external/send_suppression_list_update_to_campaign_monitor_v1/query.py	2024-09-23 21:13:48.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/marketing_suppression_list_external/send_suppression_list_update_to_campaign_monitor_v1/query.py	2024-09-23 21:13:52.000000000 +0000
@@ -76,11 +76,13 @@
 @click.command
 @click.option(
     "--api_key",
+    envvar='CAMPAIGN_MONITOR_API_KEY',
     required=True,
     help="Campaign Monitor API key to use for authentication.",
 )
 @click.option(
     "--client_id",
+    envvar='CAMPAIGN_MONITOR_CLIENT_ID',
     required=True,
     help="Client Id for Campaign Monitor.",
 )

Link to full diff

Use secrets in bqetl_marketing_suppression_list DAG

e39003a

scholtzan marked this pull request as ready for review September 23, 2024 21:14

scholtzan requested a review from chelseybeck September 23, 2024 21:15

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use secrets in bqetl_marketing_suppression_list DAG #6247

Use secrets in bqetl_marketing_suppression_list DAG #6247

scholtzan commented Sep 23, 2024 •

edited by data-sync-user

Loading

dataops-ci-bot commented Sep 23, 2024

Use secrets in bqetl_marketing_suppression_list DAG #6247

Are you sure you want to change the base?

Use secrets in bqetl_marketing_suppression_list DAG #6247

Conversation

scholtzan commented Sep 23, 2024 • edited by data-sync-user Loading

Description

dataops-ci-bot commented Sep 23, 2024

Integration report for "Use secrets in bqetl_marketing_suppression_list DAG"

sql.diff

scholtzan commented Sep 23, 2024 •

edited by data-sync-user

Loading

`sql.diff`