CalTrans Snowflake Terraform Provider Upgrade to 0.97

[ram-kishore-odi]

Can you please review the initial changes related to this upgrade ?

The upgrade needs to happen in to two stages.

Step 1 - Upgrade from version 0.88 to 0.92 - Successful for development environment. Related changes are in this PR
Step 2 - Upgrade from version 0.92 to 0.97 - WIP. This requires code changes to convert some of the resource definitions to conform to the 0.97 requirements.

The key ask of this PR is to review the initial conversion changes and provide your feedback.
I felt that the conversion is not straight forward as I could not find any relevant samples in official documentation to match what is needed for us.

Terraform validate succeeds but the plan output is complaining about bugs in the provider. So I feel more changes are needed. Please see the partial output from the commands below -

** validate **
(infra) ram.kishore@180 dev % terraform validate
Success! The configuration is valid.

** Plan ***

module.elt.module.transforming["S"].snowflake_warehouse.this will be created

• resource "snowflake_warehouse" "this" {
  • auto_resume = "true"
  • auto_suspend = 300
  • comment = "Primary warehouse for transforming data. Analytics engineers and automated transformation tools should use this warehouse"
  • enable_query_acceleration = "default"
  • fully_qualified_name = (known after apply)
  • id = (known after apply)
  • initially_suspended = true
  • max_concurrency_level = (known after apply)
  • name = "TRANSFORMING_S_DEV"
  • parameters = (known after apply)
  • query_acceleration_max_scale_factor = -1
  • show_output = (known after apply)
  • statement_queued_timeout_in_seconds = (known after apply)
  • statement_timeout_in_seconds = (known after apply)
  • warehouse_size = "SMALL"
    }

module.elt.module.transforming["XL"].snowflake_account_role.this will be created

• resource "snowflake_account_role" "this" {
  • comment = "Monitoring, usage, and operating permissions for the TRANSFORMING_XL_DEV warehouse"
  • fully_qualified_name = (known after apply)
  • id = (known after apply)
  • name = "TRANSFORMING_XL_DEV_WH_MOU"
  • show_output = (known after apply)
    }

module.elt.module.transforming["XL"].snowflake_role.this will be destroyed

(because snowflake_role.this is not in configuration)

• resource "snowflake_role" "this" {
  • comment = "Monitoring, usage, and operating permissions for the TRANSFORMING_XL_DEV warehouse" -> null
  • id = "TRANSFORMING_XL_DEV_WH_MOU" -> null
  • name = "TRANSFORMING_XL_DEV_WH_MOU" -> null
    }

module.elt.module.transforming["XL"].snowflake_warehouse.this will be updated in-place

~ resource "snowflake_warehouse" "this" {
~ enable_query_acceleration = "false" -> "default"
id = "TRANSFORMING_XL_DEV"
- max_cluster_count = 1 -> null
~ max_concurrency_level = 8 -> (known after apply)
- min_cluster_count = 1 -> null
name = "TRANSFORMING_XL_DEV"
~ query_acceleration_max_scale_factor = 8 -> -1
- scaling_policy = "STANDARD" -> null
~ show_output = [
- {
- auto_resume = true
- auto_suspend = 300
- available = 0
- comment = "Primary warehouse for transforming data. Analytics engineers and automated transformation tools should use this warehouse"
- created_on = "2023-10-18 11:11:28.402 -0700 PDT"
- enable_query_acceleration = false
- is_current = false
- is_default = false
- max_cluster_count = 1
- min_cluster_count = 1
- name = "TRANSFORMING_XL_DEV"
- other = 0
- owner = "SYSADMIN"
- owner_role_type = "ROLE"
- provisioning = 0
- query_acceleration_max_scale_factor = 8
- queued = 0
- quiescing = 0
- resumed_on = "2024-11-26 14:33:29.565 -0800 PST"
- running = 0
- scaling_policy = "STANDARD"
- size = "XLARGE"
- started_clusters = 0
- state = "SUSPENDED"
- type = "STANDARD"
- updated_on = "2024-11-26 14:33:29.565 -0800 PST"
# (1 unchanged attribute hidden)
},
] -> (known after apply)
~ statement_queued_timeout_in_seconds = 0 -> (known after apply)
~ statement_timeout_in_seconds = 172800 -> (known after apply)
- warehouse_type = "STANDARD" -> null
# (7 unchanged attributes hidden)
}

module.elt.module.transforming["XS"].snowflake_account_role.this will be created

• resource "snowflake_account_role" "this" {
  • comment = "Monitoring, usage, and operating permissions for the TRANSFORMING_XS_DEV warehouse"
  • fully_qualified_name = (known after apply)
  • id = (known after apply)
  • name = "TRANSFORMING_XS_DEV_WH_MOU"
  • show_output = (known after apply)
    }

module.elt.module.transforming["XS"].snowflake_role.this will be destroyed

(because snowflake_role.this is not in configuration)

• resource "snowflake_role" "this" {
  • comment = "Monitoring, usage, and operating permissions for the TRANSFORMING_XS_DEV warehouse" -> null
  • id = "TRANSFORMING_XS_DEV_WH_MOU" -> null
  • name = "TRANSFORMING_XS_DEV_WH_MOU" -> null
    }

module.elt.module.transforming["XS"].snowflake_warehouse.this will be updated in-place

~ resource "snowflake_warehouse" "this" {
~ enable_query_acceleration = "false" -> "default"
id = "TRANSFORMING_XS_DEV"
- max_cluster_count = 1 -> null
~ max_concurrency_level = 8 -> (known after apply)
- min_cluster_count = 1 -> null
name = "TRANSFORMING_XS_DEV"
~ query_acceleration_max_scale_factor = 8 -> -1
- scaling_policy = "STANDARD" -> null
~ show_output = [
- {
- auto_resume = true
- auto_suspend = 300
- available = 0
- comment = "Primary warehouse for transforming data. Analytics engineers and automated transformation tools should use this warehouse"
- created_on = "2023-10-18 11:11:28.272 -0700 PDT"
- enable_query_acceleration = false
- is_current = false
- is_default = false
- max_cluster_count = 1
- min_cluster_count = 1
- name = "TRANSFORMING_XS_DEV"
- other = 0
- owner = "SYSADMIN"
- owner_role_type = "ROLE"
- provisioning = 0
- query_acceleration_max_scale_factor = 8
- queued = 0
- quiescing = 0
- resumed_on = "2024-11-27 13:09:48.271 -0800 PST"
- running = 0
- scaling_policy = "STANDARD"
- size = "XSMALL"
- started_clusters = 0
- state = "SUSPENDED"
- type = "STANDARD"
- updated_on = "2024-11-27 13:09:48.271 -0800 PST"
# (1 unchanged attribute hidden)
},
] -> (known after apply)
~ statement_queued_timeout_in_seconds = 0 -> (known after apply)
~ statement_timeout_in_seconds = 172800 -> (known after apply)
- warehouse_type = "STANDARD" -> null
# (7 unchanged attributes hidden)
}

Plan: 122 to add, 20 to change, 38 to destroy.
╷
│ Warning: Missing resource schema from provider
│
│ No resource schema found for snowflake_stage_grant when decoding prior state
╵
╷
│ Warning: Missing resource schema from provider
│
│ No resource schema found for snowflake_integration_grant when decoding prior state
╵
╷
│ Warning: Missing resource schema from provider
│
│ No resource schema found for snowflake_stage_grant when decoding prior state
╵
╷
│ Error: no schema available for module.snowflake_clearinghouse.snowflake_integration_grant.pems_storage_integration_to_sysadmin while reading state; this is a bug in Terraform and should be reported
│
│
╵
╷
│ Error: no schema available for module.snowflake_clearinghouse.snowflake_stage_grant.pems_marts while reading state; this is a bug in Terraform and should be reported
│
│
╵
╷
│ Error: no schema available for module.snowflake_clearinghouse.snowflake_stage_grant.pems_raw while reading state; this is a bug in Terraform and should be reported

[ian-r-rose]

The code changes look good to me @ram-kishore-odi, please merge when ready!

For the terraform plan: I see that there are some roles which are planned to be destroyed and recreated. Can you do a state rm and import to avoid recreating them?

[ram-kishore-odi]

Sure @ian-r-rose. Yes I will identify the appropriate resources and perform the state rm and import operations before applying the changes to development environment.

[ram-kishore-odi]

Pushed updated lock files and production config updates. After review, I plan to merge and apply changes to production from main branch. Please let me know if you have any questions

[ian-r-rose]

That sounds good! The dbt cloud error looks unrelated, I'll bet if you merge/rebase it will go away.

The main thing is to be triply sure that we do terraform state rm and import for the snowflake_schema resources, as recreating them would have pretty negative consequences.

[ram-kishore-odi]

Thank you Ian ! Will merge into main and take action if I run into any dbt Cloud or other issues.

[ian-r-rose]

I meant merge main into here so that CI checks would pass