Adding support for Iceberg tables

[bjornm82]

Terraform CLI and Provider Versions

CLI: 1.0.11
Provider: 0.78.0

Use Cases or Problem Statement

As of the public preview of ICEBERG TABLES (https://docs.snowflake.com/en/user-guide/tables-iceberg) is available, it would be great to support the complete ability to manage ICEBERG TABLES with the terraform provider.

Proposal

Not sure if this would be an option to only support the CATALOG INTEGRATION and EXTERNAL VOLUME initially on AWS and followup to support Azure and Google.

Adding support for EXTERNAL VOLUME, CATALOG INTEGRATION and ICEBERG TABLE including DESC EXTERNAL VOLUME.

It seems like all of the resources are new.

How much impact is this issue causing?

High

Additional Information

#2248

[bjornm82]

@sfc-gh-swinkler , could you verify this request? I'm unsure about adding public preview features in this repo and whether it's possible to only add the AWS support over Google and Azure (even though it seems only some fields differ) and not supporting Glue initially.

[sfc-gh-asawicki]

Hey @bjornm82. Thanks for reaching out.

Currently, we are focusing on stabilizing the provider and providing support for the GA features. Ultimately we will support PuPr ones, but we are not there yet.

Iceberg tables are in PuPr so they are not high on our priority list.

We have recently released a unsafe_execute resource that can be used to manage not implemented features for the time being. Be aware that this is a dangerous resource that should be used with caution.

[bjornm82]

unsafe_execute does the trick right now! I would like to keep this ticket open in order to get the full functionality properly. Does that make sense?

[sfc-gh-asawicki]

Yes, let's keep it open. :)

[timvw]

Here is an example of how you can leverage the unsafe_execute resource to create an external volume (s3) and glue catalog.
It parses the query_results to output the aws_iam_user_arns and external_ids needed to setup cross account access) ...

https://gist.github.com/timvw/52f742553f691c92e69c33fc3102c2a6

[sfc-gh-swinkler]

@timvw good job, that looks nice. We do plan to add support for iceberg tables, but it will take some time for us to properly implement. Recommend using the unsafe execute resource for now, and when iceberg tables do get added you can simply import the object using the identifier.

[caseywhoop]

@bjornm82 I have been using the terraform provider where possible to manage iceberg tables, and I am having grant issues outlined in this bug ticket. Have you encounter similar issues?

[nicolamarangoni]

Hello, I'm using the snowflake_unsafe_execute resource for iceberg stuff too, however I always get the deprecation warning.
Is snowflake_unsafe_execute going to be dropped somewhen or replaced with something else? I think it is very useful to be able to perform generic statements during a terraform-deployment.

[sfc-gh-jcieslak]

Hey @nicolamarangoni please, take a look here.

[nicolamarangoni]

Hi @sfc-gh-swinkler , Thanks! I suppose this is the real answer:

We have had many discussions regarding this resource, and we will keep a "safer" version of it inside the provider. I can't give you any timeline now, but I can guarantee that we will not remove the unsafe resource without a suitable alternative.

I look forward to get a safe version of it!

[abdulkaderf]

Now that the iceberg tables are in GA, can we get an estimation on when we can expect those resources to be available in the provider? thanks!

[sfc-gh-asawicki]

Hey. The new features going GA are not changing our V1 scope. We know that Iceberg tables are important to many of you and they are the first prio after ESSENTIAL_GA_OBJECTS.MD and REMAINING_GA_OBJECTS.MD. For now, we can't commit to any sooner date. We are open to contributions, though. https://github.com/Snowflake-Labs/terraform-provider-snowflake/blob/main/CONTRIBUTING.md

[jdoldis]

Hi @sfc-gh-jcieslak , we're interested in getting these resources too, and I would have time to work on a contribution in the next couple of weeks. Before starting on that, I just wanted to check that nobody is already doing this?

[sfc-gh-asawicki]

Hey @jdoldis. No one from our team is currently working on iceberg tables. I am also unaware of any external contributors working on it (no one has contacted us).

[jdoldis]

Great, I'll get a start on that 👍

[sfc-gh-asawicki]

That's great to hear, @jdoldis!

Please remember our guidelines (https://github.com/Snowflake-Labs/terraform-provider-snowflake/blob/main/CONTRIBUTING.md).

We are also open to discussing the change before implementation, so please reach out to us with any questions you might have (we can also have a call if this works better for you).

[jdoldis]

Awesome thanks @sfc-gh-asawicki ! I've just sent you an email to connect on that.

[harukitake]

@sfc-gh-asawicki Hi!

I tried to create an external volume using the code below.
It failed because the role didn't have CREATE EXTERNAL VOLUME privilege.
But this was not indicated in the error message.

resource "snowflake_external_volume" "this" {
  name = "ICEBERG_LAB_EXTERNAL_VOLUME"
  storage_location {
    storage_location_name = "s3-aaaa"
    storage_provider      = "S3"
    storage_base_url      = "s3://aaaa/"
    storage_aws_role_arn  = "arn:aws:iam::aaaaa:role/bbbbbb"
  }
}

terraform error

 Warning: Failed to query external volume. Marking the resource as removed.
│
│   with snowflake_external_volume.this,
│   on main.tf line 62, in resource "snowflake_external_volume" "this":
│   62: resource "snowflake_external_volume" "this" {
│
│ External Volume: "ICEBERG_LAB_EXTERNAL_VOLUME", Err: object does not exist
╵
╷
│ Error: Provider produced inconsistent result after apply
│
│ When applying changes to snowflake_external_volume.this, provider "provider[\"registry.terraform.io/snowflake-labs/snowflake\"]" produced an unexpected
│ new value: Root object was present, but now absent.
│
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.

failed query and result

CREATE EXTERNAL VOLUME "ICEBERG_LAB_EXTERNAL_VOLUME" STORAGE_LOCATIONS = ((NAME = 's3-aaaa STORAGE_PROVIDER = 'S3' STORAGE_AWS_ROLE_ARN = 'arn:aws:iam::aaaaa:role/bbbbb' STORAGE_BASE_URL = 's3://aaaa/'))

SQL access control error: Insufficient privileges to operate on account 'account id'

Terraform version 1.9.5
Provider version 0.98.0

[sfc-gh-asawicki]

Hey @harukitake. I will add this to our checklist for external volumes. Note, that this resource will be in preview for the V1 (https://github.com/Snowflake-Labs/terraform-provider-snowflake/blob/main/v1-preparations/LIST_OF_PREVIEW_FEATURES_FOR_V1.md).

[trevormarburger]

Hey, @jdoldis 👋 - do you have any updates on the Iceberg table resource feature?

[jdoldis]

Hey @trevormarburger , as you can see above we managed to implement external volumes which are in preview for v1, the Snowflake team will finish those up at some point in the new year. For the next couple of months it doesn’t look like I’ll have time to implement catalog integrations or iceberg tables, there’s a few other topics keeping me busy. If I don’t get to it I believe the Snowflake team will pick them up later in the year.

[sfc-gh-asawicki]

Hey @trevormarburger. I can confirm what @jdoldis posted: we are aware that iceberg resources are highly anticipated. We will take this into account while preparing the resource prioritization early next year (as shared in the newest roadmap entry).