Hi @maxfirman . Thanks for taking the time to add this functionality in.

One of the team will spend some time reviewing it and get back to you.

Hi @glsdown, thanks for taking the time to review my PR.

I can see that my changes caused a regression for upload_sources and dim_dbt__snapshots. I've just pushed a commit that will hopefully resolve those failures.

@glsdown thanks very for taking a look at this. I appreciate its a reasonable amount of work to set up integration testing, and I'm happy to help as much as I can.

I've tested your latest changes. I had to make a small patch, but I can now build the test project locally without errors.

There are a couple of ways to go in terms of setting up a Dremio instance for testing. The most straightforward approach, assuming you have access to an AWS account, would be to spin up a Dremio Cloud account and link it to your AWS account. You would then need to connect to a metastore in order to be able to create Apache Iceberg tables. Your two options would be either AWS Glue Catalog or Dremio's Arctic metastore. I don't have much experience with the later, so would probably suggest going with AWS Glue catalog. In either case you will also need to spin up an s3 bucket to store the actual data, and configure to be the hive.metastore.warehouse.dir.

The alternative approach would be to go totally DIY and use the standalone dremio-oss Docker image. The problem with this is that you would also need to spin up a Minio object store and a Hive Metastore in order to be able to create Iceberg tables. Getting everything configured and talking to each other which would probably require more effort compared to Dremio Cloud / AWS approach.