[Deploy] Add Terraform Deployer

[bzub]

I'm working on a Deployer implementation for Terraform modules and would like to get early input for the design. Initial proposal follows. Questions and comments are highly encouraged!

Skaffold Terraform Deployer

Uses terraform cli to deploy Kubernetes resources via Terraform module(s).

Use Cases / Goals

• Rapid development of apps in an environment where Terraform is used significantly to manage Kubernetes clusters and resources.
• Infrastructure and environment details can be consumed from a remote Terraform state and incorporated in an app's development/deployment dynamically.
• Apps developed with Skaffold + Terraform Deployer can be tested and then released as Terraform modules for use in more complex Terraform environments.

Design

Scope

• I believe the Skaffold Terraform Deployer will be relatively simple, providing a thin layer of automation on top of simply running terraform apply and terraform destroy on one or more root modules.
• Terraform modules, and variables settings fed to said modules, would be the primary (only?) configurations available at the Skaffold level/config.
• Dynamic values managed by Skaffold like image tags would be passed into Terraform modules as variable inputs.

Skaffold Config

Best represented with an example:

deploy:
  terraform:
    modules:
    - path: tf/module1
      variables:
        variable_name: value
    - path: tf/module2

In the example above two Terraform modules backed by two separate Terraform states will be used by the Skaffold project. This config results in (roughly) the following CLI commands being run by Skaffold:

• tf/module1

terraform [apply|delete] -state=~/tf/states/module1/terraform.state -var 'variable_name=value' tf/module1

• tf/module2

terraform [apply|delete] -state=tf/module2/terraform.state tf/module2

[balopat]

Hi @bzub, thank you for opening this. We are thinking about how Skaffold could help in setting up the infrastructure around a deployment. However the Deployer interface does not seem like the best place to start in my mind, it was meant to be for a Kubernetes (maybe Knative in the future) specific application deployment that for example should be triggered every time there is a source code change in skaffold dev. Were you thinking about Kubernetes modules only in Terraform? Doing infrastructure work on every code change seems excessive.

[bzub]

Hi @balopat, I'm glad you brought this up. My use-case is limited to deploying Kubernetes resources via the Kubernetes Terraform Provider. Any examples of using this proposed deployer would reflect that theme.

That said, I had not intended on enforcing this as a restriction in the deployer. It is possible to use the terraform --target flag to limit use to only kubernetes resources, however that quickly becomes fragile due to the possibility to use multiple instances of the kubernetes provider via aliases.

So in the end I thought this Skaffold deployer would assume the user is deploying Kubernetes resources via Terraform configs. If users go beyond this assumption, it will not get in their way. Since this deployer should be a fairly thin wrapper over Terraform I think it will be rare for a problem with Terraform or a provider other than Kubernetes to be conflated as a Skaffold issue.

[bzub]

From a design perspective, deploying other infrastructure with this deployer could actually be fairly intuitive. Assuming this deployer tracks what Kubernetes resources it creates accurately, and deletes those resources from a cluster accurately, when a user does a skaffold deploy again their non-Kubernetes resources would not be re-applied, since Terraform will detect that only the Kubernetes specific resources are not in their desired state.

[balopat]

Hi @bzub - for now I would put this idea on hold and close this issue. We thought about this, and are not convinced that this belongs to the scope of Skaffold. If we see a lot of demand, we can reopen it.

[adrien-barret]

Hi @balopat ,

I got the same pretty usecase than Bzub but I understand the mindest behind terraform is not the same as skaffold is doing. So question, how to do work at hashicorp to get your live debugging into a kubernetes but keeping deploying with TF ?

[vladdy]

I have another use case where the app instance goes beyond kubernetes as it requires some AWS resources (i.e, KMS keys, IAM roles for service accounts). Having Terraform as a deployer or some kind of hook to run it along with existing deployers would make it possible to use skaffold in such development requirements.

[MoSattler]

Same here - a terraform deployer would be immensely helpful

[bbhuston]

@balopat Would love to +100 the usefulness of skaffold support for terraform here as well.

To point to one immediate example use-case, the various Google Cloud blueprint modules are currently a total mess when it comes to tooling. Users are forced use multiple, disparate toolchains (e.g., terraform, kubectl, helm, kustomize, kpt, etc) to complete the installation of any blueprint that is even moderately complicated.

Skaffold already supports 4 of the 5 CLIs that I mentioned, so if it were to support terraform as well, then GCP blueprint authors could use skaffold as the top-level abstraction that end-users interact with and push all the "tooling glue" into skaffold.yaml files. This would insulate blueprint end-users from a bunch implementation details and also allow authors to migrate logic between tools (say, from terraform to kpt) over time without anyone knowing.

Finally, in light of the cool skaffold-powered release-management functionality that is being baked into products like Cloud Deploy, having skaffold serve as the "uber-abstraction" for GCP blueprints would allow blueprint authors to provide release-management functionality 'for free' for IaC use-cases.

[tejal29]

Currently there is no plan to support terraform deployer in skaffold. We will think more about this internally and would give an update soon

[bbhuston]

@tejal29 Circling back to your comment from last May, would your team happened to reached a go/no-go decision on support for a terraform deployer? Would appreciate any update that you can share.

[aaron-prindle]

@bbhuston this discussion is still ongoing unfortunately. The team very much wants to support terraform and believes it would help users and expand Skaffold's usability for CI/CD but we are still talking through the prioritization of this

TLDR; we want to support this but the work hasn't been able to be prioritized

[8ball030]

What would need to be done here?

Would love to see this integration, it would save a lot of time for me!

Let me know if there is anything i can do to help!

[podleaper]

echoing @8ball030 and the others. My team and company devops teams use would greatly benefit from Skaffold-Terraform integration.

[abergmeier]

So took a stab at OpenTofu integration here: #9591