Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add native support for updating .terraform.lock.hcl #90

Merged
merged 32 commits into from
Jul 4, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
9a44923
Split file per registry API
minamijoyo Jun 6, 2023
d184fe3
Implement ProviderPackageMetadata API
minamijoyo Jun 6, 2023
b38c67b
Implement downloading provider
minamijoyo Jun 7, 2023
06704f5
Test ProviderDownload() with a mocked Registry API
minamijoyo Jun 7, 2023
222cba4
Validate sha256 checksum for zip data
minamijoyo Jun 7, 2023
dd3df2b
Validate SHASumsData
minamijoyo Jun 7, 2023
48abf5c
Implement an index of provider versions for caching
minamijoyo Jun 9, 2023
912c079
Return an error if provider address is implicit legacy things
minamijoyo Jun 15, 2023
0585339
Add tests for buildProviderVersion
minamijoyo Jun 15, 2023
350423f
Add tests for providerIndex.getOrCreateProviderVersion
minamijoyo Jun 16, 2023
80face0
Add tests for Index.GetOrCreateProviderVersion
minamijoyo Jun 16, 2023
3d07683
Implement version constraint detection
minamijoyo Jun 21, 2023
721888c
Fix lint issues
minamijoyo Jun 21, 2023
679f233
Inject a ModuleContext to Updater
minamijoyo Jun 22, 2023
ae1f023
Add native support for updating .terraform.lock.hcl
minamijoyo Jun 23, 2023
3dbe933
Use filename instead of platform for keys of hash
minamijoyo Jun 26, 2023
ae823c8
Fix lint issues
minamijoyo Jun 28, 2023
2899273
Inject a Go's standard context outside of the tfupdate package
minamijoyo Jun 28, 2023
8dd093c
Add more debug logs
minamijoyo Jun 28, 2023
e6b979d
Move a lock index instance into LockUpdater
minamijoyo Jun 28, 2023
8a81eea
Remove the shorthand flag for platform
minamijoyo Jun 28, 2023
cb4a4bd
Parameterise the mock response
minamijoyo Jun 29, 2023
ba90f8e
Add more tests for LockUpdater
minamijoyo Jun 29, 2023
c05e01e
Add tests for hclwrite helpers
minamijoyo Jun 29, 2023
54d666a
Setup acceptance tests
minamijoyo Jun 30, 2023
b0cbd5d
Fix the git permission issue
minamijoyo Jun 30, 2023
4c5160f
Update README for the tfupdate lock command
minamijoyo Jun 30, 2023
7e130e5
Fix tests for index
minamijoyo Jul 3, 2023
2f5aa16
Mention that provider packages could be distributed from elsewhere
minamijoyo Jul 3, 2023
0efebb6
Fix an error message for ProviderVersion.Merge
minamijoyo Jul 3, 2023
f18d544
Fix typo in testacc
minamijoyo Jul 3, 2023
54ab92d
Fix typo
minamijoyo Jul 4, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .envrc.sample
Original file line number Diff line number Diff line change
@@ -1 +1 @@
export GO111MODULE=on
export TERRAFORM_VERSION=1.5.2
18 changes: 18 additions & 0 deletions .github/workflows/test.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -24,3 +24,21 @@ jobs:
go-version-file: '.go-version'
- name: test
run: make test
testacc:
runs-on: ubuntu-latest
timeout-minutes: 5
strategy:
matrix:
terraform:
- 1.5.2
- 0.14.11
env:
TERRAFORM_VERSION: ${{ matrix.terraform }}
steps:
- uses: actions/checkout@v3
- name: docker build
run: docker-compose build
- name: terraform --version
run: docker compose run --rm tfupdate terraform --version
- name: testacc
run: docker compose run --rm tfupdate make testacc
27 changes: 27 additions & 0 deletions Dockerfile.dev
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
# terraform
ARG TERRAFORM_VERSION=latest
FROM hashicorp/terraform:$TERRAFORM_VERSION AS terraform

# tfupdate
FROM golang:1.20-alpine3.18 AS tfupdate
RUN apk --no-cache add make git

# A workaround for a permission issue of git.
# Since UIDs are different between host and container,
# the .git directory is untrusted by default.
# We need to allow it explicitly.
# https://github.com/actions/checkout/issues/760
RUN git config --global --add safe.directory /work

# for testing
RUN apk add --no-cache bash
COPY --from=terraform /bin/terraform /usr/local/bin/

WORKDIR /work

COPY go.mod go.sum ./
RUN go mod download

COPY . .
RUN make install

15 changes: 15 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -22,5 +22,20 @@ lint:
test: build
go test ./...

.PHONY: testacc
testacc: install testacc-lock-simple

.PHONY: testacc-lock-simple
testacc-lock-simple: install
scripts/testacc/lock.sh run simple

.PHONY: testacc-lock-debug
testacc-lock-debug: install
scripts/testacc/lock.sh $(ARG)

.PHONY: testacc-all
testacc-all: install
scripts/testacc/all.sh

.PHONY: check
check: lint test
138 changes: 136 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,8 @@
## Features

- Update version constraints of Terraform core, providers, and modules
- Update all your Terraform configurations recursively under a given directory
- Update dependency lock files (.terraform.lock.hcl) without Terraform CLI
- Update all your Terraform configurations and lock files recursively under a given directory
- Get the latest release version from the GitHub, GitLab, or Terraform Registry
- Terraform v0.12+ support

Expand Down Expand Up @@ -64,10 +65,11 @@ $ docker run -it --rm minamijoyo/tfupdate --version
## Usage

```
tfupdate --help
$ tfupdate --help
Usage: tfupdate [--version] [--help] <command> [<args>]

Available commands are:
lock Update dependency lock files
module Update version constraints for module
provider Update version constraints for provider
release Get release version information
Expand Down Expand Up @@ -203,6 +205,8 @@ terraform {
}
```

For updating the dependency lock file (.terraform.lock.hcl), use the `tfupdate lock` command.

### module

```
Expand Down Expand Up @@ -358,6 +362,136 @@ $ tfupdate release list -n 5 hashicorp/terraform
0.12.21
```

### lock

The tfupdate lock command updates the dependency lock file (.terraform.lock.hcl).
For more information on the dependency lock file, see the official Terraform documentation:
https://developer.hashicorp.com/terraform/language/files/dependency-lock

```
$ tfupdate lock --help
Usage: tfupdate lock [options] <PATH>

Arguments
PATH A path of directory to update

Options:
--platform Specify a platform to update dependency lock files.
At least one or more --platform flags must be specified.
Use this option multiple times to include checksums for multiple target systems.
Target platform names consist of an operating system and a CPU architecture.
(e.g. linux_amd64, darwin_amd64, darwin_arm64)
-r --recursive Check a directory recursively (default: false)
-i --ignore-path A regular expression for path to ignore
If you want to ignore multiple directories, set the flag multiple times.
```

Given the following configuration:

```
$ cat test-fixtures/lock/simple/main.tf
terraform {
required_providers {
null = {
source = "hashicorp/null"
version = "3.1.1"
}
}
}
```

As you know, you can generate the dependency lock file by the terraform providers lock command:

```
$ terraform -chdir=test-fixtures/lock/simple providers lock -platform=linux_amd64 -platform=darwin_amd64 -platform=darwin_arm64
```

```
$ cat test-fixtures/lock/simple/.terraform.lock.hcl
# This file is maintained automatically by "terraform init".
# Manual edits may be lost in future updates.

provider "registry.terraform.io/hashicorp/null" {
version = "3.1.1"
constraints = "3.1.1"
hashes = [
"h1:71sNUDvmiJcijsvfXpiLCz0lXIBSsEJjMxljt7hxMhw=",
"h1:Pctug/s/2Hg5FJqjYcTM0kPyx3AoYK1MpRWO0T9V2ns=",
"h1:YvH6gTaQzGdNv+SKTZujU1O0bO+Pw6vJHOPhqgN8XNs=",
"zh:063466f41f1d9fd0dd93722840c1314f046d8760b1812fa67c34de0afcba5597",
"zh:08c058e367de6debdad35fc24d97131c7cf75103baec8279aba3506a08b53faf",
"zh:73ce6dff935150d6ddc6ac4a10071e02647d10175c173cfe5dca81f3d13d8afe",
"zh:78d5eefdd9e494defcb3c68d282b8f96630502cac21d1ea161f53cfe9bb483b3",
"zh:8fdd792a626413502e68c195f2097352bdc6a0df694f7df350ed784741eb587e",
"zh:976bbaf268cb497400fd5b3c774d218f3933271864345f18deebe4dcbfcd6afa",
"zh:b21b78ca581f98f4cdb7a366b03ae9db23a73dfa7df12c533d7c19b68e9e72e5",
"zh:b7fc0c1615dbdb1d6fd4abb9c7dc7da286631f7ca2299fb9cd4664258ccfbff4",
"zh:d1efc942b2c44345e0c29bc976594cb7278c38cfb8897b344669eafbc3cddf46",
"zh:e356c245b3cd9d4789bab010893566acace682d7db877e52d40fc4ca34a50924",
"zh:ea98802ba92fcfa8cf12cbce2e9e7ebe999afbf8ed47fa45fc847a098d89468b",
"zh:eff8872458806499889f6927b5d954560f3d74bf20b6043409edf94d26cd906f",
]
}
```

When updating provider version, the lock file must also be updated:

```
$ tfupdate provider null -v 3.2.1 ./test-fixtures/lock/simple/
```

```
$ cat test-fixtures/lock/simple/main.tf
terraform {
required_providers {
null = {
source = "hashicorp/null"
version = "3.2.1"
}
}
}
```

You can update the lock file by the tfupdate lock command without Terraform CLI:

```
$ tfupdate lock --platform=linux_amd64 --platform=darwin_amd64 --platform=darwin_arm64 ./test-fixtures/lock/simple/
```

Note that unlike the terraform providers lock command, the `--platform` flag requires two hyphens.

```
$ cat test-fixtures/lock/simple/.terraform.lock.hcl
# This file is maintained automatically by "terraform init".
# Manual edits may be lost in future updates.

provider "registry.terraform.io/hashicorp/null" {
version = "3.2.1"
constraints = "3.2.1"
hashes = [
"h1:FbGfc+muBsC17Ohy5g806iuI1hQc4SIexpYCrQHQd8w=",
"h1:tSj1mL6OQ8ILGqR2mDu7OYYYWf+hoir0pf9KAQ8IzO8=",
"h1:ydA0/SNRVB1o95btfshvYsmxA+jZFRZcvKzZSB+4S1M=",
"zh:58ed64389620cc7b82f01332e27723856422820cfd302e304b5f6c3436fb9840",
"zh:62a5cc82c3b2ddef7ef3a6f2fedb7b9b3deff4ab7b414938b08e51d6e8be87cb",
"zh:63cff4de03af983175a7e37e52d4bd89d990be256b16b5c7f919aff5ad485aa5",
"zh:74cb22c6700e48486b7cabefa10b33b801dfcab56f1a6ac9b6624531f3d36ea3",
"zh:78d5eefdd9e494defcb3c68d282b8f96630502cac21d1ea161f53cfe9bb483b3",
"zh:79e553aff77f1cfa9012a2218b8238dd672ea5e1b2924775ac9ac24d2a75c238",
"zh:a1e06ddda0b5ac48f7e7c7d59e1ab5a4073bbcf876c73c0299e4610ed53859dc",
"zh:c37a97090f1a82222925d45d84483b2aa702ef7ab66532af6cbcfb567818b970",
"zh:e4453fbebf90c53ca3323a92e7ca0f9961427d2f0ce0d2b65523cc04d5d999c2",
"zh:e80a746921946d8b6761e77305b752ad188da60688cfd2059322875d363be5f5",
"zh:fbdb892d9822ed0e4cb60f2fedbdbb556e4da0d88d3b942ae963ed6ff091e48f",
"zh:fca01a623d90d0cad0843102f9b8b9fe0d3ff8244593bd817f126582b52dd694",
]
}
```

The tfupdate lock command parses the `required_providers` block in your configuration, downloads provider packages and calculates hash values under the hood. The most important point is that it caches calculated hash values in memory, which gives us a huge performance advantage when updating multiple directories at once using the `-r (--recursive)` option.

To skip terraform init, we assume that all dependencies are pinned to a specific version in the required_providers block of the root module. Note that version constraint expressions or indirect dependencies via modules are not supported and ignored.

## Keep your dependencies up-to-date

If you integrate tfupdate with your favorite CI or job scheduler, you can check the latest release daily and create a Pull Request automatically.
Expand Down
94 changes: 94 additions & 0 deletions command/lock.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,94 @@
package command

import (
"context"
"fmt"
"log"
"strings"

"github.com/minamijoyo/tfupdate/tfupdate"
flag "github.com/spf13/pflag"
)

// LockCommand is a command which update dependency lock files.
type LockCommand struct {
Meta
platforms []string
path string
recursive bool
ignorePaths []string
}

// Run runs the procedure of this command.
func (c *LockCommand) Run(args []string) int {
cmdFlags := flag.NewFlagSet("lock", flag.ContinueOnError)
cmdFlags.StringArrayVar(&c.platforms, "platform", []string{}, "A target platform for dependecy lock file")
cmdFlags.BoolVarP(&c.recursive, "recursive", "r", false, "Check a directory recursively")
cmdFlags.StringArrayVarP(&c.ignorePaths, "ignore-path", "i", []string{}, "A regular expression for path to ignore")

if err := cmdFlags.Parse(args); err != nil {
c.UI.Error(fmt.Sprintf("failed to parse arguments: %s", err))
return 1
}

if len(cmdFlags.Args()) != 1 {
c.UI.Error(fmt.Sprintf("The command expects 1 arguments, but got %d", len(cmdFlags.Args())))
c.UI.Error(c.Help())
return 1
}

c.path = cmdFlags.Arg(0)

if len(c.platforms) == 0 {
c.UI.Error("The --platform flag is required")
c.UI.Error(c.Help())
return 1
}

log.Println("[INFO] Update dependency lock files")
option, err := tfupdate.NewOption("lock", "", "", c.platforms, c.recursive, c.ignorePaths)
if err != nil {
c.UI.Error(err.Error())
return 1
}

gc, err := tfupdate.NewGlobalContext(c.Fs, option)
if err != nil {
c.UI.Error(err.Error())
return 1
}

err = tfupdate.UpdateFileOrDir(context.Background(), gc, c.path)
if err != nil {
c.UI.Error(err.Error())
return 1
}

return 0
}

// Help returns long-form help text.
func (c *LockCommand) Help() string {
helpText := `
Usage: tfupdate lock [options] <PATH>

Arguments
PATH A path of directory to update

Options:
--platform Specify a platform to update dependency lock files.
At least one or more --platform flags must be specified.
Use this option multiple times to include checksums for multiple target systems.
Target platform names consist of an operating system and a CPU architecture.
(e.g. linux_amd64, darwin_amd64, darwin_arm64)
-r --recursive Check a directory recursively (default: false)
-i --ignore-path A regular expression for path to ignore
If you want to ignore multiple directories, set the flag multiple times.
`
return strings.TrimSpace(helpText)
}

// Synopsis returns one-line help text.
func (c *LockCommand) Synopsis() string {
return "Update dependency lock files"
}
11 changes: 9 additions & 2 deletions command/module.go
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
package command

import (
"context"
"fmt"
"log"
"strings"
Expand Down Expand Up @@ -49,13 +50,19 @@ func (c *ModuleCommand) Run(args []string) int {
}

log.Printf("[INFO] Update module %s to %s", c.name, v)
option, err := tfupdate.NewOption("module", c.name, v, c.recursive, c.ignorePaths)
option, err := tfupdate.NewOption("module", c.name, v, []string{}, c.recursive, c.ignorePaths)
if err != nil {
c.UI.Error(err.Error())
return 1
}

err = tfupdate.UpdateFileOrDir(c.Fs, c.path, option)
gc, err := tfupdate.NewGlobalContext(c.Fs, option)
if err != nil {
c.UI.Error(err.Error())
return 1
}

err = tfupdate.UpdateFileOrDir(context.Background(), gc, c.path)
if err != nil {
c.UI.Error(err.Error())
return 1
Expand Down
10 changes: 8 additions & 2 deletions command/provider.go
Original file line number Diff line number Diff line change
Expand Up @@ -59,13 +59,19 @@ func (c *ProviderCommand) Run(args []string) int {
}

log.Printf("[INFO] Update provider %s to %s", c.name, v)
option, err := tfupdate.NewOption("provider", c.name, v, c.recursive, c.ignorePaths)
option, err := tfupdate.NewOption("provider", c.name, v, []string{}, c.recursive, c.ignorePaths)
if err != nil {
c.UI.Error(err.Error())
return 1
}

err = tfupdate.UpdateFileOrDir(c.Fs, c.path, option)
gc, err := tfupdate.NewGlobalContext(c.Fs, option)
if err != nil {
c.UI.Error(err.Error())
return 1
}

err = tfupdate.UpdateFileOrDir(context.Background(), gc, c.path)
if err != nil {
c.UI.Error(err.Error())
return 1
Expand Down
Loading