Terraform lock files

[viceice]

What would you like Renovate to be able to do?

Since terraform will write a lockfile (.terraform.lock.hcl) since v0.14. so if dependencies are updated by renovate, we need to update the lockfile too.

Did you already have any implementation ideas?

One idea it to simply run terraform init after update, so the lockfile will begenerated / updated.

.terraform.lock.hcl sample

# This file is maintained automatically by "terraform init".
# Manual edits may be lost in future updates.

provider "registry.terraform.io/hashicorp/helm" {
  version     = "1.3.2"
  constraints = "1.3.2"
  hashes = [
    "h1:lNIvu0axcxUHBEsZX5CceT2ArFvlckTXoGWS9286U9s=",
    "zh:0d8e1293cb99b61d3aefbab3f1c1e258e121d860312115e23b240e8acb92b855",
    "zh:17524fac3f1eb46901a27ecef0c054c8b5390994b2f0bd48746a5eb47e42fad5",
    "zh:74ff2d471fc934d0e65452f914248c23394937a9b4cfb560ce920d5c42568303",
    "zh:855255f4afe7b86d88744f5615b6b6a6172fa7fc28c24d8fb5838b715e3b8a97",
    "zh:8b3bb0f0e2e6908c3d41ee183451cb388a80cf576b0953ad3d1e06cb4de22842",
    "zh:b46d607cedfefc94460bbff8a9d46e50f7dce364dc4050a2df81357159e07f81",
    "zh:c9b9f3b0e6aaec7081230df257f89e00e522a3a283197126d88ae646551cde6e",
    "zh:ccb0b341351df79773367aa6d895b89647ceb9a75fff1c434ee480513515d112",
    "zh:e39f56174f61556f2937fe50035703346833555cb83f2a880e3a4d832262120e",
    "zh:f792f8b620551198807bed0752453ff0574b1b7b03ec9d9a580177b84049c700",
  ]
}

provider "registry.terraform.io/hashicorp/kubernetes" {
  version     = "1.13.3"
  constraints = "1.13.3"
  hashes = [
    "h1:0OglC1HmT3NqMn4A83dXbeLp3D6Qhv1FtlwcwyzjeBY=",
    "zh:11fd58df9c297a4f3cab82ce8eb2f54f1be27f78fa23be2273ecd545ab254b91",
    "zh:5b9e6f352c5666d791e2658a1d18bf0990f3ab70c99c916c393a2ee7f385364c",
    "zh:5c94f1350471a5c8e8ee6675874608c506a0bfd3164bdd91b802842723547e2c",
    "zh:5d9c5c44dba9addbb86491339012096e74778bb4ea93b70f12333bffba3d05e6",
    "zh:6336f9cbb0b580f247cebb97fb7d4cc5e7fe9cc734d8d958d84c4ea3f1e24041",
    "zh:bca3b9d4dcbe6f804f5611a83add371dc03b5aa92271f60ebdc2216bfedfab28",
    "zh:cbcdc87a593090f490f7899f4f2d302e0c7023155591fcf65e6fadd69f5452f0",
    "zh:ec2886a1adbfe3c861b2deb9446369111b9c6116701ae73ef372dc7df5bb3c9e",
    "zh:edb5b4172610672bb4d7425511961fda2047b8a00675b99ae6887cd2ece4bda9",
    "zh:ff7ea7743246181ea739643d7751c37041c4016eb6bbc39beb1e3b4e99629112",
  ]
}

provider "registry.terraform.io/kreuzwerker/docker" {
  version     = "2.8.0"
  constraints = "2.8.0"
  hashes = [
    "h1:gH4EPJhzOh+3tt4uWwByFfbCIiVK5rRxEmq9wA2oIUk=",
    "zh:001846931fa93e874d65512ac72e36f79464b86dd9769753acfff871d6acb1fd",
    "zh:093f017ad043e0035c1ed62ebb4daf5f4e978e49534a24fba88f9f3fe206a0e0",
    "zh:7ad122a675216450ade497acebd411277a875cfeac8ebb4bb21aaee16dbcc22a",
    "zh:821c154735007592e3b656e359c1590ede80fae7bb3e55305e1b511973926bb1",
    "zh:a58f8357a21f4243990f5b006eef5ea0ddf8beb91eae402f7af7a0da899f9feb",
    "zh:b4e97b3923b3f115a040a7a84f7681bfb774e801162ec2fc834135444be50f93",
    "zh:b4f6e3fa9314ea8c254ee001b272c0fd58372942e3177c318faa1a000d06fd67",
    "zh:b7ec7c4fb28efbcdab62cbe11860d32915b1e5b19a75f4725aa3f64cc8e01a83",
    "zh:c86b0ae710ee9747df3d37a365eed7dd7a2f7e240d414bd8c5c7ce3a84f808d2",
    "zh:dff347b82a1fa7c492c1d546c09cb94e797cee33c759de2be17fb999f173238a",
    "zh:f338c82c1e7ed8c2fffee5a15dac174d365ea2f22a976049aaba046237ad2aab",
  ]
}

https://github.com/hashicorp/terraform/releases/tag/v0.14.0

[rarkins]

Yep, that looks too complicated for us to try to replicate/reverse engineer - let's run terraform init.

Can we install terraform dynamically at runtime or will we need renovate/terraform images pre-built?

[viceice]

I think we can download it from https://releases.hashicorp.com/terraform/0.14.0/.

It's provied as zip file (~30MB), so we would need to extract. Creating the required docker images is easy for me. So not sure which solution is best

[viceice]

versions.hcl

terraform {
  required_providers {
    docker = {
      source  = "kreuzwerker/docker"
      version = "2.8.0"
    }
    helm = {
      source  = "hashicorp/helm"
      version = "1.3.2"
    }
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = "1.13.3"
    }
  }
  required_version = ">= 0.13"
}

This is the corresponding config file. Best practise is to use versions.tf as this is the autogenerated file by terraform. But it's not required to have that name, can be any hcl-file name. 😕

So with the example above we would need to extract the >= 0.13 terraform version constraint and use best matching terraform version.

[rarkins]

In that case:

• If we need to generate artifacts then we look for versions.tf and if we can't find a required version that way then we default to latest
• Users who don't want to use that file name can configure using constraints

[secustor]

I can't find any references that .hcl files are used rather than .tf ones. This is the only guide for now who explains how to use the lock files. https://learn.hashicorp.com/tutorials/terraform/provider-versioning
There they are using .tf files too.
Actually this should work with the current manager already. As we look trough all .tf files to find these definitions.

https://learn.hashicorp.com/tutorials/terraform/provider-versioning#explore-terraform-lock-hcl

Terraform will generate a new .terraform.lock.hcl file in the current working directory.
So the basics would be these:

• Analyse .tf files
• update dependencies
• check if .terraform.lock.hcl is present
• if it exists in the workdir, download or run the newest valid Terraform version as described by @viceice
• execute terraform init -upgrade

What this last step is also introducing is that all the of the backend configuration has to be present inside the execution context of renovate. This includes e.g. an access key for a blob storage.

I have opened a issue, which should solve this, at the Terraform repo. hashicorp/terraform#27161

[viceice]

Ah, sorry. yes it's versions.tf and not versions.hcl.

What this last step is also introducing is that all the of the backend configuration has to be present inside the execution context of renovate. This includes e.g. access keys for blob storage.

Oh, yes. So i will git ignore the lockfile until there is a proper solution.

[Vrtak-CZ]

What this last step is also introducing is that all the of the backend configuration has to be present inside the execution context of renovate. This includes e.g. an access key for a blob storage.

Yesterday I implemented "ugly" workaround for this.

{
  "postUpgradeTasks": {
    "commands": ["terraform get -no-color -update", "rm .terraform.lock.hcl", "terraform providers lock -no-color -platform=linux_amd64 -platform=darwin_amd64"],
    "fileFilters": [".terraform.lock.hcl"]
  }
}

The terraform get -no-color -update will download and initialize modules so it will not complain about it in future. I need to remove current lock file (this is the ugly part) because if i don't do that the next command can fail. And the terraform providers lock -no-color -platform=linux_amd64 -platform=darwin_amd64 will actually generate lock file.

[evanstoddard23]

Renovate should also be able to make updates to just the lockfile when providers are updated. For example I have:

versions.tf

terraform {
  required_version = ">= 0.14.0"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = ">= 2.0"
    }
  }
}

.terraform.lock.hcl

provider "registry.terraform.io/hashicorp/aws" {
  version     = "3.21.0"
  constraints = ">= 2.0.0"
  hashes = [
    "h1:YGpoWikUMRAqKJAzT5llGi4fE/ce/MFqfXg7m0f7VNA=",
    "zh:11b8c85d063775029245e86ebce346ed86aa77aefaa72dc558da37ea347aa77c",
    "zh:15f0f5bcbcffe1d2aeaa595f6573f0779932bd3a7647f28ad6aacf1a20f35562",
    "zh:4459e3de50fa9ff64ce50b812ddbcfe7f6fe5fcdd1c23c35a85f11bda5ee8cdb",
    "zh:4a146a958ff5997dca61c016330ed0546093873cce2791cdb727ef897b3d5bdd",
    "zh:698717e66ad6cd58842b696c46c1f4d97c0e3cdad1b872665d79f74e6bbc4310",
    "zh:7e270b37e275d17195993c0b13221e400a67eb26850517977a0587de092a055f",
    "zh:7f14438403ca2ebbe39561f13b78d44bad83d519e907414c2e1f1dc9c2059c0f",
    "zh:9799606edd2079c92e181e8a0f022ac69f2e7093bb23cc9348dc9db9b76aa7da",
    "zh:b947e1a3768650aab0d4679a5202d17464f4ad8c429a57627f9cadf5b78843fc",
    "zh:f7fbd8827afcbab1c5da907283acde543cd0ad8513cec0eea14afb7b926b5a5a",
  ]
}

Running terraform init -upgrade gives me this .terraform.lock.hcl without any other source code changes:

provider "registry.terraform.io/hashicorp/aws" {
  version     = "3.22.0"
  constraints = ">= 2.0.0"
  hashes = [
    "h1:f/Tz8zv1Zb78ZaiyJkQ0MGIViZwbYrLuQk3kojPM91c=",
    "zh:4a9a66caf1964cdd3b61fb3ebb0da417195a5529cb8e496f266b0778335d11c8",
    "zh:514f2f006ae68db715d86781673faf9483292deab235c7402ff306e0e92ea11a",
    "zh:5277b61109fddb9011728f6650ef01a639a0590aeffe34ed7de7ba10d0c31803",
    "zh:67784dc8c8375ab37103eea1258c3334ee92be6de033c2b37e3a2a65d0005142",
    "zh:76d4c8be2ca4a3294fb51fb58de1fe03361d3bc403820270cc8e71a04c5fa806",
    "zh:8f90b1cfdcf6e8fb1a9d0382ecaa5056a3a84c94e313fbf9e92c89de271cdede",
    "zh:d0ac346519d0df124df89be2d803eb53f373434890f6ee3fb37112802f9eac59",
    "zh:d6256feedada82cbfb3b1dd6dd9ad02048f23120ab50e6146a541cb11a108cc1",
    "zh:db2fe0d2e77c02e9a74e1ed694aa352295a50283f9a1cf896e5be252af14e9f4",
    "zh:eda61e889b579bd90046939a5b40cf5dc9031fb5a819fc3e4667a78bd432bdb2",
  ]
}

[wyardley]

We have the same issue. Personally, since we’re already pinning to specific versions exactly and using renovate to update versions (we had scripts to update before renovate supported it; love that Renovate supports this now, btw!), I prefer the old behavior, and we (currently) .gitignore the lockfiles.

if there were an easy way to skip requiring tf init -upgrade (that is, to essentially allow an explicit pin to update the version on a regular init), I’d prefer to do that and continue not checking in the lockfiles, and I agree that having renovate manage this lockfile could be challenging, especially since they release new versions often, and people may not be using the same version that renovate does.

[iiro]

@wyardley we've done the same here - and completely agree with you - I also don't believe it's possible to support the Lock File feature properly with Renovate. And, we've also already very pleased with Renovate's support for Terraform (as you as well). But, will follow-up this issue of course...

[MPV]

If it's of any help, here's how a tfupdate (a Terraform-specific equivalent to Renovate) interprets this situation: minamijoyo/tfupdate#32 (including links from there to reported upstream issues with terraform)

[mercuriete]

Can somebody add a little more information on how to opt in to this feature?

I am a GitHub marketplace user.

Thanks for implementing It.

[rarkins]

@mercuriete initially this will only be possible for self-hosted users as it's enabled using an environment variable:

## RENOVATE_X_TERRAFORM_LOCK_FILE

If set to any value, Renovate will update Terraform lock files and allow lockfile maintenance.

Hopefully others who were waiting for this feature are on self-hosted and can report back on whether it's working as intended after they add that flag. If so then we can make it a user-configurable flag so that it works via the hosted app you're using.

[rarkins]

FYI we needed to revert this due to windows problems. @secustor could you re-raise the PR and we'll add Windows tests to the branch to verify it before merging the second time?

[secustor]

I have re raised the PR #10469

[viceice]

@secustor Please add commits from #10461 and #10463 to that pr

[renovate-release]

🎉 This issue has been resolved in version 25.46.0 🎉

The release is available on:

• GitHub release
• 25.46.0

Your semantic-release bot 📦🚀

[hojerst]

It seems this experimental feature currently doesn't work with community providers. I'm currently running a upgrade for Telmate/proxmox from 2.6.7 to 2.7.1. However, renovate is not able to find the zips for hash creation: Failed to retrieve builds for terraform-provider-proxmox 2.7.1 (from renovates log).

[rarkins]

@hojerst can you create this as a bug report with reproduction repo?

[hojerst]

Sure: #10550

[ekristen]

Any idea when this will make it in as a user config option instead of env var only?

[xanonid]

The experimental feature RENOVATE_X_TERRAFORM_LOCK_FILE seems to need more recent node.js version. fs.rm was introduced with v14.14.0

With node.js v12, I get:

       "err": {
         "message": "fs.rm is not a function",
         "stack": "TypeError: fs.rm is not a function\n    at Object.rm (/node_modules/renovate/dist/util/fs/proxies.js:79:15)\n    at hashOfZipContent (/node_modules/renovate/dist/manager/terraform/lockfile/hash.js:70:14)\n    at async p_map_1.default.concurrency (/node_modules/renovate/dist/manager/terraform/lockfile/hash.js:89:20)\n    at async //node_modules/p-map/index.js:57:22"
       },

[rarkins]

Are you able to use node 14? I think we'd probably prefer to deprecate node 12 in our next major release (v26) seeing as this is experimental.

[viceice]

The experimental feature RENOVATE_X_TERRAFORM_LOCK_FILE seems to need more recent node.js version. fs.rm was introduced with v14.14.0

With node.js v12, I get:

       "err": {
         "message": "fs.rm is not a function",
         "stack": "TypeError: fs.rm is not a function\n    at Object.rm (/node_modules/renovate/dist/util/fs/proxies.js:79:15)\n    at hashOfZipContent (/node_modules/renovate/dist/manager/terraform/lockfile/hash.js:70:14)\n    at async p_map_1.default.concurrency (/node_modules/renovate/dist/manager/terraform/lockfile/hash.js:89:20)\n    at async //node_modules/p-map/index.js:57:22"
       },

Renovate official supports only node v14. V12 is only accepted. So to support legacy V12 we need community support.

renovate/package.json

Line 122 in cb752db

"node": ">=14.15.0"

renovate/package.json

Line 122 in cb752db

"node": ">=14.15.0"

[xanonid]

@viceice @rarkins
Thanks for the promptly clarification.