Terraform generates different hashes for providers when use -plugin-dir parameter

[S0meth1ng85]

Terraform Version

Terraform v1.8.5
on windows_amd64

Terraform Configuration Files

terraform {
  backend "azurerm" {}
  required_providers {
    kubernetes = {
      source = "hashicorp/kubernetes"
      version = "=2.30.0"
    }
    azurerm = {
      source = "hashicorp/azurerm"
      version = "=3.114.0"
    }
  }
  required_version = ">= 1.8.5"
}

Debug Output

terraform init -upgrade --backend=false
https://gist.github.com/S0meth1ng85/4729352ef8ce74b650c936ac36e3a5e5

# This file is maintained automatically by "terraform init".
# Manual edits may be lost in future updates.

provider "registry.terraform.io/hashicorp/azurerm" {
  version     = "3.114.0"
  constraints = "3.114.0"
  hashes = [
    "h1:af8gzp2nuiJVXGW2v3Ch9+W/SjbwFCTpWaylAhbiby4=",
    "zh:016b6f4662d1cfcddbe968624e899c1a20c6df0ed5014cdeed19c3e945ea80ee",
    "zh:08448eeaaa9e9e84a2887282f9524faa2bb000fbdfcdac610c088a74e36e6911",
    "zh:17975bb18d0ad3e2530261773e4fbfae078bfc4db4e0a5458b823b3ec79642e1",
    "zh:3030ad1b13fe487ce791c851c6b5f3035af08f60b335d7be5ce6ce76af43062f",
    "zh:68b2914edae1049506aab9f2c11c5b2b2c8d01aa3e0ad53e07ce75ae58906a45",
    "zh:cffa9af324a0c621317b6d33f80a28159d01706846877d5784d37dad76635d78",
    "zh:d36d44617b890a8a6d404a016c10428c3393e072d484addfb56334183893998b",
    "zh:d5c217d7a24b32b18cb9ad47544050c5ec9e6b40ce3f34ff37be5e2d232b4dad",
    "zh:d5cd83a9701a9bcd17bbd86beb5accdc6c487fcfa472b868bc581e4d5b67d59d",
    "zh:f4ba0bd65d9a10f8185e163217e10e5fa91e386c68e6773c188881b088315477",
    "zh:f569b65999264a9416862bca5cd2a6177d94ccb0424f3a4ef424428912b9cb3c",
    "zh:f807554e5e08e38e6526e363641219e89ad9eda0b24ec09f25e61c74eece2490",
  ]
}

provider "registry.terraform.io/hashicorp/kubernetes" {
  version     = "2.30.0"
  constraints = "2.30.0"
  hashes = [
    "h1:KFBOyOGlS+BGrDfbuVdBhTIRefDo+vJEO/IooUR6T4g=",
    "zh:06531333a72fe6d2829f37a328e08a3fc4ed66226344a003b62418a834ac6c69",
    "zh:34480263939ef5007ce65c9f4945df5cab363f91e5260ae552bcd9f2ffeed444",
    "zh:59e71f9177da570c33507c44828288264c082d512138c5755800f2cd706c62bc",
    "zh:6e979b0c07326f9c8d1999096a920322d22261ca61d346b3a9775283d00a2fa5",
    "zh:73e3f228de0077b5c0a84ec5b1ada507fbb3456cba35a6b5758723f77715b7af",
    "zh:79e0de985159c056f001cc47a654620d51f5d55f554bcbcde1fe7d52f667db40",
    "zh:8accb9100f609377db42e3ced42cc9d5c36065a06644dfb21d3893bb8d4797fd",
    "zh:9f99aa0bf5caa4223a7dbf5d22d71c16083e782c4eea4b0130abfd6e6f1cec18",
    "zh:bcb2ad76ad05ec23f8da62231a2360d1f70bbcd28abd06b8458a9e2f17da7873",
    "zh:bce317d7790c2d3c4e724726dc78070db28daf7d861faa646fc891fe28842a29",
    "zh:ed0a8e7fa8a1c419a19840b421d18200c3a63cf16ccbcbc400cb375d5397f615",
    "zh:f569b65999264a9416862bca5cd2a6177d94ccb0424f3a4ef424428912b9cb3c",
  ]
}

terraform init -plugin-dir="path_to_providers" -upgrade --backend=false
https://gist.github.com/S0meth1ng85/26a07c61e9cdceb4d0c14db55158d8b3

# This file is maintained automatically by "terraform init".
# Manual edits may be lost in future updates.

provider "registry.terraform.io/hashicorp/azurerm" {
  version     = "3.114.0"
  constraints = "3.114.0"
  hashes = [
    "h1:Wwoq/9/PI3/HFAK3b7Wy02XL8ndxHHQft4Pr2mkbWy8=",
  ]
}

provider "registry.terraform.io/hashicorp/kubernetes" {
  version     = "2.30.0"
  constraints = "2.30.0"
  hashes = [
    "h1:jYQ5GNISKevc83YSsw+iR79DdUc1xbhAYOe0lPNJly8=",
  ]
}

Expected Behavior

terraform init -plugin-dir="path_to_providers" -upgrade --backend=false and terraform init -upgrade --backend=false should generate the same h1 hashes in terraform.lock.hcl file

Actual Behavior

When I run terraform init -upgrade --backend=false in Windows, Terraform downloads binaries for the providers configured in versions.tf and generates terraform.lock.hcl with h1 and zh hashes.
Then I copy terraform providers binaries to a separate folder, delete .terraform system folder, delete terraform.lock.hcl and trying to run terraform init -plugin-dir="path_to_providers" -upgrade --backend=false
In that case terraform generates a new terraform.lock.hcl file which has different h1 hashes for the same providers versions and the same binary files which have been downloaded previously by Terraform. And only this terraform.lock.hcl then works during pipeline run.

We have Windows pipeline agents which prebuild with appropriate terraform version and providers binaries which we use. Git repository of the project includes terraform.lock.hcl for each part of out Terraform blueprint.
Everything is ok with the rest of terraform.lock.hcl files which we generate using terraform init -upgrade --backend=false from Linux, for example. But one part of blueprint which we run on Windows agents do not work with terraform.lock.hcl and instead we need to generate this file using terraform init -plugin-dir="path_to_providers" -upgrade --backend=false.

I am 100% sure that the binaries are the same. We copy binaries from .terraform/providers folder

Steps to Reproduce

1. Configure versions.tf for example to use "hashicorp/kubernetes" in version 2.30.0 and "hashicorp/azurerm" in version "=3.114.0"
2. terraform init -upgrade --backend=false in Windows
3. Check generated terraform.lock.hcl and hashes for h1
4. Copy downloaded terraform providers binaries to a separate folder
5. Remove .terraform folder and terraform.lock.hcl
6. terraform init -plugin-dir="path_to_providers_binaries" -upgrade --backend=false
7. Check terraform.lock.hcl
8. h1 hashes are differ then using terraform init -upgrade --backend=false

Additional Context

No response

References

No response

[jbardin]

Hi @S0meth1ng85,

Thanks for filing the issue. The log output you included looks as it should be expected, so I'm not sure exactly what we're looking for here. Can you show the actual lock files with the different hashes?

[S0meth1ng85]

@jbardin , I have updated my original post and added terraform.lock.hcl files for both variants

[jbardin]

Hi @S0meth1ng85,

I don't see any reason the provider data should return a different hash depending on the location, given that both locations contain the exact same data. Exactly how are you creating the directory of providers?

[S0meth1ng85]

Hi @S0meth1ng85,

I don't see any reason the provider data should return a different hash depending on the location, given that both locations contain the exact same data. Exactly how are you creating the directory of providers?

I have such folders structure:

And I pass this providers folder using -plugin-dir

Yesterday, Ive checked many times. I do not know what happened. But Ive copied whole providers folder downloaded previously by terraform to my providers location and used terraform init -upgrade -plugin-dir and hashes looks good know.

I really do not know what is wrong with my binaries..

I think, for now we can close this issue. Sorry for disturbing you

[jbardin]

@S0meth1ng85,

It doesn't look like you copied the entire directory structure, newer provider releases contain a LICENSE.txt file which is missing in your tree. The hashes are based on the entire contents of the archive, because providers may package multiple files into their distribution.