Terraform 1.6.0 - Host Key Verification Failed when using git ssh regression

[marvelous-melanie]

Terraform Version

Terraform v1.6.0

Terraform Configuration Files

module "XXXXXXXXXX" {
  source = "git@example.com:site/module.git"
  name   = "XXXXXXXXXX"
}

Debug Output

on linux_amd64
Initializing plugins and modules...

Initializing Terraform Cloud...
Initializing modules...
Downloading git::ssh://git@github.com/company/repo-name.git for module-name...
╷
│ Error: Failed to download module
│ 
│   on module_name.tf line 25:
│   25: module "module_name" {
│ 
│ Could not download module "module_name" (module_name.tf:25)
│ source code from
│ "git::ssh://git@github.com/company/repo-name.git": error
│ downloading 'ssh://git@github.com/company/repo-name.git':
│ /usr/bin/git exited with 128: Cloning into
│ '.terraform/modules/module_name'...
│ Host key verification failed.
│ fatal: Could not read from remote repository.
│ 
│ Please make sure you have the correct access rights
│ and the repository exists.
│

Expected Behavior

This module should successfully download. We have multiple other workspaces running with the same exact configuration (same SSH key, specified in the same way, accessing the same remote module using the exact same module source line) but lower versions of Terraform.

Those all work perfectly, and this workspace also worked perfectly until starting to fail yesterday, again with no configuration change on our end. Reverting this workspace to 1.5.7 caused the module to begin downloading properly again.

Actual Behavior

The module fails to download.

Steps to Reproduce

1. Using a v1.6.0 remote Terraform Cloud workspace, configure Terraform Cloud to use a GitHub SSH key
2. Create a module with a remote git source
3. terraform init
4. terraform plan

Additional Context

No response

References

I believe this bug was previously opened and closed in this issue, but given that we started experiencing it again a few days ago, I suspect it has been reintroduced.

[jbardin]

Hi @marvelous-melanie,

That linked issue was fixed after version v1.6.0, you should be using the latest patch release, which is v1.6.6. Can you confirm if the problem still exists in a current release?

Thanks!

[marvelous-melanie]

Hi @jbardin , to my understanding, the issue was fixed with a backport to all of 1.6? We have been running on 1.6.0 for months and the issue just started occurring yesterday/today, hence my thinking that this is a new issue

[jbardin]

@marvelous-melanie, there is no way to retroactively change code that has already been released. In order to get a new patched version of the binary, you would need to download that patched version. The latest patch release for v1.6 is here:v1.6.6, which will include the fix from that issue. If that does not solve your problem, then it is a different issue and we can investigate further.

[josiahwitheford]

Our pipelines started failing today with this issue. Setting the tf client version to 1.6.6 did indeed resolve the issue.

I tested with the latest 1.7.0 and it seems that was causing the host key verification error. Has the fix perhaps not made it into that release yet? Interesting 1.6.0 works fine for us.

[jbardin]

@josiahwitheford, Terraform v1.6 and v1.7 use the exact same go-getter code which calls git, so I would first look for some other discrepancy between the two systems.

[levid0s]

I'm having issues too, although first time configuring this, I've been trying now for two days to set up Git SSH between TFC and Gitlab using Terraform.

Edit: I realised my error is coming from the Gitlab runner and not the TFC runner. It's confusing, because terraform init clones the modules on the Gitlab runner, but terraform plan will do another clone of the modules on the TFC runner side. So the git authentication to the module repos has to be set up on both places.

I fixed the error on the Gitlab side by following this guide:

- eval $(ssh-agent -s) > /dev/null
- ssh-add <(echo "$GITLAB_CI_READER_SSH_KEY") > /dev/null
- mkdir -p ~/.ssh
- echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config

I was getting this error on the TFC runner side too, but for that, I opened a separate issue.

[marvelous-melanie]

I want to reiterate that we did not change anything with our configuration when these failures started. The timeline was as follows:

1. In October 2023, we set up a Terraform Cloud workspace running 1.6.0. This workspace was immediately configured to pull a remote module from a private GitHub repository.
2. Runs on this workspace succeeded and the remote module successfully pulled.
3. 9 days ago, on Friday, January 12, a plan was run and succeeded on our Terraform config, including pulling the remote module.
4. 5 days ago, with absolutely no code changes between the two runs, same commit, same config, not a single setting on TF Cloud or GitHub touched, the remote module failed to pull with the info in this bug report.

This seems to indicate to me that something on Terrraform's end has changed which is why I opened the issue. Is it possible something changed with the way Terraform Cloud uses go-getter or sets up SSH keys? I do understand your response here, but the inconsistent behavior isn't really explained.

[brandonc]

When initiating a run on Terraform Cloud using terraform CLI 1.7.0, the .terraform/modules directory was being unintentionally excluded from the configuration upload and in turn Terraform Cloud would then attempt to download module sources. This could cause new authentication errors during initialization. This is fixed in hashicorp/go-slug#54 and should be addressed in the next terraform 1.7 patch release.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.