Loading modules via Git, terraform ignores GIT_SSH environment variable

[standardizer]

Terraform Version

v0.11.10

Terraform Configuration Files

module "rds_mymodule" {
  source = "git::ssh://git@gitlab.my-site.org/devops/terraform-aws/modules/mymodule.git?ref=v0.0.5"
  …
}
  

Expected Behavior

Running terraform init, which ist loading the module mymodule.git using git internally, the executable plink.exe should be used, like git clone does it right here: https://gist.github.com/standardizer/da751da4c325af97ca9a18247dd2e53d
The SSH key is delivered by Pageant (part of PuTTY), which is called by plink.exe.

Actual Behavior

Running terraform init, which ist loading the module mymodule.git using git internally, the executable ssh is used: https://gist.github.com/standardizer/09fda4ca1e246e4e43868e84ce7a6c30
The module cannot be cloned, because ssh is not prepared to use the SSH key, as it should not be called at all.

Steps to Reproduce

terraform init

Additional Context

Windows 10 Pro x64 Version 18.03

Environment variables set:
SET GIT_SSH=C:\Program Files\PuTTY\plink.exe

Git configuration (additionally set, but not needed for git clone working properly):
git config --global ssh.variant plink

References

• Add SSH agent support for Windows #4323 (“The Windows support is limited to the Pageant SSH authentication agent.”)
• Add Support for OpenSSH on Windows 10 #17866 (“… to extend the existing SSH on Windows support (Pageant only) …”)
• terraform source not able to pull github enterprise repository. #18541 (here using ssh-agent, but did not work for me as well)

[standardizer]

After doing some further inspection in the terraform source code I understand, that the environment variable GIT_SSH ist not interpreted.
You have to set GIT_SSH_COMMAND="C:\Program Files\PuTTY\plink.exe" -v (for verbose output, and if you want to use Pageant) instead.
Do we see this as a bug, as the current Git for Windows installer only sets GIT_SSH, but not GIT_SSH_COMMAND?

[mildwonkey]

Hi @standardizer, you've got it correct in your second comment. The git environment variables are not parsed by terraform at all but instead used directly by the installed git client.

Thanks!

[standardizer]

Hi @mildwonkey, you’re assuming the Git client is responsible for handling GIT_SSH_COMMAND and ignoring GIT_SSH environment variable, right?
But git clone ssh://git@gitlab.my-site.org/devops/terraform-aws/modules/mymodule.git works with having set GIT_SSH only (and leaving GIT_SSH_COMMAND empty).
Having a look into the terraform source code, I found GIT_SSH_COMMAND in different files, but not GIT_SSH. The decision to handle GIT_SSH_COMMAND and to ignore GIT_SSH seems to be terraform specific. I think that is not a good idea, because current Git for Windows installer only sets GIT_SSH, but not GIT_SSH_COMMAND. What do you think?

[mildwonkey]

Hmm, thank you for the information @standardizer - I'll re-open this and label it appropriately so we can take a look.

(If you are interested in contributing to terraform, a PR would also be welcome!)

[apparentlymart]

I think I remember encountering this before, and finding that Terraform was handling the environment variable itself here because it wanted to impose its own special default handling in the case where the variables aren't set. I think it was to handle a situation related to running Git on Windows.

I don't remember exactly where this code is, but treating GIT_SSH and GIT_SSH_COMMAND as synonyms seems like a reasonable idea. If Git itself supports both then we should try to implement the same priority ordering too, so that our behavior is consistent with Git's.

[jabba2324]

I'm having a similar issue, I've set GIT_SSH and GIT_SSH_COMMAND env variables to C:\Windows\System32\OpenSSH\ssh.exe

I've also set my global git config to do the same with git config --global core.sshCommand "'C:\Windows\System32\OpenSSH\ssh.exe'"

But terraform init will not use the given ssh agent, using git clone directly in cmd works fine.

[bbooth]

I was able to get this working by escaping the GIT_SSH_COMMAND environment variable. C:\\Windows\\System32\\OpenSSH\\ssh.exe

[derekrprice]

I don't think that treating GIT_SSH & GIT_SSH_COMMAND as synonyms is sufficient to fix this problem. Terraform shouldn't be overriding the configured git behavior at all. Since it appears to be checking an environment variable directly and trying to override normal git behavior regarding SSH when it sees that it isn't set, the core.sshCommand key, which can be set in a user's .gitconfig or in the system git config, is also ignored. I worked around this by setting GIT_SSH_COMMAND in the environment, but it certainly came as a surprise to me when git clone of my modules repository worked just fine from the command line but git run via Terraform reports that I have a pubkey with an invalid format (in fact, it appears to be trying to read my private key for some reason).

[mhvelplund]

I was able to get this working by escaping the GIT_SSH_COMMAND environment variable. C:\\Windows\\System32\\OpenSSH\\ssh.exe

Additionally; if the path to plink has a space in it, this won't work.

[tynril]

Just encountered that issue, which I worked around by setting GIT_SSH_COMMAND to an escaped path to ssh.exe, as suggested by @derekrprice.

Additionally, while researching this error, I've found several issues (such as this one) that were opened with similar errors, which were caused by an ambiguity in parsing module source between "SCP style" and "URL style".

While that bug has been fixed in the current version of Terraform, because the symptom is the same as for the bug described here (failure to clone a module from Git via SSH), it made it challenging to find this workaround.

[snolan-uturn]

I'm pretty sure this is because when Terraform runs a git clone, Git automatically uses the OpenSSH client installed with Git for Windows - when you run Git on its own it works because the SSH client prompts you to enter the passphrase for the private key (if running in a Git Bash and the ssh-agent is running, the SSH client stores the identity in the ssh-agent so that you are not prompted for the passphrase again until the ssh-agent stops). When Terraform uses Git for Windows the prompt does not work, so your identity is not known and the error above occurs (debugging shows that the prompt gets lost with a message similar to debug1: read_passphrase: can't open /dev/tty - the SSH client included with Git for Windows is a Cygwin client that I suspect simply doesn't work well with the Terraform Windows executable).

What happens when you set GIT_SSH_COMMAND as people have suggested here is that you are telling Git for Windows to use the OpenSSH-Win32 client that is installed in Windows recent versions (hence the C:\Windows\System32\OpenSSH\ssh.exe path) and it works because it is compiled to work with Windows - internally it uses Windows paths rather than Unix paths, and it has a Windows service implementation of ssh-agent for storing the identity after prompting for a private key passphrase.

Some have said that GIT_SSH_COMMAND doesn't work if there is a space in the path - it does work, but I think that if you've tried it with a path with a space you've probably tried to set the path to point to C:\Program Files\Git\usr\bin\ssh.exe - but that is the SSH client installed with Git for Windows, which as I've explained doesn't work with Terraform.

Here's what I do in Windows, works well for me:

1. Get latest OpenSSH-Win64.zip from https://github.com/PowerShell/Win32-OpenSSH/releases - there are much newer releases than the release that installs in Windows Features

2. Extract to C:\Program Files\OpenSSH

3. Add C:\Program Files\OpenSSH to PATH

4. Run install-sshd.ps1 in C:\Program Files\OpenSSH (need this for the ssh-agent even if not using the server)

5. Start the ssh-agent service with PowerShell

   Start-Service ssh-agent

   Optionally, set the service to start automatically

   Set-Service ssh-agent -StartupType Automatic

6. Get a list of stored identities

   ssh-add -l

7. Add an identity, for example to add the custom private SSH key C:\Users\your_user\.ssh\id_rsa_myfile

   ssh-add C:\Users\your_user\.ssh\id_rsa_myfile

   You will be prompted for the passphrase for the private key, and then the identity for key will be stored in the ssh-agent and available for use with the OpenSSH Client.

8. Finally, to have Git use the OpenSSH-Win32 client use the GIT_SSH_COMMAND environment variable to specify the path to ssh.exe, the path needs to be enclosed with quotes because it contains a space. You can set the value for the current session only with:

   $env:GIT_SSH_COMMAND='"C:\Program Files\OpenSSH\ssh.exe"'

   Or to set the value permanently:

   [System.Environment]::SetEnvironmentVariable('GIT_SSH_COMMAND', '"C:\Program Files\OpenSSH\ssh.exe"', [System.EnvironmentVariableTarget]::User)

You may need to close all Visual Studio Code open windows after setting the environment variable as Visual Studio Code sometimes does not see new environment variables even after opening a new Terminal.

Full instructions for installing OpenSSH-Win32 are here: https://github.com/PowerShell/Win32-OpenSSH/wiki/Install-Win32-OpenSSH

More info here too: https://docs.microsoft.com/en-us/windows-server/administration/openssh/openssh_keymanagement

[marcortw]

For the record: I am using a Windows 10 pageant/plink based git+ssh setup. All I had to do is explicitly declare GIT_SSH_COMMAND with surrounding double quotes:

[snolan-uturn]

For the record: I am using a Windows 10 pageant/plink based git+ssh setup. All I had to do is explicitly declare GIT_SSH_COMMAND with surrounding double quotes:

Yes, that will work for the same reason as OpenSSH-Win32 ssh.exe, ie. you're doing what I described above but using PuTTY plink.exe instead of the OpenSSH-Win32 ssh.exe that I'm using. As you've seen the space in the path works fine for plink.exe too because the space is not the issue that people are seeing, the issue is caused by using the CygWin version of ssh.exe.

[marcortw]

What I wanted to highlight was the fact that I had to explicitly add the double quotes in the variable value, Without the quotes it didn't work for me.

[danieldreier]

I want to apologize for the slow response time on this issue, and also let you know that I am bulk-closing all issues exclusively reported against Terraform 0.11.x, including this issue, because we are no longer investigating issues reported against Terraform 0.11.x. In most cases, when we try to reproduce issues reported against 0.11, we either can't reproduce them anymore, or the reporter has moved on, so we believe we can better support the Terraform user community by prioritizing more recent issues.

Terraform 0.12 has been available since May of 2019, and there are really significant benefits to adopting it. I know that migrating from 0.11 to versions past 0.12 can require a bit of effort, but it really is worth it, and the upgrade path is pretty well understood in the community by now. 0.14 is available and stable, and we are quickly approaching an 0.15 release.

We have made a significant effort in the last year to stay on top of bug reports; we have triaged almost all new bug reports within 1-2 weeks for 6+ months now. If you are still experiencing this problem, please submit a new bug report with a reproduction case that works on 0.14.x, link this old issue for context, and we will triage it.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.