Packer rate-limited by github API, does not log issue by default

[sparrc]

When filing a bug, please include the following headings if possible. Any
example text in this template can be deleted.

Overview of the Issue

Hello, we are using packer to build some AWS AMIs. We have found that packer is getting rate-limited by github, but don't have any clear way to resolve that issue, as it appears that these are github API requests that are being made by packer itself, and we haven't been able to find a way to give packer a github "personal access token" to use.

It would also be nice if packer could log when it encounters errors in packer init, so that we wouldnt have to use PACKER_LOG=1 to debug this issue.

We run the packer build in a CI system, and we started noticing that the builds were frequently failing when we called packer init and would display just this error message:

no release version found for the github.com/hashicorp/amazon plugin matching the constraint(s): ">= 1.0.0"

After setting PACKER_LOG=1, we discovered that this failure is being caused by being throttled by github:

+ PACKER_TYPE=amazon-ebs.al2
+ ./packer --version
1.7.4
+ PACKER_LOG=1
+ ./packer init .
2021/09/09 17:49:42 [INFO] Packer version: 1.7.4 [go1.16.6 linux amd64]
2021/09/09 17:49:42 Detected config directory from env var: /tmp/tmp.GZ133EOGJO
2021/09/09 17:49:42 [TRACE] discovering plugins in /local/REDACTED-nobody/00484ceb-6d6a-4d80-a6b3-7906e12d0835/env/REDACTED-1.0/test-runtime
2021/09/09 17:49:42 Detected config directory from env var: /tmp/tmp.GZ133EOGJO
2021/09/09 17:49:42 [TRACE] discovering plugins in /tmp/tmp.GZ133EOGJO/.packer.d/plugins
2021/09/09 17:49:42 [TRACE] discovering plugins in .
2021/09/09 17:49:42 [INFO] PACKER_CONFIG env var not set; checking the default config file path
2021/09/09 17:49:42 Detected config directory from env var: /tmp/tmp.GZ133EOGJO
2021/09/09 17:49:42 [INFO] PACKER_CONFIG env var set; attempting to open config file: /tmp/tmp.GZ133EOGJO/.packerconfig
2021/09/09 17:49:42 [WARN] Config file doesn't exist: /tmp/tmp.GZ133EOGJO/.packerconfig
2021/09/09 17:49:42 Detected config directory from env var: /tmp/tmp.GZ133EOGJO
2021/09/09 17:49:42 [INFO] Setting cache directory: /local/REDACTED-nobody/00484ceb-6d6a-4d80-a6b3-7906e12d0835/src/REDACTED/test-runtime/packer_cache
2021/09/09 17:49:42 Detected config directory from env var: /tmp/tmp.GZ133EOGJO
e: Running in background, not using a TTY
2021/09/09 17:49:42 [TRACE] init: plugingetter.ListInstallationsOptions{FromFolders:[]string{"/local/REDACTED-nobody/00484ceb-6d6a-4d80-a6b3-7906e12d0835/env/REDACTED-1.0/test-runtime/packer", ".", "/tmp/tmp.GZ133EOGJO/.packer.d/plugins"}, BinaryInstallationOptions:plugingetter.BinaryInstallationOptions{APIVersionMajor:"5", APIVersionMinor:"0", OS:"linux", ARCH:"amd64", Ext:"", Checksummers:[]plugingetter.Checksummer{plugingetter.Checksummer{Type:"sha256", Hash:(*sha256.digest)(0xc0000e4680)}}}}
2021/09/09 17:49:42 [TRACE] listing potential installations for "github.com/hashicorp/amazon" that match ">= 1.0.0". plugingetter.ListInstallationsOptions{FromFolders:[]string{"/local/REDACTED-nobody/00484ceb-6d6a-4d80-a6b3-7906e12d0835/env/REDACTED-1.0/test-runtime/packer", ".", "/tmp/tmp.GZ133EOGJO/.packer.d/plugins"}, BinaryInstallationOptions:plugingetter.BinaryInstallationOptions{APIVersionMajor:"5", APIVersionMinor:"0", OS:"linux", ARCH:"amd64", Ext:"", Checksummers:[]plugingetter.Checksummer{plugingetter.Checksummer{Type:"sha256", Hash:(*sha256.digest)(0xc0000e4680)}}}}
2021/09/09 17:49:42 [TRACE] for plugin github.com/hashicorp/amazon found 0 matching installation(s)
2021/09/09 17:49:42 [TRACE] getting available versions for the github.com/hashicorp/amazon plugin
2021/09/09 17:49:42 [DEBUG] github-getter: getting "https://api.github.com/repos/hashicorp/packer-plugin-amazon/git/matching-refs/tags"
2021/09/09 17:49:42 [TRACE] &{%!q(*github.Client=&{{0 0} 0xc00044b650 0xc000459c20 0xc000459cb0 packer-getter-github-1.7.4 {0 0} [{60 0 {{0 63766808924 0x8e1d720}}} {0 0 {{0 0 <nil>}}}] {0xc00068d340} 0xc00068d3c8 0xc00068d3c8 0xc00068d3c8 0xc00068d3c8 0xc00068d3c8 0xc00068d3c8 0xc00068d3c8 0xc00068d3c8 0xc00068d3c8 0xc00068d3c8 0xc00068d3c8 0xc00068d3c8 0xc00068d3c8 0xc00068d3c8 0xc00068d3c8 0xc00062ef10 0xc00068d3c8 0xc00068d3c8 0xc00068d3c8 0xc00068d3c8 0xc00068d3c8 0xc00068d3c8 0xc00068d3c8 0xc00068d3c8 0xc00068d3c8}) "packer-getter-github-1.7.4"} getter could not get release: GET https://api.github.com/repos/hashicorp/packer-plugin-amazon/git/matching-refs/tags: 403 API rate limit exceeded for 205.XXX.XXX.XXX. (But here's the good news: Authenticated requests get a higher rate limit. Check out the documentation for more details.) [rate reset in 39m01s]
2021/09/09 17:49:42 [DEBUG] will try to install: []
2021/09/09 17:49:42 [INFO] (telemetry) Finalizing.
no release version found for the github.com/hashicorp/amazon plugin matching the constraint(s): ">= 1.0.0"
2021/09/09 17:49:43 waiting for all plugin processes to complete...

Reproduction Steps

Using a single IP address, recreate and destroy the packer environment many times and run packer init . each time.

We also have this in our hcl file:

packer {
  required_plugins {
    amazon = {
      version = ">= 1.0.0"
      source  = "github.com/hashicorp/amazon"
    }
  }
}

Packer version

1.7.4

Simplified Packer Buildfile

the basic packer example build file in the aws tutorial should suffice: https://learn.hashicorp.com/tutorials/packer/aws-get-started-build-image?in=packer/aws-get-started

Operating system and Environment details

AL2 linux, x86_64

[sparrc]

From what I can tell we would need to add the ability to configure and then specify a github token here: XXX, I will look into seeing if I can submit a PR for this.

Looks like we can use PACKER_GITHUB_API_TOKEN, though I think this issue should stay open to improve logging around this, and maybe document this env var as I dont see it officially documented anywhere.

[azr]

related: #11223

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.