Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix build error due to deleted module. Add pause_before to packer-basic-example. Shave other yaks. #775

Merged
merged 6 commits into from
Feb 4, 2021

Conversation

brikis98
Copy link
Member

@brikis98 brikis98 commented Feb 1, 2021

This PR fixes multiple issues:

  1. Fix a build failure where a module we rely on in CI was deleted.

  2. Add pause_before to packer-basic-example. Without this, we sometimes get build failures such as:

    TestBuildPackerArtifactWithGit/ref-sha 2021-01-29T15:45:17Z logger.go:66: 1611935117,,ui,message,    amazon-ebs: Reading package lists...
    TestBuildPackerArtifactWithGit/ref-sha 2021-01-29T15:45:17Z logger.go:66: 1611935117,,ui,message,    amazon-ebs: Reading package lists...
    TestBuildPackerArtifactWithGit/ref-sha 2021-01-29T15:45:17Z logger.go:66: 1611935117,,ui,message,    amazon-ebs: Building dependency tree...
    TestBuildPackerArtifactWithGit/ref-sha 2021-01-29T15:45:17Z logger.go:66: 1611935117,,ui,message,    amazon-ebs: Reading state information...
    TestBuildPackerArtifactWithGit/ref-sha 2021-01-29T15:45:17Z logger.go:66: 1611935117,,ui,message,    amazon-ebs: Calculating upgrade...
    TestBuildPackerArtifactWithGit/ref-sha 2021-01-29T15:45:17Z logger.go:66: 1611935117,,ui,message,    amazon-ebs: The following packages have been kept back:
    TestBuildPackerArtifactWithGit/ref-sha 2021-01-29T15:45:17Z logger.go:66: 1611935117,,ui,message,    amazon-ebs:   linux-aws linux-headers-aws linux-image-aws
    TestBuildPackerArtifactWithGit/ref-sha 2021-01-29T15:45:18Z logger.go:66: 1611935118,,ui,message,    amazon-ebs: The following packages will be upgraded:
    TestBuildPackerArtifactWithGit/ref-sha 2021-01-29T15:45:18Z logger.go:66: 1611935118,,ui,message,    amazon-ebs:   sudo tzdata
    TestBuildPackerArtifactWithGit/ref-sha 2021-01-29T15:45:18Z logger.go:66: 1611935118,,ui,error,==> amazon-ebs: E: Could not open file /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_xenial-backports_universe_i18n_Translation-en - open (2: No such file or directory)
    TestBuildPackerArtifactWithGit/ref-sha 2021-01-29T15:45:18Z logger.go:66: 1611935118,,ui,error,==> amazon-ebs: E: Could not open file /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_xenial-backports_universe_binary-amd64_Packages - open (2: No such file or directory)
    TestBuildPackerArtifactWithGit/ref-sha 2021-01-29T15:45:18Z logger.go:66: 1611935118,,ui,message,    amazon-ebs: 2 upgraded%!!(MISSING)(PACKER_COMMA) 0 newly installed%!!(MISSING)(PACKER_COMMA) 0 to remove and 3 not upgraded.
    TestBuildPackerArtifactWithGit/ref-sha 2021-01-29T15:45:18Z logger.go:66: 1611935118,,ui,error,==> amazon-ebs: E: Could not open file /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_xenial-backports_main_i18n_Translation-en - open (2: No such file or directory)
    TestBuildPackerArtifactWithGit/ref-sha 2021-01-29T15:45:18Z logger.go:66: 1611935118,,ui,error,==> amazon-ebs: E: Could not open file /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_xenial-backports_main_binary-amd64_Packages - open (2: No such file or directory)
    TestBuildPackerArtifactWithGit/ref-sha 2021-01-29T15:45:18Z logger.go:66: 1611935118,,ui,error,==> amazon-ebs: E: Could not open file 
    

    I think this is due to the issue where you run apt too soon after boot...

  3. Update gruntwork-installer version to pick up a new version of fetch that fixes a bug with the --branch argument.

  4. Pick instance types dynamically for Packer builds, rather than hard-coding t2.micro, as t2.micro isn't available in all regions.

  5. Add more retries for yet more transient provider download issues.

Without this, we sometimes get build failures such as:

```
TestBuildPackerArtifactWithGit/ref-sha 2021-01-29T15:45:17Z logger.go:66: 1611935117,,ui,message,    amazon-ebs: Reading package lists...
TestBuildPackerArtifactWithGit/ref-sha 2021-01-29T15:45:17Z logger.go:66: 1611935117,,ui,message,    amazon-ebs: Reading package lists...
TestBuildPackerArtifactWithGit/ref-sha 2021-01-29T15:45:17Z logger.go:66: 1611935117,,ui,message,    amazon-ebs: Building dependency tree...
TestBuildPackerArtifactWithGit/ref-sha 2021-01-29T15:45:17Z logger.go:66: 1611935117,,ui,message,    amazon-ebs: Reading state information...
TestBuildPackerArtifactWithGit/ref-sha 2021-01-29T15:45:17Z logger.go:66: 1611935117,,ui,message,    amazon-ebs: Calculating upgrade...
TestBuildPackerArtifactWithGit/ref-sha 2021-01-29T15:45:17Z logger.go:66: 1611935117,,ui,message,    amazon-ebs: The following packages have been kept back:
TestBuildPackerArtifactWithGit/ref-sha 2021-01-29T15:45:17Z logger.go:66: 1611935117,,ui,message,    amazon-ebs:   linux-aws linux-headers-aws linux-image-aws
TestBuildPackerArtifactWithGit/ref-sha 2021-01-29T15:45:18Z logger.go:66: 1611935118,,ui,message,    amazon-ebs: The following packages will be upgraded:
TestBuildPackerArtifactWithGit/ref-sha 2021-01-29T15:45:18Z logger.go:66: 1611935118,,ui,message,    amazon-ebs:   sudo tzdata
TestBuildPackerArtifactWithGit/ref-sha 2021-01-29T15:45:18Z logger.go:66: 1611935118,,ui,error,==> amazon-ebs: E: Could not open file /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_xenial-backports_universe_i18n_Translation-en - open (2: No such file or directory)
TestBuildPackerArtifactWithGit/ref-sha 2021-01-29T15:45:18Z logger.go:66: 1611935118,,ui,error,==> amazon-ebs: E: Could not open file /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_xenial-backports_universe_binary-amd64_Packages - open (2: No such file or directory)
TestBuildPackerArtifactWithGit/ref-sha 2021-01-29T15:45:18Z logger.go:66: 1611935118,,ui,message,    amazon-ebs: 2 upgraded%!!(MISSING)(PACKER_COMMA) 0 newly installed%!!(MISSING)(PACKER_COMMA) 0 to remove and 3 not upgraded.
TestBuildPackerArtifactWithGit/ref-sha 2021-01-29T15:45:18Z logger.go:66: 1611935118,,ui,error,==> amazon-ebs: E: Could not open file /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_xenial-backports_main_i18n_Translation-en - open (2: No such file or directory)
TestBuildPackerArtifactWithGit/ref-sha 2021-01-29T15:45:18Z logger.go:66: 1611935118,,ui,error,==> amazon-ebs: E: Could not open file /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_xenial-backports_main_binary-amd64_Packages - open (2: No such file or directory)
TestBuildPackerArtifactWithGit/ref-sha 2021-01-29T15:45:18Z logger.go:66: 1611935118,,ui,error,==> amazon-ebs: E: Could not open file 
```

I _think_ this is due to the issue where you run `apt` too soon after boot...
@brikis98
Copy link
Member Author

brikis98 commented Feb 1, 2021

Build failed because we were relying on the terraform-aws-ci version of gcp-helpers, but these were moved to the terraform-google-ci repo.

@brikis98 brikis98 changed the title Add pause_before to packer-basic-example Fix build error due to deleted module. Add pause_before to packer-basic-example. Feb 1, 2021
@brikis98
Copy link
Member Author

brikis98 commented Feb 3, 2021

This is currently blocked due to gruntwork-io/fetch#87.

@robmorgan
Copy link
Contributor

FYI - gruntwork-io/fetch#87 has been fixed and released.

This pulls in the latest version of fetch, which has a fix for the --branch param.
@brikis98
Copy link
Member Author

brikis98 commented Feb 4, 2021

FYI - gruntwork-io/fetch#87 has been fixed and released.

Indeed! I just released https://github.com/gruntwork-io/gruntwork-installer/releases/tag/v0.0.33 too, and now have updated this PR to use it to see if it fixes the build.

@brikis98 brikis98 changed the title Fix build error due to deleted module. Add pause_before to packer-basic-example. Fix build error due to deleted module. Add pause_before to packer-basic-example. Shave other yaks. Feb 4, 2021
@brikis98
Copy link
Member Author

brikis98 commented Feb 4, 2021

Yay, tests passed!

Copy link
Contributor

@infraredgirl infraredgirl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@brikis98
Copy link
Member Author

brikis98 commented Feb 4, 2021

Thanks for the review! Merging now.

@brikis98 brikis98 merged commit 78a1263 into master Feb 4, 2021
@brikis98 brikis98 deleted the brikis98-patch-1 branch February 4, 2021 14:38
@robmorgan
Copy link
Contributor

Yay!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants