Plugin Cache Not Being Updated after Failed Download

[philnielsen]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request. Searching for pre-existing feature requests helps us consolidate datapoints for identical requirements into a single place, thank you!
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.

---

Overview of the Issue

We had a plugin fail to download because of disk space issues on our atlantis persistant volume. possibly related to #916 Error for that is as follows:

Initializing the backend...

Successfully configured the backend "s3"! Terraform will automatically
use this backend unless the backend configuration changes.

Initializing provider plugins...
- Reusing previous version of hashicorp/aws from the dependency lock file
- Installing hashicorp/aws v4.17.1...

Error: Failed to install provider

Error while installing hashicorp/aws v4.17.1: write
/atlantis-data/plugin-cache/registry.terraform.io/hashicorp/aws/4.17.1/linux_amd64/terraform-provider-aws_v4.17.1_x5:
no space left on device

Now whenever we try to use that provider (in other branches, new PRs, etc) it tries to use the cache but it can't find that provider version with the following error.

Initializing the backend...

Successfully configured the backend "s3"! Terraform will automatically
use this backend unless the backend configuration changes.

Initializing provider plugins...
- Reusing previous version of hashicorp/aws from the dependency lock file
- Using hashicorp/aws v4.17.1 from the shared cache directory

Error: Failed to install provider from shared cache

Error while importing hashicorp/aws v4.17.1 from the shared cache directory:
the provider cache at .terraform/providers has a copy of
registry.terraform.io/hashicorp/aws 4.17.1 that doesn't match any of the
checksums recorded in the dependency lock file.

Is there a symlink somewhere to that that I am missing that I need to cleanup? I tried to go manually into the atlantis container and running TF_PLUGIN_CACHE_DIR=/atlantis-data/plugin-cache/ /atlantis-data/bin/terraform0.14.10 init and that fixes it for the current PR but doesn't fix it for any other PRs.

Happy to provide any more details as needed!

Reproduction Steps

This is using terraform 0.14.10 (but I have tried with the latest tf version, 1.2.2, and got the same error.) and Atlantis 0.18.5. I haven't been able to reproduce this on my test atlantis running the latest version because I can't get the download to fail again 😢

Logs

Environment details

Additional Context

[gruzewski]

Looks like it is related to Terraform rather than Atlantis (see issue 27530 and issue 27388).

In my case, updating lock file for MacOs ARM and Linux 64-bit didn't help and I couldn't get Atlantis applying my changes. The solution that worked for me, suggested in this comment, was to:

1. Generate a hash for the platform on which Atlantis was running on (terraform providers lock -platform=linux_amd64) then push my changes and let Atlantis to do the plan.
2. Generate hashes for all the remaining platforms (terraform providers lock -platform=linux_amd64 -platform=darwin_arm64), push it and then apply.

[jamengual]

TF related