cli: Opt-in local cache directory for plugins #16000
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
This needs some documentation and some more thorough testing on Windows, but I've run out of time for today so I'm sharing this for some initial review. |
apparentlymart
force-pushed
the
f-plugin-cache
branch
4 times, most recently
from
507b93a
to
cce5b95
Compare
|
While I was testing more thoroughly on both Windows and Linux I found a few minor bugs which I've now fixed; the approach is still exactly the same. Once I started trying to document this I found myself having to reorganize some of the docs a bit, so the doc changes here ended up being a lot more elaborate than I originally planned. @jbardin given how much doc stuff I've added since you reviewed this, I think it probably deserves a second look before I merge it, if you're willing. |
apparentlymart
force-pushed
the
f-plugin-cache
branch
from
cce5b95
to
c0ceddc
Compare
|
This would be rad 👍 |
jbardin
approved these changes
Sep 29, 2017
This, in principle, allows us to make use of configuration information when we populate the Meta structure, though we won't actually make use of that until a subsequent commit.
If we encounter something that isn't a file -- for example, a dangling symlink whose referent has been deleted -- we'll ignore it so that we can either later produce a "no such plugin" error or auto-install a plugin that will actually work.
For users that have metered or slow internet connections it is annoying to have Terraform constantly re-downloading the same files when they initialize many separate directories. To help such users, here we add an opt-in mechanism to use a local directory as a read-through cache. When enabled, any plugin download will be skipped if a suitable file already exists in the cache directory. If the desired plugin isn't in the cache, it will be downloaded into the cache for use next time. This mechanism also serves to reduce total disk usage by allowing plugin files to be shared between many configurations, as long as the target system isn't Windows and supports either hardlinks or symlinks.
Here we add a new caller-settable field to command.Meta that activates a read-through cache directory for plugin installation.
Either the environment variable TF_PLUGIN_CACHE_DIR or a setting in the CLI config file (~/.terraformrc on Unix) allow opting in to the plugin caching behavior. This is opt-in because for new users we don't want to pollute their system with extra directories they don't know about. By opting in to caching, the user is assuming the responsibility to occasionally prune the cache over time as older plugins become stale and unused.
This is a tough one to unit tests because the behavior is tangled up in the code that hits releases.hashicorp.com, so we'll add this e2etest as some extra insurance that this works end-to-end.
Previously this file was mentioned in passing in a few locations. It now has enough functionality to warrant its own page.
Previously we described inline here where to put the .terraformrc file, but now we have a separate page all about this file that gives us more room to describe in more detail where the file is placed and what else it can do.
This mechanism for configuring plugins is now deprecated, since it's not capable of declaring plugin versions. Instead, we recommend just placing plugins into a particular directory, which is now documented on the main providers documentation page and linked from the more detailed docs on plugins in general.
Since we don't currently auto-install provisioner plugins this is currently placed on the providers documentation page and referred to as the "Provider Plugin Cache". In future this mechanism may also apply to provisioners, in which case we'll figure out at that point where better to place this information so it can be referenced from both the provider and provisioner documentation pages.
apparentlymart
force-pushed
the
f-plugin-cache
branch
from
c0ceddc
to
2208f7d
Compare
|
I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further. |
ghost
locked and limited conversation to collaborators
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
To avoid surprising users by creating a global plugin directory they don't know about, we elected to auto-install plugins into the local
.terraformdirectory to keep things contained.However, an implication of this decision is that by default a fresh set of plugins must be downloaded for each distinct working directory. This is bothersome for those on metered or slow Internet connections.
As a compromise, here we add an opt-in mechanism to treat a local directory as a read-through cache of plugins. When the cache directory is enabled, Terraform will still fetch metadata from releases.hashicorp.com (to learn if new versions are available), but once a particular version is selected the local cache will be consulted first to avoid re-downloading the plugin distribution zip file.
After opting in, it's the user's responsibility to manage this cache directory: Terraform will only add new files to it, and never clean up existing files. Since it is just a cache, it doesn't matter if the user "over-prunes" it since plugins can be re-downloaded automatically when they are next needed.
The cache directory is maintained separately from the auto-install directory to avoid it interfering with our mechanisms to detect when a newer plugin version is available. Unlike the local plugin search directories, which can prevent Terraform for looking for newer versions altogether, the cache directory is consulted only after a specific version of a plugin has been selected, and only for that specific version.
If installing a plugin from cache, Terraform will try to create either a hardlink or a symlink to the cache directory to minimize local disk usage when many configurations require the same plugin. If this can't be done (or if we're on Windows) the file is instead just copied into place.
The cache directory is enabled either from a setting in the
terraformrcfile (plugin_cache_dir) or from theTF_PLUGIN_CACHE_DIRenvironment variable, with the latter taking precedence.This change does not attempt to solve for the use-case where Terraform can't access the releases server at all. This mechanism is intended for those who wish to use the auto-install feature but wish to reduce the amount of data transfer required.
The existing
-plugin-dirmechanism is intended to enable working offline, and in a future change we may allow it to be set viaterraformrctoo, so it's easier to "set and forget". That is, however, outside the scope of this change.