Copy hidden directories using CopyTerraformFolderToTemp

[ztki]

Currently - CopyTerraformFolderToTemp allows us to copy around terraform modules to temp locations - which is particularly nice for running a bunch of parallel tests, however - it doesn't copy hidden directories such as .terraform which prevents me from trying to cache both my plugins and modules prior to running my tests.

Something I've run into lately when running some tests in parallel, when running InitAndApply is hitting infrequent - but enough to be annoying transient network errors for fetching the various modules used that are basically Failed to download module type errors.

I ended up hitting a wall with causing the errors to go away since it's out of my immediate control, my first approach to solving this issue was to run a single terraform init prior to running my tests to populate .terraform/plugins and .terraform/modules prior to running the tests and just use those - reducing my number of Inits, and also improving the efficiency of the pipeline slightly.

I ended up finding out however that CopyTerraformFolderToTemp doesn't end up copying the .terraform directory (and seems to exclude all hidden directories here). While ultimately I'm able to use RetryableTerraformErrors to just retry the step to get around the problem for my for specific use-case, I still wanted to make the suggestion to potentially allow CopyTerraformFolderToTemp to copy hidden directories (or just .terraform) to the temp directories.

While Terraform's native functionality does offer the ability to have a separate directory for the plugin-cache by setting a plugin_cache_dir env var - there (currently) exists no matching functionality to be able to manage a separate modules-cache type directory - so all downloaded modules are expected to be in .terraform/modules (see hashicorp/terraform#16268).

I think a solution built into CopyTerraformFolderToTemp, or a separate function to allow this would be more desirable than a hacky workaround where we copy modules around manually for those who want to make use of a module cache while still being able to use CopyTerraformFolderToTemp

I don't mind (attempting to) implement something, but wanted to put the idea out there to collect any sort of feedback prior to making any sort of attempts.

Thanks!

[brikis98]

This has come up a few times. We should probably add a new function called something like CopyTerraformFolderToTempWithFilter, which allows you to filter what does/doesn't get copied. Under the hood, the current CopyTerraformFolderToTempWithFilter could call CopyTerraformFolderToTempWithFilter with a filter that works as it does today. A PR to add this is very welcome!

[brikis98]

See also #963.

[AgustinRamiroDiaz]

Hi! First of all thank you very much for this great package!

I've been hitting this one, and it'd be great to add this feature. Can I work on it?

[james03160927]

It looks like this functionality is already supported. You can work around the issue using a pattern like the one below:

filter := func(path string) bool {
	if PathContainsTerraformStateOrVars(path) {
		return false
	}
	return true
}

CopyFolderToDest(folderpath, os.TempDir(), tempFolderPrefix, filter)