-
Notifications
You must be signed in to change notification settings - Fork 1.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add support for parallel plans #926
Add support for parallel plans #926
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hey sorry it's taken so long to get to this. I've left a couple of comments, but I know it's been a while so no worries if you can't get back to me.
I was thinking about having this be on automatically if the plans are in separate workspaces and skip the config option.
@lkysow thanks! I'll address the comments this week |
@Fauzyy FYI I was looking at enabling this for all projects, not just workspaces. If we add a queuing system to the working dir locker then we could lock it while it's doing the clone but otherwise we'd be fine to run You wouldn't have to do it in this PR, I could pick it up once you've got the per-workspace code ready. |
@lkysow that sounds great 👍. The only case I'd be concerned about is the one where multiple atlantis projects point to the same directory (through some kind of |
741c892
to
4d1899f
Compare
@adnankobir that sounds reasonable yeah, I can make plans default parallel but add a flag for applies. Let me know what you think, @lkysow |
+ Default parallel plans breaks in a lot of autoplan scenarios i.e. with shared workspaces + Added a set of e2e tests to check parallel plans and applies + Also added an option to the Terraform client to turn off the plugin cache. This was breaking the e2e test suite (never seen this error in practice)
For some reason the e2e test is running when it shouldn't be:
I also added back the flag to disable parallel plans (off by default), for now. It's breaking a lot of e2e tests when you're not using workspaces. Feel free to change this after! |
It looks like your segmentio circleci instance is running on this PR and somehow reporting checks back. |
3060bb0
to
0e22d2b
Compare
Oops, should be fixed now 👍 |
Codecov Report
@@ Coverage Diff @@
## master #926 +/- ##
==========================================
- Coverage 71.96% 71.50% -0.46%
==========================================
Files 65 65
Lines 5429 5489 +60
==========================================
+ Hits 3907 3925 +18
- Misses 1215 1254 +39
- Partials 307 310 +3
Continue to review full report at Codecov.
|
How do we enable parallel plan using repos.yaml |
@@ -65,6 +65,9 @@ type DefaultClient struct { | |||
|
|||
// versionsLock is used to ensure versions isn't being concurrently written to. | |||
versionsLock *sync.Mutex | |||
|
|||
// usePluginCache determines whether or not to set the TF_PLUGIN_CACHE_DIR env var | |||
usePluginCache bool |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Are you going to expose this via a config file flag? Curious as to your use case for why you want to be able to turn this off.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I could definitely expose this as a config file flag, this was just to fix the e2e test suite for parallel plans. They would transiently fail with the plugin cache enabled, we never saw this error in production however. The error mentions that the null_resource provider is missing when a plan is run, I wonder if the nature of these tests exacerbates the problem
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ahh okay
Amazing work, sorry this took so long. I'm planning to take steps to avoid this long a delay in the future. We will need to add docs for this. |
Has the documentation not been updated for this feature? I don't see it here: https://www.runatlantis.io/docs/repo-level-atlantis-yaml.html#example-using-all-keys |
No it has not. PRs welcome! |
Is this only valid if using terraform workspaces? |
@nwsparks yes this is terraform workspaces only I believe. |
We do not use terraform workspaces, but use parallelization. There is a difference between Atlantis workspaces and terraform workspaces |
Issue: #260
We've been running a simple repo config implementation of parallel plans for a couple weeks in our fork and it's working well
Caveat: because Atlantis has internal locks on the workspace name, this only works if all projects in the repo use Terraform Workspaces.
This PR also adds parallel apply as well.
Example config: