Dedicated Development Resources?

[thegranddesign]

I have a general question. It seems as though this is one of the core pillars in the Hashicorp ecosystem, however it appears as though (but may not be the case), that although response to issues is quick (tagging, organization, etc), actually fixing those issues is not. Just since I've been using Terraform over the last couple months, the open issues count has risen from 900 to 1,050.

Some of those issues (of which I've encountered many) are fairly critical to even using Terraform in a way that is more complex than spinning up a couple servers (eg not being able to use arrays in configuration, not being able to use an AWS policy from a template file without it reapplying on every apply, not being able to conditionally apply resources and attributes, not being able to create custom interpolation helpers, and many more).

I understand that this is open source, but it also seems to be a core pillar of the Hashicorp story and I was curious if there is active/dedicated manpower being devoted to it, if resources are being devoted elsewhere and the plan is to come back to Terraform, etc.

This is fairly critical to whether I (and I'm assuming many others) continue to use what could be an amazing piece of software.

Thanks to everyone who has put in the time and energy thus far, by the way, overall I'm enjoying the experience, and I fully understand that this is an ambitious task, however I don't want to put a bunch of manpower into something that isn't going to end up being able to meet our needs.

Thank you for your time.

[phinze]

Hi @thegranddesign, thanks for this thoughtful and well articulated question.

I definitely understand the concerns you have about the status of the project. It can be tough to get a read on what is happening in a big piece of OSS like this one. And that becomes especially concerning when it feels like the issues affecting you are being neglected!

For an overall read on the activity on the project, I'd point to the GitHub pulse view.

Here's a snapshot from this past month:

Looking at that view, it probably wouldn't surprise you to learn that @jen20, @catsby, and myself are all full time Terraform-dedicated HashiCorp employees. Terraform also gets regular contributions from other HC employees and a ton of help our team of wonderful open source collaborators.

But - as you're noting - even with all that effort, the Terraform community still manages to produce an overwhelming amount of content! As hard as we work on this project every day, we're still struggling to stay in good communication on all of the active conversations across GitHub, the mailing list, and IRC - all this while also working on diagnosis, fixes, and features for Terraform itself. This is an issue we're acutely aware of and working to address on several fronts - including hiring, improving our community guidelines around issues & PRs, and iterating on our own internal processes for communication and prioritization.

To address the examples you cited directly:

• First class support for lists and maps is a deep and complex improvement that @jen20 has been working on full time for several weeks now. He's nearly there, and this is one of the big features that will be out with Terraform 0.7. (You can check out the WIP on the dev-0.7 branch if you like!)
• Many of the issues with AWS Policies we're hitting have to do with AWS "normalizing" the content we send it - declaratively managing an API that doesn't let you read your own writes is tricky. We've got a couple of initiatives in play (e.g. aws_iam_policy_document logical resource #3124) that attempt to allow us to generate policy documents in precisely the format the API does to so we avoid spurious diffs.
• Conditionals are a topic that we're intentionally approaching very slowly, since the nature of conditional branching in a declarative language has a lot of potential pitfalls. I put out a call for uses cases in Support use cases with conditional logic #1604 that generated a lot of good community input, and from there we have a general direction for where we want to go next.
• Arbitrary interpolation helpers is not a request I have seen often, but I can see the appeal of such a feature. I'm guessing this context comes from Embed lua interpreter in terraform #5278 which has a lot of good discussion and detail from @mitchellh on Terraform's philosophy and direction here.

I hope this helps give you a bit more context on these issues. We (both the community and HashiCorp employees) are definitely working as hard as we can to improve Terraform every day!

[u2mejc]

@phinze Thank you for taking the time to comment and to speak candidly. I had previously started using Terraform a year ago and stepped away for a short while. It was refreshing to come back and find a bunch of pain points now resolved! Please keep up the great work! 👍

I wanted to follow @thegranddesign 's comments, but instead of addressing issues, addressing PRs. I'm sure you familiar with drive by contributions, or contributors who are intensely passionate about just a singular feature inside the larger project. Has your team thought about how they can nurture that contributor's brief energy / enthusiasm and guide them to a PR that matches not just the style guides but the general direction of the project with code that is maintainable after they move on? IMHO this free labor is very beneficial to Hashicorp, and being timely about merging PRs is equally important to capture that brief enthusiasm, incase refactoring is required before merging.

[phinze]

Hi @u2mejc, you make great points about PRs and the nature of the time and effort behind them. This is 100% in line with our thinking and the conversations we're having about how us HashiCorp employees can better help the Terraform community.

In fact, we elected as a team to focus on PR pulse as an internal priority precisely because we want to do everything we can to acknowledge and respect the volunteered efforts of our contributors. You'll notice in the screenshot I captured above that we're well into the positive on PR pulse - this is a metric we now review weekly with the goal of keeping it consistently positive. We've got a deficit to make up, but we're dedicated to keeping the trend lines going in the right direction.

So to answer you're question directly: yes, we are most definitely thinking about how we can ensure that PR contributors have a consistently good experience with the project! This is a big priority for us.

In a project as active as this one, there is always going to be a mismatch between the volume of content (issues, ML posts, PRs, etc.) that the community can produce and the core team's ability to process that content. As I mentioned above, we're actively working on improvements to our processes so that our community engagement can be more consistent and predictable - ultimately so that fewer things fall through the cracks! We posted early this year about a set of changes we made to start down this path of improvement, and we're now at the point of evaluating the impact and effectiveness of that pass so we can determine our next steps. We'll continue to share any updates and progress we make on this front via docs and in conversations like this one.

We are dedicated to ensuring that the Terraform project remains happy and healthy, and we're working every day to get better!

[thegranddesign]

@phinze thank you for the amazing response. I'm looking forward to seeing what you all come up with. :) I'll close this for now.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.