WIP: support for opsworks for chef automate #7817

kitchen · 2019-03-05T22:51:33Z

Fixes #403

Changes proposed in this pull request:

Add support for provisioning OpsWorks for Chef Automate instances
Tests and documentation for above

Output from acceptance testing:

$ make testacc TESTARGS='-run=TestAccAWSAvailabilityZones'

WIP WIP WIP WIP WIP
...

It's missing a bunch of stuff, for sure, like parameters, tests, docs, etc, and I haven't actually run it against aws, but I have what I think is all of the boilerplate required, so it's a start

I grabbed all of these from the API docs: https://docs.aws.amazon.com/sdk-for-go/api/service/opsworkscm/#CreateServerInput It's still not entirely there, and I'll need to document the instance/service profile ARN stuff, and the security groups since I think the security groups need to be made outside and not automanaged by the server resource on AWS's side

everything is there but the EngineAttributes bit and validation. also debating still on whether some of those required params should be optional. I'd like them to be required, but I'll have to ask upstream their thoughts

They already are required because of the way I made the Create function, so while it's still an open question whether or not they should actually be required required, I at least want to make it consistent at the moment

kitchen

Obviously I also need to write tests and docs, but I wanted to get the code up first for comments on the questions I have.

Any advice would be greatly appreciated.

Also, the underlying API here is the opsworkscm, which actually manages both opsworks for chef and opsworks for puppet. I'm not sure style-wise whether this resource should be made generic for both or if opsworks for puppet would have its own resource that uses the same underlying api. I'm a fan of having them be separate, for a number of reasons, but I defer to y'all's judgement.

Thanks!

kitchen · 2019-03-05T22:53:02Z

aws/resource_aws_opsworks_chef.go

+
+		Schema: map[string]*schema.Schema{
+			// TODO: do we want this? I think this is another "magic" thing that happens
+			// I think we can still associate a public IP later if we want? :shrug:


I'm curious to know what y'all think about this. I'm on the side of not having magic happen that the user might not be expecting, meaning this attribute would go away and we'd just hardcode it to no behind the scenes.

kitchen · 2019-03-05T22:56:30Z

aws/resource_aws_opsworks_chef.go

+
+			// TODO:
+			// I kinda want to make this required because how else are we going to get at that info?
+			// Also, since these are write-only and not returned by the API, we'll probably need a custom diff function that ignores them?


Also interested in y'all's thoughts on this:

What's happening here is these are optional things to the API, and amazon will generate values for you and return them in the CreateServerResponse, but you'll never see it again.

I don't know if there's a way then to take the output of that (data provider-like?) and shove it into another resource like a vault password or something. And I'm fairly certain the whole thing would be pretty useless if you threw away these values. So I'm leaning on the side of making these required up front.

"these" being the chef_pivotal_key and chef_delivery_admin_password attributes, both of which behave this way.

Note, this is just the pubkey component. Due to sensitive information leak in state the privkey should ideally be kept out of terraform - making this argument required.

Also needs to be base64 encoded, which differs from what is provided in your testcase (which would include -----BEGIN...)

kitchen · 2019-03-05T23:00:00Z

aws/resource_aws_opsworks_chef.go

+
+			// TODO:
+			// default and only valid. I'll leave it configurable for future proofing but
+			// validation failure, maybe?


These are required parameters to the API call, but they only have one valid value at the moment.

Should I remove these from the resource and hardcode the values, leave them on the resource but enforce them in validation, or leave them on the resource, make them optional with these values as the default, and let the API handle validation? Or something else entirely?

kitchen · 2019-03-05T23:00:53Z

aws/resource_aws_opsworks_chef.go

+			// TODO:
+			// if you don't specify this, it'll create one. I'm of the opinion that users should
+			// create it themselves. Or maybe some sort of wrapper module? I dunno.
+			// if it's optional, will they be computed? if so, is that something we can express here?


Here's another "magic happens" thing. If you don't specify security groups, it'll make one for you. Is this something we want to allow or should we require folks to specify the security groups explicitly? I'm leaning on the side of explicitly.

kitchen · 2019-03-05T23:01:38Z

aws/resource_aws_opsworks_chef.go

+			// your EC2 instances are created in a default subnet that is selected by Amazon
+			// EC2. If you specify subnet IDs, the VPC must have "Auto Assign Public IP"
+			// enabled.
+			// if it's optional, will they be computed? if so, is that something we can express here?


All sorts of :wat: here.

I'm thinking we should force people to specify subnet IDs. But that's just me.

Make subnets required argument.

The subnet only needs "Auto Assign Public IP" if associate_public_ip_address == true, otherwise it won't be internet accessible. AWS docs may have drifted, you can now make chef servers with private ips.

This, together with EC2-Classic requirements should just be covered in documentation.

kitchen · 2019-03-05T23:04:04Z

aws/resource_aws_opsworks_chef.go

+}
+
+func resourceAwsOpsworksChefValidate(d *schema.ResourceData) error {
+	// TODO: validation function


For validation, how much should I replicate the validation that amazon is going to do on the backend here?

The maintenance and backup windows have a simple format that could be validated, and then there's the chef admin password. Oh, and the chef_pivotal_key. How much should I validate before just leaving it to the API to explode when the input is wrong?

ok so window format is already a thing that is validated elsewhere so I'll validate it too. I wanted to wrap the existing validators but wasn't really sure how to do it so I made another that did the needful. Definitely open to any suggestions for improvement :)

and nevermind, I found https://github.com/hashicorp/terraform/blob/a95d97f0665f905411d4e6ab400136c50539d6c6/helper/validation/validation.go#L33-L47 and am incorporating :)

annnnd nevermind, that's in a newer version of terraform than this package is pulling in (I think, I'm not super familiar with golang development tbh). I'll use the one in this provider: https://github.com/terraform-providers/terraform-provider-aws/blob/85e58f8cf3c79621d32db94f657e8a258ee44fd3/aws/validators.go#L27-L42

kitchen · 2019-03-05T23:05:50Z

aws/resource_aws_opsworks_chef.go

+		resp, cerr = client.CreateServer(req)
+		if cerr != nil {
+			if opserr, ok := cerr.(awserr.Error); ok {
+				// TODO: does this also happen with this resource?


Should I err on the side of "we can add this back later if it turns out it's a problem" or should I try to recreate the circumstances described to see if it needs to stay?

kitchen · 2019-03-05T23:08:57Z

aws/resource_aws_opsworks_chef.go

+	}
+
+	// TODO: these are the only values that are allowed to be changed
+	// according to the API, is there a way to express this in the provider somehow?


question here: only some of the values are able to be updated. The others are write-once. How to handle?

There are actually 2 more that can be updated: the chef_pivotal_key and chef_delivery_admin_password, but those are actually set using a different API call (easy enough to handle, though) but the question there is how do we know if those values need to change. Is there a way we can store the output of the createserver in a hashed form somehow so we can compare the inputs to what we last knew the values were on the server and update them? Or something else?

Seems like ForceNew on the attribute is the Schema is what I'm after for the first part of that question.

followup question on the ForceNew: will that propagate to any node associations that exist? Also, I should probably create a resource for those.

https://docs.aws.amazon.com/sdk-for-go/api/service/opsworkscm/#OpsWorksCM.AssociateNode

hmm. Now that I think of it, the AssociateNode is probably going to be done outside of terraform scope, which means terraform wouldn't have the info needed to be able to recreate the associations. So a ForceNew would be a pretty destructive thing.

I'm curious how this is even handled on AWS side. If instance type is one of the forcenew params, then I'm assuming they're not able to roll the backend chef machine(s) to new sizes, which makes me wonder if they even have multiples or if it's HA or whatever, what happens when one dies? I'm probably reading too much into that part of things, but I might contact amazon support and ask for more details.

Ok, I filed a support request with amazon to see what they have to say about this. For now, I'll leave ForceNew on, but I actually think forcing new here would be a really bad idea, so I'm hoping there are better ways to handle that situation :) I'm open to any suggestions.

I think this is the baseline requirements for setting up the opsworks_chef instance. It's ... verbose. But this is also a fairly complex thing to be creating, so that's kinda to be expected, methinks.

per discussion in the PR, I think ForceNew here is probably a bad idea because it will almost certainly destroy any node associations, but at least it won't silently do nothing :)

since this doesn't seem to be running yet (I think this only runs in acceptance tests?) it doesn't explode, but I'm testing something like it elsewhere

atward · 2019-04-18T04:45:26Z

aws/resource_aws_opsworks_chef.go

+				Optional: true,
+				Default:  1,
+				// https://docs.aws.amazon.com/opsworks/latest/userguide/opscm-chef-backup.html
+				// I could swear I found this 1-30 limitation elsewhere, but I can't seem to find it now other than there


Found docs here

A maximum of 30 backups are kept; when the maximum is reached, AWS OpsWorks for Chef Automate deletes the oldest backups to make room for new ones.
It's also on the console wizard

terraboops · 2019-09-03T16:46:22Z

Hello! Just curious if this PR is still being worked on; it would be a great benefit to me. 🎉

Is there anything I can do to assist -- or just shut up and wait 🤐?

Thanks!

kitchen · 2019-09-04T00:03:47Z

It is not currently being worked on, by me anyways, sorry :( I don’t remember quite where I left off with it but I imagine it’s missing docs, some fixes on the comments I had and had gotten replies to, and maybe a few other things. I will be out of town for 2 weeks starting in a few days so I won’t have much of an opportunity to work in it for a while.

…

On Sep 3, 2019, at 9:46 AM, Tyler Mauthe ***@***.***> wrote: Hello! Just curious if this PR is still being worked on; it would be a great benefit to me. 🎉 Is there anything I can do to assist -- or just shut up and wait 🤐? Thanks! — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

ewbankkit · 2019-12-19T13:01:42Z

You can now add tags to an AWS OpsWorks for Chef Automate server or an AWS OpsWorks for Puppet Enterprise master, or to server backups.
Announcement.

Requires AWS SDK v1.26.5:

Update module aws/aws-sdk-go to v1.26.5 #11353

teamterraform · 2020-08-18T19:44:55Z

Notification of Recent and Upcoming Changes to Contributions

Thank you for this contribution! There have been a few recent development changes that affect this pull request. We apologize for the inconvenience, especially if there have been long review delays up until now. Please note that this is automated message from an unmonitored account. See the FAQ for additional information on the maintainer team and review prioritization.

If you are unable to complete these updates, please leave a comment for the community and maintainers so someone can potentially continue the work. The maintainers will encourage other contributors to use the existing contribution as the base for additional changes as appropriate. Otherwise, contributions that do not receive updated code or comments from the original contributor may be closed in the future so the maintainers can focus on active items.

For the most up to date information about Terraform AWS Provider development, see the Contributing Guide. Additional technical debt changes can be tracked with the technical-debt label on issues.

As part of updating a pull request with these changes, the most current unit testing and linting will run. These may report issues that were not previously reported.

Action Required: Terraform 0.12 Syntax

Reference: #8950
Reference: #14417

Version 3 and later of the Terraform AWS Provider, which all existing contributions would potentially be added, only supports Terraform 0.12 and later. Certain syntax elements of Terraform 0.11 and earlier show deprecation warnings during runs with Terraform 0.12. Documentation and test configurations, such as those including deprecated string interpolations (some_attribute = "${aws_service_thing.example.id}") should be updated to the newer syntax (some_attribute = aws_service_thing.example.id). Contribution testing will automatically fail on older syntax in the near future. Please see the referenced issues for additional information.

Action Required: Terraform Plugin SDK Version 2

Reference: #14551

The Terraform AWS Provider has been upgraded to the latest version of the Terraform Plugin SDK. Generally, most changes to contributions should only involve updating Go import paths in source code files. Please see the referenced issue for additional information.

Removal of website/aws.erb File

Reference: #14712

Any changes to the website/aws.erb file are no longer necessary and should be removed from this contribution to prevent merge issues in the near future when the file is removed from the repository. Please see the referenced issue for additional information.

Upcoming Change of Git Branch Naming

Reference: #14292

Development environments will need their upstream Git branch updated from master to main in the near future. Please see the referenced issue for additional information and scheduling.

Upcoming Change of GitHub Organization

Reference: #14715

This repository will be migrating from https://github.com/terraform-providers/terraform-provider-aws to https://github.com/hashicorp/terraform-provider-aws. No practitioner or developer action is anticipated and most GitHub functionality will automatically redirect to the new location. Go import paths including terraform-providers can remain for now. Please see the referenced issue for additional information and scheduling.

ewbankkit · 2021-08-30T14:14:57Z

Closing as stale due to inactivity.

github-actions · 2022-06-24T02:36:48Z

I'm going to lock this pull request because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

kitchen added 4 commits March 4, 2019 16:05

it compiles

05af368

It's missing a bunch of stuff, for sure, like parameters, tests, docs, etc, and I haven't actually run it against aws, but I have what I think is all of the boilerplate required, so it's a start

send the parameters to the api

a4b72ae

everything is there but the EngineAttributes bit and validation. also debating still on whether some of those required params should be optional. I'd like them to be required, but I'll have to ask upstream their thoughts

set the pivotal key and admin password in the create call

3cc6329

ghost added size/XL Managed by automation to categorize the size of a PR. provider Pertains to the provider itself, rather than any interaction with AWS. service/opsworks Issues and PRs that pertain to the opsworks service. labels Mar 5, 2019

make these required

cac53cb

They already are required because of the way I made the Create function, so while it's still an open question whether or not they should actually be required required, I at least want to make it consistent at the moment

kitchen commented Mar 5, 2019

View reviewed changes

kitchen added 4 commits March 5, 2019 15:24

actually set the server name when creating the server :)

cdf44d2

starting to work on tests

2641e0d

I think this is the baseline requirements for setting up the opsworks_chef instance. It's ... verbose. But this is also a fairly complex thing to be creating, so that's kinda to be expected, methinks.

ForceNew on these

0cad028

per discussion in the PR, I think ForceNew here is probably a bad idea because it will almost certainly destroy any node associations, but at least it won't silently do nothing :)

don't typecast a string to a string

94b271c

ghost added the tests PRs: expanded test coverage. Issues: expanded coverage, enhancements to test infrastructure. label Mar 6, 2019

kitchen added 7 commits March 6, 2019 15:09

add some validation for some parameters

6603197

add the destroy check function

ec3a977

register the resource

a0cce4d

validation stuff

61fb889

need to Elem for TypeList

206e2f9

some fixes for the actual test code

71fa201

since this doesn't seem to be running yet (I think this only runs in acceptance tests?) it doesn't explode, but I'm testing something like it elsewhere

some fixes to make it actually work in real life

96f3661

atward reviewed Apr 18, 2019

View reviewed changes

aeschright requested a review from a team June 26, 2019 00:46

Base automatically changed from master to main January 23, 2021 00:55

ewbankkit closed this Aug 30, 2021

github-actions bot locked as resolved and limited conversation to collaborators Jun 24, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP: support for opsworks for chef automate #7817

WIP: support for opsworks for chef automate #7817

kitchen commented Mar 5, 2019

kitchen left a comment

kitchen Mar 5, 2019

kitchen Mar 5, 2019

atward Apr 9, 2019

kitchen Mar 5, 2019

kitchen Mar 5, 2019

kitchen Mar 5, 2019

atward Apr 9, 2019

kitchen Mar 5, 2019

kitchen Mar 6, 2019

kitchen Mar 6, 2019

kitchen Mar 6, 2019

kitchen Mar 5, 2019

kitchen Mar 5, 2019

kitchen Mar 5, 2019

kitchen Mar 5, 2019

kitchen Mar 5, 2019

kitchen Mar 6, 2019

atward Apr 18, 2019

terraboops commented Sep 3, 2019

kitchen commented Sep 4, 2019 via email

ewbankkit commented Dec 19, 2019 •

edited

Loading

teamterraform commented Aug 18, 2020

ewbankkit commented Aug 30, 2021

github-actions bot commented Jun 24, 2022

WIP: support for opsworks for chef automate #7817

WIP: support for opsworks for chef automate #7817

Conversation

kitchen commented Mar 5, 2019

kitchen left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

terraboops commented Sep 3, 2019

kitchen commented Sep 4, 2019 via email

ewbankkit commented Dec 19, 2019 • edited Loading

teamterraform commented Aug 18, 2020

Notification of Recent and Upcoming Changes to Contributions

Action Required: Terraform 0.12 Syntax

Action Required: Terraform Plugin SDK Version 2

Removal of website/aws.erb File

Upcoming Change of Git Branch Naming

Upcoming Change of GitHub Organization

ewbankkit commented Aug 30, 2021

github-actions bot commented Jun 24, 2022

ewbankkit commented Dec 19, 2019 •

edited

Loading