Distributed or dynamic management of security groups or rules

[bitglue]

Use case: I'm running a CoreOS cluster, which dynamically schedules docker containers around a cluster of EC2 instances. Some of those docker containers need to expose ports to load balancers or the internet, but which ports, and on what hosts the containers will be running, isn't known at deployment time, since containers can be added, removed, and moved at runtime by Fleet (CoreOS's scheduler).

What I'd like to do is have some startup code in my container which uses the AWS API to assign an additional security group to the instance running it. This way, wherever the container ends up running, security groups will be set appropriately, dynamically. The trouble: when Terraform runs again, it will undo these changes.

Essentially, sometimes I want to say "this instance must have exactly these security groups", while other times I mean "this instance must have at least these security groups". Or maybe I don't want Terraform to do anything with security groups, which is like saying "this instance must have at least (the empty set) security groups". However, Terraform allows only some of these things to be expressed, depending on the provider implementation.

The same problem exists for any other kind of 1:N relationship:

• security groups to inbound rules
• ELBs to their members
• IAM users to policies (Implement a subset of IAM resources #939)

In the case of ELBs, if I create the ELB in Terraform without listing any members, then I can dynamically add them later and Terraform won't try to remove them. I'm using elb-presence to add and remove hosts from the load balancer as containers are scheduled and it works fine.

In the case of IAM users, I implemented the 1:N relationships as separate resources. That is, there is a first-class iam_user_policy, rather than listing policies as sub-blocks under iam_user. This has the nice added benefit of allowing an iam_user to be declared in one module, while some other module can add policies to it. Such a pattern isn't possible with, for example, the inbound rules of security groups.

Thus, there are many behaviors present in Terraform, each which may or may not be the behavior that's desired by the user. I think the Terraform language needs some more explicit syntax to specify how a 1:N relationship should be managed.

[mitchellh]

Yes, I've been considering something in core that tells Terraform where or not to manage a specific feature of a resource completely. I'm going to tag this as thinking so I can think more (heh) and formalize my thoughts.

[bitglue]

Another thought: the way schema works now, if you want sub-blocks in your resources (like aws_security_group has ingress, or tags on some of the EC2 resources), then you put a schema.Schema inside another schema.Schema. I remember reading somewhere that these sub-schemas might have lifecycles of their own, although I don't remember ever seeing how that's described, or seeing a provider actually do that.

If those sub-schemas actually do have their own lifecycles, then it becomes a lot easier to move the management of them around, so for example, a security group is created it one module, and rules added to it in another. It also eliminates a lot of the places where partial state is needed. Essentially, those sub-resources actually become first-class resources, and nesting them inside another resource is just optional syntactic sugar. Could be an elegant way to proceed.

[josephholsten]

@mitchellh how's the thinking going?

[stevehorsfield]

We have a similar interest for security group rules. We'd like to dynamically configure security group rules based on runtime logic, but we don't want Terraform to reset the rules to an earlier state. In other words, rules should only be set on group creation, other resource changes don't affect rules.

This seems to be very similar to the issues described above.

[apparentlymart]

Hi everyone! Thanks for the great discussion here, and sorry for the long silence.

A while back the ignore_changes lifecycle flag was added to address situations like this generically. This causes Terraform to disregard changes to certain attributes when building a diff.

Along with this, the default behavior for some resources was changed to ignore certain attributes for diffing when they are not specified in config. For example, the ingress and egress blocks in aws_security_group now behave in this way, so if no blocks are specified at all in config Terraform will not try to un-set any rules that exist in the API, and the aws_security_group_rule resource can be used to manage individual rules on a security group without interfering with others.

Between both of these I think this has addressed the main use-cases discussed in this issue, so I'm going to close it. If there are more specific cases that are not catered for by these changes we'd welcome new top-level issues (either in this repository or in the separate provider repositories as appropriate) that describe those use-cases specifically, since it's easier to deal with issues that have a single well-defined goal.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.