-
Notifications
You must be signed in to change notification settings - Fork 9.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Data sources should allow empty results without failing #16380
Comments
We desperately needed this for a long time, couldn't find an effective way to deal with such resources that could be created only once (for example, security group in AWS) and used later in other configurations (while creating a separate configurations for such shared resources are not reasonable this case). |
i am sitting right at the same problem. would really be helpful if empty returns where possible! |
Agree that this would be useful. I would like to use "aws_ebs_snapshot" data source to look for a snapshot that matches on some filters and create a volume from it. But if none is found, then I would like to create a volume from a default snapshot id. |
Could this be a valid solution: #11782 ? |
@shazChaudhry that appears to only work on maps, yeah? |
I have hit this a few times. But my latest use case is in trying to get around the way that Plan cannot tell that a resource will be created in the apply when the filter values are computed, in particular when using modules. Like I am trying to build a security group list from a map using names and use a security group data source filtering on name to find the corresponding IDs. If a security group is one that will be created in the same Apply, it fails the Plan to "no matching SecurityGroup found". I have a fair understanding of the terraform model and that the plan needs to create the graph to know what will really happen in Apply, but overall it continues to be difficult to create dynamic and DRY terraform "code" to deal with more complex/sophisticated provisioning scenarios. |
I have the same problem:
This sequence of events happens:
Now, I could use a However, if I am creating the environment for the first time this is not possible. This does not work if you are creating the environment for the first time, but it will on subsequent runs:
I worked around it by using a templating library (mustache) to generate my terraform resources and it only includes the It would appear that I'd submit a PR for that behaviour myself but I'd like to first see if there's a difference between how I'm using the software and how the maintainers intended it to be used. Perhaps my use-case is too weird for Terraform to cover it, or perhaps I'm meant to integrate Terraform more deeply in my CI pipeline. Aside from that, it's done everything else quite marvellously so far. [0] Amazon Elastic Container Service |
I was actually able to work around this problem using Terraform and I no longer need to use a templating library to do what I described. I need to add a
This has the downside that every time I run I'm a bit confused about why I explicitly have to write In light of this, perhaps no change is required. |
That is an interesting find on depends_on working in the data block. In my case and I believe others in the thread, the creation of the resource is not necessarily happening in the same root or module and the data block is using more variable data for the look-up. So I think supporting this kind of behavior (allowing not found) should still be pursued. |
I have the same issue. Even if NULL were not an option, it would be good if I could simply specify a default resultset or value for the data source to return if it finds no results. I could then use this value in combination with ternary logic elsewhere to make some decision. |
Hitting similar issue - creating a new ASG to take over a previous one, I set the desired count to the number of instances currently in the 'to be deposed' ASG pulled from 'aws_instances' data source. |
Bump. This is a truly needed feature to deal with already created infrastructure: ` resource "aws_vpn_gateway" "vpn_gw" { tags { }` This should create the resource if it cannot find the data source. Any better way of doing this? |
@mightystoosh - for already created infrastructure you can import the entity to your Terraform state. If you've only got a couple of environments to worry about that's a perfectly fine workflow and might work for you. Given you're talking about a NAT gateway I guess it will. Then, you just treat it as any old resource. This issue is about situations where a resource may or may not exist and either possibility is totally fine. However, in the situation I described, AWS quietly added a feature to force a redeployment of a service in ECS so this issue no longer affects me personally. That said, I think it would still be useful, such as in @trilitheus' situation. |
What is best practices for adding infrastructure? Basically I want to setup a bunch of VPN connections based on a list and connect it to a single VGW. It'd be nice if I could iterate over a list in Terraform or even run terraform multiple times using I'm new to Terraform and it's nice, but I don't really see a way of adding multiple endpoints easily without copying of code. |
@mightystoosh your workaround example above is feasible in some cases as what I understand from the example is you are guaranteeing at least one item returned and then testing for more than one to know if the desired one is returned. It does requires that there is one resource you know will be there as @steverukuts alluded to. However, several data resources only allow one item to be returned, which then wouldn't fit this scenario. |
I'm in the 'AMI lookup' camp. What I want to do is this: |
I won't bother prefacing with all the usual reasons why this hack I'm about to paste is hacky and tightly coupled and will most likely need refactored out once Terraform has a native solution which is in the framework, etc. But maybe someone can use my duct tape for automation testing, non-critical items like I am doing, or stimulate some better creative solution.
This submodule verifies if a certain SNS topic subscription is present by running a bash script inside a conditional to detect if the subscription resource exists. The script is using the AWS CLI along with jq to select for specific dynamic attributes which might be module variables or other values, and also to produce JSON, so Terraform can handle it. The CLI call is wrapped in a simple if/then conditional to manufacture a true/false output based on the returned result. One functional obstacle for me is that external data sources run immediately on Apply, so the mixture of null_resources using local-exec and the external data sources and their referenced interpolated outputs is used to keep everything in dependency order. |
thanks @BlaineBradbury ! i was running into the same kind of issue, start a RDS instance from a previously created final snapshot if it exists... so i did something like that: create
then in my template :
|
Bump. Would be super helpful for determining whether to create an autoscaling group or not. |
I have a similar issue but with security group. While creating a new environment, I get "data.aws_security_group.XXXXXX_sg: no matching SecurityGroup found". |
I just hit this issue too. My use case is for running a job that needs to sync a load balancer with instances that have been created independently, and that may not even exist. I know there are other ways to solve this but this would have been a nice way to do it. data "aws_instances" "servers" {
instance_tags {
X-Service = "my-server"
}
}
resource "aws_lb_target_group_attachment" "servers" {
count = "${length(data.servers.ids)}"
target_group_arn = "${aws_alb_target_group.www_lb.arn}"
target_id = "${data.aws_instances.servers.ids[count.index]}"
port = 80
} |
We just hit this issue too, while hoping to provide a way to add dns records to zones that already exist to give fully working dns to terraform created load balancers, but create the zone if none exists already. |
A related issue. I am trying to restrict access to kms:decrypt on an iam policy. The resource arn needs to have kms key arn based on specific environments (not all regions are part of every environment. For example, integration env regions - us-west-2, eu-west-1; test env - us-west-2; production - us-west-2, eu-west-1, us-east-1. data "aws_caller_identity" "current" {} data "aws_region" "current" {} data "aws_kms_key" "kms-us-west-2" { data "aws_kms_key" "kms-eu-west-1" { ... resource "aws_iam_policy" "lambda_execute_policy" { } Wondering if there is a way to dynamically create this array? |
I found a hack that worked for me: data "openstack_networking_network_v2" "vlan_network" {
count = "${var.vlan_network_enabled != "0" ? 1 : 0}"
name = "foo"
}
resource "openstack_networking_network_v2" "vxlan_network" {
count = "${var.vlan_network_enabled == "0" ? 1 : 0}"
name = "foo"
admin_state_up = "true"
}
locals {
# Hack to work around https://github.com/hashicorp/terraform/issues/15605 and https://github.com/hashicorp/terraform/issues/16380
network_id = "${var.vlan_network_enabled == "0" ?
element(concat(openstack_networking_network_v2.vxlan_network.*.id, list("")), 0) :
element(concat(data.openstack_networking_network_v2.vlan_network.*.id, list("")), 0)}"
} Explanation: I needed to conditionally select between a network created outside of Terraform, and one that Terraform provisions. The |
My use case is AWS RDS can create a final snapshot when you destroy a database or cluster. I want to be able to restore from that final snapshot if it exists, else create a blank database. But if you setup a data source to look for the final snapshot it fails because it doesn’t exist. Being able to specify a default set of values or even a Boolean (to use in count metadata) would solve this! Otherwise you have to pass in var’s (as per lblackstone’s solution above) but that means you have to know in advance if the backup exists. |
@4dz see my comment above i have a solution to this. Only issue we face with that solution is that when the final snapshot exists at destroy time, the destroy will fail, but we're not doing that on a regular basis. Just like |
Thanks @earzur . I really want to avoid the external script although I agree it will work for many use cases! The reason being that,
I'm working on a helper module to address exactly that problem! It will work with rds instances but not clusters because there is no |
The simplest we can do here is at least not Error if a data filter is not found. Then we can select something different. Is this so hard ? Use case: Check if a certificate is uploaded in acm or upload and use a self signed using iam... Data Source
If the acm certificate is missing we have the following error
So it would be ideal if we can continue and deal with an empty return somewhere else i.e
|
+1 Use Case: Find all aws instances for a particular environment and add route 53 dns entries.
|
Just asking if this will possible to have in the next 0.12 version? |
This will not be in Terraform 0.12... it needs considerably more design, prototyping, and thought before we could move forward with it, since it's not clear that the sort of highly-dynamic behavior people want to achieve here will fit in well with other Terraform features. In particular, lots of requesters here seem to want not just the ability for a data source to return the lack of items but also to return possibly multiple items matching a query, which is a pretty major change to how data sources work today and would require the feature to be redesigned. In the mean time, the current approach is to add "plural" data sources to providers where there is a clear use-case, such as 0.12 will also make it easier to use dependency inversion to get similar results in a different way. We'll be writing a guide on this and some other new patterns that the 0.12 features allow as we get closer to the release. |
Hey @ostmike I need the same
Replacing data "aws_ami" with custom module to query awscli e.g. https://github.com/matti/terraform-shell-resource |
bump: I have a similar question as @trilitheus and @charlesarnaudo. When updating an AMI in an existing launch configuration that was created in a previous terraform run, I want to be able to set the "desired" to the same value as the previous ASG so we don't cause an outage when rolling to the new AMI. |
I understand where @apparentlymart is coming from, but I would ask that returning empty results be considered separately from all of the other use-cases mentioned. How about an "allow_empty" property? |
I would also suggest to add a possibility to set default values when empty result is returned. |
+1 bump from another user with the same use case as @earzur #16380 (comment)
|
Bump. Would be very helpful |
+1 |
Hi all, |
Terraform `data` resources fetch the current state of the live system, rather than relying on the Terraform state. This means we can create new task definitions outside of Terraform and have it pick them up in a plan. Our build and deploy process does this, which was causing an issue where `terraform apply`s were rolling back to earlier versions if there was no change to the task definition. This may cause issues on a fresh system deployment, as there is no matching task definition at that point. hashicorp/terraform#16380 (comment)
@apparentlymart do you have any links to examples of the above now that 0.12 has been released? |
Another use case: When creating an ebs volume, I was to lookup if there is a snapshot available if not create the ebs volume without a snapshot otherwise create with a snapshot. Having the ability to create from a snapshot when available is quite critical to our infrastructure as initially we won't have a snapshot but then we have a dlm that will snapshot daily so if something goes wrong we would like to just run terraform and the ebs will get the latest snapshot. |
@emmaLP I just came across https://registry.terraform.io/modules/connect-group/lambda-exec in #2756 which looks like a novel, albeit slightly complicated, way to solve what you’re trying to do. One of the suggested use cases is close to what you’re after as well. |
Hello All, So I'm having this issue I did terraform destroy. Afterwards I ran terraform plan hopping everything would be up again.... but no. After everything got destroyed (and there was NOTHING of this data in state) still was giving me the error: data.aws_lb.load_balancer: data.aws_lb.load_balancer: Search returned 0 results, please revise so only one is returned My code:
So what I had to do to fix this is to add a depends_on into "aws_lb"."load_balancer" and change syntax in outputs so this could ran with empty which is a bad practice because it causes DNS records to be recreated on every terraform run. Check hashicorp/terraform-provider-aws#8541 After that this route53 records.... started to give errors because they couldn't use DATA so then I tried to add counts and depends_on over the records but it was getting messy... then I had to put the route53 stuff in another module getting from state... quite messy! Please fix this... |
Hi folks, The sort of dynamic decision-making that is being requested here runs counter to Terraform's design goals, since it makes the configuration a description of what might possibly be rather than what is. We are not going to change the behavior of data sources when the response is empty or 404, and will close this issue. We feel that there are better patterns to follow when managing the differences between environments: composing different modules in different ways, rather than trying to use the same modules in all environments but make them take different actions based on data source results. Please take a look at the module composition documentation - specifically, the section that describes conditional creation of objects - for a full description. I realize that this will be an unsatisfying response to some. I encourage you to visit the community forum if you have any questions about restructuring your configuration to achieve the results you need without this feature. Thank you! |
I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further. |
This is an enhancement request. Data sources should allow empty results without failing the plan or apply. This would allow for fallback or fallthrough scenarios that are needed in certain situations. An optional
allow_empty_result
attribute on the data source would support many use cases where a datasource value is not explicitly required. The specific attribute name isn't important to me.As an example, the configuration below currently fails because the
aws_ami
datasource will not find any results. However I still want the apply to continue using a fallback AMI.(This is just an example so please don't get hung up on this specific use case.)
The text was updated successfully, but these errors were encountered: