Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Status=429 Code="RetryableError" Message="A retryable error occured." #7986

Closed
zot24 opened this issue Aug 4, 2016 · 23 comments · Fixed by #13637 or #14365
Closed

Status=429 Code="RetryableError" Message="A retryable error occured." #7986

zot24 opened this issue Aug 4, 2016 · 23 comments · Fixed by #13637 or #14365

Comments

@zot24
Copy link

zot24 commented Aug 4, 2016

I think this might be related with this issue #7153 I'm open it to provide more information about it and to keep separate from that issue.

Terraform Version

Terraform v0.7.0

Affected Resource(s)

  • azurerm_subnet

Terraform Configuration Files

link

Debug Output

link

Standard Output

Error applying plan:

1 error(s) occurred:

* azurerm_subnet.mod: network.SubnetsClient#CreateOrUpdate: Failure responding to request: StatusCode=429 -- Original Error: autorest/azure: Service returned an error. Status=429 Code="RetryableError" Message="A retryable error occured." Details=[{"code":"ReferencedResourceNotProvisioned","message":"Cannot proceed with operation because resource /subscriptions/614e4618-7e10-4847-be48-a61acdbadff1/resourceGroups/testm1_resource_group_1/providers/Microsoft.Network/networkSecurityGroups/testm1_network_security_group_1 used by resource /subscriptions/614e4618-7e10-4847-be48-a61acdbadff1/resourceGroups/testm1_resource_group_1/providers/Microsoft.Network/virtualNetworks/testm1_virtual_network_1/subnets/testm1_subnet_1 is not in Succeeded state. Resource is in Updating state and the last operation that updated/is updating the resource is PutSecurityRuleOperation."}]

Azure Error

    "properties": {
        "statusCode": "429",
        "serviceRequestId": "fa3f97c4-3118-4a93-b625-6a0948b5d9f4",
        "statusMessage": "{\"error\":{\"code\":\"RetryableError\",\"message\":\"A retryable error occured.\",\"details\":[{\"code\":\"ReferencedResourceNotProvisioned\",\"message\":\"Cannot proceed with operation because resource /subscriptions/614e4618-7e10-4847-be48-a61acdbadff1/resourceGroups/testm1_resource_group_1/providers/Microsoft.Network/networkSecurityGroups/testm1_network_security_group_1 used by resource /subscriptions/614e4618-7e10-4847-be48-a61acdbadff1/resourceGroups/testm1_resource_group_1/providers/Microsoft.Network/virtualNetworks/testm1_virtual_network_1/subnets/testm1_subnet_1 is not in Succeeded state. Resource is in Updating state and the last operation that updated/is updating the resource is PutSecurityRuleOperation.\"}]}}"
    },
    "resourceGroupName": "testm1_resource_group_1",
    "resourceId": "/subscriptions/614e4618-7e10-4847-be48-a61acdbadff1/resourceGroups/testm1_resource_group_1/providers/Microsoft.Network/virtualNetworks/testm1_virtual_network_1/subnets/testm1_subnet_1",
    "resourceProviderName": {
        "localizedValue": "Microsoft.Network",
        "value": "Microsoft.Network"
    },
@colemickens
Copy link

This occurs on more deployments than it doesn't. In fact, it's bad enough that I'm ignoring the return value from terraform apply on the first execution and then just running it again: https://github.com/kubernetes/kubernetes-anywhere/blob/690418e84bd0567330b956345f424c698787c2d8/phase1/azure/do#L21

@colemickens
Copy link

cc: @stack72

I'm still getting this on 0.7.1 (which includes the updated SDK as far as I can tell from looking at the vendoring history).

Is there a proper dependency between the NSG and Subnet established in the Terraform plan? One needs to wait on the other to finish provisioning.

@edevil
Copy link
Contributor

edevil commented Oct 3, 2016

I also get these, have to retry everytime:

* azurerm_network_interface.etcdNIC.0: network.InterfacesClient#CreateOrUpdate: Failure responding to request: StatusCode=429 -- Original Error: autorest/azure: Service returned an error. Status=429 Code="RetryableError" Message="A retryable error occured." Details=[{"code":"ReferencedResourceNotProvisioned","message":"Cannot proceed with operation because resource /subscriptions/4df9b22d-85c6-440a-a1ab-f230d65e619c/resourceGroups/kuberneteslabs/providers/Microsoft.Network/virtualNetworks/KubeVNET/subnets/etcd used by resource /subscriptions/4df9b22d-85c6-440a-a1ab-f230d65e619c/resourceGroups/kuberneteslabs/providers/Microsoft.Network/networkInterfaces/etcd-0-nic is not in Succeeded state. Resource is in Updating state and the last operation that updated/is updating the resource is PutSubnetOperation."}]
* azurerm_network_interface.etcdNIC.1: network.InterfacesClient#CreateOrUpdate: Failure responding to request: StatusCode=429 -- Original Error: autorest/azure: Service returned an error. Status=429 Code="RetryableError" Message="A retryable error occured." Details=[{"code":"ReferencedResourceNotProvisioned","message":"Cannot proceed with operation because resource /subscriptions/4df9b22d-85c6-440a-a1ab-f230d65e619c/resourceGroups/kuberneteslabs/providers/Microsoft.Network/virtualNetworks/KubeVNET/subnets/etcd used by resource /subscriptions/4df9b22d-85c6-440a-a1ab-f230d65e619c/resourceGroups/kuberneteslabs/providers/Microsoft.Network/networkInterfaces/etcd-1-nic is not in Succeeded state. Resource is in Updating state and the last operation that updated/is updating the resource is PutSubnetOperation."}]
* azurerm_network_interface.master1NIC: network.InterfacesClient#CreateOrUpdate: Failure responding to request: StatusCode=429 -- Original Error: autorest/azure: Service returned an error. Status=429 Code="RetryableError" Message="A retryable error occured." Details=[{"code":"ReferencedResourceNotProvisioned","message":"Cannot proceed with operation because resource /subscriptions/4df9b22d-85c6-440a-a1ab-f230d65e619c/resourceGroups/kuberneteslabs/providers/Microsoft.Network/virtualNetworks/KubeVNET/subnets/master used by resource /subscriptions/4df9b22d-85c6-440a-a1ab-f230d65e619c/resourceGroups/kuberneteslabs/providers/Microsoft.Network/networkInterfaces/master1nic is not in Succeeded state. Resource is in Updating state and the last operation that updated/is updating the resource is PutSubnetOperation."}]
* azurerm_network_interface.jboxNIC: network.InterfacesClient#CreateOrUpdate: Failure responding to request: StatusCode=429 -- Original Error: autorest/azure: Service returned an error. Status=429 Code="RetryableError" Message="A retryable error occured." Details=[{"code":"ReferencedResourceNotProvisioned","message":"Cannot proceed with operation because resource /subscriptions/4df9b22d-85c6-440a-a1ab-f230d65e619c/resourceGroups/kuberneteslabs/providers/Microsoft.Network/virtualNetworks/KubeVNET/subnets/management used by resource /subscriptions/4df9b22d-85c6-440a-a1ab-f230d65e619c/resourceGroups/kuberneteslabs/providers/Microsoft.Network/networkInterfaces/jumpboxnic is not in Succeeded state. Resource is in Updating state and the last operation that updated/is updating the resource is PutSubnetOperation."}]

It seems it tries to create the NICs before the subnets are ready, probably because it is applying a security group or route table.

Terraform v0.7.4

@edevil
Copy link
Contributor

edevil commented Nov 2, 2016

Just tried with v0.7.8, still have to retry the apply:

Error applying plan:

5 error(s) occurred:

* azurerm_network_interface.etcdNIC.2: network.InterfacesClient#CreateOrUpdate: Failure responding to request: StatusCode=429 -- Original Error: autorest/azure: Service returned an error. Status=429 Code="RetryableError" Message="A retryable error occured." Details=[{"code":"ReferencedResourceNotProvisioned","message":"Cannot proceed with operation because resource /subscriptions/4df9b22d-85c6-440a-a1ab-f230d65e619c/resourceGroups/kuberneteslabs/providers/Microsoft.Network/virtualNetworks/KubeVNET/subnets/etcd used by resource /subscriptions/4df9b22d-85c6-440a-a1ab-f230d65e619c/resourceGroups/kuberneteslabs/providers/Microsoft.Network/networkInterfaces/etcd-2-nic is not in Succeeded state. Resource is in Updating state and the last operation that updated/is updating the resource is PutSubnetOperation."}]
* azurerm_network_interface.etcdNIC.1: network.InterfacesClient#CreateOrUpdate: Failure responding to request: StatusCode=429 -- Original Error: autorest/azure: Service returned an error. Status=429 Code="RetryableError" Message="A retryable error occured." Details=[{"code":"ReferencedResourceNotProvisioned","message":"Cannot proceed with operation because resource /subscriptions/4df9b22d-85c6-440a-a1ab-f230d65e619c/resourceGroups/kuberneteslabs/providers/Microsoft.Network/virtualNetworks/KubeVNET/subnets/etcd used by resource /subscriptions/4df9b22d-85c6-440a-a1ab-f230d65e619c/resourceGroups/kuberneteslabs/providers/Microsoft.Network/networkInterfaces/etcd-1-nic is not in Succeeded state. Resource is in Updating state and the last operation that updated/is updating the resource is PutSubnetOperation."}]
* azurerm_network_interface.etcdNIC.0: network.InterfacesClient#CreateOrUpdate: Failure responding to request: StatusCode=429 -- Original Error: autorest/azure: Service returned an error. Status=429 Code="RetryableError" Message="A retryable error occured." Details=[{"code":"ReferencedResourceNotProvisioned","message":"Cannot proceed with operation because resource /subscriptions/4df9b22d-85c6-440a-a1ab-f230d65e619c/resourceGroups/kuberneteslabs/providers/Microsoft.Network/virtualNetworks/KubeVNET/subnets/etcd used by resource /subscriptions/4df9b22d-85c6-440a-a1ab-f230d65e619c/resourceGroups/kuberneteslabs/providers/Microsoft.Network/networkInterfaces/etcd-0-nic is not in Succeeded state. Resource is in Updating state and the last operation that updated/is updating the resource is PutSubnetOperation."}]
* azurerm_network_interface.masterNIC.0: network.InterfacesClient#CreateOrUpdate: Failure responding to request: StatusCode=429 -- Original Error: autorest/azure: Service returned an error. Status=429 Code="RetryableError" Message="A retryable error occured." Details=[{"code":"ReferencedResourceNotProvisioned","message":"Cannot proceed with operation because resource /subscriptions/4df9b22d-85c6-440a-a1ab-f230d65e619c/resourceGroups/kuberneteslabs/providers/Microsoft.Network/virtualNetworks/KubeVNET/subnets/master used by resource /subscriptions/4df9b22d-85c6-440a-a1ab-f230d65e619c/resourceGroups/kuberneteslabs/providers/Microsoft.Network/networkInterfaces/master-0-nic is not in Succeeded state. Resource is in Updating state and the last operation that updated/is updating the resource is PutSubnetOperation."}]
* azurerm_network_interface.masterNIC.1: network.InterfacesClient#CreateOrUpdate: Failure responding to request: StatusCode=429 -- Original Error: autorest/azure: Service returned an error. Status=429 Code="RetryableError" Message="A retryable error occured." Details=[{"code":"ReferencedResourceNotProvisioned","message":"Cannot proceed with operation because resource /subscriptions/4df9b22d-85c6-440a-a1ab-f230d65e619c/resourceGroups/kuberneteslabs/providers/Microsoft.Network/virtualNetworks/KubeVNET/subnets/master used by resource /subscriptions/4df9b22d-85c6-440a-a1ab-f230d65e619c/resourceGroups/kuberneteslabs/providers/Microsoft.Network/networkInterfaces/master-1-nic is not in Succeeded state. Resource is in Updating state and the last operation that updated/is updating the resource is PutSubnetOperation."}]

Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.

@DavidR91
Copy link

DavidR91 commented Nov 4, 2016

Yep, I'm constantly seeing 429s in 0.7.8 as well (without additional detail, so assuming they are rate limits?)

* azurerm_virtual_network.test: network.VirtualNetworksClient#CreateOrUpdate: 
Failure responding to request: StatusCode=429 --
 Original Error: autorest/azure: Service returned an error. Status=429 Code="RetryableError" Message="A retryable error occured." 
Details=[{"Code":"GatewayError","Message":"Error occured in resource provider infrastructure services.","Target":null}]

This is after trying --parallelism=0

For me at least, additional attempts at apply aren't actually succeeding either. It is making terraform basically unusable.

What I do not understand is that destroy seems to work fine, even leaving the default parallelism setting.

I should note too that my .tf is just a rip of the example/test one in the docs, so I'm reasonably sure the dependencies etc. are correct

@DavidR91
Copy link

DavidR91 commented Nov 4, 2016

This is the output when I have TF_LOG set to debug:

2016/11/04 17:13:09 [DEBUG] plugin: terraform.exe: HTTP/1.1 429
2016/11/04 17:13:09 [DEBUG] plugin: terraform.exe: Connection: close
2016/11/04 17:13:09 [DEBUG] plugin: terraform.exe: Content-Length: 2545
2016/11/04 17:13:09 [DEBUG] plugin: terraform.exe: Cache-Control: no-cache
2016/11/04 17:13:09 [DEBUG] plugin: terraform.exe: Content-Type: application/json; charset=utf-8
2016/11/04 17:13:09 [DEBUG] plugin: terraform.exe: Date: Fri, 04 Nov 2016 17:14:29 GMT
2016/11/04 17:13:09 [DEBUG] plugin: terraform.exe: Expires: -1
2016/11/04 17:13:09 [DEBUG] plugin: terraform.exe: Pragma: no-cache
2016/11/04 17:13:09 [DEBUG] plugin: terraform.exe: Server: Microsoft-HTTPAPI/2.0
2016/11/04 17:13:09 [DEBUG] plugin: terraform.exe: Strict-Transport-Security: max-age=31536000; includeSubDomains
2016/11/04 17:13:09 [DEBUG] plugin: terraform.exe: X-Ms-Correlation-Request-Id: fff4004c-aa55-46e5-b33f-d762703aa447
2016/11/04 17:13:09 [DEBUG] plugin: terraform.exe: X-Ms-Ratelimit-Remaining-Subscription-Writes: 1160
2016/11/04 17:13:09 [DEBUG] plugin: terraform.exe: X-Ms-Request-Id: fff4004c-aa55-46e5-b33f-d762703aa447
2016/11/04 17:13:09 [DEBUG] plugin: terraform.exe: X-Ms-Routing-Request-Id: UKNORTH:20161104T171430Z:fff4004c-aa55-46e5-b33f-d762703aa447
2016/11/04 17:13:09 [DEBUG] plugin: terraform.exe:
2016/11/04 17:13:09 [DEBUG] plugin: terraform.exe: {"Error":{"Details":[{"Code":"GatewayError","Message":"Error occured in resource provider infrastructure services.","Target":null}],"InnerError":"Microsoft.WindowsAzure.Networking.Nrp.Frontend.Common.NrpException: A retryable error occured. ---> Microsoft.WindowsAzure.Networking.Nrp.Frontend.Common.NrpException: Error occured in resource provider infrastructure services. ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()\r\n   at Microsoft.WindowsAzure.Networking.Nrp.Common.ReliableHttpClient.HttpClientWrapper.<SendAsync>d__4.MoveNext() in X:\\bt\\746499\\repo\\src\\sources\\Common\\ReliableHttpClient\\HttpClientWrapper.cs:line 80\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()\r\n   at Microsoft.WindowsAzure.Networking.Nrp.GatewayServiceCommon.GatewayServiceRoutingController.<CallNrpAndGetResponseWithRetries>d__22.MoveNext() in X:\\bt\\746499\\repo\\src\\sources\\GatewayService\\GatewayCommon\\GatewayServiceRoutingController.cs:line 399\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()\r\n   at Microsoft.WindowsAzure.Networking.Nrp.GatewayServiceCommon.GatewayServiceRoutingController.<ForwardMessageToNrpAndProcessResponseAsync>d__19.MoveNext() in X:\\bt\\746499\\repo\\src\\sources\\GatewayService\\GatewayCommon\\GatewayServiceRoutingController.cs:line 159\r\n   --- End of inner exception stack trace ---\r\n   --- End of inner exception stack trace ---","Code":"RetryableError","Message":"A retryable error occured.","Target":null}}
2016/11/04 17:13:09 [DEBUG] root: eval: *terraform.EvalWriteState
2016/11/04 17:13:09 [DEBUG] root: eval: *terraform.EvalApplyProvisioners
2016/11/04 17:13:09 [DEBUG] root: eval: *terraform.EvalIf
2016/11/04 17:13:09 [DEBUG] root: eval: *terraform.EvalWriteState
2016/11/04 17:13:09 [DEBUG] root: eval: *terraform.EvalWriteDiff
2016/11/04 17:13:09 [DEBUG] root: eval: *terraform.EvalApplyPost
2016/11/04 17:13:09 [ERROR] root: eval: *terraform.EvalApplyPost, err: 1 error(s) occurred:

* azurerm_virtual_network.test: network.VirtualNetworksClient#CreateOrUpdate: Failure responding to request: StatusCode=429 -- Original Error: autorest/azure: Service returned an error. Status=429 Code="RetryableError" Message="A retryable error occured." Details=[{"Code":"GatewayError","Message":"Error occured in resource provider infrastructure services.","Target":null}]

Does 'GatewayError' point to a different problem here? (I can't see anything that implies I can change/resolve with diff. options etc. though?)

Seems to connect it back to #7153 (comment) perhaps?

@colemickens
Copy link

@DavidR91 Yes, "GatewayError" indicates it's different.

If you start from a fresh tfstate and change the resource group you're deploying to, do you still get the same error? If so, I can follow-up and see if there's a service incident going on.

@DavidR91
Copy link

DavidR91 commented Nov 4, 2016

@colemickens you're absolutely right, seems to be a service problem. Trying with a new resource group worked straight away

(and the old resource group took an exceptionally long time to destroy, seems to get hung up on storage containers especially)

@colemickens
Copy link

Okay, if it's not reproing on a new run then I'm not too terribly worried about it. Feel free to follow up with Support if you want them to look into why you were getting that error reliably on that other RG -- looked like a platform issue to me anyway.

@leowmjw
Copy link

leowmjw commented Nov 14, 2016

Just another data point to add for this issue; I also consistently see the same behavior.

1 error(s) occurred:

* azurerm_network_interface.network_interface: network.InterfacesClient#CreateOrUpdate:
Failure responding to request: StatusCode=429 -- 
Original Error: autorest/azure: Service returned an error. 
Status=429 Code="RetryableError" 
Message="A retryable error occured."

Details=
[{
"code":"ReferencedResourceNotProvisioned",

"message":"Cannot proceed with operation because resource 
/subscriptions/018c30e9-6edb-4a53-9ad0-9aa28e841f25/resourceGroups/development/providers/Microsoft.Network/virtualNetworks/developmentNetwork/subnets/subnet1 
used by resource 
/subscriptions/018c30e9-6edb-4a53-9ad0-9aa28e841f25/resourceGroups/development/providers/Microsoft.Network/networkInterfaces/developmentNetworkInterface 
is not in Succeeded state. 

Resource is in Updating state and the last operation that updated/is updating the 
resource is PutSubnetOperation."
}]

@edevil
Copy link
Contributor

edevil commented Mar 13, 2017

I just tried with the latest master and I always get this error. Retrying fixes the issue but it seems there's some kind of dependency missing and a NIC is being created before the subnet is finished:

Error applying plan:

9 error(s) occurred:

* azurerm_network_interface.etcdNIC[0]: 1 error(s) occurred:

* azurerm_network_interface.etcdNIC.0: network.InterfacesClient#CreateOrUpdate: Failure responding to request: StatusCode=429 -- Original Error: autorest/azure: Service returned an error. Status=429 Code="RetryableError" Message="A retryable error occured." Details=[{"code":"ReferencedResourceNotProvisioned","message":"Cannot proceed with operation because resource /subscriptions/4df9b22d-85c6-440a-a1ab-f230d65e619c/resourceGroups/kuberneteslabs/providers/Microsoft.Network/virtualNetworks/KubeVNET/subnets/etcd used by resource /subscriptions/4df9b22d-85c6-440a-a1ab-f230d65e619c/resourceGroups/kuberneteslabs/providers/Microsoft.Network/networkInterfaces/etcd-0-nic is not in Succeeded state. Resource is in Updating state and the last operation that updated/is updating the resource is PutSubnetOperation."}]
* azurerm_network_interface.nodeNIC[0]: 1 error(s) occurred:

* azurerm_network_interface.nodeNIC.0: network.InterfacesClient#CreateOrUpdate: Failure responding to request: StatusCode=429 -- Original Error: autorest/azure: Service returned an error. Status=429 Code="RetryableError" Message="A retryable error occured." Details=[{"code":"ReferencedResourceNotProvisioned","message":"Cannot proceed with operation because resource /subscriptions/4df9b22d-85c6-440a-a1ab-f230d65e619c/resourceGroups/kuberneteslabs/providers/Microsoft.Network/virtualNetworks/KubeVNET/subnets/node used by resource /subscriptions/4df9b22d-85c6-440a-a1ab-f230d65e619c/resourceGroups/kuberneteslabs/providers/Microsoft.Network/networkInterfaces/node-0-nic is not in Succeeded state. Resource is in Updating state and the last operation that updated/is updating the resource is PutSubnetOperation."}]
* azurerm_network_interface.jboxNIC: 1 error(s) occurred:

* azurerm_network_interface.jboxNIC: network.InterfacesClient#CreateOrUpdate: Failure responding to request: StatusCode=429 -- Original Error: autorest/azure: Service returned an error. Status=429 Code="RetryableError" Message="A retryable error occured." Details=[{"code":"ReferencedResourceNotProvisioned","message":"Cannot proceed with operation because resource /subscriptions/4df9b22d-85c6-440a-a1ab-f230d65e619c/resourceGroups/kuberneteslabs/providers/Microsoft.Network/virtualNetworks/KubeVNET/subnets/management used by resource /subscriptions/4df9b22d-85c6-440a-a1ab-f230d65e619c/resourceGroups/kuberneteslabs/providers/Microsoft.Network/networkInterfaces/jumpboxnic is not in Succeeded state. Resource is in Updating state and the last operation that updated/is updating the resource is PutSubnetOperation."}]
* azurerm_network_interface.nodeNIC[1]: 1 error(s) occurred:

* azurerm_network_interface.nodeNIC.1: network.InterfacesClient#CreateOrUpdate: Failure responding to request: StatusCode=429 -- Original Error: autorest/azure: Service returned an error. Status=429 Code="RetryableError" Message="A retryable error occured." Details=[{"code":"ReferencedResourceNotProvisioned","message":"Cannot proceed with operation because resource /subscriptions/4df9b22d-85c6-440a-a1ab-f230d65e619c/resourceGroups/kuberneteslabs/providers/Microsoft.Network/virtualNetworks/KubeVNET/subnets/node used by resource /subscriptions/4df9b22d-85c6-440a-a1ab-f230d65e619c/resourceGroups/kuberneteslabs/providers/Microsoft.Network/networkInterfaces/node-1-nic is not in Succeeded state. Resource is in Updating state and the last operation that updated/is updating the resource is PutSubnetOperation."}]
* azurerm_network_interface.nodeNIC[3]: 1 error(s) occurred:

* azurerm_network_interface.nodeNIC.3: network.InterfacesClient#CreateOrUpdate: Failure responding to request: StatusCode=429 -- Original Error: autorest/azure: Service returned an error. Status=429 Code="RetryableError" Message="A retryable error occured." Details=[{"code":"ReferencedResourceNotProvisioned","message":"Cannot proceed with operation because resource /subscriptions/4df9b22d-85c6-440a-a1ab-f230d65e619c/resourceGroups/kuberneteslabs/providers/Microsoft.Network/virtualNetworks/KubeVNET/subnets/node used by resource /subscriptions/4df9b22d-85c6-440a-a1ab-f230d65e619c/resourceGroups/kuberneteslabs/providers/Microsoft.Network/networkInterfaces/node-3-nic is not in Succeeded state. Resource is in Updating state and the last operation that updated/is updating the resource is PutSubnetOperation."}]
* azurerm_network_interface.nodeNIC[2]: 1 error(s) occurred:

* azurerm_network_interface.nodeNIC.2: network.InterfacesClient#CreateOrUpdate: Failure responding to request: StatusCode=429 -- Original Error: autorest/azure: Service returned an error. Status=429 Code="RetryableError" Message="A retryable error occured." Details=[{"code":"ReferencedResourceNotProvisioned","message":"Cannot proceed with operation because resource /subscriptions/4df9b22d-85c6-440a-a1ab-f230d65e619c/resourceGroups/kuberneteslabs/providers/Microsoft.Network/virtualNetworks/KubeVNET/subnets/node used by resource /subscriptions/4df9b22d-85c6-440a-a1ab-f230d65e619c/resourceGroups/kuberneteslabs/providers/Microsoft.Network/networkInterfaces/node-2-nic is not in Succeeded state. Resource is in Updating state and the last operation that updated/is updating the resource is PutSubnetOperation."}]
* azurerm_network_interface.etcdNIC[1]: 1 error(s) occurred:

* azurerm_network_interface.etcdNIC.1: network.InterfacesClient#CreateOrUpdate: Failure responding to request: StatusCode=429 -- Original Error: autorest/azure: Service returned an error. Status=429 Code="RetryableError" Message="A retryable error occured." Details=[{"code":"ReferencedResourceNotProvisioned","message":"Cannot proceed with operation because resource /subscriptions/4df9b22d-85c6-440a-a1ab-f230d65e619c/resourceGroups/kuberneteslabs/providers/Microsoft.Network/virtualNetworks/KubeVNET/subnets/etcd used by resource /subscriptions/4df9b22d-85c6-440a-a1ab-f230d65e619c/resourceGroups/kuberneteslabs/providers/Microsoft.Network/networkInterfaces/etcd-1-nic is not in Succeeded state. Resource is in Updating state and the last operation that updated/is updating the resource is PutSubnetOperation."}]
* azurerm_network_interface.nodeNIC[4]: 1 error(s) occurred:

* azurerm_network_interface.nodeNIC.4: network.InterfacesClient#CreateOrUpdate: Failure responding to request: StatusCode=429 -- Original Error: autorest/azure: Service returned an error. Status=429 Code="RetryableError" Message="A retryable error occured." Details=[{"code":"ReferencedResourceNotProvisioned","message":"Cannot proceed with operation because resource /subscriptions/4df9b22d-85c6-440a-a1ab-f230d65e619c/resourceGroups/kuberneteslabs/providers/Microsoft.Network/virtualNetworks/KubeVNET/subnets/node used by resource /subscriptions/4df9b22d-85c6-440a-a1ab-f230d65e619c/resourceGroups/kuberneteslabs/providers/Microsoft.Network/networkInterfaces/node-4-nic is not in Succeeded state. Resource is in Updating state and the last operation that updated/is updating the resource is PutSubnetOperation."}]
* azurerm_network_interface.etcdNIC[2]: 1 error(s) occurred:

* azurerm_network_interface.etcdNIC.2: network.InterfacesClient#CreateOrUpdate: Failure responding to request: StatusCode=429 -- Original Error: autorest/azure: Service returned an error. Status=429 Code="RetryableError" Message="A retryable error occured." Details=[{"code":"ReferencedResourceNotProvisioned","message":"Cannot proceed with operation because resource /subscriptions/4df9b22d-85c6-440a-a1ab-f230d65e619c/resourceGroups/kuberneteslabs/providers/Microsoft.Network/virtualNetworks/KubeVNET/subnets/etcd used by resource /subscriptions/4df9b22d-85c6-440a-a1ab-f230d65e619c/resourceGroups/kuberneteslabs/providers/Microsoft.Network/networkInterfaces/etcd-2-nic is not in Succeeded state. Resource is in Updating state and the last operation that updated/is updating the resource is PutSubnetOperation."}]

@clstokes
Copy link
Contributor

I'm getting this sporadically on v0.8.6 for other network related resources.


* azurerm_network_interface.nat.0: network.InterfacesClient#CreateOrUpdate: Failure responding to request: StatusCode=429 -- Original Error: autorest/azure: Service returned an error. Status=429 Code="RetryableError" Message="A retryable error occured." Details=[{"code":"ReferencedResourceNotProvisioned","message":"Cannot proceed with operation because resource /subscriptions/db0bd227-fcee-440f-a225-ce1cd1c848c8/resourceGroups/vpc-foundation/providers/Microsoft.Network/virtualNetworks/vpc-foundation/subnets/vpc-foundation-public-0 used by resource /subscriptions/db0bd227-fcee-440f-a225-ce1cd1c848c8/resourceGroups/vpc-foundation/providers/Microsoft.Network/networkInterfaces/vpc-foundation-nat-0 is not in Succeeded state. Resource is in Updating state and the last operation that updated/is updating the resource is PutSubnetOperation."}]

I think this is similar to errors Terraform has had with other providers (AWS) where Terraform just needs to handle the 429 as a 404 and retry the same operation (with backoff).

@luisamador
Copy link
Contributor

luisamador commented Apr 6, 2017

I'm seeing this error on v0.9.2 when deploying a network interface that is in a specific subnet:

`* azurerm_network_interface.bastion: 1 error(s) occurred:

  • azurerm_network_interface.bastion: network.InterfacesClient#CreateOrUpdate: Failure responding to request: StatusCode=429 -- Original Error: autorest/azure: Service returned an error. Status=429 Code="RetryableError" Message="A retryable error occured." Details=[{"code":"ReferencedResourceNotProvisioned","message":"Cannot proceed with operation because resource /subscriptions/d460e9c0-da40-4234-ad00-b48c6bec32ae/resourceGroups/net-rg/providers/Microsoft.Network/virtualNetworks/test-vnet/subnets/test-dmz1-sn used by resource /subscriptions/d460e9c0-da40-4234-ad00-b48c6bec32ae/resourceGroups/test-rg/providers/Microsoft.Network/networkInterfaces/test-bastion-netif is not in Succeeded state. Resource is in Updating state and the last operation that updated/is updating the resource is PutSubnetOperation."}]`

How come this is still happening on version 0.9.2? This issue was opened in Aug 2016...

Any chance this will get fixed soon?

Thanks

@colemickens
Copy link

It's actually been since June. cc: @tombuildsstuff

@baannee
Copy link

baannee commented Apr 8, 2017

I am getting the same issue when I am trying to associate NSG to NIC

_azurerm_network_interface.web-srv-interface: network.InterfacesClient#CreateOrUpdate: Failure responding to reques
StatusCode=429 -- Original Error: autorest/azure: Service returned an error. Status=429 Code="RetryableError" Mes
ge="A retryable error occured." Details=[{"code":"ReferencedResourceNotProvisioned","message":"Cannot proceed with
peration because resource /subscriptions/d34f6ad4-64cd-4ece-8822-ebd8ea5187fc/resourceGroups/IoT-ConnectedCar/prov
ers/Microsoft.Network/networkSecurityGroups/SG_Public_Web used by resource /subscriptions/d34f6ad4-64cd-4ece-8822-
d8ea5187fc/resourceGroups/IoT-ConnectedCar/providers/Microsoft.Network/networkInterfaces/NI_Web_Server is not in S
ceeded state. Resource is in Updating state and the last operation that updated/is updating the resource is PutSec
ityRuleOperation."}]
azurerm_network_interface.bas-srv-interface: 1 error(s) occurred: _

@sheerun
Copy link
Contributor

sheerun commented Apr 9, 2017

@colemickens @tombuildsstuff I get the same when creating network interface, if at the same time there are rules added to network security group (rendering network security group in "Updated" state). I need to retry every time as well. Here's an error.

  • module.kubernetes.module.node1_network_interface.azurerm_network_interface.master: 1 error(s) occurred:

  • azurerm_network_interface.master: network.InterfacesClient#CreateOrUpdate: Failure responding to request: StatusCode=429 -- Original Error: autorest/azure: Service returned an error. Status=429 Code="RetryableError" Message="A retryable error occured." Details=[{"code":"ReferencedResourceNotProvisioned","message":"Cannot proceed with operation because resource /subscriptions/xxx/resourceGroups/kubegreen/providers/Microsoft.Network/networkSecurityGroups/kubegreen-security-group used by resource /subscriptions/xxx/resourceGroups/kubegreen/providers/Microsoft.Network/networkInterfaces/kubegreen-node1-interface is not in Succeeded state. Resource is in Updating state and the last operation that updated/is updating the resource is PutSecurityRuleOperation."}]

  • module.kubernetes.module.master1_network_interface.azurerm_network_interface.master: 1 error(s) occurred:

  • azurerm_network_interface.master: network.InterfacesClient#CreateOrUpdate: Failure responding to request: StatusCode=429 -- Original Error: autorest/azure: Service returned an error. Status=429 Code="RetryableError" Message="A retryable error occured." Details=[{"code":"ReferencedResourceNotProvisioned","message":"Cannot proceed with operation because resource /subscriptions/xxx/resourceGroups/kubegreen/providers/Microsoft.Network/networkSecurityGroups/kubegreen-security-group used by resource /subscriptions/xxx/resourceGroups/kubegreen/providers/Microsoft.Network/networkInterfaces/kubegreen-master1-interface is not in Succeeded state. Resource is in Updating state and the last operation that updated/is updating the resource is PutSecurityRuleOperation."}]

The solution would be an auto-dependency of azurerm_network_interface on all azurerm_network_security_rule of azurerm_network_security_group it is attached to to.

In short the error happens because azurerm_network_security_rule of given azurerm_network_security_group are created at the same time azurerm_network_interface is created that is attached to given azurerm_network_security_group

@squasta
Copy link

squasta commented Apr 25, 2017

Hi, I'am using Terraform 0.9.3 and have also the same issue

  • azurerm_network_interface.Terra-NIC1-OCPNA1[1]: 1 error(s) occurred:
  • azurerm_network_interface.Terra-NIC1-OCPNA1.1: network.InterfacesClient#CreateOrUpdate: Failure responding to request: StatusCode=429 -- Original Error: autorest/azure: Service returned an error. Status=429 Code="RetryableError" Message="A retryable error occured." Details=[{"code":"ReferencedResourceNotProvisioned","message":"Cannot proceed with operation because resource /subscriptions/f885b031-XXXX-40e4-9240-eb77ae16cc26/resourceGroups/rg-openshift/providers/Microsoft.Network/virtualNetworks/VNet-OCP/subnets/Subnet-OCPAppliNode used by resource /subscriptions/f885b031-XXXX-40e4-9240-eb77ae16cc26/resourceGroups/rg-openshift/providers/Microsoft.Network/networkInterfaces/NIC0-OCPNA-1 is not in Succeeded state. Resource is in Updating state and the last operation that updated/is updating the resource is PutSubnetOperation."}

I am deploying several VM (>12) connected to many Subnet with NSG applied on Subnet.

This issue occurs every time on my deployment (if i execute on more time Terraform apply, deployement can finish but it appears to me not ready for production)

@tombuildsstuff
Copy link
Contributor

Hi @squasta

I believe this issue has been fixed in Terraform 0.9.4 - would it be possible for you to confirm using this version?

Thanks! :)

@edevil
Copy link
Contributor

edevil commented May 4, 2017

Not yet for me:

Error applying plan:

6 error(s) occurred:

* azurerm_network_interface.etcdNIC[0]: 1 error(s) occurred:

* azurerm_network_interface.etcdNIC.0: network.InterfacesClient#CreateOrUpdate: Failure responding to request: StatusCode=429 -- Original Error: autorest/azure: Service returned an error. Status=429 Code="RetryableError" Message="A retryable error occured." Details=[{"code":"ReferencedResourceNotProvisioned","message":"Cannot proceed with operation because resource /subscriptions/4df9b22d-85c6-440a-a1ab-f230d65e619c/resourceGroups/kuberneteslabs/providers/Microsoft.Network/virtualNetworks/KubeVNET/subnets/etcd used by resource /subscriptions/4df9b22d-85c6-440a-a1ab-f230d65e619c/resourceGroups/kuberneteslabs/providers/Microsoft.Network/networkInterfaces/etcd-0-nic is not in Succeeded state. Resource is in Updating state and the last operation that updated/is updating the resource is PutSubnetOperation."}]
* azurerm_network_interface.jboxNIC: 1 error(s) occurred:

* azurerm_network_interface.jboxNIC: network.InterfacesClient#CreateOrUpdate: Failure responding to request: StatusCode=429 -- Original Error: autorest/azure: Service returned an error. Status=429 Code="RetryableError" Message="A retryable error occured." Details=[{"code":"ReferencedResourceNotProvisioned","message":"Cannot proceed with operation because resource /subscriptions/4df9b22d-85c6-440a-a1ab-f230d65e619c/resourceGroups/kuberneteslabs/providers/Microsoft.Network/virtualNetworks/KubeVNET/subnets/management used by resource /subscriptions/4df9b22d-85c6-440a-a1ab-f230d65e619c/resourceGroups/kuberneteslabs/providers/Microsoft.Network/networkInterfaces/jumpboxnic is not in Succeeded state. Resource is in Updating state and the last operation that updated/is updating the resource is PutSubnetOperation."}]
* azurerm_network_interface.etcdNIC[1]: 1 error(s) occurred:

* azurerm_network_interface.etcdNIC.1: network.InterfacesClient#CreateOrUpdate: Failure responding to request: StatusCode=429 -- Original Error: autorest/azure: Service returned an error. Status=429 Code="RetryableError" Message="A retryable error occured." Details=[{"code":"ReferencedResourceNotProvisioned","message":"Cannot proceed with operation because resource /subscriptions/4df9b22d-85c6-440a-a1ab-f230d65e619c/resourceGroups/kuberneteslabs/providers/Microsoft.Network/virtualNetworks/KubeVNET/subnets/etcd used by resource /subscriptions/4df9b22d-85c6-440a-a1ab-f230d65e619c/resourceGroups/kuberneteslabs/providers/Microsoft.Network/networkInterfaces/etcd-1-nic is not in Succeeded state. Resource is in Updating state and the last operation that updated/is updating the resource is PutSubnetOperation."}]
* azurerm_network_interface.masterNIC[0]: 1 error(s) occurred:

* azurerm_network_interface.masterNIC.0: network.InterfacesClient#CreateOrUpdate: Failure responding to request: StatusCode=429 -- Original Error: autorest/azure: Service returned an error. Status=429 Code="RetryableError" Message="A retryable error occured." Details=[{"code":"ReferencedResourceNotProvisioned","message":"Cannot proceed with operation because resource /subscriptions/4df9b22d-85c6-440a-a1ab-f230d65e619c/resourceGroups/kuberneteslabs/providers/Microsoft.Network/virtualNetworks/KubeVNET/subnets/master used by resource /subscriptions/4df9b22d-85c6-440a-a1ab-f230d65e619c/resourceGroups/kuberneteslabs/providers/Microsoft.Network/networkInterfaces/master-0-nic is not in Succeeded state. Resource is in Updating state and the last operation that updated/is updating the resource is PutSubnetOperation."}]
* azurerm_network_interface.masterNIC[1]: 1 error(s) occurred:

* azurerm_network_interface.masterNIC.1: network.InterfacesClient#CreateOrUpdate: Failure responding to request: StatusCode=429 -- Original Error: autorest/azure: Service returned an error. Status=429 Code="RetryableError" Message="A retryable error occured." Details=[{"code":"ReferencedResourceNotProvisioned","message":"Cannot proceed with operation because resource /subscriptions/4df9b22d-85c6-440a-a1ab-f230d65e619c/resourceGroups/kuberneteslabs/providers/Microsoft.Network/virtualNetworks/KubeVNET/subnets/master used by resource /subscriptions/4df9b22d-85c6-440a-a1ab-f230d65e619c/resourceGroups/kuberneteslabs/providers/Microsoft.Network/networkInterfaces/master-1-nic is not in Succeeded state. Resource is in Updating state and the last operation that updated/is updating the resource is PutSubnetOperation."}]
* azurerm_network_interface.etcdNIC[2]: 1 error(s) occurred:

* azurerm_network_interface.etcdNIC.2: network.InterfacesClient#CreateOrUpdate: Failure responding to request: StatusCode=429 -- Original Error: autorest/azure: Service returned an error. Status=429 Code="RetryableError" Message="A retryable error occured." Details=[{"code":"ReferencedResourceNotProvisioned","message":"Cannot proceed with operation because resource /subscriptions/4df9b22d-85c6-440a-a1ab-f230d65e619c/resourceGroups/kuberneteslabs/providers/Microsoft.Network/virtualNetworks/KubeVNET/subnets/etcd used by resource /subscriptions/4df9b22d-85c6-440a-a1ab-f230d65e619c/resourceGroups/kuberneteslabs/providers/Microsoft.Network/networkInterfaces/etcd-2-nic is not in Succeeded state. Resource is in Updating state and the last operation that updated/is updating the resource is PutSubnetOperation."}]

Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.

@tombuildsstuff
Copy link
Contributor

Hi @edevil

Thanks for letting us know, apologies about this - I've taken another look at this and managed to isolate this down to creating multiple NIC's in different Subnets within the same Virtual Network at the same time in this minimal Terraform configuration as my test case.

I've opened PR #14365 which I believe fixes this - and tested it using the configuration posted above, and the Terraform configuration generated from phase1 of the kubernetes-anywhere project and it looks good across multiple runs :)

Thanks!

cc @colemickens @clstokes

@orelhinhas
Copy link

I experienced a similar problem with LB creation. After the error occured if I run apply again the resource is created normal. When I want destroy the same behavior occurs.
I'm using Terraform 0.9.6

Follow the error

  • module.frontend.azurerm_lb.lb_frontend[1]: 1 error(s) occurred:

  • azurerm_lb.lb_frontend.1: Error Creating/Updating LoadBalancer network.LoadBalancersClient#CreateOrUpdate: Failure responding to request: StatusCode=429 -- Original Error: autorest/azure: Service returned an error. Status=429 Code="RetryableError" Message="A retryable error occured." Details=[{"code":"RetryableErrorDueToAnotherOperation","message":"Operation PutLoadBalancerOperation (26a1fe7e-38e0-4807-b5a7-08a1d1b0b95c) is updating resource /subscriptions/7550e848-cbc3-43a3-a8f1-39a157d5f88e/resourceGroups/leroleroFrontend/providers/Microsoft.Network/loadBalancers/lb_sharepoint. The call can be retried in 12 seconds."}]

@orelhinhas
Copy link

Never mind, Azure may has a rate limit on their api because when I set -parallelism=2 for example no errors occurred.

@ghost
Copy link

ghost commented Apr 11, 2020

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

@ghost ghost locked and limited conversation to collaborators Apr 11, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.