429 Retryable error when creating Subnet with Route Table

[whiskeyjay]

1. Create a Route Table using the RouteTablesClient.CreateOrUpdate() method. Verified the op’s status was “Succeeded”.
2. Create a Route using the RoutesClient.CreateOrUpdate() method, which adds the Route to the Route Table created in step 1. Verified the op’s status was “Succeeded”.
3. Create a Subnet using the SubnetClient.CreateOrUpdate() method, with Route Table ID from step 1.

Intermittently, I get the following error from step 3:

Failure responding to request: StatusCode=429 -- Original Error: autorest/azure: Service returned an error. Status=429 Code="RetryableError" Message="A retryable error occured." Details=[{"code":"ReferencedResourceNotProvisioned","message":"Cannot proceed with operation because resource /subscriptions/c3adb315-00d7-4bd0-96be-c96210ced312/resourceGroups/acctestRG/providers/Microsoft.Network/routeTables/rtTable2-RT used by resource /subscriptions/c3adb315-00d7-4bd0-96be-c96210ced312/resourceGroups/acctestRG/providers/Microsoft.Network/virtualNetworks/acctestvirtnet/subnets/acctestsubnet is not in Succeeded state. Resource is in Updating state and the last operation that updated/is updating the resource is PutRouteOperation."}]

The interesting thing is, here code 429 doesn’t seem to mean “too many requests” as we usually see; the message says the Route Table I was trying to associated with the Subnet is in “Updating” state and that causes the creation of the Subnet to fail.

Questions:

• In step 1, it is already verified that the RouteTable creations status was “Succeeded”, why in step 3 it became “Updating”? Is it because the op in step 2?
• For this kind of error, can GO SDK do something like automatic retry?
• Or maybe in step 2, the method should not only check the status of the Route, but also check the status of its containing Route Table before returning?

[mcardosos]

This SDK has automatic retries, yes. But not for that status code.
This is one of those cases in which writing a custom CreateOrUpdateSender func is the best thing to do. In that custom Sender you can add 429 as a valid status code for retrying.

[marstr]

We're going to leave this issue open until the issue above (Azure/azure-rest-api-specs#1124) has a resolution. At the moment, without figuring out what is the right behavior up the stack, there isn't really anything actionable to do here.

[tombuildsstuff]

👋

An issue's been opened in the Terraform Repository about a similar issue in the network package which I believe is related. Out of interest, is this status code reused across the networking API's?

Thanks :)

[marstr]

Good question, @tombuildsstuff. @DeepakRajendranMsft, is that something you can speak to?

[marstr]

Alright, we've recently had this come up again, and while it's top of mind I want to refresh this thread. We've come to decide that there is actually a pretty good answer to what to do in all cases for 429:

• 429 with a Retry-After header - Instead of waiting the normal amount of time as calculated by our normal back-off strategy, wait however long the response told you to wait.
• 429 without a Retry-After header - Stop retrying immediately, if you continue (this is not a known Azure behavior, just a thought experiment,) you may be labeled a DDOS attacker.

You may notice that this means that our retry logic need to be somewhat more sophisticated than what is currently present in go-autorest at time of writing. We'll be needing to rewrite some of our retry infrastructure to accomodate this. It'll likely be a relatively big change, so that probably won't land until go-autorest v9.0.

[mcardosos]

Fixed on Azure/go-autorest#155
@tombuildsstuff @whiskeyjay