BUG: Create PrivateEndpoint using PrivateLink's alias may fail in concurrent creation scenario #20289

lonegunmanb · 2022-08-18T09:39:15Z

When creating multiple PrivateEndpoints with PrivateLink's alias may fail with the following error message:

PrivateLinkServiceId Invalid private link service id: {0} passed in private link service connection: {1} passed in input is invalid

The put request to create the PrivateLink resource:

PUT https://management.azure.com/subscriptions/xxxxxxxxxxxxx/resourceGroups/acctestRG-privatelink-220818165005420368/providers/Microsoft.Network/privateLinkServices/acctestPLS-220818165005420368?api-version=2021-08-01 HTTP/1.1
Host: management.azure.com
User-Agent: Go/go1.19 (amd64-windows) go-autorest/v14.2.1 Azure-SDK-For-Go/v65.0.0 network/2021-08-01 HashiCorp Terraform/1.2.6 (+https://www.terraform.io) Terraform Plugin SDK/2.10.1 terraform-provider-azurerm/dev pid-222c6c49-1b0a-5959-a213-6608f9eb8820
Content-Length: 962
Authorization: Bearer 
Content-Type: application/json; charset=utf-8
X-Ms-Authorization-Auxiliary: 
X-Ms-Correlation-Request-Id: 0c5863fa-fb25-5b75-eab0-4802ca30531e
Accept-Encoding: gzip

The request's body is:

{
    "location": "centralus",
    "properties": {
        "autoApproval": {
            "subscriptions": [
                "xxxxxxxxxxxxx"
            ]
        },
        "enableProxyProtocol": false,
        "fqdns": [],
        "ipConfigurations": [
            {
                "name": "primaryIpConfiguration-220818165005420368",
                "properties": {
                    "primary": true,
                    "privateIPAddress": "",
                    "privateIPAddressVersion": "IPv4",
                    "privateIPAllocationMethod": "Dynamic",
                    "subnet": {
                        "id": "/subscriptions/xxxxxxxxxxxxx/resourceGroups/acctestRG-privatelink-220818165005420368/providers/Microsoft.Network/virtualNetworks/acctestvnet-220818165005420368/subnets/acctestsnetservice-220818165005420368"
                    }
                }
            }
        ],
        "loadBalancerFrontendIpConfigurations": [
            {
                "id": "/subscriptions/xxxxxxxxxxxxx/resourceGroups/acctestRG-privatelink-220818165005420368/providers/Microsoft.Network/loadBalancers/acctestlb-220818165005420368/frontendIPConfigurations/acctestpip-220818165005420368"
            }
        ],
        "visibility": {
            "subscriptions": [
                "xxxxxxxxxxxxx"
            ]
        }
    },
    "tags": {}
}

The response is:

HTTP/1.1 201 Created
Cache-Control: no-cache
Pragma: no-cache
Content-Length: 2585
Content-Type: application/json; charset=utf-8
Expires: -1
Retry-After: 10
x-ms-request-id: 2da20903-2d40-4b7b-b545-50dfb9602eab
Azure-AsyncOperation: https://management.azure.com/subscriptions/xxxxxxxxxxxxx/providers/Microsoft.Network/locations/centralus/operations/2da20903-2d40-4b7b-b545-50dfb9602eab?api-version=2021-08-01
x-ms-correlation-request-id: 0c5863fa-fb25-5b75-eab0-4802ca30531e
Azure-AsyncNotification: Enabled
x-ms-arm-service-request-id: 4bc3f756-b4da-4b91-95b9-522032985f74
Strict-Transport-Security: max-age=31536000; includeSubDomains
Server: Microsoft-HTTPAPI/2.0
Server: Microsoft-HTTPAPI/2.0
x-ms-ratelimit-remaining-subscription-writes: 1195
x-ms-routing-request-id: SOUTHEASTASIA:20220818T085145Z:33bc72f4-61e9-47bc-854d-ec7d23274951
X-Content-Type-Options: nosniff
Date: Thu, 18 Aug 2022 08:51:44 GMT

{
  "name": "acctestPLS-220818165005420368",
  "id": "/subscriptions/xxxxxxxxxxxxx/resourceGroups/acctestRG-privatelink-220818165005420368/providers/Microsoft.Network/privateLinkServices/acctestPLS-220818165005420368",
  "etag": "W/\"067e8de9-b44f-473a-901d-0ad58301f3cc\"",
  "type": "Microsoft.Network/privateLinkServices",
  "location": "centralus",
  "tags": {},
  "properties": {
    "provisioningState": "Updating",
    "resourceGuid": "e2736d14-8e82-4051-98d0-cbff65b71fae",
    "fqdns": [],
    "alias": "acctestpls-220818165005420368.b08a765d-0bcb-41bf-a0ec-755d43737dbc.centralus.azure.privatelinkservice",
    "visibility": {
      "subscriptions": [
        "xxxxxxxxxxxxx"
      ]
    },
    "autoApproval": {
      "subscriptions": [
        "xxxxxxxxxxxxx"
      ]
    },
    "enableProxyProtocol": false,
    "loadBalancerFrontendIpConfigurations": [
      {
        "id": "/subscriptions/xxxxxxxxxxxxx/resourceGroups/acctestRG-privatelink-220818165005420368/providers/Microsoft.Network/loadBalancers/acctestlb-220818165005420368/frontendIPConfigurations/acctestpip-220818165005420368"
      }
    ],
    "ipConfigurations": [
      {
        "name": "primaryIpConfiguration-220818165005420368",
        "id": "/subscriptions/xxxxxxxxxxxxx/resourceGroups/acctestRG-privatelink-220818165005420368/providers/Microsoft.Network/privateLinkServices/acctestPLS-220818165005420368/ipConfigurations/primaryIpConfiguration-220818165005420368",
        "etag": "W/\"067e8de9-b44f-473a-901d-0ad58301f3cc\"",
        "type": "Microsoft.Network/privateLinkServices/ipConfigurations",
        "properties": {
          "provisioningState": "Succeeded",
          "privateIPAllocationMethod": "Dynamic",
          "subnet": {
            "id": "/subscriptions/xxxxxxxxxxxxx/resourceGroups/acctestRG-privatelink-220818165005420368/providers/Microsoft.Network/virtualNetworks/acctestvnet-220818165005420368/subnets/acctestsnetservice-220818165005420368"
          },
          "primary": true,
          "privateIPAddressVersion": "IPv4"
        }
      }
    ],
    "privateEndpointConnections": [],
    "networkInterfaces": [
      {
        "id": "/subscriptions/xxxxxxxxxxxxx/resourceGroups/acctestRG-privatelink-220818165005420368/providers/Microsoft.Network/networkInterfaces/acctestPLS-220818165005420368.nic.fd797a3e-c0fd-4464-a8cc-32a3f1e21992"
      }
    ]
  }
}

As we can see, the PrivateLink resource acctestPLS-220818165005420368 was created successfully, it's alias was acctestpls-220818165005420368.b08a765d-0bcb-41bf-a0ec-755d43737dbc.centralus.azure.privatelinkservice

Then we sent a request to create a new PrivateEndpoint using this alias:

PUT https://management.azure.com/subscriptions/xxxxxxxxxxxxx/resourceGroups/acctestRG-privatelink-220818165005420368/providers/Microsoft.Network/privateEndpoints/acctest-privatelink-220818165005420368?api-version=2021-08-01 HTTP/1.1
Host: management.azure.com
User-Agent: Go/go1.19 (amd64-windows) go-autorest/v14.2.1 Azure-SDK-For-Go/v65.0.0 network/2021-08-01 HashiCorp Terraform/1.2.6 (+https://www.terraform.io) Terraform Plugin SDK/2.10.1 terraform-provider-azurerm/dev pid-222c6c49-1b0a-5959-a213-6608f9eb8820
Content-Length: 593
Authorization: Bearer 
Content-Type: application/json; charset=utf-8
X-Ms-Authorization-Auxiliary: 
X-Ms-Correlation-Request-Id: 0c5863fa-fb25-5b75-eab0-4802ca30531e
Accept-Encoding: gzip

The body was:

{
    "location": "centralus",
    "properties": {
        "manualPrivateLinkServiceConnections": [
            {
                "name": "acctestPLS-220818165005420368",
                "properties": {
                    "groupIds": [],
                    "privateLinkServiceId": "acctestpls-220818165005420368.b08a765d-0bcb-41bf-a0ec-755d43737dbc.centralus.azure.privatelinkservice",
                    "requestMessage": "test"
                }
            }
        ],
        "privateLinkServiceConnections": [],
        "subnet": {
            "id": "/subscriptions/xxxxxxxxxxxxx/resourceGroups/acctestRG-privatelink-220818165005420368/providers/Microsoft.Network/virtualNetworks/acctestvnet-220818165005420368/subnets/acctestsnetendpoint-220818165005420368"
        }
    },
    "tags": {}
}

We used the PrivateLink's alias here, and the response was:

HTTP/1.1 400 Bad Request
Cache-Control: no-cache
Pragma: no-cache
Content-Length: 237
Content-Type: application/json; charset=utf-8
Expires: -1
x-ms-request-id: d8c3b7fa-7169-4ce5-b2ae-4ecc7629d94c
x-ms-correlation-request-id: 0c5863fa-fb25-5b75-eab0-4802ca30531e
x-ms-arm-service-request-id: 3edd9ac3-4ceb-42b1-b1d2-92ca978897df
Strict-Transport-Security: max-age=31536000; includeSubDomains
Server: Microsoft-HTTPAPI/2.0
Server: Microsoft-HTTPAPI/2.0
x-ms-ratelimit-remaining-subscription-writes: 1196
x-ms-routing-request-id: SOUTHEASTASIA: 20220818T085206Z:c744abc8-4534-4774-845a-a588a6856e90
X-Content-Type-Options: nosniff
Date: Thu,
18 Aug 2022 08: 52: 06 GMT

{
    "error": {
        "code": "InvalidPrivateLinkServiceId",
        "message": "PrivateLinkServiceId Invalid private link service id: {0} passed in private link service connection: {1} passed in input is invalid",
        "details": []
    }
}

We believe this issue is caused by some cache-related issue on service side because:

It only happens when we create multiple PrivateLinks concurrently
If we retry the creation of PrivateEndpoint, we'll get a successful creation eventually.

We can reproduce this issue by creationg multiple PrivateLinks concurrently, then create a PrivateEndpoint with PrivateLink's alias. Can anyone help us solve this issue? Thanks!

The text was updated successfully, but these errors were encountered:

ghost · 2022-08-22T09:52:25Z

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @privlinksuppgithub.

Issue Details

When creating multiple PrivateEndpoints with PrivateLink's alias may fail with the following error message:

PrivateLinkServiceId Invalid private link service id: {0} passed in private link service connection: {1} passed in input is invalid

The put request to create the PrivateLink resource:

PUT https://management.azure.com/subscriptions/xxxxxxxxxxxxx/resourceGroups/acctestRG-privatelink-220818165005420368/providers/Microsoft.Network/privateLinkServices/acctestPLS-220818165005420368?api-version=2021-08-01 HTTP/1.1
Host: management.azure.com
User-Agent: Go/go1.19 (amd64-windows) go-autorest/v14.2.1 Azure-SDK-For-Go/v65.0.0 network/2021-08-01 HashiCorp Terraform/1.2.6 (+https://www.terraform.io) Terraform Plugin SDK/2.10.1 terraform-provider-azurerm/dev pid-222c6c49-1b0a-5959-a213-6608f9eb8820
Content-Length: 962
Authorization: Bearer 
Content-Type: application/json; charset=utf-8
X-Ms-Authorization-Auxiliary: 
X-Ms-Correlation-Request-Id: 0c5863fa-fb25-5b75-eab0-4802ca30531e
Accept-Encoding: gzip

The request's body is:

{
    "location": "centralus",
    "properties": {
        "autoApproval": {
            "subscriptions": [
                "xxxxxxxxxxxxx"
            ]
        },
        "enableProxyProtocol": false,
        "fqdns": [],
        "ipConfigurations": [
            {
                "name": "primaryIpConfiguration-220818165005420368",
                "properties": {
                    "primary": true,
                    "privateIPAddress": "",
                    "privateIPAddressVersion": "IPv4",
                    "privateIPAllocationMethod": "Dynamic",
                    "subnet": {
                        "id": "/subscriptions/xxxxxxxxxxxxx/resourceGroups/acctestRG-privatelink-220818165005420368/providers/Microsoft.Network/virtualNetworks/acctestvnet-220818165005420368/subnets/acctestsnetservice-220818165005420368"
                    }
                }
            }
        ],
        "loadBalancerFrontendIpConfigurations": [
            {
                "id": "/subscriptions/xxxxxxxxxxxxx/resourceGroups/acctestRG-privatelink-220818165005420368/providers/Microsoft.Network/loadBalancers/acctestlb-220818165005420368/frontendIPConfigurations/acctestpip-220818165005420368"
            }
        ],
        "visibility": {
            "subscriptions": [
                "xxxxxxxxxxxxx"
            ]
        }
    },
    "tags": {}
}

The response is:

HTTP/1.1 201 Created
Cache-Control: no-cache
Pragma: no-cache
Content-Length: 2585
Content-Type: application/json; charset=utf-8
Expires: -1
Retry-After: 10
x-ms-request-id: 2da20903-2d40-4b7b-b545-50dfb9602eab
Azure-AsyncOperation: https://management.azure.com/subscriptions/xxxxxxxxxxxxx/providers/Microsoft.Network/locations/centralus/operations/2da20903-2d40-4b7b-b545-50dfb9602eab?api-version=2021-08-01
x-ms-correlation-request-id: 0c5863fa-fb25-5b75-eab0-4802ca30531e
Azure-AsyncNotification: Enabled
x-ms-arm-service-request-id: 4bc3f756-b4da-4b91-95b9-522032985f74
Strict-Transport-Security: max-age=31536000; includeSubDomains
Server: Microsoft-HTTPAPI/2.0
Server: Microsoft-HTTPAPI/2.0
x-ms-ratelimit-remaining-subscription-writes: 1195
x-ms-routing-request-id: SOUTHEASTASIA:20220818T085145Z:33bc72f4-61e9-47bc-854d-ec7d23274951
X-Content-Type-Options: nosniff
Date: Thu, 18 Aug 2022 08:51:44 GMT

{
  "name": "acctestPLS-220818165005420368",
  "id": "/subscriptions/xxxxxxxxxxxxx/resourceGroups/acctestRG-privatelink-220818165005420368/providers/Microsoft.Network/privateLinkServices/acctestPLS-220818165005420368",
  "etag": "W/\"067e8de9-b44f-473a-901d-0ad58301f3cc\"",
  "type": "Microsoft.Network/privateLinkServices",
  "location": "centralus",
  "tags": {},
  "properties": {
    "provisioningState": "Updating",
    "resourceGuid": "e2736d14-8e82-4051-98d0-cbff65b71fae",
    "fqdns": [],
    "alias": "acctestpls-220818165005420368.b08a765d-0bcb-41bf-a0ec-755d43737dbc.centralus.azure.privatelinkservice",
    "visibility": {
      "subscriptions": [
        "xxxxxxxxxxxxx"
      ]
    },
    "autoApproval": {
      "subscriptions": [
        "xxxxxxxxxxxxx"
      ]
    },
    "enableProxyProtocol": false,
    "loadBalancerFrontendIpConfigurations": [
      {
        "id": "/subscriptions/xxxxxxxxxxxxx/resourceGroups/acctestRG-privatelink-220818165005420368/providers/Microsoft.Network/loadBalancers/acctestlb-220818165005420368/frontendIPConfigurations/acctestpip-220818165005420368"
      }
    ],
    "ipConfigurations": [
      {
        "name": "primaryIpConfiguration-220818165005420368",
        "id": "/subscriptions/xxxxxxxxxxxxx/resourceGroups/acctestRG-privatelink-220818165005420368/providers/Microsoft.Network/privateLinkServices/acctestPLS-220818165005420368/ipConfigurations/primaryIpConfiguration-220818165005420368",
        "etag": "W/\"067e8de9-b44f-473a-901d-0ad58301f3cc\"",
        "type": "Microsoft.Network/privateLinkServices/ipConfigurations",
        "properties": {
          "provisioningState": "Succeeded",
          "privateIPAllocationMethod": "Dynamic",
          "subnet": {
            "id": "/subscriptions/xxxxxxxxxxxxx/resourceGroups/acctestRG-privatelink-220818165005420368/providers/Microsoft.Network/virtualNetworks/acctestvnet-220818165005420368/subnets/acctestsnetservice-220818165005420368"
          },
          "primary": true,
          "privateIPAddressVersion": "IPv4"
        }
      }
    ],
    "privateEndpointConnections": [],
    "networkInterfaces": [
      {
        "id": "/subscriptions/xxxxxxxxxxxxx/resourceGroups/acctestRG-privatelink-220818165005420368/providers/Microsoft.Network/networkInterfaces/acctestPLS-220818165005420368.nic.fd797a3e-c0fd-4464-a8cc-32a3f1e21992"
      }
    ]
  }
}

As we can see, the PrivateLink resource acctestPLS-220818165005420368 was created successfully, it's alias was acctestpls-220818165005420368.b08a765d-0bcb-41bf-a0ec-755d43737dbc.centralus.azure.privatelinkservice

Then we sent a request to create a new PrivateEndpoint using this alias:

PUT https://management.azure.com/subscriptions/xxxxxxxxxxxxx/resourceGroups/acctestRG-privatelink-220818165005420368/providers/Microsoft.Network/privateEndpoints/acctest-privatelink-220818165005420368?api-version=2021-08-01 HTTP/1.1
Host: management.azure.com
User-Agent: Go/go1.19 (amd64-windows) go-autorest/v14.2.1 Azure-SDK-For-Go/v65.0.0 network/2021-08-01 HashiCorp Terraform/1.2.6 (+https://www.terraform.io) Terraform Plugin SDK/2.10.1 terraform-provider-azurerm/dev pid-222c6c49-1b0a-5959-a213-6608f9eb8820
Content-Length: 593
Authorization: Bearer 
Content-Type: application/json; charset=utf-8
X-Ms-Authorization-Auxiliary: 
X-Ms-Correlation-Request-Id: 0c5863fa-fb25-5b75-eab0-4802ca30531e
Accept-Encoding: gzip

The body was:

{
    "location": "centralus",
    "properties": {
        "manualPrivateLinkServiceConnections": [
            {
                "name": "acctestPLS-220818165005420368",
                "properties": {
                    "groupIds": [],
                    "privateLinkServiceId": "acctestpls-220818165005420368.b08a765d-0bcb-41bf-a0ec-755d43737dbc.centralus.azure.privatelinkservice",
                    "requestMessage": "test"
                }
            }
        ],
        "privateLinkServiceConnections": [],
        "subnet": {
            "id": "/subscriptions/xxxxxxxxxxxxx/resourceGroups/acctestRG-privatelink-220818165005420368/providers/Microsoft.Network/virtualNetworks/acctestvnet-220818165005420368/subnets/acctestsnetendpoint-220818165005420368"
        }
    },
    "tags": {}
}

We used the PrivateLink's alias here, and the response was:

HTTP/1.1 400 Bad Request
Cache-Control: no-cache
Pragma: no-cache
Content-Length: 237
Content-Type: application/json; charset=utf-8
Expires: -1
x-ms-request-id: d8c3b7fa-7169-4ce5-b2ae-4ecc7629d94c
x-ms-correlation-request-id: 0c5863fa-fb25-5b75-eab0-4802ca30531e
x-ms-arm-service-request-id: 3edd9ac3-4ceb-42b1-b1d2-92ca978897df
Strict-Transport-Security: max-age=31536000; includeSubDomains
Server: Microsoft-HTTPAPI/2.0
Server: Microsoft-HTTPAPI/2.0
x-ms-ratelimit-remaining-subscription-writes: 1196
x-ms-routing-request-id: SOUTHEASTASIA: 20220818T085206Z:c744abc8-4534-4774-845a-a588a6856e90
X-Content-Type-Options: nosniff
Date: Thu,
18 Aug 2022 08: 52: 06 GMT

{
    "error": {
        "code": "InvalidPrivateLinkServiceId",
        "message": "PrivateLinkServiceId Invalid private link service id: {0} passed in private link service connection: {1} passed in input is invalid",
        "details": []
    }
}

We believe this issue is caused by some cache-related issue on service side because:

It only happens when we create multiple PrivateLinks concurrently
If we retry the creation of PrivateEndpoint, we'll get a successful creation eventually.

We can reproduce this issue by creationg multiple PrivateLinks concurrently, then create a PrivateEndpoint with PrivateLink's alias. Can anyone help us solve this issue? Thanks!

Author:	lonegunmanb
Assignees:	-
Labels:	`Service Attention`, `needs-triage`, `Network - Private Link`
Milestone:	-

…e issue Azure/azure-rest-api-specs#20289 no longer exist under API version 2023-09-01

ghost added the needs-triage Workflow: This is a new issue that needs to be triaged to the appropriate team. label Aug 18, 2022

lonegunmanb mentioned this issue Aug 18, 2022

azurerm_private_endpoint Try to add retry on creation hashicorp/terraform-provider-azurerm#16315

Merged

ankhyk added Service Attention Workflow: This issue is responsible by Azure service team. Network - Private Link labels Aug 22, 2022

ghost removed the needs-triage Workflow: This is a new issue that needs to be triaged to the appropriate team. label Aug 22, 2022

lonegunmanb mentioned this issue May 17, 2024

Support for creating multiple private endpoints simultaneously hashicorp/terraform-provider-azurerm#26005

Closed

1 task

lonegunmanb added a commit to lonegunmanb/terraform-provider-azurerm that referenced this issue May 17, 2024

remove internal lock on subnet while processing private endpoint sinc…

bbc40b0

…e issue Azure/azure-rest-api-specs#20289 no longer exist under API version 2023-09-01

lonegunmanb mentioned this issue May 17, 2024

azurerm_private_endpoint - Remove internal lock on subnet while processing private endpoint hashicorp/terraform-provider-azurerm#26006

Merged

14 tasks

lonegunmanb added a commit to lonegunmanb/terraform-provider-azurerm that referenced this issue Jun 7, 2024

remove internal lock on subnet while processing private endpoint sinc…

6220136

…e issue Azure/azure-rest-api-specs#20289 no longer exist under API version 2023-09-01

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BUG: Create PrivateEndpoint using PrivateLink's alias may fail in concurrent creation scenario #20289

BUG: Create PrivateEndpoint using PrivateLink's alias may fail in concurrent creation scenario #20289

lonegunmanb commented Aug 18, 2022 •

edited

Loading

ghost commented Aug 22, 2022

BUG: Create PrivateEndpoint using PrivateLink's alias may fail in concurrent creation scenario #20289

BUG: Create PrivateEndpoint using PrivateLink's alias may fail in concurrent creation scenario #20289

Comments

lonegunmanb commented Aug 18, 2022 • edited Loading

ghost commented Aug 22, 2022

lonegunmanb commented Aug 18, 2022 •

edited

Loading