Using Vnet Peerings across subscriptions

[Chris-Gray94]

I am not entirely sure which category this question should appear in.
We are trying to implement Vnet Peerings across multiple subscriptions. Each subscription has its own Service Principle. We have granted contributor permissions for one of the Service Principals to another subscription in which we are trying to create the Vnet Peering too. However, that Service Principal cannot find the resource group of the Vnet in which we want the Vnet Peering to peer too. This is causing us issues with us automating this process in Terraform. I have the code I believe I need for this to work, but I think we are having issues with Service Principals and permissions to get this working. The code I have works fine within the same subscription, just not across different subscriptions.

Can someone please advise?

Here is the code I have currently to do this:

resource "azurerm_virtual_network_peering" "dev-to-test" {
  name                      = "dev-to-test"
  resource_group_name       = "gl-dev-rg"
  virtual_network_name      = "gl-dev-vnet"
  remote_virtual_network_id = "/subscriptions/subscriptionid/resourceGroups/gl-test-rg/providers/Microsoft.Network/virtualNetworks/gl-test-vnet"
  allow_virtual_network_access = true
  allow_forwarded_traffic   = true
}

resource "azurerm_virtual_network_peering" "test-to-dev" {
  name                      = "test-to-dev"
  resource_group_name       = "gl-test-rg"
  virtual_network_name      = "gl-test-vnet"
  remote_virtual_network_id = "/subscriptions/subscriptionid/resourceGroups/gl-dev-rg/providers/Microsoft.Network/virtualNetworks/gl-dev-vnet"
  allow_virtual_network_access = true
  allow_forwarded_traffic   = true
}

The code above is an example of the vnet peerings I have. When I try to go to the vnet ID for the peerings I have in my code, within the portal, it goes straight to that particular vnet.

Many Thanks,
Chris

[Chris-Gray94]

@tombuildsstuff I have updated the question with config. Hope this helps

[tombuildsstuff]

hey @Chris-Gray94

Thanks for opening this issue :)

In the example above both resources are being allocated from the same version of the Provider (which in turn assumes permission over both subscriptions) - where you've got two Service Principals (one per subscription) - you need to create multiple versions of the provider with alias them (here's the Terraform documentation on that) and then alias them on the resources; for example here's an example of how to do this with the AzureRM Provider.

This would mean in your specific case the configuration would look something like the following:

provider "azurerm" {
  version = "=1.6.0"
  alias = "dev"

  client_id = "..."
  client_secret = "..."
  subscription_id = "..."
  tenant_id = "..."
}

provider "azurerm" {
  version = "=1.6.0"
  alias = "test"

  client_id = "..."
  client_secret = "..."
  subscription_id = "..."
  tenant_id = "..."
}

data "azurerm_virtual_network" "dev" {
  name = "gl-dev-vnet"
  resource_group_name = "gl-dev-rg"
  provider = "azurerm.dev"
}

data "azurerm_virtual_network" "test" {
  name = "gl-test-vnet"
  resource_group_name = "gl-test-rg"
  provider = "azurerm.test"
}

resource "azurerm_virtual_network_peering" "dev-to-test" {
  name = "dev-to-test"
  resource_group_name = "${data.azurerm_virtual_network.test.resource_group_name}"
  virtual_network_name = "${data.azurerm_virtual_network.test.name}"
  remote_virtual_network_id = "${data.azurerm_virtual_network.test.id}"
  allow_virtual_network_access = true
  allow_forwarded_traffic = true
  provider = "azurerm.dev"
}

resource "azurerm_virtual_network_peering" "test-to-dev" {
  name = "test-to-dev"
  resource_group_name = "${data.azurerm_virtual_network.dev.resource_group_name}"
  virtual_network_name = "${data.azurerm_virtual_network.dev.name}"
  remote_virtual_network_id = "${data.azurerm_virtual_network.dev.id}"
  allow_virtual_network_access = true
  allow_forwarded_traffic = true
  provider = "azurerm.test"
}

Would you be able to take a look and see if that solves your issue?

Thanks!

[Chris-Gray94]

Hi @tombuildsstuff,

Thank you for your response and suggestion. However, this doesn't seem to have worked.
We have our Virtual Network etc in a module. I have defined the provider at the module level, but it is saying it does not have access to the remote virtual network because of its service principle.

I had to define the provider where the module is being called, as that was an error I was getting.

Here is an example of the code I have tried:

provider "azurerm" {
version = "=1.6.0"
alias = "dev"

client_id = "..."
client_secret = "..."
subscription_id = "..."
tenant_id = "..."
}

provider "azurerm" {
version = "=1.6.0"
alias = "test"

client_id = "..."
client_secret = "..."
subscription_id = "..."
tenant_id = "..."
}

module "virtual_network" {
source = "../../modules/virtual-network"
providers = {
azurerm = "azurerm.dev"
}

Within the module I have the following code:

resource "azurerm_virtual_network_peering" "dev-to-test" {
name = "dev-test"
resource_group_name = "${var.dev_rg_name}"
virtual_network_name = "${var.dev_vnet_name}"
remote_virtual_network_id = "${var.test_vnet_id}"
allow_virtual_network_access = true
allow_forwarded_traffic = true
}

resource "azurerm_virtual_network_peering" test-to-dev" {
name = "test-dev"
resource_group_name = "${var.test_rg_name}"
virtual_network_name = "${var.test_vnet_name}"
remote_virtual_network_id = "${var.dev_vnet_id}"
allow_virtual_network_access = true
allow_forwarded_traffic = true
}

[webbj62]

Hi @Chris-Gray94

I'm using Azure Provider version 1.6.0 and I used the code below to create Virtual Network Peering across two Azure subscriptions without any issues. I hope the code below works for you situation.

provider "azurerm" {
alias = "sub1"
subscription_id = "${var.subscription1_id}"
tenant_id = "${var.tenant_id}"
client_id = "${var.client_id}"
client_secret = "${var.client_secret}"
}

provider "azurerm" {
alias = "sub2"
subscription_id = "${var.subscription2_id}"
tenant_id = "${var.tenant_id}"
client_id = "${var.client_id}"
client_secret = "${var.client_secret}"
}

data "azurerm_resource_group" "rg1" {
provider = "azurerm.sub1"
name = "${var.resource_group_name1}"
}

data "azurerm_resource_group" "rg2" {
provider = "azurerm.sub2"
name = "${var.resource_group_name2}"
}

data "azurerm_virtual_network" "vnet1" {
provider = "azurerm.sub1"
name = "${var.virtual_network_name1}"
resource_group_name = "${data.azurerm_resource_group.rg1.name}"
}

data "azurerm_virtual_network" "vnet2" {
provider = "azurerm.sub2"
name = "${var.virtual_network_name2}"
resource_group_name = "${data.azurerm_resource_group.rg2.name}"
}

resource "azurerm_virtual_network_peering" "vnet-peer-1" {
provider = "azurerm.sub1"
name = "vnet1-vnet2"
resource_group_name = "${data.azurerm_resource_group.rg1.name}"
virtual_network_name = "${data.azurerm_virtual_network.vnet1.name}"
remote_virtual_network_id = "${data.azurerm_virtual_network.vnet2.id}"
allow_virtual_network_access = "true"
allow_forwarded_traffic = "true"
}

resource "azurerm_virtual_network_peering" "vnet-peer-2" {
provider = "azurerm.sub2"
name = "vnet2-vnet1"
resource_group_name = "${data.azurerm_resource_group.rg2.name}"
virtual_network_name = "${data.azurerm_virtual_network.vnet2.name}"
remote_virtual_network_id = "${data.azurerm_virtual_network.vnet1.id}"
allow_virtual_network_access = "true"
allow_forwarded_traffic = "true"
}

[Chris-Gray94]

@webbj62 Is there any reason why you are creating the Vnets and Resource Groups using data instead of resource?

[webbj62]

@Chris-Gray94 I'm using data because the resources were already created in our environment prior to executing the script that I provided as an example. You should be able to create the resources at the same time. I would include "depends_on" in the "azurerm_virtual_network_peering" settings to ensure the virtual networks are created first. (Ex. depends_on["azurerm_virtual_network.vnet1", "azurerm_virtual_network.vnet2"] )

[tombuildsstuff]

@Chris-Gray94 in the module you've posted above the same service principal (dev) is being used to create the Pairing between Dev and Test; is that intentional?

[cripth]

i am having the same issue here.
@webbj62 can you confirm that you are using same clientID for both subscriptions ?

[Chris-Gray94]

@tombuildsstuff Hi Tom, I left them both as Dev because Dev would have permissions in Test sub too. When I have tried creating a peering across to a subscription, I continually hit an issue where it does not have any permission over the other subscription. This happens regardless of whether I try the way you have suggested or not.

[webbj62]

@cripth Yes - I using the same clientID for both subscriptions because the principal is in the same Azure Active Directory. I assuming if you are using two different service principals in two different AAD accounts, then you would have pass different set service credentials. If they are in the same AAD account, make sure your service principal has contributor role access to both subscriptions.

[cerocool1203]

Hi all, I have come to same issue, even if I try using 2 providers it does not work. It comes with the Az login to basically re-login. Not sure if anyone has the same issue or if anyone has got it working. My SPN has contributor to both Subs but not sure if you can login in parallel as per providers.

[cripth]

@cerocool1203, have you follow sample given by @webbj62.

It should be straight forward thing, and can you post your code here

[cerocool1203]

@cripth , It did work as expected. However I had to create a new state file as when I tried to use the one I had and it did not let me use the 2nd provider. However, it worked like a charm!!!. Thanks again and kudos to @webbj62 for the solution :)

[tombuildsstuff]

👋

I'm going to close this question since it appears to have been resolved - if you're still seeing issues with this, as mentioned by @webbj62 in this comment please ensure the Service Principal you're using either has permissions to both Subscriptions; or that a different Service Principal is used for each Provider block (with the associated permissions) - as shown below:

provider "azurerm" {
  version         = "=1.16.0"
  tenant_id       = "aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa"
  subscription_id = "aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa"
  client_id       = "aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa"
  client_secret   = "aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa"
  alias           = "dev"
}

provider "azurerm" {
  version         = "=1.16.0"
  tenant_id       = "bbbbbbbb-bbbb-bbbb-bbbb-bbbbbbbbbbbb"
  subscription_id = "bbbbbbbb-bbbb-bbbb-bbbb-bbbbbbbbbbbb"
  client_id       = "bbbbbbbb-bbbb-bbbb-bbbb-bbbbbbbbbbbb"
  client_secret   = "bbbbbbbb-bbbb-bbbb-bbbb-bbbbbbbbbbbb"
  alias           = "test"
}

data "azurerm_virtual_network" "dev" {
  name                = "dev-network"
  resource_group_name = "dev-network-rg"
  provider            = "azurerm.dev"
}

data "azurerm_virtual_network" "test" {
  name                = "test-network"
  resource_group_name = "test-network-rg"
  provider            = "azurerm.test"
}

resource "azurerm_virtual_network_peering" "dev-to-test" {
  name                         = "dev-to-test"
  resource_group_name          = "${data.azurerm_virtual_network.test.resource_group_name}"
  virtual_network_name         = "${data.azurerm_virtual_network.test.name}"
  remote_virtual_network_id    = "${data.azurerm_virtual_network.test.id}"
  allow_virtual_network_access = true
  allow_forwarded_traffic      = true
  provider                     = "azurerm.dev"
}

resource "azurerm_virtual_network_peering" "test-to-dev" {
  name                         = "test-to-dev"
  resource_group_name          = "${data.azurerm_virtual_network.dev.resource_group_name}"
  virtual_network_name         = "${data.azurerm_virtual_network.dev.name}"
  remote_virtual_network_id    = "${data.azurerm_virtual_network.dev.id}"
  allow_virtual_network_access = true
  allow_forwarded_traffic      = true
  provider                     = "azurerm.test"
}

Thanks!

[nycjay01]

@tombuildsstuff this worked for me a few weeks back (you helped me out at MSFT community gardening day). We recently set up a new tenant with new subscriptions and I cant get the code above to work. I went and created a new role:
az ad sp create-for-rbac --role="Contributor" --scopes="/subscriptions/aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa

and
az ad sp create-for-rbac --role="Contributor" --scopes="/subscriptionsbbbbbbbb-bbbb-bbbb-bbbb-bbbbbbbbbbbb"

and copied down the app IDs and added it to the provider block how you have it above and I am getting the error below:

My plan is to build:
RSG
Vnet
subnet
nsg
peer1
peer2

Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.


data.azurerm_virtual_network.subscription2: Refreshing state...

Error: Error refreshing state: 1 error(s) occurred:

* provider.azurerm.subscription1: Unable to list provider registration status, it is possible that this is due to invalid credentials or the service principal does not have permission to use the Resource Manager API, Azure error: azure.BearerAuthorizer#WithAuthorization: Failed to refresh the Token for request to https://management.azure.com/subscriptions/aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa/providers?api-version=2017-05-10: StatusCode=400 -- Original Error: adal: Refresh request failed. Status Code = '400'. Response body: {"error":"invalid_request","error_description":"AADSTS90002: Tenant 63c899bd-49d9-42d2-8bd1-19c20b525bdnot found. This may happen if there are no active subscriptions for the tenant. Check with your subscription administrator.\r\nTrace ID: ad444f3d-17ac-4000-b024-c8aa9dfe8b00\r\nCorrelation ID: f62d0da7-5f70-4c87-966f-1b63cc7bc22f\r\nTimestamp: 2018-10-12 19:48:36Z","error_codes":[90002],"timestamp":"2018-10-12 19:48:36Z","trace_id":"ad444f3d-17ac-4000-b024-c8aa9dfe8b00","correlation_id":"f62d0da7-5f70-4c87-966f-1b63cc7bc22f"}

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 hashibot-feedback@hashicorp.com. Thanks!