Add support for customer provided encryption keys for disks.

docs/advanced/managed-disks/README.md

@@ -34,15 +34,11 @@ Below are behavior changes with a new deployment:
 
 ### Before the Migration
 
-You should **NOT** migrate the deployment if any of the following conditions is true. You should leave `use_managed_disks` as `false` in the manifest file in this case.
-
-  * The region does not support managed disks. You can see [Azure Products by Region](https://azure.microsoft.com/en-us/regions/services/) for the availability of the Managed Disks feature.
-
 You need to review the following checklist to prevent predictable migration failures.
 
   * The default storage account is used to store stemcells uploaded by CPI. In CPI v20 or older, it's specified by `azure.storage_account_name` in the global configurations. In CPI v20+, this property is optional. However, in the migration scenario, please make sure the default storage account is specified by `azure.storage_account_name` in the global configurations. Otherwise, CPI won't find your default storage account, which causes that all the uploaded stemcells can't be re-used.
 
-  * The maximum number of fault domains of managed availability sets varies by region - either two or three managed disk fault domains per region. The [table](https://docs.microsoft.com/en-us/azure/virtual-machines/windows/manage-availability#use-managed-disks-for-vms-in-an-availability-set) shows the number per region. If your existing deploymeng is using 3 fault domains, you need to check whether the region supports 3 managed disk fault domains. Please see details [here](#with-availability-sets).
+  * As of CPI v52.0.0, the maximum number of fault domains of managed availability sets will be configured to the maximum fault domains supported by the region or the currently configured value for the availability set, whichever is lower. CPI versions before this will fail if the region does not support the number of fault domains currently configured on the availability set and you will first need to manually adjust the fault domain count in Azure.
 
   * Unmanaged snapshots cannot be migrated to managed version, and it may cause migration failure, so you need to delete all snapshots and disable snapshots in `bosh.yml` before the migration, if you enabled snapshots in the existing deployment. You can enable snapshots after full migration if you want.
 
@@ -134,14 +130,7 @@ Delete all storage accounts **without** below tags in the resource group. Please
 
 Only managed availability set can host VMs with managed disks. However, [the maximum number of fault domains of managed availability sets varies by region - either two or three managed disk fault domains per region.](https://docs.microsoft.com/en-us/azure/virtual-machines/virtual-machines-windows-manage-availability#configure-multiple-virtual-machines-in-an-availability-set-for-redundancy)
 
-By default, CPI will migrate the old unmanaged availability sets into managed availability sets automatically, and during the migration, the fault domain number can't be changed. However due to the deployment schedule, the new managed availability set may not support the same maximum number of fault domain as the old unmanaged availability set; in this case, the migration might be blocked. You need to wait till the region supports the same maximum number of fault domains.
-
-Let's assume that the fault domain number is set to `3` in your existing deployment.
-
-* For the regions which support 3 FDs, the migration will succeed.
-* For the regions which only support 2 FDs
-  * If the existing deployment doesn't use load balancer, the migration will secceed. You need to specify a new availability set name in `resource_pools`. Then CPI will create new VMs with managed disks in new availability sets (managed) one by one. After migration, you can delete the old unmanaged availability sets manually.
-  * If the existing deployment is using load balancer, the migration will fail because the VMs behind a load balancer have to be in a same availability set. This prevents CPI creating VMs in the new availability set one by one. You should not use managed disks feature until the region supports 3 FDs.
+As of CPI version v52.0.0, if the availability set is configured with a higher fault domains count than the region supports, the availability set will be migrated to match the lower number supported by the region.
 
 #### The default storage account's location is different from the resource group location
 

src/bosh_azure_cpi/lib/cloud/azure/cloud.rb

@@ -358,9 +358,10 @@ def create_disk(size, cloud_properties, vm_cid = nil)
             caching = cloud_properties.fetch('caching', 'None')
             iops = cloud_properties.fetch('iops', nil)
             mbps = cloud_properties.fetch('mbps', nil)
+            disk_encryption_set_name = cloud_properties.fetch('disk_encryption_set_name', nil)
             validate_disk_caching(caching)
             disk_id = DiskId.create(caching, true, resource_group_name: resource_group_name)
-            @disk_manager2.create_disk(disk_id, location, size / 1024, storage_account_type, zone, iops, mbps)
+            @disk_manager2.create_disk(disk_id, location, size / 1024, storage_account_type, zone, iops, mbps, disk_encryption_set_name: disk_encryption_set_name)
           else
             storage_account_name = _azure_config.storage_account_name
             caching = cloud_properties.fetch('caching', 'None')

src/bosh_azure_cpi/lib/cloud/azure/disk/disk_manager2.rb

@@ -21,7 +21,7 @@ def initialize(azure_client)
     # @param [String] zone                   Zone number in string. Possible values: "1", "2" or "3".
     #
     # @return [void]
-    def create_disk(disk_id, location, size, storage_account_type, zone = nil, iops = nil, mbps = nil)
+    def create_disk(disk_id, location, size, storage_account_type, zone = nil, iops = nil, mbps = nil, disk_encryption_set_name: nil)
       @logger.info("create_disk(#{disk_id}, #{location}, #{size}, #{storage_account_type}, #{zone})")
       resource_group_name = disk_id.resource_group_name
       disk_name = disk_id.disk_name
@@ -34,7 +34,8 @@ def create_disk(disk_id, location, size, storage_account_type, zone = nil, iops
         location: location,
         tags: tags,
         disk_size: size,
-        account_type: storage_account_type
+        account_type: storage_account_type,
+        disk_encryption_set_name: disk_encryption_set_name,
       }
 
       disk_params[:zone] = zone unless zone.nil?
@@ -185,19 +186,20 @@ def os_disk_placement(placement)
       end
     end
 
-    def os_disk(vm_name, stemcell_info, size, caching, use_root_disk_as_ephemeral)
+    def os_disk(vm_name, stemcell_info, size, caching, use_root_disk_as_ephemeral, disk_encryption_set_name: nil)
       validate_disk_caching(caching)
 
       disk_size = get_os_disk_size(size, stemcell_info, use_root_disk_as_ephemeral)
 
       {
         disk_name: generate_os_disk_name(vm_name),
         disk_size: disk_size,
-        disk_caching: caching
+        disk_caching: caching,
+        disk_encryption_set_name: disk_encryption_set_name
       }
     end
 
-    def ephemeral_os_disk(vm_name, stemcell_info, root_disk_size, ephemeral_disk_size, use_root_disk_as_ephemeral, placement)
+    def ephemeral_os_disk(vm_name, stemcell_info, root_disk_size, ephemeral_disk_size, use_root_disk_as_ephemeral, placement, disk_encryption_set_name: nil)
       disk_size = if use_root_disk_as_ephemeral && !ephemeral_disk_size.nil? && root_disk_size.nil?
                     # when no size was specified at the root disk, we have to use the default stemcell image size based on the os type. For linux we will use 3g and 128gb for windows.
                     stemcell_info.image_size / 1024
@@ -213,11 +215,12 @@ def ephemeral_os_disk(vm_name, stemcell_info, root_disk_size, ephemeral_disk_siz
         disk_name: generate_os_disk_name(vm_name),
         disk_size: disk_size,
         disk_caching: 'ReadOnly',
-        disk_placement: disk_placement
+        disk_placement: disk_placement,
+        disk_encryption_set_name: disk_encryption_set_name
       }
     end
 
-    def ephemeral_disk(vm_name, instance_type, size, type, use_root_disk_as_ephemeral, caching, iops, mbps)
+    def ephemeral_disk(vm_name, instance_type, size, type, use_root_disk_as_ephemeral, caching, iops, mbps, disk_encryption_set_name: nil)
       return nil if use_root_disk_as_ephemeral
 
       disk_info = DiskInfo.for(instance_type)
@@ -235,7 +238,8 @@ def ephemeral_disk(vm_name, instance_type, size, type, use_root_disk_as_ephemera
         disk_caching: caching,
         disk_type: type,
         iops: iops,
-        mbps: mbps
+        mbps: mbps,
+        disk_encryption_set_name: disk_encryption_set_name
       }
     end
 

src/bosh_azure_cpi/lib/cloud/azure/models/ephemeral_disk.rb

@@ -2,15 +2,16 @@
 
 module Bosh::AzureCloud
   class EphemeralDisk
-    attr_reader :use_root_disk, :size, :type, :caching, :iops, :mbps
+    attr_reader :use_root_disk, :size, :type, :caching, :iops, :mbps, :disk_encryption_set_name
 
-    def initialize(use_root_disk, size, type, caching, iops, mbps)
+    def initialize(use_root_disk, size, type, caching, iops, mbps, disk_encryption_set_name: nil)
       @use_root_disk = use_root_disk
       @size = size
       @type = type
       @caching = caching
       @iops = iops
       @mbps = mbps
+      @disk_encryption_set_name = disk_encryption_set_name
     end
   end
 end

src/bosh_azure_cpi/lib/cloud/azure/models/root_disk.rb

@@ -2,12 +2,13 @@
 
 module Bosh::AzureCloud
   class RootDisk
-    attr_reader :size, :type, :placement
+    attr_reader :size, :type, :placement, :disk_encryption_set_name
 
-    def initialize(size, type, placement)
+    def initialize(size, type, placement, disk_encryption_set_name: nil)
       @size = size
       @type = type
       @placement = placement
+      @disk_encryption_set_name = disk_encryption_set_name
     end
   end
 end

src/bosh_azure_cpi/lib/cloud/azure/models/vm_cloud_props.rb

@@ -44,7 +44,8 @@ def initialize(vm_properties, global_azure_config)
       @root_disk = Bosh::AzureCloud::RootDisk.new(
         root_disk_hash['size'],
         root_disk_hash['type'],
-        _default_root_disk_placement(root_disk_hash['placement'])
+        _default_root_disk_placement(root_disk_hash['placement']),
+        disk_encryption_set_name: root_disk_hash['disk_encryption_set_name']
       )
 
       cloud_error("Only one of 'type' and 'placement' is allowed to be configured for the root_disk when 'placement' is not set to persistent") if @root_disk.placement != 'remote' && !@root_disk.type.nil? && !global_azure_config.use_managed_disks
@@ -56,7 +57,8 @@ def initialize(vm_properties, global_azure_config)
         ephemeral_disk_hash['type'],
         ephemeral_disk_hash['caching'],
         ephemeral_disk_hash['iops'],
-        ephemeral_disk_hash['mbps']
+        ephemeral_disk_hash['mbps'],
+        disk_encryption_set_name: ephemeral_disk_hash['disk_encryption_set_name']
       )
 
       @caching = vm_properties.fetch('caching', 'ReadWrite')

src/bosh_azure_cpi/lib/cloud/azure/restapi/azure_client.rb

@@ -83,6 +83,7 @@ class AzureClient
     REST_API_VIRTUAL_MACHINES            = 'virtualMachines'
     REST_API_AVAILABILITY_SETS           = 'availabilitySets'
     REST_API_DISKS                       = 'disks'
+    REST_API_DISK_ENCRYPTION_SETS        = 'diskEncryptionSets'
     REST_API_IMAGES                      = 'images'
     REST_API_SNAPSHOTS                   = 'snapshots'
     REST_API_VM_IMAGE                    = 'vmimage'
@@ -254,16 +255,18 @@ def get_resource_group(resource_group_name)
     # * +:managed+              - Boolean. Needs to be true to create managed disk VMs. Default value is nil.
     #
     #   When managed is true, below parameters are required
-    # * +:image_id+             - String. The id of the image to create the virtual machine.
-    # * +:os_disk+              - Hash. OS Disk for the virtual machine instance.
-    # *   +:disk_name+          - String. The name of the OS disk.
-    # *   +:disk_caching+       - String. The caching option of the OS disk. Possible values: None, ReadOnly or ReadWrite.
-    # *   +:disk_size+          - Integer. The size in GiB of the OS disk. It could be nil.
-    # * +:ephemeral_disk+       - Hash. Ephemeral Disk for the virtual machine instance. It could be nil.
-    # *   +:disk_name+          - String. The name of the ephemeral disk.
-    # *   +:disk_caching+       - String. The caching option of the ephemeral disk. Possible values: None, ReadOnly or ReadWrite.
-    # *   +:disk_size+          - Integer. The size in GiB of the ephemeral disk.
-    # *   +:disk_type+          - String. The disk type of the ephemeral disk.
+    # * +:image_id+                   - String. The id of the image to create the virtual machine.
+    # * +:os_disk+                    - Hash. OS Disk for the virtual machine instance.
+    # *   +:disk_name+                - String. The name of the OS disk.
+    # *   +:disk_caching+             - String. The caching option of the OS disk. Possible values: None, ReadOnly or ReadWrite.
+    # *   +:disk_size+                - Integer. The size in GiB of the OS disk. It could be nil.
+    # *   +:disk_encryption_set_name+ - String. If specified, encrypted the os_disk with the customer provided encryption key used in the provided disk encryption set.
+    # * +:ephemeral_disk+             - Hash. Ephemeral Disk for the virtual machine instance. It could be nil.
+    # *   +:disk_name+                - String. The name of the ephemeral disk.
+    # *   +:disk_caching+             - String. The caching option of the ephemeral disk. Possible values: None, ReadOnly or ReadWrite.
+    # *   +:disk_size+                - Integer. The size in GiB of the ephemeral disk.
+    # *   +:disk_type+                - String. The disk type of the ephemeral disk.
+    # *   +:disk_encryption_set_name+ - String. If specified, encrypted the ephemeral_disk with the customer provided encryption key used in the provided disk encryption set.
     #
     #   When managed is true and root_disk type is not 'remote' below parameters are required
     # * +:image_id+             - String. The id of the image to create the virtual machine.
@@ -357,6 +360,10 @@ def create_virtual_machine(resource_group_name, vm_params, network_interfaces, a
           'caching' => vm_params[:os_disk][:disk_caching]
         }
         os_disk['diskSizeGB'] = vm_params[:os_disk][:disk_size] unless vm_params[:os_disk][:disk_size].nil?
+        if vm_params[:os_disk][:disk_encryption_set_name]
+          disk_encryption_set_id = rest_api_url(REST_API_PROVIDER_COMPUTE, REST_API_DISK_ENCRYPTION_SETS, name: vm_params[:os_disk][:disk_encryption_set_name])
+          os_disk['managedDisk'] = { 'diskEncryptionSet' => { 'id' => disk_encryption_set_id } }
+        end
       end
 
       unless vm_params[:ephemeral_os_disk].nil?
@@ -370,6 +377,10 @@ def create_virtual_machine(resource_group_name, vm_params, network_interfaces, a
           'name' => vm_params[:ephemeral_os_disk][:disk_name]
         }
         os_disk['diskSizeGB'] = vm_params[:ephemeral_os_disk][:disk_size] unless vm_params[:ephemeral_os_disk][:disk_size].nil?
+        if vm_params[:ephemeral_os_disk][:disk_encryption_set_name]
+          disk_encryption_set_id = rest_api_url(REST_API_PROVIDER_COMPUTE, REST_API_DISK_ENCRYPTION_SETS, name: vm_params[:ephemeral_os_disk][:disk_encryption_set_name])
+          os_disk['managedDisk'] = { 'diskEncryptionSet' => { 'id' => disk_encryption_set_id } }
+        end
       end
 
       if vm_params[:image_reference].nil?
@@ -423,10 +434,13 @@ def create_virtual_machine(resource_group_name, vm_params, network_interfaces, a
           'caching' => vm_params[:ephemeral_disk][:disk_caching]
         }]
         if vm_params[:managed]
+          managed_disk = vm['properties']['storageProfile']['dataDisks'][0]['managedDisk'] = {}
           if vm_params[:ephemeral_disk][:disk_type]
-            vm['properties']['storageProfile']['dataDisks'][0]['managedDisk'] = {
-              'storageAccountType' => vm_params[:ephemeral_disk][:disk_type]
-            }
+            managed_disk['storageAccountType'] = vm_params[:ephemeral_disk][:disk_type]
+          end
+          if vm_params[:ephemeral_disk][:disk_encryption_set_name]
+            disk_encryption_set_id = rest_api_url(REST_API_PROVIDER_COMPUTE, REST_API_DISK_ENCRYPTION_SETS, name: vm_params[:ephemeral_disk][:disk_encryption_set_name])
+            managed_disk['diskEncryptionSet'] = {'id' => disk_encryption_set_id}
           end
         else
           vm['properties']['storageProfile']['dataDisks'][0]['vhd'] = {
@@ -897,6 +911,7 @@ def delete_availability_set(resource_group_name, name)
     # * +:disk_size+                    - Integer. Specifies the size in GB of the empty managed disk.
     # * +:account_type+                 - String. Specifies the account type of the empty managed disk.
     #                                     Optional values: Standard_LRS, StandardSSD_LRS, Premium_LRS.
+    # * +:disk_encryption_set_name+     - String. If specified, encrypted the disk with the customer provided encryption key used in the provided disk encryption set.
     # When disk is in a zone
     # * +:zone+                         - String. Zone number in string.
     #
@@ -921,6 +936,13 @@ def create_empty_managed_disk(resource_group_name, params)
       disk['zones'] = [params[:zone]] unless params[:zone].nil?
       disk['properties']['diskIOPSReadWrite'] = params[:iops] unless params[:iops].nil?
       disk['properties']['diskMBpsReadWrite'] = params[:mbps] unless params[:mbps].nil?
+      if params[:disk_encryption_set_name]
+        disk_encryption_set_id = rest_api_url(REST_API_PROVIDER_COMPUTE, REST_API_DISK_ENCRYPTION_SETS, name: params[:disk_encryption_set_name])
+        disk['properties']['encryption'] = {
+          'diskEncryptionSetId' => disk_encryption_set_id,
+          'type' => 'EncryptionAtRestWithCustomerKey'
+        }
+      end
       http_put(url, disk)
     end
 

src/bosh_azure_cpi/lib/cloud/azure/vms/vm_manager_disk.rb

@@ -8,14 +8,14 @@ def _build_disks(instance_id, stemcell_info, vm_props)
       if @use_managed_disks
         ephemeral_disk = @disk_manager2.ephemeral_disk(instance_id.vm_name, vm_props.instance_type, vm_props.ephemeral_disk.size,
                                                        vm_props.ephemeral_disk.type, vm_props.ephemeral_disk.use_root_disk, vm_props.ephemeral_disk.caching,
-                                                       vm_props.ephemeral_disk.iops, vm_props.ephemeral_disk.mbps)
+                                                       vm_props.ephemeral_disk.iops, vm_props.ephemeral_disk.mbps, disk_encryption_set_name: vm_props.ephemeral_disk.disk_encryption_set_name)
 
         # check if disk has the default behavior
         if vm_props.root_disk.placement == 'remote'
-          os_disk = @disk_manager2.os_disk(instance_id.vm_name, stemcell_info, vm_props.root_disk.size, vm_props.caching, vm_props.ephemeral_disk.use_root_disk)
+          os_disk = @disk_manager2.os_disk(instance_id.vm_name, stemcell_info, vm_props.root_disk.size, vm_props.caching, vm_props.ephemeral_disk.use_root_disk, disk_encryption_set_name: vm_props.root_disk.disk_encryption_set_name)
         else
           ephemeral_os_disk = @disk_manager2.ephemeral_os_disk(instance_id.vm_name, stemcell_info, vm_props.root_disk.size, vm_props.ephemeral_disk.size,
-                                                               vm_props.ephemeral_disk.use_root_disk, vm_props.root_disk.placement)
+                                                               vm_props.ephemeral_disk.use_root_disk, vm_props.root_disk.placement, disk_encryption_set_name: vm_props.root_disk.disk_encryption_set_name)
         end
       else
         storage_account_name = instance_id.storage_account_name