Help needed to understand Copy session Target device

[mukeshguptakc]

Hello,
We are using PyU4V= 9.1.3.0. with PowerMax.

Steps followed:

1. Create snapshot of a storage group with volumes.
2. Create and link the new storage group with the snapshot created.
3. Create masking view id and attach it to the host.

Now, while we were testing the above code in at scale, we observed that after the very first create and link of the storage with snapshot is completed.
we observe the following error traceback:

ERROR - Request failed unexpectedly" "Traceback (most recent call last):" " File ""/tmp/libs/powermax/lib/PyU4V/replication.py"", line 422, in modify_storage_group_snapshot" " return self.modify_resource(" " File ""/tmp/libs/powermax/lib/PyU4V/common.py"", line 409, in modify_resource" " self.check_status_code_success(operation, status_code, message)" " File ""/tmp/libs/powermax/lib/PyU4V/common.py"", line 181, in check_status_code_success" " raise exception.VolumeBackendAPIException(" "PyU4V.utils.exception.VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error PUT symmetrix resource. The status code received is 500 and the message is {'message': 'A problem occurred modifying the snapshot generation resource: The Target device cannot be manipulated because it is already a Copy session Target device'}."

So, the error message : A problem occurred modifying the snapshot generation resource: The Target device cannot be manipulated because it is already a Copy session Target device'

This occurs as soon as the first storage group is created and linked.

1. can you please point out to docs regarding different session and how to leverage it.
2. since the operation was in scale we maintained one session, do we need to terminate each session post every call.
3. Do we need to specify any wait or interval between operations any docuementations on that would also be helpful.
4. And lastly any scale numbers that are recommended for scale operations.

Anything else apart from the above that you want to suggest would be really helpful.

[rawstorage]

Hi, The problem appears to be that you are attempting to link a snapshot to a volume that is alrrady linked to a previous generation. In this case you should unlink or relink. You should be able to link again after unlink without any wait period. The operations for snapvx operations are called out in the Solutions enabler documents. This is not an issue with scale simply the order in which you are doing things. The following document should also help. One thing to consider is that the latest code release uses snapset id to identify snapshots rather than generation, this is much easier to work with at scale as the snapid doesn't change if another snapshot with the same name is created. Details are also in the tech note https://www.delltechnologies.com/resources/en-us/asset/white-papers/products/storage/h13697-dell-emc-powermax-vmax-all-flash-timefinder-snapvx-local-replication.pdf Hope this helps. Get Outlook for Android<https://aka.ms/ghei36>

…

________________________________ From: Mukesh Gupta <notifications@github.com> Sent: Monday, December 7, 2020 6:17:03 PM To: dell/PyU4V <PyU4V@noreply.github.com> Cc: Subscribed <subscribed@noreply.github.com> Subject: [dell/PyU4V] Help needed to understand Copy session Target device (#107) [EXTERNAL EMAIL] Hello, We are using PyU4V= 9.1.3.0. with PowerMax. Steps followed: 1. Create snapshot of a storage group with volumes. 2. Create and link the new storage group with the snapshot created. 3. Create masking view id and attach it to the host. Now, while we were testing the above code in at scale, we observed that after the very first create and link of the storage with snapshot is completed. we observe the following error traceback: ERROR - Request failed unexpectedly" "Traceback (most recent call last):" " File ""/tmp/libs/powermax/lib/PyU4V/replication.py"", line 422, in modify_storage_group_snapshot" " return self.modify_resource(" " File ""/tmp/libs/powermax/lib/PyU4V/common.py"", line 409, in modify_resource" " self.check_status_code_success(operation, status_code, message)" " File ""/tmp/libs/powermax/lib/PyU4V/common.py"", line 181, in check_status_code_success" " raise exception.VolumeBackendAPIException(" "PyU4V.utils.exception.VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error PUT symmetrix resource. The status code received is 500 and the message is {'message': 'A problem occurred modifying the snapshot generation resource: The Target device cannot be manipulated because it is already a Copy session Target device'}." So, the error message : A problem occurred modifying the snapshot generation resource: The Target device cannot be manipulated because it is already a Copy session Target device' This occurs as soon as the first storage group is created and linked. 1. can you please point out to docs regarding different session and how to leverage it. 2. since the operation was in scale we maintained one session, do we need to terminate each session post every call. 3. Do we need to specify any wait or interval between operations any docuementations on that would also be helpful. 4. And lastly any scale numbers that are recommended for scale operations. Anything else apart from the above that you want to suggest would be really helpful. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub<https://github.com/dell/PyU4V/issues/107>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AF7FWPJEXTATPNY3PP4K6VLSTULZ7ANCNFSM4UQ2QEFQ>.

[mukeshguptakc]

Hi Paul,

Thanks, for pointing it out, in our configuration we had Parent and Child Storage groups created and Once the first child storage group snapshot is linked to another storage group the rest child storage group snapshot linking to other storage group were failing with the above error.

I updated the config and have created standalone Storage group for scale and now when we try to create and link a storage group from a snapshot, we observed the following traceback:

raise exception.VolumeBackendAPIException("
"PyU4V.utils.exception.VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error PUT symmetrix resource. The status code received is 500 and the message is {'message': 'A problem occurred modifying the snapshot generation resource: Cannot select enough Target devices to perform this operation'}." 

This occurs intermittently post 20-25 storage groups are linked to snapshots and the rest create and linking storage group snapshots fails.

API used:

self.replication.modify_storage_group_snapshot(
                src_storage_grp_id=storage_group, tgt_storage_grp_id=
                target_storage_group, snap_name=snapshot_name,
                gen_num=gen_num, link=True

[rawstorage]

The error seems pretty clear to me. The failure is due to the fact that you don't have enough devices in the target storage group. You need to have enough devices to link all the snaps to, the devices need to be the same size or larger.

First time you do this operation if the target storage group doesn't exist the API will create one with the correct number of devices and size to match the source storage group. If you modify the number of the devices in the source storage group by adding more then you will need to remember to update any linked target storage groups with more devices also.

[mukeshguptakc]

Hi Paul,

Yes, agreed the failure seems to be of not having enough devices in the target storage group.

Here, we are trying to create a new storage group from the snapshot and with each operation and not add devices to an existing target storage group.

Based, on the earlier tests what we observed is the target storage group is getting created but the linking with snapshot is failing with the error (Cannot select enough Target devices to perform this operation).

So, with each operation a new target storage group is created and linking fails.

Steps, we followed:

1. Create 50 storage with devices/disks, no previous snapshots maintained.
2. Data being updated on all the mounted volumes.
3. Trigger Snapshot operation for all 50 Storage group, -> Which succeeds.
4. create a new target storage group out of the snapshot and link it for all 50 storage groups.
   --> In which we are currently observing exception traceback mentioned above.
5. We restore the snapshots. -> which succeeds.
6. we remove mapping of the target storage group, created in step 4 and do unlink and clean-up.
7. Finally, we delete the snapshot created.

Here's the snippet of creating new target storage group and mapping it to host.

def export_to_host(self, storage_group, target_storage_group,
                   snapshot_name, gen_num, initiator_info):
   """
   The method is sub-divided into 4 parts.
   1. Create a storage group and link it snapshot.
   2. Fetch Host id from the array.
   3. Fetch Port Group id from the array.
   4. Create Masking view group based on storage group, host_id, port-group_id
   """

   def create_and_link_storage_group():
       try:
           export_response = self.replication.modify_storage_group_snapshot(
               src_storage_grp_id=storage_group, tgt_storage_grp_id=
               target_storage_group, snap_name=snapshot_name,
               gen_num=gen_num, link=True
           )
           return export_response
       except exception.PyU4VException as err:
           raise Exception(err)

   def get_host_id():
       initiator_list = []
       # Fetch Host id and related Array Initiators
       # Return the same
       return host_id, initiator_list

   def get_port_group_id():
       # based on initiator list fetch port_group_id
       return port_group_id

   def create_masking_view():
       """
       Create masking view and attach storage group to provide host_id
       """
       masking_view_id = "{}_{}".format("VG", str(target_storage_group))
       try:
           attach_response = self.attach_storage_group(
               port_group_id, masking_view_id, target_storage_group, host_id)
           return attach_response
       except exception.PyU4VException as err:
           raise Exception(err)
   # Create and link target storage group
   create_response = create_and_link_storage_group()
   # Get Host_id to attach the target storage group
   host_id, initiator_list = get_host_id()
   # Get Port_group_id to attach the target storage group
   port_group_id = get_port_group_id()
   # Create a masking view to attach the target storage group.
   attach_response = create_masking_view()
   return create_response, attach_response

And also we are getting our Unipshere upgraded to 9.2 to leverage the latest PyU4V, while browsing through the SDK i could find references to using of generation number for deleting a snapshot.
I did find references to deleting a snapshot by (name and gen_num) and also deleting a snapshot by (name and snapid).

Please, if you could also correct my understanding of using snapsetId(in documentation) is similar to snapid(in PyU4V SDK).

[rawstorage]

Manoj,
after restoring a snapshot you need to terminate the restore session. There is an example of this in base.py in the ci_tests folder.

def delete_sg_snapshot(self, sg_name, snap_name):
    """Delete a test storage group snapshot.

    :param sg_name: storage group name - - str
    :param snap_name: snapvx name - - str
    """
    snap_id = self.replication.get_storage_group_snapshot_snap_id_list(
        sg_name, snap_name)[0]
    snap_details = self.replication.get_snapshot_snap_id_details(
        sg_name, snap_name, snap_id)
    if 'Restored' in snap_details.get('state'):
        try:
            self.replication.delete_storage_group_snapshot_by_snap_id(
                sg_name, snap_name, snap_id)
        except Exception:
            pass
    self.replication.delete_storage_group_snapshot_by_snap_id(
        sg_name, snap_name, snap_id)

Note although this is using the delete storage_group_snapshot function it only deletes the restore session.. The snapshot should still be around. (Trust me, but please also verify on a test environment as I'm working from memory)

on the snapid you are correct, it is snapsetid in cli documents I but it's referring to the same thing. Note in the docs for PyU4V it also states This is for arrays with microcode less than 5978.669.669. Please use delete_storage_group_snapshot_by_snap_id for microcode greater than 5978.669.669. So you need to be at Unsiphere 9.2 and have the required micorode to use this functionality.

[mukeshguptakc]

Hi Paul,

The failure we observed occurs prior to we make a call to restore and delete snapshot.
And yes i do remeber in one our earlier threads you have mentioned to terminate a restore session, Post restore call.

The Error we observed above occurs when:

1. Snapshot operation for all 50 Storage group is created , -> Which succeeds.
   then
2. create a new target storage group from the snapshot and link it for all 50 storage groups.
   Using method def export_to_host
   --> In which we are currently observing Exception/Error traceback.

raise exception.VolumeBackendAPIException("
"PyU4V.utils.exception.VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error PUT symmetrix resource. The status code received is 500 and the message is {'message': 'A problem occurred modifying the snapshot generation resource: Cannot select enough Target devices to perform this operation'}." 

Now at step2: following method is executed.

def export_to_host is the block which performing following steps:
The method is sub-divided into 4 parts.

1. Creates a new storage group and link it snapshot.
2. Fetch Host id from the array.
3. Fetch Port Group id from the array.
4. Create Masking view group based on new storage group, host_id, port-group_id

Few Observations:

1. we create a new storage group for snapshot and link to it.
2. The Target storage group is created, but fails while linking with snapshot.

Post Step2 of scale operations:, we go ahead and call Restore and Delete Snapshot methods which is successful for all 50 storage groups snapshots.

[rawstorage]

Mukesh,
please email me direct, paule.martinAtdell.com, I'd like to take a look at some logs from your system. Pretty sure this is procedural.

[212850a]

@rawstorage, am I right if I say that Relink is not implemented in PyU4V yet?
Additionally as far as I see session mode is set to 'nocopy' always as you use copy=False in payload everywhere. Isn't it?

[rawstorage]

looks like that is an oversight, it should absolutely be there, we'll create an enhancement request for this and put it in a future release. Copy being set to false is intentional to ensure that data savings are achieved.

if you want to do this right now you can relink by creating a custom function. https://pyu4v.readthedocs.io/en/latest/quick_start.html#perform-a-custom-rest-call-in-pyu4v

[212850a]

Yep, I've just implemented it with a help of existing modify_resource function, just formed manually the following payload before:

payload = ({'action': 'Relink', 'relink': {
    'storage_group_name': TARGET_SG_NAME,
    'copy': True}})