Add Force checkbox to Live Migrate Dialog #66

igerlster · 2018-05-22T21:43:37Z

Hi,

live migration from e.g. xeon v3 to xeon v4 fails because of cpu mismatch.. xe vm-migrate command line with force it mostly works fine...

Maybe a "Force Live Migration" Checkbox could be added ?

Thanks
Sebastian

olivierlambert · 2018-05-22T21:59:47Z

It is doing a force migration by default (without force, it would be refused). We use basically the same API calls than what you did in CLI.

igerlster · 2018-05-22T22:33:23Z

Oh.. i only tested it on XenSever and thought it would be the same...

Thanks !

olivierlambert · 2018-05-22T22:45:14Z

Oh sorry misread your request: you mean you can't force it in XCP-ng Center?

igerlster · 2018-05-23T13:55:54Z

On XenServer Center i HAVE to use the command line since GUI doesn't have a force button... if XCP-ng Center uses force by default it should be fine...

olivierlambert · 2018-05-23T22:02:57Z

Okay I was confused: it was late and I think you asked for Xen Orchestra UI! In Xen Orchestra, we force it by default.

In XenCenter, it's likely that's it's NOT forced by default (hence your experience with it) and indeed, if you don't see a force checkbox, it's not exposed.

@borzel This is probably something you could add and push against mainline too ;)

borzel · 2018-07-11T21:14:25Z

@igerlster Can you provide a bit more context? Are you livemigrating in a pool or from host to host? Are you using the migration wizard? Can you maybe post a screenshot?

At this stage it's not clear to me, what you are exactly doing.

I looked into XCP-ng Center source code, but was not able to find the migration code or the mechanic to trigger a migration. It's a bit undocumented :-/

borzel · 2018-07-11T22:14:30Z

@igerlster I managed to find the migration code, can you please test this fix?: https://builds.schulzalex.de/xcp-ng/fix-migrate-with-force

igerlster · 2018-07-13T10:54:16Z

Hi,
Sorry for my late reply.

unfortunately no success, i'm trying to migrate a vm from XeonE3 to slightly different XeonE3 and get CPU mismatch. Using force on the command line works fine. And it shouldn't be a problem most of the time since the only feature missing is a on chip gpu.

https://i.imgur.com/DMRnCOD.png

If this Dialog would give a warning "CPU mismatch, force live migrate ?" that would be very nice :-)

Thanks,

borzel · 2018-07-13T13:21:43Z

Can you please provide the title of the dialog? Its a generic copy wizard which has many modes. I'll try to find your specific issue first :-)

igerlster · 2018-07-13T13:32:57Z

Sure : right click on a VM -> Migrate to Sever -> Migrate VM wizard... -> Dialog with Name "Migrate VM" opens

:-)

borzel · 2018-07-13T14:24:30Z

I assume you are livemigrating from host to host? Not from poolmember to poolmember?

igerlster · 2018-07-13T14:28:28Z

Yes, from Host to Host....

borzel · 2018-07-13T14:37:49Z

Ok, wish me luck... Maybe this weekend I have another testversion :-)

igerlster · 2018-07-13T15:08:40Z

OK :-) nice .-) Good luck !

borzel · 2018-07-14T20:30:47Z

I did look deeper into this ... it's complicated...

The wizard ask's the server, wether it can migrate the vm:

hmm....

The hard part is not to force migrate, the hard part is to intercept the "error" message and to ignore/overwrite it.

igerlster · 2018-07-14T20:39:02Z

hmm. maybe you can add "force=true" in _options ? so the answer the server gives should be "yes"... maybe :-)

borzel · 2018-07-14T21:00:44Z

good hint... I did not see this parameter 🙈

borzel · 2018-07-14T21:25:01Z

You are right, XAPI checks if there is a "force" option
(https://github.com/xapi-project/xen-api/blob/a9eb7cd0977ef7ea55ca81246078c456bd839ebb/ocaml/xapi/xapi_vm_migrate.ml#L1167)

borzel · 2018-07-14T21:27:40Z

It's there :-)

igerlster · 2018-07-14T21:31:43Z

Oh wow .. Nice :)

igerlster · 2018-07-14T21:34:37Z

Now passing force to the the part of the code asking if migrate is possible ... Hmm...

borzel · 2018-07-14T21:36:51Z

buildserver is allready building

borzel · 2018-07-14T21:41:35Z

Please carefully check with this build: https://builds.schulzalex.de/xcp-ng/fix-migrate-with-force/
I did just implement the checking, the actual migration may fail. Please test not with systems in production. Test with VMs of which you have a backup!

igerlster · 2018-07-14T22:11:03Z

Oh nice, thanks! :-) now i get the option to start the migration. So seems to be working. I don't have any non production systems right now to test if migration would start :-(

borzel · 2018-07-14T22:22:07Z

I even don't have production systems with this scenario, so I'm not able to test too :-(

borzel · 2018-07-14T22:22:57Z

⚠️ Call to the world ⚠️

Please can someone test on non production systems?
https://builds.schulzalex.de/xcp-ng/fix-migrate-with-force/

unusual-aspect · 2018-07-16T08:47:09Z

I have two servers that have 'similar' CPU, and have that problem, so i can't move VMs live ... from one server to another, only solution was to shutdown VM, move and then run again.

Error (or more warning), was showing that server CPU don't have features on remote ... where both server have Intel i5 cpus - only difference is generation ;)

Installed your nightly version, and can confirm that moving VM is possible now! Thanks, have no idea what is changed, but is possible to move from one server to another - on live session.

Edit: One more thing ... XCP-ng tools are not installed on VM.

unusual-aspect · 2018-07-16T09:04:41Z

After further testing ... is not that easy as i was think, some VM failed to move:

"Failed","Migrating VM 'Scripts' from 'two-xenserver' to 'xenserver'
Internal error: Xenops_interface.Internal_error("(Unix.Unix_error "Connection reset by peer" write "")")
Time: 00:08:12","xenserver","Jul 16, 2018 9:45 AM"

"Failed","Resuming VM 'Scripts'
VM cannot be resumed because it has no suspend VDI","two-xenserver","Jul 16, 2018 9:53 AM"

borzel · 2018-07-16T20:26:40Z

I try to build another improved nightly

borzel · 2018-07-16T21:39:44Z

I added another menu item to "Force Migrate" VMs:

please try again with build 99.99.99.36
https://builds.schulzalex.de/xcp-ng/fix-migrate-with-force/

unusual-aspect · 2018-07-17T09:01:18Z

"Failed","Migrating VM 'Scripts' from 'two-xenserver' to 'xenserver'
Internal error: Xenops_interface.Internal_error("(Unix.Unix_error "Connection reset by peer" write "")")
Time: 00:08:02","xenserver","Jul 17, 2018 9:18 AM"

Nope ;) - Force flag for some reason not working :/ - when i was try to migrate in 'normal' way have same result.

Only difference that i notice is 'progress bar' in events ... first (attempt) was when migrate vm, and that was fine, then progress was reset to 0, then i believe was try to startup vm? So progress is moving, and then failure message appear.

With one is weird anyway, as i can migrate easily from 'xenserver' to 'two-xenserver'

PS: Sorry it was i7, i was think that they are i5 ;)

[root@xenserver ~]# xe host-cpu-info
cpu_count       : 8
    socket_count: 1
          vendor: GenuineIntel
           speed: 2793.074
       modelname: Intel(R) Core(TM) i7 CPU         860  @ 2.80GHz
          family: 6
           model: 30
        stepping: 5
           flags: fpu de tsc msr pae mce cx8 apic sep mca cmov pat clflush acpi mmx fxsr sse sse2 ht syscall nx lm constant_tsc arch_perfmon rep_good nopl nonstop_tsc pni monitor est ssse3 cx16 sse4_1 sse4_2 popcnt hypervisor lahf_lm ida dtherm
        features: 0098e3fd-bfebfbff-00000001-28100800
     features_pv: 17c9cbf5-80b82201-2191cbf5-00000003-00000000-00000000-00000000-00000000-00000000-00000000
    features_hvm: 17cbfbff-80b82221-2993fbff-00000003-00000000-00000000-00000000-00000000-00000000-00000000
[root@xenserver ~]#

[root@two-xenserver ~]# xe host-cpu-info
cpu_count       : 8
    socket_count: 1
          vendor: GenuineIntel
           speed: 3591.766
       modelname: Intel(R) Core(TM) i7-4790 CPU @ 3.60GHz
          family: 6
           model: 60
        stepping: 3
           flags: fpu de tsc msr pae mce cx8 apic sep mca cmov pat clflush acpi mmx fxsr sse sse2 ht syscall nx lm constant_tsc arch_perfmon rep_good nopl nonstop_tsc eagerfpu pni pclmulqdq monitor est ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm ida arat epb pln pts dtherm fsgsbase bmi1 avx2 bmi2 erms xsaveopt
        features: 7ffafbff-bfebfbff-00000021-2c100800
     features_pv: 17c9cbf5-f6f83203-2191cbf5-00000023-00000001-00000329-00000000-00000000-00000000-00000000
    features_hvm: 17cbfbff-f7fa3223-2d93fbff-00000023-00000001-000007ab-00000000-00000000-00000000-00000000
[root@two-xenserver ~]#

Last thing, like i believe will be helpful is that VM OS is Ubuntu 16.04.4 LTS

unusual-aspect · 2018-07-17T09:07:06Z

So in short, i can move vm up in cpu features (stepping)? and cannot be downgraded? I mean when run vm on better cpu, you cannot be moved on worst one?

borzel · 2018-07-17T09:11:50Z

That's the allways the case if you use single servers with 'different' CPU's.

If you build a pool from your single Servers, the CPU-features are used from the "worst" CPU; but some CPU's don't have the masking feature so even this isn't possible.

(In my option the live migration from host to host (outside a pool) should only be needed in migration/maintenance scenarios.)

unusual-aspect · 2018-07-17T09:52:18Z

Well i have only two servers, and i use them to move on server maintenance, and on balancing resources.

I will try to create pool and migrate vm's between servers in one pool, to have look that will work, as i was never try before.

borzel · 2018-07-25T21:37:01Z

to sum up, you can now migrate with XCP-ng Center the same way like with cli-commands?
So the issue is resolved with my changes?

cometmth · 2018-07-25T23:17:04Z

Based on my understanding of the issue, I think it's resolved. My test environment has an i3-4130 and an i7-3370. Llive migrating from the newer i3 to the older i7 required using the new 'Force Migrate VM wizard...' The regular migration wizard was showing the "..does not have some of the CPU features..." message in this direction. The live migration was successful. I was able to migrate back from the i7 to the i3 using the standard migrate wizard.

The i3 is in a pool by itself (only member), and the i7 is a standalone server. Both are on xcp-ng 7.4. I tested with xcp-ng center linked above (https://builds.schulzalex.de/xcp-ng/fix-migrate-with-force/)

igerlster closed this as completed May 22, 2018

borzel reopened this May 23, 2018

borzel added enhancement New feature or request not reproduced labels May 23, 2018

borzel added this to the Version 7.5.1 milestone Jul 7, 2018

borzel self-assigned this Jul 11, 2018

borzel added the in progress 🏃‍♂️ Someone is working on that label Jul 11, 2018

borzel added ...waiting... and removed not reproduced labels Jul 12, 2018

borzel removed the ...waiting... label Jul 13, 2018

borzel added help wanted Extra attention is needed do PR to upstream labels Jul 14, 2018

borzel modified the milestones: Version 7.5.1, Version 7.6 Jul 26, 2018

borzel mentioned this issue Jul 26, 2018

Enhancement: migrate with force #85

Merged

borzel closed this as completed Jul 26, 2018

Add Force checkbox to Live Migrate Dialog #66

Add Force checkbox to Live Migrate Dialog #66

Comments

igerlster commented May 22, 2018

olivierlambert commented May 22, 2018

igerlster commented May 22, 2018

olivierlambert commented May 22, 2018

igerlster commented May 23, 2018

olivierlambert commented May 23, 2018

borzel commented Jul 11, 2018

borzel commented Jul 11, 2018 • edited Loading

igerlster commented Jul 13, 2018

borzel commented Jul 13, 2018

igerlster commented Jul 13, 2018

borzel commented Jul 13, 2018 • edited Loading

igerlster commented Jul 13, 2018

borzel commented Jul 13, 2018 • edited Loading

igerlster commented Jul 13, 2018

borzel commented Jul 14, 2018 • edited Loading

igerlster commented Jul 14, 2018 • edited Loading

borzel commented Jul 14, 2018

borzel commented Jul 14, 2018 • edited Loading

borzel commented Jul 14, 2018

igerlster commented Jul 14, 2018

igerlster commented Jul 14, 2018

borzel commented Jul 14, 2018

borzel commented Jul 14, 2018 • edited Loading

igerlster commented Jul 14, 2018

borzel commented Jul 14, 2018

borzel commented Jul 14, 2018 • edited Loading

unusual-aspect commented Jul 16, 2018 • edited Loading

unusual-aspect commented Jul 16, 2018 • edited Loading

borzel commented Jul 16, 2018

borzel commented Jul 16, 2018

unusual-aspect commented Jul 17, 2018 • edited Loading

unusual-aspect commented Jul 17, 2018

borzel commented Jul 17, 2018 • edited Loading

unusual-aspect commented Jul 17, 2018

borzel commented Jul 25, 2018

cometmth commented Jul 25, 2018

borzel commented Jul 11, 2018 •

edited

Loading

borzel commented Jul 13, 2018 •

edited

Loading

borzel commented Jul 13, 2018 •

edited

Loading

borzel commented Jul 14, 2018 •

edited

Loading

igerlster commented Jul 14, 2018 •

edited

Loading

borzel commented Jul 14, 2018 •

edited

Loading

borzel commented Jul 14, 2018 •

edited

Loading

borzel commented Jul 14, 2018 •

edited

Loading

unusual-aspect commented Jul 16, 2018 •

edited

Loading

unusual-aspect commented Jul 16, 2018 •

edited

Loading

unusual-aspect commented Jul 17, 2018 •

edited

Loading

borzel commented Jul 17, 2018 •

edited

Loading