More than 4 VMs booting simultanously yield "lab.conf Lock error"

[schlomo]

When too many VMs boot at the same time then the locking for the lab.conf runs into a timeout. Example log output proving the point:

/usr/lib/lml/pxelinux.pl:86|10307|3022 milliseconds
/usr/lib/lml/pxelinux.pl:86|12146|5031 milliseconds
/usr/lib/lml/pxelinux.pl:86|12138|5032 milliseconds
/usr/lib/lml/pxelinux.pl:86|19197|9058 milliseconds
/usr/lib/lml/tools/lml-maintenance.pl:56|3867|3025 milliseconds
/usr/lib/lml/pxelinux.pl:86|15920|6038 milliseconds
/usr/lib/lml/pxelinux.pl:86|26445|17104 milliseconds
/usr/lib/lml/pxelinux.pl:86|17305|8079 milliseconds
/usr/lib/lml/pxelinux.pl:86|37438|28259 milliseconds
/usr/lib/lml/pxelinux.pl:86|17194|8051 milliseconds
/usr/lib/lml/pxelinux.pl:86|13390|4028 milliseconds
/usr/lib/lml/pxelinux.pl:86|19446|10062 milliseconds
/usr/lib/lml/pxelinux.pl:86|35315|25131 milliseconds
/usr/lib/lml/pxelinux.pl:86|24832|14075 milliseconds
/usr/lib/lml/tools/lml-maintenance.pl:56|26955|26134 milliseconds

And

[Tue Oct 29 12:14:23 2013] [error] [client 10.13.255.108] I could not get a lock for '/nfs/tuvfls02/nic/nic/tuvnic/lml/lab.conf': [Timed out waiting for blocking lock] at /usr/lib/lml/pxelinux.pl line 86
[Tue Oct 29 12:22:14 2013] [error] [client 10.13.255.132] I could not get a lock for '/nfs/tuvfls02/nic/nic/tuvnic/lml/lab.conf': [Timed out waiting for blocking lock] at /usr/lib/lml/pxelinux.pl line 86
[Tue Oct 29 12:22:20 2013] [error] [client 10.100.115.151] I could not get a lock for '/nfs/tuvfls02/nic/nic/tuvnic/lml/lab.conf': [Timed out waiting for blocking lock] at /usr/lib/lml/api/vm_overview.pl line 22, referer: http://tuvnic01.dev.is24.loc/lml/
[Tue Oct 29 12:22:37 2013] [error] [client 10.13.255.148] I could not get a lock for '/nfs/tuvfls02/nic/nic/tuvnic/lml/lab.conf': [Timed out waiting for blocking lock] at /usr/lib/lml/pxelinux.pl line 86
[Tue Oct 29 12:22:37 2013] [error] [client 10.13.255.189] I could not get a lock for '/nfs/tuvfls02/nic/nic/tuvnic/lml/lab.conf': [Timed out waiting for blocking lock] at /usr/lib/lml/pxelinux.pl line 86
[Tue Oct 29 12:22:56 2013] [error] [client 10.13.255.122] I could not get a lock for '/nfs/tuvfls02/nic/nic/tuvnic/lml/lab.conf': [Timed out waiting for blocking lock] at /usr/lib/lml/pxelinux.pl line 86

Possible Solutions

• Split lab.conf into many small files so that booting a VM will only touch a few of them.
• Use a persistent connection to vSphere. This will reduce script execution time by at least 3 seconds and also reduce the load on the vSphere server.
• Separate VM boot from network, datastore, host and folder updates.

Probably we should do all 3 of these.