VMware: Support parsing DEFAULT-RUN-POST-CUST-SCRIPT

[xiaofengw-vmware]

The flag "enable-custom-scripts" in VM Tools configuration is used to determine whether script is allowed to run. If this flag is not set in VM Tools configuration, we will take default value "false" for security purpose. vCloud product complain this break their regression behavior. To resolve this, we introduce flag "DEFAULT-RUN-POST-CUST-SCRIPT" so that the default value of "enable-custom-scripts" is "yes" in case "enable-custom-scripts" is absent. If "enable-custom-scripts" is explicitly set, then we still follow its value to enable/disable the script.

[xiaofengw-vmware]

@smoser @raharper Please help to assign this request and review, thanks a lot.

[mitechie]

Thanks, I'll get it on the queue for review.

[blackboxsw]

@blackboxsw grabs this for review today

[blackboxsw]

@xiaofengw-vmware, sorry to rehash this earlier discussion, but I need a bit of clarification on the deployment 'flow' and a vmware user launching a vm in vCloud (not vmware admin) would follow in vCloud product.

I have a couple of questions about this architecture that hopefully you can clarift, because this doesn't seem to meet this concerns @smoser mentioned on the previous PR #426.

Please correct any misunderstanding I have about vCenter, vSphere, or vCloud products. If something sounds incorrect to you, it probably is due to my lack of understanding of VMware products.

1. Is the intent of this branch is force the post-cloud-init-customization script to run of VM Tools configuration does not provide a value of true for "enable-custom-scripts" from the utility "vmware-toolbox-cmd". (I see the following doc reference about enable-custom-scripts)

2. Does a user launching each VM in vCloud have the opportunity to override this config option to "false" on the command line and/or UI during each vm creation of their images? What does that look like in vCloud/vSphere world?

3. I believe the concern @smoser raised may have been getting at is the following:

   • a given vCenter/vSphere or vCloud admin chooses a to default post-install script behavior to run on all VMs launched on that vCenter or vCloud deployment through VMTools configuration or through vCloud default_run_post_script setting
   • it would be ideal if during each vm creation, the user launching an image/vm could specify an override in cloud-init which takes precedence over both vm tools config and vCloud default_run_post_script setting.

In this case we consider both VM Tools config and vCloud default_run_post_script to be "cloud platform default behavior" but we need a way for a cloud-init consumer to disable such behavior if desired.

One way to do that is to provide datasource-configuration options for DataSourceOVF which are read from the class attribute self.ds_cfg and is already being done for the flag vmware_cust_file_max_wait . This would allow a customer to create a custom image that would override any default script opinions that a vCenter|vSphere/vCloud platform had chosen.

[blackboxsw]

If we provide the capability to override vmware default_run_post_script via cloud-init datasource config, we would need to document that override option in the ovf datasource docs (doc/rtd/topics/datasources/ovf.rst) which gets published here https://cloudinit.readthedocs.io/en/latest/topics/datasources/ovf.html
Other datasources document their config options: Azure, Ec2, OpenStack

[xiaofengw-vmware]

Thanks again @xiaofengw-vmware for working with us on pings here. This is a little sensitive in general as it has security implications and we'd like to surface the ability for users to "lock down" their instance with user-data if possible. And generally in cloud-init space, we are preferring the end-user's configuration preferences (via cloud-init user-data) over any of the platform

But if user doesn't set "enable-custom-scripts" explicitly, the default value of "enable-custom-scripts" is false. This fix only wants to change the default value of "enable-custom-scripts" to true if "enable-custom-scripts" is not set in
VM Tools configuration. User still could disable the script by setting "enable-custom-scripts" to false.

[blackboxsw] This feature/behavior makes sense to me , thanks for the clarification. If unset in VMtools, set true to enable custom scripts via DEFAULT-RUN-POST-CUST-SCRIPT if configured.

2. Does a user launching each VM in vCloud have the opportunity to override this config option to "false" on the command line and/or UI during each vm creation of their images? What does that look like in vCloud/vSphere world?

[xiaofengw-vmware]: Yes, user could disable the script by setting "enable-custom-scripts" to false. Then even DEFAULT-RUN-POST-CUST-SCRIPT is set to true, the script still couldn't run. DEFAULT-RUN-POST-CUST-SCRIPT only works if "enable-custom-scripts" is not set.

[blackboxsw]: This too seems clear from a vmware-tooling only point of view.

....

One way to do that is to provide datasource-configuration options for DataSourceOVF which are read from the class attribute self.ds_cfg and is already being done for the flag vmware_cust_file_max_wait . This would allow a customer to create a custom image that would override any default script opinions that a vCenter|vSphere/vCloud platform had chosen.
[xiaofengw-vmware]: I agree with you that we could add a such configuration in datasource-configuration. The problem is: vmware has its own engine to initialize the VM, by default cloud-init is disabled. After a VM / image is deployed, the vCenter or vCD administrator doesn't know whether cloud-init is enabled on VM or not. He just collects the
desired network setting / hostname and send this configuration to VM. Then VM Tools will check whether cloud-init is enabled. If no, vmware default engine will customize the VM per the configuration. Otherwise, DataSourceOVF of cloud-init will read meta data from the configuration and customize the VM following cloud-init work flow. So:

• We add this flag "DEFAULT-RUN-POST-CUST-SCRIPT" into the configuration, this flag could works for both vmware default engine and cloud-init.

[blackboxsw]: Agreed, and in the above statement, that "metadata" the vmware provides contains a setting that says, I expect this vm to run a custom script by default. As with any configuration directives from a cloud metadata service, cloud-init would like to have sensible defaults to "let the metadata service to everything it wants by default".

But, we also want to provide cloud-init users with the ability to opt-out of some configuration directives, especially if they lead to things like running arbitrary custom scripts on the vm during the initial boot process. There will be users that want to ability to avoid running this customization script, which is probably also why the VMTools surface a config option to set/unset "enable-custom-scripts". Cloud-init user-data would like the same option to ignore "enable-custom-scripts".

If there is a case where an image creator is providing custom cloud-init user-data, and doesn't use VMTools, providing them with a user-data option in cloud-init proper is important. If the custom vm image doesn't get properly set up because it missed running the recommended run post custom script, that ultimately should be the users problem as they opted out of the "default" configuration option, not a VMware tooling issue.

• If we add this flag into datasource configuration, then we couldn't handle vmware default engine and cloud-init in the same way. Also, vCenter and vCD administrator need to know whether cloud-init or vmware default engine is enabled. Currently they don't need to know this.

[blackboxsw]: I might be missing the point here. Why does vCenter of vCD need to know if cloud-init decided to follow the suggested custom-script guidelines. I assume cloud-init user-data is opaque to vmware's engine and I am suggesting we preserve that "opaqueness" because vCenter and vCD shouldn't have to know how to parse cloud-config user-data.

What will "break" in vmware's default engine if cloud-init's opaque user-data decides not run the post-cust-script? Wouldn't that same bit of logic break if VMTools set enable-custom-script "no"? The reason I ask these questions is leading to the "why does the platform care" if an individual turns off a setting from inside an opaque system setup/config? Those users may break themselves as they ignored defaults, or they may have a special use-case that they want to lock down as they may not trust the custom scripts that were config'd on that platform.

• After this fix, the behavior will be:
  1. If VM administrator doesn't want others to execute a script on this VM, he could set "enable-custom-scripts" to false from the utility "vmware-toolbox-cmd".
  2. If VM administrator doesn't set value to "enable-custom-scripts", then by default this script is disabled for security purpose.
  3. For vCD product, they prefer to enable the script if "enable-custom-scripts" is not set. So vCD will generate a configuration file with "DEFAULT-RUN-POST-CUST-SCRIPT" set to true. This flag works for both vmware customization engine and cloud-init.

[blackboxsw] The only thing I would add to this is a fourth config option (opaque to vmware tooling if possible)

4. cloud-init datasource config for DataSourceOVF honoring the following config datasource config option:

datasources:
   DataSourceOVF:
        enable-custom-scripts: false

This would allow DataSourceOVF logic to read:

    disabled_msg = ""
    if not util.get_cfg_option_bool(self.ds_cfg, "enable-custom-scripts", True):
          disabled_msg = "Custom script is disabled by cloud-init user-data"
    custScriptConfig = get_tools_config(                        
        CONFGROUPNAME_GUESTCUSTOMIZATION,                       
        GUESTCUSTOMIZATION_ENABLE_CUST_SCRIPTS,                 
         defVal)
    if  custScriptConfig.lower() != "true":  
        disabled_msg = "Custom script is disabled by VM Administrator"
   if disabled_msg:
        LOG.debug(msg)
        set_customization_status(
                GuestCustStateEnum.GUESTCUST_STATE_RUNNING,         
                GuestCustErrorEnum.GUESTCUST_ERROR_SCRIPT_DISABLED)
        raise RuntimeError(msg)                                 

[xiaofengw-vmware]

@blackboxsw, could you review my reply and check whether it's ok for you? Thanks.

[xiaofengw-vmware]

@blackboxsw @raharper Hi Chad, Ryan, please help to review my comments and check whether this design is ok. Thanks for your time.

[blackboxsw]

@xiaofengw-vmware sorry, just top-posted a response/question/suggestion two comments up

[xiaofengw-vmware]

@xiaofengw-vmware sorry, just top-posted a response/question/suggestion two comments up

@blackboxsw, Thanks Chad for the detail comments, I think I have totally understood your concern. As to the new switch "enable-custom-scripts" in datasourceOVF you suggested, I think it's a little duplicated with the one in VMTools configuration. The reason why I say that is that: The VMTools and datasourceOVF must work together. The VMTools is help to fetch the meta data and custom script, then put them under a folder. datasourceOVF start a loop to check that folder and apply the meta data/custom script once it find the meta data/custom script under that folder. For each cloud-init datasourceOVF user, he must install VMTools and be very clear with the VMTools configuration. I think it's a little confused if we add "enable-custom-script" for both VMTools configuration and datasourceOVF configuration. I wonder whether we could setup a meeting to have a quick sync and ensure there is no misunderstanding. Please let me know when you are available. My timezone is GMT+8 and I could attend the meeting in the morning or evening. Thanks for your time in advance.

[blackboxsw]

Thanks @xiaofengw-vmware I think if this is a requirement of cloud-init support in VMWare, we should probably document that detail in a datasource document in docs at cloud-init/doc/rtd/topics/datasources/ovf.rst

It would be nice to have some semblance of OVF documentation for VMWare that states VMTools is required for cloud-init use of and maybe a list of the VMTools config properties that coud-init consumes.

This context could be added to a new Configuration section in the ovf.rst. An example of this format is
../cloud-init/doc/rtd/topics/datasources/ec2.rst

Configuration
-------------
On VMware platforms, VMtools use is required for OVF datasource configuration settings as well as vCloud and vSphere admin configuration. To override config settings when launching individual VMs from an ova, do the following....

The following VMTools configuration options affect cloud-init's behavior on a booted VM:
 * a 
 * b
 * c 

VMWare admin can set the [config properties named in IMC config](https://github.com/canonical/cloud-init/blob/master/cloudinit/sources/helpers/vmware/imc/config.py)

For more information, see [this VMtools  documention](some-url) and specific VMTools parameters consumed

some thing like the above (obviously corrected as you have more details than we do) would give a little more context to how cloud-init interacts with VMware and VMTools.

[xiaofengw-vmware]

Thanks @xiaofengw-vmware I think if this is a requirement of cloud-init support in VMWare, we should probably document that detail in a datasource document in docs at cloud-init/doc/rtd/topics/datasources/ovf.rst

It would be nice to have some semblance of OVF documentation for VMWare that states VMTools is required for cloud-init use of and maybe a list of the VMTools config properties that coud-init consumes.

This context could be added to a new Configuration section in the ovf.rst. An example of this format is
../cloud-init/doc/rtd/topics/datasources/ec2.rst

Configuration
-------------
On VMware platforms, VMtools use is required for OVF datasource configuration settings as well as vCloud and vSphere admin configuration. To override config settings when launching individual VMs from an ova, do the following....

The following VMTools configuration options affect cloud-init's behavior on a booted VM:
 * a 
 * b
 * c 

VMWare admin can set the [config properties named in IMC config](https://github.com/canonical/cloud-init/blob/master/cloudinit/sources/helpers/vmware/imc/config.py)

For more information, see [this VMtools  documention](some-url) and specific VMTools parameters consumed

some thing like the above (obviously corrected as you have more details than we do) would give a little more context to how cloud-init interacts with VMware and VMTools.

@blackboxsw Thanks Chad, the configuration options are added into document, PTAL.

[mitechie]

I think you need to rebase the PR as it's not really possible to review in the current state with all the commit history tagged into the review. Once that's up I'll give it another look for sure.

Thank you!

[blackboxsw]

I think you need to rebase the PR as it's not really possible to review in the current state with all the commit history tagged into the review. Once that's up I'll give it another look for sure.

Thank you!

@xiaofengw-vmware
What this means, is you'll likely need to

git fetch upstream or git fetch origin and git rebase -i origin/master to rebase your local branch changes on top of existing upstream (or "origin" however you named your cloud-init remote). We don't want to see "91 files changed" or "46 commits" in your PR, just the files you locally changed

[xiaofengw-vmware]

I think you need to rebase the PR as it's not really possible to review in the current state with all the commit history tagged into the review. Once that's up I'll give it another look for sure.
Thank you!

@xiaofengw-vmware
What this means, is you'll likely need to

git fetch upstream or git fetch origin and git rebase -i origin/master to rebase your local branch changes on top of existing upstream (or "origin" however you named your cloud-init remote). We don't want to see "91 files changed" or "46 commits" in your PR, just the files you locally changed

Thanks, fixed.

[xiaofengw-vmware]

I think you need to rebase the PR as it's not really possible to review in the current state with all the commit history tagged into the review. Once that's up I'll give it another look for sure.

Thank you!

Thanks, this issue is fixed. PTAL.

[blackboxsw]

@xiaofengw-vmware thanks so much for working through this with us to make things a bit better and better documented.

[xiaofengw-vmware]

@xiaofengw-vmware thanks so much for working through this with us to make things a bit better and better documented.

Thank you all for your time.

[blackboxsw]

@xiaofengw-vmware if you have a chance. This fix has landed in -proposed for each ubuntu release and is undergoing testing. If you get a chance. Please confirm the fix works for your deployment use-case by following these steps below to enable install and test cloud-init from the -proposed apt repo.

https://cloudinit.readthedocs.io/en/latest/topics/debugging.html#stable-release-updates-sru-testing-for-cloud-init

If you get a chance to confirm behavior of VMWare. please do add a comment to that affect on our SRU verification bug https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1893064