provider/vSphere: Support for vSphere via VMC on AWS #150
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This patch introduces support for VMware vSphere and VMware Cloud (VMC) on Amazon Web Services (AWS).
This patch provides the vSphere Terraform necessary to complete support for the vSphere provider. Currently the design is dependent on a VMware Cloud on AWS (VMC) (https://cloud.vmware.com/vmc-aws) backend, but the intent is to work on vanilla vSphere as well. New config templates have been provided to sit alongside the existing Cloud-Init config templates. The new templates are based on the CoreOS Container Linux Config / Ignition format (https://coreos.com/os/docs/latest/provisioning.html). Notes: * The load balancer module has not yet been implemented. * There is an SSH key included for debugging purposes. It will be removed prior to submitting this work as a PR. * The public master IP address(es) and the master IP address(es) will be handled prior to submitting this work as a PR. Fixes crosscloudci#147
This patch configures the master VMs with static IP addresses from a pool, beginning with 192.168.1.101. NAT is configured via VMC, linking 192.168.1.101 with a public IP address in order to expose the K8s API server. This configuration will eventually be deprecated in favor of an AWS Elastic Load Balancer (ELB).
This patch updates the names and default values of the environment variables used to configure the vSphere provider.
This patch adds support for an AWS load balancer to the vSphere provider.
This patch will likely be squashed prior to a PR. This patch: - Updates the Dockerfile for the vSphere provider - Updates the vSphere provider's TF vars - Updates the vSphere provider's env vars
This patch updates the LB port from the K8s API default 6443 to the one used by Cross-Cloud, 443.
This patch updates the vSphere provider configuration so that master nodes leverage DHCP for dynamic IP addresses.
This patch changes the way the load balancer works for the vSphere provider. The LB now uses an elastic IP so that an IP address can be assigned to the DNS module's "public_master_ips" field.
This patch updates the Dockerfile to use an ENTRYPOINT command that points to the file "/cncf/provision.sh" instead of the CMD directive. The provision script has been updated to shell exec "/bin/bash" if the first argument to the script is "shell". Otherwise the provision script reads all of the same environment variables that used to be passed to the script as command-line arguments from the Dockerfile. Additionally, the script still accepts the same command-line arguments in the same order. Since environment variables cannot be used in a Dockerfile's CMD or ENTRYPOINT directives without using them in a second "sh" (or other shell) directive, they are no longer given to ENTRYPOINT or CMD as the default values for the provision script. Instead, as documented above, the values are parsed directly in the provision script.
This patch updates the vSphere provider's AWS resources with a tag "Environment" that has a value of the environment's name. This makes it easy to determine which AWS resources are related to which Cross-Cloud environment.
This patch updates the vSphere "docker run" examples to include the flag "--rm" which tells Docker to automatically remove the container once its execution has completed.
This patch updates the default resource pool in which new VMs are created. The name of the new pool is "CNCF Cross-Cloud". This pool has a max CPU resource of 256GHz and a max Mem resource of 512GB. Each maximum represents two, concurrent environments based on the Cross-Cloud FAQ regarding recommended specs per node. The CPU resource was calculated by assuming each recommended core to be 2GHz.
This patch introduces support for a resource pool per deployed
environment. Each RP is created as a child of the "CNCF Cross-Cloud"
resource pool. The "CNCF Cross-Cloud" pool has max CPU and mem settings
to allow for up to two, concurrent environments per the Cross-Cloud
team's recommended resource requirements.
Each child pool introduced by this change has a max CPU and mem setting
to prevent a single environment from consuming more than the Cross-Cloud
team's recommended resource requirements for a single environment:
CPU Max: 128GHz (4 nodes @ 16 cores / node w 2GHz / core)
Mem Max: 256GB (4 nodes @ 64GB / node)
This patch introduces support for per-environment folder structure. VMs
are now created in the following folder (by default): "Workloads/CNCF
Cross-Cloud/${name}".
This patch renames the root use of "lb" to refer to the load balancer module to the more specific "load_balancer" for the sake of clarity.
This patch updates the *.tf files at the root of the vSphere provider by running "terraform fmt" on them.
This patch merges the files modules_*.tf and input_*.tf at the root of the vSphere provider into the monolithic modules.tf and input.tf files.
This patch updates all of the vSphere provider's Terraform files with the "terraform fmt" command in order to ensure the sources follow the TF formatting convention.
This patch updates the way TF_VAR/Env Var relationships are handled in "provision.sh" for the vSphere provider. Now the TF_VAR variants are only exported if their env var counterparts are not null.
This patch reverts the master and worker templates for v1.10 back to the versions on the master branch.
This patch introduces support to the vSphere provider for the Cross-Cloud project's cloud-init config files. Cloud-init is bootstrapped via CoreOS Ignition.
This patch removes the custom Ignition transpiler Terraform provider, CT, from the Dockerfile and the vSphere Terraform module.
This patch adds "deploy.sh" and "destroy.sh" to the vSphere provider. These are helpful scripts to quickly deploy and destroy an environment with the vSphere provider.
This patch sets the minimum required version for the vSphere provider for Terraform to 1.6. This is the version of the provider in which the Terraform resource, ResourcePool, was introduced.
This patch removes all of the Terraform variables that are no longer necessary since static IPs are no longer used in favor of the AWS load balancer. This patch also removes the static configuration of gateway and DNS for virtual machines.
This patch updates the README files at the root of the project and in the vSphere module to reflect the fact that VMC on AWS is now required. This is due to Terraform not supporting the dynamic loading of modules or providers (hashicorp/terraform#953).
This patch parameterizes the allow_unverified_ssl option for the vSphere provider used by both Terraform vSphere provider and the K8s vSphere cloud provider. This patch enables the configuration of the allow_unverified_ssl option by users at runtime. The default value is "true". Setting the value to false will cause TLS connections to reject endpoints where the peer certificate cannot be validated.
This patch reverts the vSphere module's DNS service IP to 100.64.0.10, the same value used by the other provider modules. The value was temporarily changed to a custom IP address during testing. Many thanks to @figo for discovering this issue.
|
Hi @taylor, Could you please confirm you've received and are able to decrypt the credentials? Thank you! |
5 tasks
This patch sets the master and worker VM sizes. Based on the CNCF Cross-Cloud FAQ (https://github.com/crosscloudci/cross-cloud/blob/master/FAQ.md), the master and worker VMs have default settings of: - 16 vCPU (8 cores/socket) - 64 GB memory All of the settings are configurable via Terraform input variables.
|
Hi @taylor, Is it possible for you to provide an ETA on this PR being reviewed and possibly merged? Thank you! |
|
@akutz |
|
@denverwilliams thank you |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR introduces support for deploying a K8s cluster to vSphere running on VMware Cloud (VMC) on AWS. Credentials have been provided to @taylor. The vSphere directory's
README.mdfile has instructions on how to use the information to deploy and environment.Many thanks to everyone who helped make this PR a reality! I may not be tagging everyone due to saving them from the eventual deluge of e-mail triggered by comments on this PR, but suffice it to say that Hui and I could not have managed this without so many of the fine folks at Vulk :)
cc @figo
Fixes #147