vSphere/VMC Support

[akutz]

Hi,

My team at VMware is working on adding support for vSphere/VMC to the CNCF cross cloud dashboard. We have a few questions that we hope you can answer:

1. Are there concurrent builds? Several designs are currently being reviewed, and they hinge on whether or not we must support multiple, concurrent build environments.
2. How often are builds run? It looks like builds are triggered, but certainly not with every commit?
3. How is sensitive information handled and stored? It appears we will need to provide you with credentials to access our environment. We would like some more information on how that data is stored.
4. Where, when, and how is the sensitive information available? It appears that it must be available as environment variables to the process that runs provision.sh. We need to know if such information is available to provision.sh as well as Terraform.
5. Is there any caller-based state between the provision and destroy steps? For example, can provision.sh write information to disk during a deploy in order to reuse said information during the destroy step?

Thank you for your time. We've found the existing code and PRs to be useful in getting our work up and running. However, the following would be incredibly useful for us and people in the future:

• A walkthrough on how to add support for a new provider
• A high-level overview of how the environments are provisioned, tests are run, etc.

Thank you again!

[lixuna]

Hi, @akutz!

1. While builds are concurrent, they happen before provisioning on a different system. The cncf.ci dashboard provisions stable and head releases of Kubernetes concurrently to each cloud provider.

2. The dashboard is updated daily at 3 AM EST.

3. We use GitLab's secret environment variables for cloud provider credentials. The database is not accessible by the public.

4. GitLab passes the cloud provider credentials to terraform via environment variables in the GitLab runner. The credentials are not logged or shown in the GitLab console.

5. There is no caller-base state saved for the cloud provider credentials.

• While we do not yet have a walk-through for adding a new cloud provider, the team presented a deep-dive session at KubeCon EU in May 2018 to show how the provisioner works on a cloud provider.

• The team also presented an intro to the CNCF Cross-cloud CI project. The recording and slides for both presentations and additional high-level overview can be found in the crosscloudci README at: https://github.com/crosscloudci/crosscloudci/blob/master/README.md#past

Thanks so much for taking the time to add vSphere/VMC to the CNCF cross cloud dashboard. I hope this note has been helpful.

[akutz]

Hi @lixuna,

Thank you so much for the response! Some quick notes:

We use GitLab's secret environment variables for cloud provider credentials. The database is not accessible by the public.

Are the environment variables available to the process that runs provision.sh? For example, could we add a step in our if block that calls out to some custom action and use an environment variable we ask you to define for which we provide sensitive information?

There is no caller-base state saved for the cloud provider credentials.

I wasn't asking about state with respect to the credentials but rather if we were creating dynamic, public IPs and needed to know which ones were created during the stand-up so we could remove them during the tear down.

While we do not yet have a walk-through for adding a new cloud provider, the team presented a deep-dive session at KubeCon EU in May 2018 to show how the provisioner works on a cloud provider.

I actually tried viewing several presentations and slide decks, but I was told I needed to request permission. I requested it, but have yet to be granted said permission. Perhaps we're referring to different slides? Do you perhaps have a link?

Thank you again for your help. If you'd like to join our Zoom meeting, please feel free to join at REDACTED. It's currently in progress. Thanks again!

[lixuna]

@taylor can you please help with the first 2 questions?

[akutz]

Hi @lixuna,

Thank you, my colleague @figo found the YouTube link https://www.youtube.com/watch?v=m-WK-pOs5TA. Thank you again! I had crossed my CIs with K8s test-infra. I had requested access to the slides mentioned in this blog post, which I now realize isn't the same as your project :)

[lixuna]

The Cross-cloud CI KubeCon slides are at:

• Intro and

• Deep Dive (anyone with link should be able to view).

The recording for the Intro is at https://youtu.be/wb7aCAk1VFU, the recording for Deep Dive is at https://youtu.be/m-WK-pOs5TA.

[akutz]

Hi @lixuna,

Is there some property that we can use to determine if the environment is running against stable or HEAD? I noticed there are a bunch of prefixes that are different based on this decision, but I'm wondering if there is a specific environment variable we should check for this?

[taylor]

Howdy, @akutz.

All of the setup and tear down should be handled by Terraform. General steps for adding a new cloud provider to the cross-cloud provisioner:

1. Determine if a terraform module will work for the new cloud environment
2. Review how terraform templates were built
3. Look at how to do auth
4. Create sub-folder for the new cloud environment

There is a vSphere terraform module available https://www.terraform.io/docs/providers/vsphere/index.html

Review the terraform templates in the cross-cloud repo. Examples

• OpenStack https://github.com/crosscloudci/cross-cloud/tree/master/openstack (contributed to project)
• AWS https://github.com/crosscloudci/cross-cloud/tree/master/aws
• Google Compute Engine https://github.com/crosscloudci/cross-cloud/tree/master/gce

After the terraform cloud provisioning is done then the kubernetes deployment itself is all handled by the rest of the cross-cloud code including using stable/HEAD etc.

In the Deep Dive video, https://youtu.be/m-WK-pOs5TA?t=156, you can see where the provisioning happens using terraform. If y'all can get the same working for vSphere/VMC then we'll be good to go.

[akutz]

Hi @taylor,

Cool beard. Makes me miss mine:

Anyway, thank you for the info. We're aware of the vSphere Terraform provider. Oh, lordy, are we aware :) One of the challenging aspects to this project has been navigating around the differences encountered when using Terraform in combination with vSphere on VMC. Some issues include:

• VMC disallows datastore access. We can circumvent this using the content library.
• VMC disallows the normal way of doing cloud-init. We are circumventing this using CoreOS and GuestInfo in combination with Ignition.

The major challenge is the public IPs. With vSphere on VMC you cannot directly assign public IPs to VMs. You can provide public access to VMs on a private network by requesting a public IP and connecting it to the private IP via NAT or using AWS Elastic Load Balancing (ELB). We're likely pursuing the latter option.

Essentially we're handling the challenges created by vSphere on VMC one step at a time. Part of this process has been coming to better understand just what is required of a provider for the Cross-Cloud project. The existing examples are helpful, but please be aware that we need to understand the reasons behind certain actions since we cannot simply connect things to an analogue property (*.publicIPs for example) in the vSphere provider for Terraform.

Thank you again!

[akutz]

Hi @taylor,

I'm watching the YouTube deep dive now. It's very helpful. Quick question, it looks like you're mapping the K8s public IPs to DNS names using a .local suffix. That's not odd. Not for use inside the deployed environment. What's odd is it looks like the kubeconfig file being written to the host on which provision.sh is executed is used by kubectl calls in provision.sh to communicate with the remote K8s endpoint. How is that possible? Are you running a DNS server in the container image where provision.sh is executed? Modifying the running container's /etc/hosts file to map the remote K8s API endpoint to something.something.local?

Thanks again!

[akutz]

Hi @taylor / @lixuna,

After reviewing all of the provided and linked materials, I still have the following, outstanding questions:

• How does terraform destroy works for the different providers. @lixuna said you do not maintain state between a provision.sh deploy and provision.sh destroy, but if you do not, how is terraform destroy able to know what environment to target and destroy? Are you sure the deployed environment's state isn't preserved in DATA_DIRECTORY and then used to perform the destroy operation?

• How is the same kubeconfig file created for use by the deployed environment used by provision.sh to communicate with the remote K8s endpoint? The host in the kubeconfig file would have a .local suffix. Are you modifying the running container's /etc/hosts file, running a local DNS server, or modifying some static DNS server referenced by the container?

• Is there some property that we can use to determine if the environment is running against stable or HEAD? I noticed from the dashboard's linked log files that there are a bunch of environment variables with values that could be parsed to determine if it's a stable or HEAD build, but I don't see an environment variable that is the obvious one to use? Am I missing something?

Thank you again for your assistance!

[figo]

Hi @taylor @lixuna @denverwilliams

Could you allow myself/colleagues to test cncf crosscloud vSphere support against this default DNS/etcd server
147.75.69.23 ?

initially, i am able to curl it, but got the following error later on:

Recv failure: Connection reset by peer

Thanks very much

[akutz]

Hi @taylor / @lixuna,

One other question I had was what version of Docker you use to build the image at the root of this project? I ask because if we need to add anything to the image, such as a custom Terraform provider, one way to do so would be to leverage Docker's multi-stage builds. I've used them before, and they're pretty nice. However, they're only supported in Docker 17.05 and higher. If you use a compatible version then we'd be able to edit the Docker file to look like:

FROM golang:latest
WORKDIR /go/src/github.com/vmware/vmc-terraform-provider/
COPY vsphere/vmc-terraform-provider/* .
RUN go get -d -v && go build

# Switch to the second stage of the build, nothing in the first stage
# is preserved except for what is explicitly copied to the following
# stage.

FROM crosscloudci/debian-go:latest
MAINTAINER "Denver Williams <denver@debian.nz>"

...

# Copy the build artifact from the first stage of the build
COPY --from=0 /go/src/github.com/vmware/vmc-terraform-provider/vmc-terraform-provider /cncf/vsphere/vmc-terraform-provider/

[akutz]

Hi @lixuna / @taylor,

Could you please provide an e-mail address of someone we could contact to discuss the aforementioned topics? We appreciate the help you've already provided, but perhaps y'all are not the correct people to whom we should be speaking? Thank you again.

cc @clintkitson

[taylor]

@figo, using the default etcd cluster we have running is fine. Any connection reset by peer should be temporary. Are you getting that every time?

[taylor]

@akutz, right folks. Just busy ;)

I'm pulling @denverwilliams into the loop.

Re: re other methods of contact

• email crosscloudci@vulk.coop
• slack cloud-native.slack.com #cncf-ci

[akutz]

Hi @taylor,

Thanks. I wasn't sure if you were the correct people or just busy. After 12 days since the last response, I just figured we were bugging the wrong group of misfits :)

[taylor]

@figo, I just tested with

curl -X PUT  http://147.75.69.23:2379/v2/keys/_test_b4adc42e-7ca7-48b6-b3f4-33aaeb756520
curl -X DELETE  http://147.75.69.23:2379/v2/keys/_test_b4adc42e-7ca7-48b6-b3f4-33aaeb756520

successfully

[akutz]

Hi @taylor,

I've asked @figo to please hold off on his question above so we can first focus on the general process. The way DNS is managed is unclear. Are your answers implicit affirmation that what @figo found is correct? Or is it one of the possibilities I outlined above? We are not necessarily going to be the people on our team (or off of it) that maintain this in the future, so I'd like to document things clearly for those future folk, be them ourselves or another group of crazed outlanders. Thanks!

[denverwilliams]

** What version of Docker you use to build the image at the root of this project?

• We are using the latest stable version of Docker, v18.x
• You should be able to leverage Docker's multi-stage builds

** How is the same kubeconfig file created for use by the deployed environment used by provision.sh to communicate with the remote K8s endpoint? The host in the kubeconfig file would have a .local suffix. Are you modifying the running container's /etc/hosts file, running a local DNS server, or modifying some static DNS server referenced by the container?

• We are using our own "etcd backed" dns server that is publicly available to all nodes in the cluster, we run terraform apply we generating a resolv.conf that configures the nodes to use "${ var.discovery_nameserver }" as the primary nameserver using the template found here https://github.com/crosscloudci/cross-cloud/blob/master/dns-etcd/dns.conf

• The next step is for us to add the required records to the etcd server so we can resolve the nodes in the cluster. This is done by the dns-etcd module https://github.com/crosscloudci/cross-cloud/tree/master/dns-etcd https://github.com/crosscloudci/cross-cloud/blob/master/dns-etcd/dns.tf#L16 is the main file that will take the private & public ip address provided to the module and create etcd/dns records for them.

• Clients external to the cluster will need to have the primary nameserver updated to "${ var.discovery_nameserver }" to be able to use the generated kubeconfig

[akutz]

Hi @denverwilliams,

Thank you very much for the response. That does answer the DNS question. We still have the following, outstanding inquiries:

• How does terraform destroy works for the different providers. @lixuna said you do not maintain state between a provision.sh deploy and provision.sh destroy, but if you do not, how is terraform destroy able to know what environment to target and destroy? Are you sure the deployed environment's state isn't preserved in DATA_DIRECTORY and then used to perform the destroy operation?
• Is there some property that we can use to determine if the environment is running against stable or HEAD? I noticed from the dashboard's linked log files that there are a bunch of environment variables with values that could be parsed to determine if it's a stable or HEAD build, but I don't see an environment variable that is the obvious one to use? Am I missing something?

If you could possibly help us with the above questions as well, I would be immensely grateful.

Finally, is there any possibility that you've looked at generating a Container Linux Config or Ignition Config in addition to the Cloud-Init config? Thanks again!

[figo]

@taylor thanks for reply,
curl -X PUT http://147.75.69.23:2379/v2/keys/_test_b4adc42e-7ca7-48b6-b3f4-33aaeb756520
it works fine with my home network public ip,
but when executing the same line through corporate network, it got error Connection reset by peer, this is not a blocking issue anymore since i can test it with home network.

[taylor]

@figo, maybe your corporate network has a proxy that is dropping those connections?

[figo]

@taylor not sure, but thanks for your reply, and we are good with continuing the work on non-corporate network, we can come back for investigation later.

[figo]

@taylor @denverwilliams
we are currently facing DNS issue that need your help greatly.

our script is able to add record successfully for address like:
vsphere-master-2.vsphere.vsphere.local:

curl -XPUT http://\"147.75.69.23:2379\"/v2/keys/skydns/local/\"vsphere\"/\"vsphere\"/\"vsphere-master-2\" 
\\\n  -d value='{\"host\":\"192.168.1.27\"}'

server confirmed with:

{"action":"set","node":{"key":"/skydns/local/vsphere/vsphere/vsphere-master-2",
"value":"{\"host\":\"192.168.1.27\"}","modifiedIndex":12995,"createdIndex":12995}}

but dig the address got SERVFAIL,
i ran this cmd from my mac:

bash-3.2$ dig A vsphere-master-2.vsphere.vsphere.local @147.75.69.23 

; <<>> DiG 9.8.3-P1 <<>> A vsphere-master-2.vsphere.vsphere.local @147.75.69.23
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 16531
;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0
;; WARNING: recursion requested but not available

;; QUESTION SECTION:
;vsphere-master-2.vsphere.vsphere.local.	IN A

;; Query time: 34 msec
;; SERVER: 147.75.69.23#53(147.75.69.23)
;; WHEN: Wed Jun  6 10:44:52 2018
;; MSG SIZE  rcvd: 56

i am new to skydns, by googling the error does not help much,
do i missed anything obviously?
Thanks

[denverwilliams]

• How does terraform destroy works for the different providers. @lixuna said you do not maintain state between a provision.sh deploy and provision.sh destroy, but if you do not, how is terraform destroy able to know what environment to target and destroy? Are you sure the deployed environment's state isn't preserved in DATA_DIRECTORY and then used to perform the destroy operation?

  • We store our state in an S3 bucket during the deploy and destroy stages. The Terraform code that does this can be found here. https://github.com/crosscloudci/cross-cloud/blob/master/provision.sh#L64

• Is there some property that we can use to determine if the environment is running against stable or HEAD? I noticed from the dashboard's linked log files that there are a bunch of environment variables with values that could be parsed to determine if it's a stable or HEAD build, but I don't see an environment variable that is the obvious one to use? Am I missing something?

  • The default Kubernetes version used during the deploy can be found here in the "artifact version" section https://github.com/crosscloudci/cross-cloud/blob/master/gce/input.tf#L31.
    These can all be overwritten by setting the variables found here https://github.com/crosscloudci/cross-cloud/blob/master/provision.sh#L22 if any of these are set they will take precedence over the defaults.

[denverwilliams]

@figo in regards to the skydns issue, i forgot to add "vsphere.local" as an accepted domain to our Coredns config. It has been added now, so hopefully you will be able to resolve names for *.vsphere.local

. {
    etcd aws.local azure.local gce.local packet.local openstack.local {
        stubzones
        path /skydns
        endpoint http://147.75.69.23:2379
    }
    loadbalance
}

[figo]

@denverwilliams a million thanks, it magically works now, this is something we will never able to figure out by ourself, it will be great to document this in some place.

[akutz]

Hi @denverwilliams,

Thank you for the reply. Per your suggestion regarding determining the environment type, you're basically recommending the same approach I outlined, correct? Use one of the existing property values and test for the prefix master- to determine if it's the HEAD or stable build?

[akutz]

Hi @denverwilliams / @taylor / @lixuna,

FYI, there's a branch that mostly supports deploying to vSphere via the Docker image:

$ docker run \
>   -v $(pwd)/data:/cncf/data \
>   -e VSPHERE_SERVER=$VSPHERE_SERVER \
>   -e VSPHERE_USER=$VSPHERE_USER \
>   -e VSPHERE_PASSWORD=$VSPHERE_PASSWORD \
>   -e VSPHERE_AWS_ACCESS_KEY_ID=$VSPHERE_AWS_ACCESS_KEY_ID \
>   -e VSPHERE_AWS_SECRET_ACCESS_KEY=$VSPHERE_AWS_SECRET_ACCESS_KEY \
>   -e VSPHERE_AWS_REGION=$VSPHERE_AWS_REGION \
>   -e CLOUD=vsphere \
>   -e COMMAND=deploy \
>   -e NAME=cross-cloud \
>   -e BACKEND=file \
>   -ti provisioning

The full log from the above command is here.

However, it hangs here:

Apply complete! Resources: 38 added, 0 changed, 0 destroyed.

The state of your infrastructure has been saved to the path
below. This state is required to modify and destroy your
infrastructure, so keep it safe. To inspect the complete state
use the `terraform show` command.

State path: /cncf/data/cross-cloud/terraform.tfstate

real	2m46.212s
user	0m5.490s
sys	0m1.820s
❤ Trying to connect to cluster with kubectl......

Apparently it cannot communicate with the cluster?

If y'all would like, I can provide you credentials in a secure manner that will let you test as well. Please e-mail me for access, thanks!

[akutz]

Hi @taylor / @denverwilliams,

Your DNS server disallows recursion, and as such the value we're providing to the public IP list for master nodes is failing to resolve beyond the .local CNAME being created:

Query Public DNS Entry

$ dig master.akutz.vsphere.local @147.75.69.23 

; <<>> DiG 9.10.6 <<>> master.akutz.vsphere.local @147.75.69.23
;; global options: +cmd
;; Got answer:
;; WARNING: .local is reserved for Multicast DNS
;; You are currently testing what happens when an mDNS query is leaked to DNS
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 38507
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;master.akutz.vsphere.local.	IN	A

;; ANSWER SECTION:
master.akutz.vsphere.local. 300	IN	CNAME	xapi-20180614203519432300000002-c6821688ccc48489.elb.us-west-2.amazonaws.com.

;; Query time: 96 msec
;; SERVER: 147.75.69.23#53(147.75.69.23)
;; WHEN: Thu Jun 14 15:38:07 CDT 2018
;; MSG SIZE  rcvd: 145

Query CNAME

$ dig  xapi-20180614201228551500000001-04a63f1f82619497.elb.us-west-2.amazonaws.com @147.75.69.23 

; <<>> DiG 9.10.6 <<>> xapi-20180614201228551500000001-04a63f1f82619497.elb.us-west-2.amazonaws.com @147.75.69.23
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 25304
;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;xapi-20180614201228551500000001-04a63f1f82619497.elb.us-west-2.amazonaws.com. IN A

;; Query time: 76 msec
;; SERVER: 147.75.69.23#53(147.75.69.23)
;; WHEN: Thu Jun 14 15:38:55 CDT 2018
;; MSG SIZE  rcvd: 105

Query Public DNS with host

$ host master.akutz.vsphere.local 147.75.69.23
Using domain server:
Name: 147.75.69.23
Address: 147.75.69.23#53
Aliases: 

master.akutz.vsphere.local is an alias for xapi-20180614203519432300000002-c6821688ccc48489.elb.us-west-2.amazonaws.com.
Host xapi-20180614203519432300000002-c6821688ccc48489.elb.us-west-2.amazonaws.com not found: 2(SERVFAIL)
Host xapi-20180614203519432300000002-c6821688ccc48489.elb.us-west-2.amazonaws.com not found: 2(SERVFAIL)

As you can see, host automatically attempts to recurse to find the IP of the AWS load balancer, while it's a two step process with dig.

Either way, it looks like your DNS server isn't happy when provided with an FQDN as opposed to an IP due to recursion being disabled.

[akutz]

Hi @lixuna,

Please ignore this comment for now. I know why it cannot connect to the K8s server. I SSHd into one of the master nodes. K8s isn't running:

core@akutz-master-1 ~ $ sudo lsof -noP | grep LISTEN
systemd      1                  root   52u     IPv6              16160       0t0        TCP *:22 (LISTEN)

We're looking into it.

[denverwilliams]

Hi @figo @akutz
To get around this i would recommend passing the "${ module.lb.host_name }" directly to the kubeconfig module https://github.com/akutz/cross-cloud/blob/feature/vsphere/vsphere/modules.tf#L222 so it looks something like this.

module "kubeconfig" {
  source = "../kubeconfig"

  data_dir = "${ var.data_dir }"
  endpoint = "${ module.lb.host_name }"
  name = "${ var.name }"
  ca = "${ module.tls.ca }"
  client = "${ module.tls.admin }"
  client_key = "${ module.tls.admin_key }"
}

This will give you kubeconfig file that is using the lb_host_name as the cluster api endpoint, which means you won't need to worry about using the dns server for external Client ---> Server Communications.

You will also need to add *.elb.us-west-2.amazonaws.com to the api cert so that the api server will accept the lb name being used directly, this would involve adding it to this line https://github.com/akutz/cross-cloud/blob/feature/vsphere/vsphere/modules.tf#L167 so it looks something like this

tls_apiserver_cert_dns_names = "kubernetes,kubernetes.default,kubernetes.default.svc,kubernetes.default.svc.cluster.local,*.${ var.name }.${ var.cloud_provider }.local,*.elb.us-west-2.amazonaws.com"

[akutz]

Hi @denverwilliams,

Thanks for the reply. We just spoke about this, and I decided to use an elastic IP with the LB so I could pass the IP instead of FQDN. That should work with the existing config, right?

[akutz]

FYI, the reason I think we should stick with IPs are:

• The cert wildcard is based on an AWS region and would have to change with the region.
• The DNS server would have to be updated to allow for lookups against some part of the LB’s FQDN

[denverwilliams]

@akutz Unfortunately you might run into a nasty caveat when using ips directly, because cloud-init data needs to be injected early it is generated before cloud resources are created (Including the certificates which need to have the ip injected into them). So to use the ip directly you would need to be able to reserve/predict what this ip is so that it is available when cloud-init/cert data is created.

In the past when we were using load balances we built out the certificate dns name based on the region by doing something like the following *.elb.${ var.aws.region }.amazonaws.com"

As for the etcd/dns server, there shouldn't need to be any changes for this to work as the LB's FQDN should be publicly resolvable on the internet. So if a client is using the etcd/dns server and a request comes in to resolve the LB's FQDN it will just forward it on to 8.8.8.8 and get a response.

[akutz]

Hi @denverwilliams,

Well, with AWS's elastic IP service we could grab the IP super early and then inform the LB which IP will be used. But even so, I'm not sure how using an IP for public_master_ips is any different than what happens with AWS or OpenStack? Those use IPs now, right? All I'm suggesting is, even sans the elastic IP service, would be supply the IP of the LB to public_master_ips instead of the LB's FQDN. How would that be different than any other provider today?

[akutz]

Hi @denverwilliams,

Okay, my branch now uses an requests an AWS elastic IP and supplies that to the LB. The elastic IP is also assigned to public_master_ips. From my local system I can use kubectl to probe the API server via the LB after I add your DNS server to my local system's /etc/resolv.conf:

$ kubectl --kubeconfig data/kubeconfig get all
NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
service/kubernetes   ClusterIP   100.64.0.1   <none>        443/TCP   15m

However, the deploy still hangs here:

Apply complete! Resources: 39 added, 0 changed, 0 destroyed.

The state of your infrastructure has been saved to the path
below. This state is required to modify and destroy your
infrastructure, so keep it safe. To inspect the complete state
use the `terraform show` command.

State path: /cncf/data/akutz/terraform.tfstate

real	3m8.952s
user	0m7.200s
sys	0m6.970s
❤ Trying to connect to cluster with kubectl......

FWIW, here's what dig returns:

$ dig master.akutz.vsphere.local @147.75.69.23 

; <<>> DiG 9.10.6 <<>> master.akutz.vsphere.local @147.75.69.23
;; global options: +cmd
;; Got answer:
;; WARNING: .local is reserved for Multicast DNS
;; You are currently testing what happens when an mDNS query is leaked to DNS
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 38507
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;master.akutz.vsphere.local.	IN	A

;; ANSWER SECTION:
master.akutz.vsphere.local. 300	IN	CNAME	xapi-20180614203519432300000002-c6821688ccc48489.elb.us-west-2.amazonaws.com.

;; Query time: 96 msec
;; SERVER: 147.75.69.23#53(147.75.69.23)
;; WHEN: Thu Jun 14 15:38:07 CDT 2018
;; MSG SIZE  rcvd: 145

[akutz]

Hi @denverwilliams,

Maybe I'm not waiting long enough for the provision to complete? How long does it generally take once it hits the step that verifies the cluster is up and running? I just ran this manually, and it returns data!

$ kubectl --kubeconfig data/kubeconfig get cs
NAME                 STATUS    MESSAGE              ERROR
scheduler            Healthy   ok                   
controller-manager   Healthy   ok                   
etcd-0               Healthy   {"health": "true"} 

That's the same command that is run from provision.sh, so perhaps I need to wait longer?

[akutz]

Hi @denverwilliams,

FYI, on a new deploy it's stuck waiting:

State path: /cncf/data/akutz/terraform.tfstate

real	2m57.824s
user	0m6.690s
sys	0m6.220s
❤ Trying to connect to cluster with kubectl.................

However, I can query the cluster locally:

$ kubectl --kubeconfig data/kubeconfig get cs
NAME                 STATUS    MESSAGE              ERROR
controller-manager   Healthy   ok                   
scheduler            Healthy   ok                   
etcd-0               Healthy   {"health": "true"}  

At this point I'm just lost as to why the query from my local system works and the one in the container is still waiting. I mean, the _retry function appears to retry every 5.2 seconds, so I don't know why it wouldn't have returned by now...

[akutz]

Hi @denverwilliams,

By the way...

As for the etcd/dns server, there shouldn't need to be any changes for this to work as the LB's FQDN should be publicly resolvable on the internet. So if a client is using the etcd/dns server and a request comes in to resolve the LB's FQDN it will just forward it on to 8.8.8.8 and get a response.

I'm not sure this is accurate based on the evidence I've presented above. Your shared DNS server disallows recursion and refuses to provide an answer when queried with the LB's FQDN. Now, maybe the caller will lookup against your DNS server, fail, and then lookup against Google (8.8.8.8), but it certainly doesn't seem to be happening on the server-side on your DNS server.

You can easily test this:

host

$ host google.com 147.75.69.23
Using domain server:
Name: 147.75.69.23
Address: 147.75.69.23#53
Aliases: 

Host google.com not found: 2(SERVFAIL)

dig

$ dig google.com @147.75.69.23

; <<>> DiG 9.10.6 <<>> google.com @147.75.69.23
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 20551
;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;google.com.			IN	A

;; Query time: 77 msec
;; SERVER: 147.75.69.23#53(147.75.69.23)
;; WHEN: Thu Jun 14 20:10:19 CDT 2018
;; MSG SIZE  rcvd: 39

Both attempts to resolve google.com against your name server fails due to your server not forwarding requests.

[akutz]

Hi @denverwilliams,

So I added this line in provision.sh:

    export KUBECONFIG=${TF_VAR_data_dir}/kubeconfig
    dig $(cat "$KUBECONFIG" | grep server: | awk '{print $2}' | cut -c9-)

Here's the output:

; <<>> DiG 9.10.3-P4-Debian <<>> master.akutz.vsphere.local
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 48652
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;master.akutz.vsphere.local.	IN	A

;; Query time: 7 msec
;; SERVER: 192.168.65.1#53(192.168.65.1)
;; WHEN: Fri Jun 15 02:15:12 UTC 2018
;; MSG SIZE  rcvd: 44

❤ Trying to connect to cluster with kubectl...................

It appears that Docker container running provision.sh is not configured with the shared DNS server, because the query does not appear to know how to resolve master.akutz.vsphere.local.

[akutz]

Hi @denverwilliams,

I've updated the Dockerfile and provision.sh to allow the script to exit to shell in order to do additional debugging. For example:

docker run \
  -v $(pwd)/data:/cncf/data \
  -e VSPHERE_SERVER=$VSPHERE_SERVER \
  -e VSPHERE_USER=$VSPHERE_USER \
  -e VSPHERE_PASSWORD=$VSPHERE_PASSWORD \
  -e VSPHERE_AWS_ACCESS_KEY_ID=$VSPHERE_AWS_ACCESS_KEY_ID \
  -e VSPHERE_AWS_SECRET_ACCESS_KEY=$VSPHERE_AWS_SECRET_ACCESS_KEY \
  -e VSPHERE_AWS_REGION=$VSPHERE_AWS_REGION \
  -e CLOUD=vsphere \
  -e COMMAND=deploy \
  -e NAME=cross-cloud \
  -e BACKEND=file \
  -ti provisioning \
  shell
root@1e15d0d266d1:/cncf#

However, when I examined /etc/resolv.conf, it revealed that the container is unaware of your shared DNS server:

# cat /etc/resolv.conf 
# Generated by dhcpcd from eth0.dhcp
# /etc/resolv.conf.head can replace this line
domain kutz
nameserver 192.168.65.1
# /etc/resolv.conf.tail can replace this line

So I'm curious how kubectl, executed from inside the container, is supposed to be able to resolve master.akutz.vsphere.local or any other host registered on your shared DNS server. Is the container normally started on a system where the host is also aware of the shared DNS server? What am I missing?

cc @figo

[akutz]

Hi @denverwilliams,

So I forgot that Docker containers inherit the DNS of the Docker daemon process, and that /etc/resolv.conf is bind mounted in the container. I finally got it working with the following:

$ docker run \
  --rm \
  --dns 147.75.69.23 --dns 8.8.8.8 \
  -v $(pwd)/data:/cncf/data \
  -e VSPHERE_SERVER=$VSPHERE_SERVER \
  -e VSPHERE_USER=$VSPHERE_USER \
  -e VSPHERE_PASSWORD=$VSPHERE_PASSWORD \
  -e VSPHERE_AWS_ACCESS_KEY_ID=$VSPHERE_AWS_ACCESS_KEY_ID \
  -e VSPHERE_AWS_SECRET_ACCESS_KEY=$VSPHERE_AWS_SECRET_ACCESS_KEY \
  -e VSPHERE_AWS_REGION=$VSPHERE_AWS_REGION \
  -e CLOUD=vsphere \
  -e COMMAND=deploy \
  -e NAME=akutz \
  -e BACKEND=file \
  -ti provisioning

And boom goes the cloud-o-mite:

State path: /cncf/data/akutz/terraform.tfstate

real	2m50.607s
user	0m5.800s
sys	0m6.090s
❤ Trying to connect to cluster with kubectl...✓
❤ Ensure that the kube-system namespaces exists.✓
❤ Ensure that ClusterRoles are available.✓
❤ Ensure that ClusterRoleBindings are available.✓

It works!

Might I recommend to @lixuna and others that the existing examples be updated to note that --dns should be required unless the host from which the provisioner is running is configured with the shared DNS server? Or is this just a Docker-on-macOS-ism?

cc @figo

[akutz]

Hi @lixuna,

FWIW, I added this to my macOS Docker Advanced Settings config file to make sure your DNS server is always present in my containers:

cc @figo

[akutz]

Hi @taylor / @lixuna,

The FAQ indicates the number of cores for each node, but that doesn't really give me an idea of what kind of compute power is expected of a core. A Xeon 2GHz? An i5 1.2GHz? An ARM? I ask because I'm working out resource shares for these nodes, and those are specified in CPU speed, not count.

[akutz]

Hi @taylor / @lixuna / @denverwilliams,

It's working :) And with cloud-init as well! Using @taylor's idea, we created an Ignition bootstrap file to run cloud-init. May I suggest considering the addition of direct Ignition support sometime in the near future? As of now, CoreOS has just deprecated support for cloud-init, but it's only a matter of time before support is removed altogether.

Here's the patch that uses Ignition to bootstrap cloud-init. Perhaps others will find it useful as well.

[lixuna]

Thank you, @akutz - that's a good idea to look into adding direct Ignition support. I've created a new ticket to the issue tracker for the team to review: #149.

[akutz]

Hi @lixuna,

Great, thank you. FYI, @figo and I are working on the docs today, but I fingers-crossed hope to open a PR by the end of the week. I plan to e-mail you (@lixuna) the two credential sets you'll require.

[akutz]

FWIW, re: Ignition support. Maybe it's not such a priority. I honestly didn't look to see what OS the other providers are using. I just assumed CoreOS is popular for this sort of thing.

[lixuna]

Good afternoon, @akutz.

We'll be on the lookout for the PR! I'll send a followup email with a few options for sharing the credentials to the team.

[denverwilliams]

@akutz

To Re-produce the failures for HEAD on cncf.ci you can deploy with the following variables set.

export KUBELET_ARTIFACT=https://gitlab.cncf.ci/kubernetes/kubernetes/-/jobs/104936/artifacts/raw/linux-amd64/kubelet
export KUBE_APISERVER_ARTIFACT=https://gitlab.cncf.ci/kubernetes/kubernetes/-/jobs/104936/artifacts/raw/linux-amd64/kube-apiserver
export KUBE_CONTROLLER_MANAGER_ARTIFACT=https://gitlab.cncf.ci/kubernetes/kubernetes/-/jobs/104936/artifacts/raw/linux-amd64/kube-controller-manager
export KUBE_SCHEDULER_ARTIFACT=https://gitlab.cncf.ci/kubernetes/kubernetes/-/jobs/104936/artifacts/raw/linux-amd64/kube-scheduler
export KUBE_APISERVER_IMAGE=registry.cncf.ci/kubernetes/kubernetes/kube-apiserver-amd64
export KUBE_APISERVER_TAG=master.21271.03a6d0bf
export KUBE_CONTROLLER_MANAGER_IMAGE=registry.cncf.ci/kubernetes/kubernetes/kube-controller-manager-amd64
export KUBE_CONTROLLER_MANAGER_TAG=master.21271.03a6d0bf
export KUBE_SCHEDULER_IMAGE=registry.cncf.ci/kubernetes/kubernetes/kube-scheduler-amd64
export KUBE_SCHEDULER_TAG=master.21271.03a6d0bf
export KUBE_PROXY_IMAGE=registry.cncf.ci/kubernetes/kubernetes/kube-proxy-amd64
export KUBE_PROXY_TAG=master.21271.03a6d0bf
export KUBE_AGGREGATOR_IMAGE=registry.cncf.ci/kubernetes/kubernetes/kube-aggregator-amd64
export KUBE_AGGREGATOR_TAG=master.21271.03a6d0bf
export CLOUD_CONTROLLER_IMAGE=registry.cncf.ci/kubernetes/kubernetes/cloud-controller-manager-amd64
export CLOUD_CONTROLLER_TAG=master.21271.03a6d0bf

[lixuna]

added in v1.4.0