Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Quick test doesn't seem to run? #624

Closed
andrewmyhre opened this issue Mar 6, 2019 · 6 comments
Closed

Quick test doesn't seem to run? #624

andrewmyhre opened this issue Mar 6, 2019 · 6 comments

Comments

@andrewmyhre
Copy link

What steps did you take and what happened:
I'm attempting to run the --mode=Quick test(s) but they don't seem to be executing in my cluster. For context, I'm in an air-gapped environment so I'm passing a flag for the sonobuoy image, and both the sonobuoy and sonobuoy-e2e-job-xxx pods do enter a Running state. However they don't appear to do anything.

The command I'm executing is:

sonobuoy run --sonobuoy-image=<our registry>/heptio-images/sonobuoy:v0.13.0 --kube-conformance-image=<our registry>/heptio-images/kube-conformance:v1.12 --mode=Quick

I then see the pods start:

kubectl -n heptio-sonobuoy get all                                                                                                        c4b301cf7495: Wed Mar  6 09:42:35 2019

NAME                                    READY   STATUS    RESTARTS   AGE
pod/sonobuoy                            1/1     Running   0          3m3s
pod/sonobuoy-e2e-job-8f78c2821ec5483c   2/2     Running   0          3m2s

NAME                      TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
service/sonobuoy-master   ClusterIP   192.168.26.218   <none>        8080/TCP   3m3s

Viewing the logs for the sonobuoy job pod:

kubectl -n heptio-sonobuoy logs -f sonobuoy-e2e-job-8f78c2821ec5483c e2e
/usr/local/bin/ginkgo --focus=Pods should be submitted and removed --skip=Alpha|\[(Disruptive|Feature:[^\]]+|Flaky)\] --noColor=true --nodes=1 /usr/local/bin/e2e.test -- --disable-log-dump --repo-root=/kubernetes --provider="local" --report-dir="/tmp/results" --kubeconfig=""
Mar  6 14:39:44.121: INFO: Overriding default scale value of zero to 1
Mar  6 14:39:44.121: INFO: Overriding default milliseconds value of zero to 5000
I0306 14:39:45.077331      14 test_context.go:385] Using a temporary kubeconfig file from in-cluster config : /tmp/kubeconfig-233140528
I0306 14:39:45.078019      14 e2e.go:304] Starting e2e run "aead8f44-401d-11e9-b086-a6c3fe8bb523" on Ginkgo node 1
Running Suite: Kubernetes e2e suite
===================================
Random Seed: 1551883183 - Will randomize all specs
Will run 1 of 1814 specs

Mar  6 14:39:45.423: INFO: >>> kubeConfig: /tmp/kubeconfig-233140528
Mar  6 14:39:45.439: INFO: Waiting up to 30m0s for all (but 0) nodes to be schedulable
kubectl -n heptio-sonobuoy logs -f sonobuoy-e2e-job-8f78c2821ec5483c sonobuoy-worker
time="2019-03-06T14:39:47Z" level=info msg="Waiting for waitfile" waitfile=/tmp/results/done

The message Waiting up to 30m0s for all (but 0) nodes to be schedulable seems relevant.

What did you expect to happen:
Not sure.. some logs indicating tests are being run? I'm new to sonobuoy so I'm not sure what to expect.

Anything else you would like to add:
All nodes including workers are healthy and schedulable.

Environment:

  • Sonobuoy version: v0.13.0
  • Kubernetes version: v1.12.3
  • Kubernetes installer & version: custom KTHW implementation
  • Cloud provider or hardware configuration: EC2
  • OS (e.g. from /etc/os-release): ubuntu 16.04
  • Sonobuoy tarball (which contains * below)
@johnSchnake
Copy link
Contributor

Yes this seems to be the same issue as #599 . There is an upstream issue created to try and allow custom taints on nodes, but otherwise each node in your cluster needs to report as schedulable unless it is labeled as a master node.

Is that accurate, that you have custom taints/labels on the nodes? You can check with kubectl get nodes -o yaml

There you can look at all the labels/spec settings. If it doesn't have the label node-role.kubernetes.io/master then it needs to be schedulable. Again, we're hoping to get this more flexible in the future (see the linked ticket which points upstream)

@andrewmyhre
Copy link
Author

Our masters have node-role.kubernetes.io/master=true and our workers have node-role.kubernetes.io/worker=true. Our masters are tainted with NoSchedule but our workers are not. When you say the nodes need to be schedulable do you mean not having the NoSchedule taint? If I understand you correctly I believe the workers satisfy the conditions.

@johnSchnake
Copy link
Contributor

That does seem OK: see https://github.com/kubernetes/kubernetes/blob/master/test/e2e/framework/util.go#L2750-L2758

The methods for checking the remaining pods check the following:

  • doesn't have "unschedulable" field set [check in kubectl get nodes -o yaml for this]
  • it's Ready condition is set to true [check in kubectl get nodes for this]
  • doesn't have NetworkUnavailable condition set to true [ I think this is also in the yaml node info]
  • a pod can be scheduled on it (it creates a fake pod and checks if it could be placed there). [I think this one is tricker to test manually, not sure the simplest route for this]

When you say the nodes need to be schedulable do you mean not having the NoSchedule taint?

That is just referring to the last item in the list above. It really just creates a fake pod and checks that it would be tolerated on that node. Effectively a pod without any special tolerations should be able to be scheduled there.

@johnSchnake
Copy link
Contributor

Could you provide the output of kubectl get nodes -o yaml? That would hold all the info necessary to rule those things out I think.

@andrewmyhre
Copy link
Author

Sure:

apiVersion: v1
items:
- apiVersion: v1
  kind: Node
  metadata:
    annotations:
      node.alpha.kubernetes.io/ttl: "0"
      projectcalico.org/IPv4Address: 10.205.232.148/22
      volumes.kubernetes.io/controller-managed-attach-detach: "true"
    creationTimestamp: "2019-03-06T13:55:17Z"
    labels:
      beta.kubernetes.io/arch: amd64
      beta.kubernetes.io/instance-type: m5.large
      beta.kubernetes.io/os: linux
      failure-domain.beta.kubernetes.io/region: us-east-1
      failure-domain.beta.kubernetes.io/zone: us-east-1a
      kubernetes.io/hostname: ip-10-205-232-148.ec2.internal
      node-role.kubernetes.io/worker: "true"
    name: ip-10-205-232-148.ec2.internal
    resourceVersion: "70544"
    selfLink: /api/v1/nodes/ip-10-205-232-148.ec2.internal
    uid: 79135417-4017-11e9-9764-0e5f588b4326
  spec:
    podCIDR: 192.168.135.0/24
    providerID: aws:///us-east-1a/i-060c78100f67af22e
  status:
    addresses:
    - address: 10.205.232.148
      type: InternalIP
    - address: ip-10-205-232-148.dqa.ourdomain.com
      type: InternalDNS
    - address: ip-10-205-232-148.ec2.internal
      type: Hostname
    allocatable:
      attachable-volumes-aws-ebs: "25"
      cpu: "2"
      ephemeral-storage: "93619943269"
      hugepages-1Gi: "0"
      hugepages-2Mi: "0"
      memory: 7770048Ki
      pods: "110"
    capacity:
      attachable-volumes-aws-ebs: "25"
      cpu: "2"
      ephemeral-storage: 101584140Ki
      hugepages-1Gi: "0"
      hugepages-2Mi: "0"
      memory: 7872448Ki
      pods: "110"
    conditions:
    - lastHeartbeatTime: "2019-03-06T19:13:10Z"
      lastTransitionTime: "2019-03-06T13:55:17Z"
      message: kubelet has sufficient disk space available
      reason: KubeletHasSufficientDisk
      status: "False"
      type: OutOfDisk
    - lastHeartbeatTime: "2019-03-06T19:13:10Z"
      lastTransitionTime: "2019-03-06T13:55:17Z"
      message: kubelet has sufficient memory available
      reason: KubeletHasSufficientMemory
      status: "False"
      type: MemoryPressure
    - lastHeartbeatTime: "2019-03-06T19:13:10Z"
      lastTransitionTime: "2019-03-06T13:55:17Z"
      message: kubelet has no disk pressure
      reason: KubeletHasNoDiskPressure
      status: "False"
      type: DiskPressure
    - lastHeartbeatTime: "2019-03-06T19:13:10Z"
      lastTransitionTime: "2019-03-06T13:55:17Z"
      message: kubelet has sufficient PID available
      reason: KubeletHasSufficientPID
      status: "False"
      type: PIDPressure
    - lastHeartbeatTime: "2019-03-06T19:13:10Z"
      lastTransitionTime: "2019-03-06T13:56:07Z"
      message: kubelet is posting ready status. AppArmor enabled
      reason: KubeletReady
      status: "True"
      type: Ready
    daemonEndpoints:
      kubeletEndpoint:
        Port: 10250
    images:
    - names:
      - our.docker.registry/heptio-images/kube-conformance@sha256:9ae818b51720e6fba16205c3b648d16ed9d92ac9b18245b0cd006df46cfaa931
      - our.docker.registry/heptio-images/kube-conformance:v1.12
      sizeBytes: 548682100
    - names:
      - our.docker.registry/co.elastic.docker/kibana-oss@sha256:e1fa3f513ce258b9b531233450c10c765d587e22b21b416ba71a651cfb2c6a1c
      - our.docker.registry/co.elastic.docker/kibana-oss:6.6.0
      sizeBytes: 459030535
    - names:
      - our.docker.registry/entkubernetes/datadog-agent@sha256:e04cc178565977c606ef8ef4bcd385d62ea800ba66d8c0be8014eb8e95f27f2e
      - our.docker.registry/entkubernetes/datadog-agent:6.9.0
      sizeBytes: 414298618
    - names:
      - our.docker.registry/entkubernetes/nginx-ingress-controller@sha256:63b48dd3ba635aa65409f7d6a319e293697810c1b9e885e6255bebf36a2506ab
      - our.docker.registry/entkubernetes/nginx-ingress-controller:0.16.2
      sizeBytes: 361632894
    - names:
      - qualys/sensor:1.1.5
      sizeBytes: 351318875
    - names:
      - qualys/sensor:1.2.4
      - qualys/sensor:latest
      sizeBytes: 292121056
    - names:
      - our.docker.registry/prom-official/prometheus@sha256:b32864cfe5f7f5d146820ddc6dcc5f78f40d8ab6d55e80decd5fc222b16344dc
      - our.docker.registry/prom-official/prometheus:v2.7.1
      sizeBytes: 101135325
    - names:
      - our.docker.registry/entkubernetes/kube-proxy-amd64@sha256:5f2d707e25ebd80b091cf50b8e3a2ce2dead89cf625e00176e1164c4479f3e90
      - our.docker.registry/entkubernetes/kube-proxy-amd64:v1.12.3
      sizeBytes: 96469663
    - names:
      - our.docker.registry/calico/node@sha256:624ea7f603aa07cd6a1ede662a79554a3b7d38cf079347fb9be6881c0ba643ca
      - our.docker.registry/calico/node:v3.2.3
      sizeBytes: 75156626
    - names:
      - our.docker.registry/calico/cni@sha256:4c637d8021e7cf0b99a104e8383c4b1c26ecaa0f55b7391dae8138bdbb5277a2
      - our.docker.registry/calico/cni:v3.2.3
      sizeBytes: 75086078
    - names:
      - our.docker.registry/entkubernetes/fluent-bit@sha256:4db3ae2dca4eace5a933a890a3ad35d2282b9640b168ef914bd7a5b07d1236ad
      - our.docker.registry/entkubernetes/fluent-bit:1.0.4
      sizeBytes: 53039438
    - names:
      - our.docker.registry/entkubernetes/node-operators@sha256:54fccfaf2e4d0a8787c828467084b239755cb60ed6b083d1a224839fef3f3888
      - our.docker.registry/entkubernetes/node-operators:0.1.76
      sizeBytes: 42878515
    - names:
      - our.docker.registry/entkubernetes/maintenance-controller@sha256:9962f6a23261465e1199f7810b25b85ac6e0af0cbb2bb274275518daae0cb8ef
      - our.docker.registry/entkubernetes/maintenance-controller:0.1.76
      sizeBytes: 39390336
    - names:
      - our.docker.registry/entkubernetes/kube2iam@sha256:3fcf3bcd2a400f9d7300a2943bdeb2e130adf693963ecd8dd82de664a9f2967b
      - our.docker.registry/entkubernetes/kube2iam:0.10.0
      sizeBytes: 34025676
    - names:
      - our.docker.registry/heptio-images/sonobuoy@sha256:f9b879fca17b050f6fb8f1e36020ed32139cda2cde339b9dbe2c1a9a9ab1eaaa
      - our.docker.registry/heptio-images/sonobuoy:v0.13.0
      sizeBytes: 33410348
    - names:
      - our.docker.registry/prom-official/configmap-reload@sha256:befec9f23d2a9da86a298d448cc9140f56a457362a7d9eecddba192db1ab489e
      - our.docker.registry/prom-official/configmap-reload:v0.2.2
      sizeBytes: 22381476
    - names:
      - our.docker.registry/prom-official/node-exporter@sha256:c390c8fea4cd362a28ad5070aedd6515aacdfdffd21de6db42ead05e332be5a9
      - our.docker.registry/prom-official/node-exporter:v0.17.0
      sizeBytes: 20982005
    - names:
      - our.docker.registry/entkubernetes/defaultbackend@sha256:5e635236018ac20db4a64f4b4a89b0ceea7867fea372984c01477b1522deea8b
      - our.docker.registry/entkubernetes/defaultbackend:1.3
      sizeBytes: 3618880
    - names:
      - our.docker.registry/public-cloud/busybox@sha256:29f5d56d12684887bdfa50dcd29fc31eea4aaf4ad3bec43daf19026a7ce69912
      - our.docker.registry/public-cloud/busybox:latest
      sizeBytes: 1093484
    - names:
      - our.docker.registry/entkubernetes/pause-amd64@sha256:fcaff905397ba63fd376d0c3019f1f1cb6e7506131389edbcb3d22719f1ae54d
      - our.docker.registry/entkubernetes/pause-amd64:3.1
      sizeBytes: 742472
    nodeInfo:
      architecture: amd64
      bootID: 4431d1b5-ffa7-4e0e-a06e-cbaf84cee1c5
      containerRuntimeVersion: docker://18.9.3
      kernelVersion: 4.4.0-1072-aws
      kubeProxyVersion: v1.12.3
      kubeletVersion: v1.12.3
      machineID: 8b9166bc791441aca27058bd8b01dd5e
      operatingSystem: linux
      osImage: Ubuntu 16.04.5 LTS
      systemUUID: EC2C6975-100D-CE87-B7F5-B82372F3FE12
    volumesAttached:
    - devicePath: /dev/xvdbr
      name: kubernetes.io/aws-ebs/aws://us-east-1a/vol-047eb43442ccaee5c
    volumesInUse:
    - kubernetes.io/aws-ebs/aws://us-east-1a/vol-047eb43442ccaee5c
- apiVersion: v1
  kind: Node
  metadata:
    annotations:
      node.alpha.kubernetes.io/ttl: "0"
      projectcalico.org/IPv4Address: 10.205.232.247/22
      volumes.kubernetes.io/controller-managed-attach-detach: "true"
    creationTimestamp: "2019-03-06T13:55:17Z"
    labels:
      beta.kubernetes.io/arch: amd64
      beta.kubernetes.io/instance-type: m5.large
      beta.kubernetes.io/os: linux
      failure-domain.beta.kubernetes.io/region: us-east-1
      failure-domain.beta.kubernetes.io/zone: us-east-1a
      kubernetes.io/hostname: ip-10-205-232-247.ec2.internal
      node-role.kubernetes.io/worker: "true"
    name: ip-10-205-232-247.ec2.internal
    resourceVersion: "70531"
    selfLink: /api/v1/nodes/ip-10-205-232-247.ec2.internal
    uid: 791f95b4-4017-11e9-9764-0e5f588b4326
  spec:
    podCIDR: 192.168.136.0/24
    providerID: aws:///us-east-1a/i-047920fa3167663fb
  status:
    addresses:
    - address: 10.205.232.247
      type: InternalIP
    - address: ip-10-205-232-247.dqa.ourdomain.com
      type: InternalDNS
    - address: ip-10-205-232-247.ec2.internal
      type: Hostname
    allocatable:
      attachable-volumes-aws-ebs: "25"
      cpu: "2"
      ephemeral-storage: "93619943269"
      hugepages-1Gi: "0"
      hugepages-2Mi: "0"
      memory: 7770048Ki
      pods: "110"
    capacity:
      attachable-volumes-aws-ebs: "25"
      cpu: "2"
      ephemeral-storage: 101584140Ki
      hugepages-1Gi: "0"
      hugepages-2Mi: "0"
      memory: 7872448Ki
      pods: "110"
    conditions:
    - lastHeartbeatTime: "2019-03-06T19:13:05Z"
      lastTransitionTime: "2019-03-06T13:55:17Z"
      message: kubelet has sufficient disk space available
      reason: KubeletHasSufficientDisk
      status: "False"
      type: OutOfDisk
    - lastHeartbeatTime: "2019-03-06T19:13:05Z"
      lastTransitionTime: "2019-03-06T13:55:17Z"
      message: kubelet has sufficient memory available
      reason: KubeletHasSufficientMemory
      status: "False"
      type: MemoryPressure
    - lastHeartbeatTime: "2019-03-06T19:13:05Z"
      lastTransitionTime: "2019-03-06T13:55:17Z"
      message: kubelet has no disk pressure
      reason: KubeletHasNoDiskPressure
      status: "False"
      type: DiskPressure
    - lastHeartbeatTime: "2019-03-06T19:13:05Z"
      lastTransitionTime: "2019-03-06T13:55:17Z"
      message: kubelet has sufficient PID available
      reason: KubeletHasSufficientPID
      status: "False"
      type: PIDPressure
    - lastHeartbeatTime: "2019-03-06T19:13:05Z"
      lastTransitionTime: "2019-03-06T13:56:07Z"
      message: kubelet is posting ready status. AppArmor enabled
      reason: KubeletReady
      status: "True"
      type: Ready
    daemonEndpoints:
      kubeletEndpoint:
        Port: 10250
    images:
    - names:
      - our.docker.registry/co.elastic.docker/elasticsearch-oss@sha256:f59f7936d018e9d2329dfa0705b305e2c8596045e80039108d67218ae5a967bc
      - our.docker.registry/co.elastic.docker/elasticsearch-oss:6.6.0
      sizeBytes: 678455631
    - names:
      - our.docker.registry/entkubernetes/datadog-agent@sha256:e04cc178565977c606ef8ef4bcd385d62ea800ba66d8c0be8014eb8e95f27f2e
      - our.docker.registry/entkubernetes/datadog-agent:6.9.0
      sizeBytes: 414298618
    - names:
      - qualys/sensor:1.1.5
      sizeBytes: 351318875
    - names:
      - qualys/sensor:1.2.4
      - qualys/sensor:latest
      sizeBytes: 292121056
    - names:
      - our.docker.registry/entkubernetes/fluentd@sha256:d018433d68720687bbe6aa5c24a4e215240b278c723b5d042d857ab3bd758651
      - our.docker.registry/entkubernetes/fluentd:v1.2.2-debian-kinesis
      sizeBytes: 252041588
    - names:
      - our.docker.registry/entkubernetes/cluster-autoscaler@sha256:7ff5a60304b344f2f29c804c7253632bbc818794f6932236a56db107a6a8f5af
      - our.docker.registry/entkubernetes/cluster-autoscaler:v1.13.1
      sizeBytes: 136618018
    - names:
      - our.docker.registry/entkubernetes/kube-proxy-amd64@sha256:5f2d707e25ebd80b091cf50b8e3a2ce2dead89cf625e00176e1164c4479f3e90
      - our.docker.registry/entkubernetes/kube-proxy-amd64:v1.12.3
      sizeBytes: 96469663
    - names:
      - our.docker.registry/entkubernetes/tiller@sha256:cab750b402d24dd7b24756858c31eae6a007cd0ee91ea802b3891e2e940d214d
      - our.docker.registry/entkubernetes/tiller:v2.12.3
      sizeBytes: 81415168
    - names:
      - our.docker.registry/entkubernetes/heapster@sha256:59fb34ffd902282d06fcc940a906df9787edf78651743f4c8c4abf8b3468c0e9
      - our.docker.registry/entkubernetes/heapster:v1.5.2
      sizeBytes: 75318342
    - names:
      - our.docker.registry/calico/node@sha256:624ea7f603aa07cd6a1ede662a79554a3b7d38cf079347fb9be6881c0ba643ca
      - our.docker.registry/calico/node:v3.2.3
      sizeBytes: 75156626
    - names:
      - our.docker.registry/calico/cni@sha256:4c637d8021e7cf0b99a104e8383c4b1c26ecaa0f55b7391dae8138bdbb5277a2
      - our.docker.registry/calico/cni:v3.2.3
      sizeBytes: 75086078
    - names:
      - our.docker.registry/entkubernetes/fluent-bit@sha256:4db3ae2dca4eace5a933a890a3ad35d2282b9640b168ef914bd7a5b07d1236ad
      - our.docker.registry/entkubernetes/fluent-bit:1.0.4
      sizeBytes: 53039438
    - names:
      - our.docker.registry/entkubernetes/node-operators@sha256:54fccfaf2e4d0a8787c828467084b239755cb60ed6b083d1a224839fef3f3888
      - our.docker.registry/entkubernetes/node-operators:0.1.76
      sizeBytes: 42878515
    - names:
      - our.docker.registry/entkubernetes/maintenance-controller@sha256:9962f6a23261465e1199f7810b25b85ac6e0af0cbb2bb274275518daae0cb8ef
      - our.docker.registry/entkubernetes/maintenance-controller:0.1.76
      sizeBytes: 39390336
    - names:
      - our.docker.registry/entkubernetes/addon-resizer@sha256:dcec9a5c2e20b8df19f3e9eeb87d9054a9e94e71479b935d5cfdbede9ce15895
      - our.docker.registry/entkubernetes/addon-resizer:1.7
      sizeBytes: 38983736
    - names:
      - our.docker.registry/prom-official/alertmanager@sha256:196af0317d3449c1300aa26ff0366f68c67d04581d2c9f8609cbb227424e226c
      - our.docker.registry/prom-official/alertmanager:v0.15.3
      sizeBytes: 34526872
    - names:
      - our.docker.registry/entkubernetes/kube2iam@sha256:3fcf3bcd2a400f9d7300a2943bdeb2e130adf693963ecd8dd82de664a9f2967b
      - our.docker.registry/entkubernetes/kube2iam:0.10.0
      sizeBytes: 34025676
    - names:
      - our.docker.registry/entkubernetes/maintenance-scheduler@sha256:cd692ac60e9a4377d721630044913df0f1ec195d674c3114b76ec19a2c5e33e9
      - our.docker.registry/entkubernetes/maintenance-scheduler:0.1.76
      sizeBytes: 32750687
    - names:
      - our.docker.registry/entkubernetes/kube-state-metrics@sha256:37c4ddfa160043a7fd73703b7cae41cf38b692ca19c1dff722293c90a386b59f
      - our.docker.registry/entkubernetes/kube-state-metrics:v1.5.0
      sizeBytes: 31825332
    - names:
      - our.docker.registry/prom-official/configmap-reload@sha256:befec9f23d2a9da86a298d448cc9140f56a457362a7d9eecddba192db1ab489e
      - our.docker.registry/prom-official/configmap-reload:v0.2.2
      sizeBytes: 22381476
    - names:
      - our.docker.registry/prom-official/node-exporter@sha256:c390c8fea4cd362a28ad5070aedd6515aacdfdffd21de6db42ead05e332be5a9
      - our.docker.registry/prom-official/node-exporter:v0.17.0
      sizeBytes: 20982005
    - names:
      - our.docker.registry/prom-official/pushgateway@sha256:04d01442ef40796ca7fc97735de8f7b9a87a9a7db8b7e81ca64d5adf44608e92
      - our.docker.registry/prom-official/pushgateway:v0.6.0
      sizeBytes: 16527231
    - names:
      - our.docker.registry/entkubernetes/ingress-default-go@sha256:fca39212ec9680f269b180d37064680a8d09ab084a81746e96079b67991593dd
      - our.docker.registry/entkubernetes/ingress-default-go:0.0.1
      sizeBytes: 6674461
    - names:
      - our.docker.registry/public-cloud/busybox@sha256:29f5d56d12684887bdfa50dcd29fc31eea4aaf4ad3bec43daf19026a7ce69912
      - our.docker.registry/public-cloud/busybox:latest
      sizeBytes: 1093484
    - names:
      - our.docker.registry/entkubernetes/pause-amd64@sha256:fcaff905397ba63fd376d0c3019f1f1cb6e7506131389edbcb3d22719f1ae54d
      - our.docker.registry/entkubernetes/pause-amd64:3.1
      sizeBytes: 742472
    nodeInfo:
      architecture: amd64
      bootID: b273e661-7c72-4532-9ba9-03997349b7e1
      containerRuntimeVersion: docker://18.9.3
      kernelVersion: 4.4.0-1072-aws
      kubeProxyVersion: v1.12.3
      kubeletVersion: v1.12.3
      machineID: 8b9166bc791441aca27058bd8b01dd5e
      operatingSystem: linux
      osImage: Ubuntu 16.04.5 LTS
      systemUUID: EC2E52CE-B0FB-3E38-C554-D8FF60FF8155
    volumesAttached:
    - devicePath: /dev/xvdbo
      name: kubernetes.io/aws-ebs/aws://us-east-1a/vol-050abe91a76c1cd6f
    - devicePath: /dev/xvdcp
      name: kubernetes.io/aws-ebs/aws://us-east-1a/vol-05bb1aac307201e93
    - devicePath: /dev/xvdbj
      name: kubernetes.io/aws-ebs/aws://us-east-1a/vol-0a2ad69d3b3f4ddfc
    - devicePath: /dev/xvdbg
      name: kubernetes.io/aws-ebs/aws://us-east-1a/vol-08ebe0dd826b96b8f
    volumesInUse:
    - kubernetes.io/aws-ebs/aws://us-east-1a/vol-050abe91a76c1cd6f
    - kubernetes.io/aws-ebs/aws://us-east-1a/vol-05bb1aac307201e93
    - kubernetes.io/aws-ebs/aws://us-east-1a/vol-08ebe0dd826b96b8f
    - kubernetes.io/aws-ebs/aws://us-east-1a/vol-0a2ad69d3b3f4ddfc
- apiVersion: v1
  kind: Node
  metadata:
    annotations:
      node.alpha.kubernetes.io/ttl: "0"
      projectcalico.org/IPv4Address: 10.205.233.23/22
      volumes.kubernetes.io/controller-managed-attach-detach: "true"
    creationTimestamp: "2019-03-06T13:54:01Z"
    labels:
      beta.kubernetes.io/arch: amd64
      beta.kubernetes.io/instance-type: c5.large
      beta.kubernetes.io/os: linux
      failure-domain.beta.kubernetes.io/region: us-east-1
      failure-domain.beta.kubernetes.io/zone: us-east-1a
      kubernetes.io/hostname: ip-10-205-233-23.ec2.internal
      node-role.kubernetes.io/etcd: "true"
    name: ip-10-205-233-23.ec2.internal
    resourceVersion: "70550"
    selfLink: /api/v1/nodes/ip-10-205-233-23.ec2.internal
    uid: 4bbe6380-4017-11e9-9764-0e5f588b4326
  spec:
    podCIDR: 192.168.130.0/24
    providerID: aws:///us-east-1a/i-0d1f86efbf8fe4f2f
    taints:
    - effect: NoSchedule
      key: dedicated
      value: master
  status:
    addresses:
    - address: 10.205.233.23
      type: InternalIP
    - address: ip-10-205-233-23.dqa.ourdomain.com
      type: InternalDNS
    - address: ip-10-205-233-23.ec2.internal
      type: Hostname
    allocatable:
      attachable-volumes-aws-ebs: "25"
      cpu: "2"
      ephemeral-storage: "93619943269"
      hugepages-1Gi: "0"
      hugepages-2Mi: "0"
      memory: 3698628Ki
      pods: "110"
    capacity:
      attachable-volumes-aws-ebs: "25"
      cpu: "2"
      ephemeral-storage: 101584140Ki
      hugepages-1Gi: "0"
      hugepages-2Mi: "0"
      memory: 3801028Ki
      pods: "110"
    conditions:
    - lastHeartbeatTime: "2019-03-06T19:13:11Z"
      lastTransitionTime: "2019-03-06T13:54:01Z"
      message: kubelet has sufficient disk space available
      reason: KubeletHasSufficientDisk
      status: "False"
      type: OutOfDisk
    - lastHeartbeatTime: "2019-03-06T19:13:11Z"
      lastTransitionTime: "2019-03-06T13:54:01Z"
      message: kubelet has sufficient memory available
      reason: KubeletHasSufficientMemory
      status: "False"
      type: MemoryPressure
    - lastHeartbeatTime: "2019-03-06T19:13:11Z"
      lastTransitionTime: "2019-03-06T13:54:01Z"
      message: kubelet has no disk pressure
      reason: KubeletHasNoDiskPressure
      status: "False"
      type: DiskPressure
    - lastHeartbeatTime: "2019-03-06T19:13:11Z"
      lastTransitionTime: "2019-03-06T13:54:01Z"
      message: kubelet has sufficient PID available
      reason: KubeletHasSufficientPID
      status: "False"
      type: PIDPressure
    - lastHeartbeatTime: "2019-03-06T19:13:11Z"
      lastTransitionTime: "2019-03-06T13:56:01Z"
      message: kubelet is posting ready status. AppArmor enabled
      reason: KubeletReady
      status: "True"
      type: Ready
    daemonEndpoints:
      kubeletEndpoint:
        Port: 10250
    images:
    - names:
      - qualys/sensor:1.1.5
      sizeBytes: 351318875
    - names:
      - qualys/sensor:1.2.4
      - qualys/sensor:latest
      sizeBytes: 292121056
    - names:
      - our.docker.registry/entkubernetes/kube-proxy-amd64@sha256:5f2d707e25ebd80b091cf50b8e3a2ce2dead89cf625e00176e1164c4479f3e90
      - our.docker.registry/entkubernetes/kube-proxy-amd64:v1.12.3
      sizeBytes: 96469663
    - names:
      - our.docker.registry/calico/node@sha256:624ea7f603aa07cd6a1ede662a79554a3b7d38cf079347fb9be6881c0ba643ca
      - our.docker.registry/calico/node:v3.2.3
      sizeBytes: 75156626
    - names:
      - our.docker.registry/calico/cni@sha256:4c637d8021e7cf0b99a104e8383c4b1c26ecaa0f55b7391dae8138bdbb5277a2
      - our.docker.registry/calico/cni:v3.2.3
      sizeBytes: 75086078
    - names:
      - our.docker.registry/entkubernetes/fluent-bit@sha256:4db3ae2dca4eace5a933a890a3ad35d2282b9640b168ef914bd7a5b07d1236ad
      - our.docker.registry/entkubernetes/fluent-bit:1.0.4
      sizeBytes: 53039438
    - names:
      - our.docker.registry/entkubernetes/etcd@sha256:4bcd880f42a7f243530943c6c707d3419d80906d60fc3f6f77b9709ef3cc5e71
      - our.docker.registry/entkubernetes/etcd:v3.3.0
      sizeBytes: 39420828
    - names:
      - our.docker.registry/entkubernetes/kube2iam@sha256:3fcf3bcd2a400f9d7300a2943bdeb2e130adf693963ecd8dd82de664a9f2967b
      - our.docker.registry/entkubernetes/kube2iam:0.10.0
      sizeBytes: 34025676
    - names:
      - our.docker.registry/entkubernetes/pause-amd64@sha256:fcaff905397ba63fd376d0c3019f1f1cb6e7506131389edbcb3d22719f1ae54d
      - our.docker.registry/entkubernetes/pause-amd64:3.1
      sizeBytes: 742472
    nodeInfo:
      architecture: amd64
      bootID: 8de5a938-12f5-4107-9aeb-671b6ec11b12
      containerRuntimeVersion: docker://18.9.3
      kernelVersion: 4.4.0-1072-aws
      kubeProxyVersion: v1.12.3
      kubeletVersion: v1.12.3
      machineID: 8b9166bc791441aca27058bd8b01dd5e
      operatingSystem: linux
      osImage: Ubuntu 16.04.5 LTS
      systemUUID: EC24DA50-4B93-699B-F885-4110875E29C8
- apiVersion: v1
  kind: Node
  metadata:
    annotations:
      node.alpha.kubernetes.io/ttl: "0"
      projectcalico.org/IPv4Address: 10.205.233.62/22
      volumes.kubernetes.io/controller-managed-attach-detach: "true"
    creationTimestamp: "2019-03-06T13:55:32Z"
    labels:
      beta.kubernetes.io/arch: amd64
      beta.kubernetes.io/instance-type: m5.large
      beta.kubernetes.io/os: linux
      failure-domain.beta.kubernetes.io/region: us-east-1
      failure-domain.beta.kubernetes.io/zone: us-east-1a
      kubernetes.io/hostname: ip-10-205-233-62.ec2.internal
      node-role.kubernetes.io/master: "true"
    name: ip-10-205-233-62.ec2.internal
    resourceVersion: "70528"
    selfLink: /api/v1/nodes/ip-10-205-233-62.ec2.internal
    uid: 82087f8c-4017-11e9-9764-0e5f588b4326
  spec:
    podCIDR: 192.168.139.0/24
    providerID: aws:///us-east-1a/i-07824f4c83a65bfc7
    taints:
    - effect: NoSchedule
      key: dedicated
      value: master
  status:
    addresses:
    - address: 10.205.233.62
      type: InternalIP
    - address: ip-10-205-233-62.dqa.ourdomain.com
      type: InternalDNS
    - address: ip-10-205-233-62.ec2.internal
      type: Hostname
    allocatable:
      attachable-volumes-aws-ebs: "25"
      cpu: "2"
      ephemeral-storage: "93619943269"
      hugepages-1Gi: "0"
      hugepages-2Mi: "0"
      memory: 7770048Ki
      pods: "110"
    capacity:
      attachable-volumes-aws-ebs: "25"
      cpu: "2"
      ephemeral-storage: 101584140Ki
      hugepages-1Gi: "0"
      hugepages-2Mi: "0"
      memory: 7872448Ki
      pods: "110"
    conditions:
    - lastHeartbeatTime: "2019-03-06T19:13:05Z"
      lastTransitionTime: "2019-03-06T13:55:32Z"
      message: kubelet has sufficient disk space available
      reason: KubeletHasSufficientDisk
      status: "False"
      type: OutOfDisk
    - lastHeartbeatTime: "2019-03-06T19:13:05Z"
      lastTransitionTime: "2019-03-06T13:55:32Z"
      message: kubelet has sufficient memory available
      reason: KubeletHasSufficientMemory
      status: "False"
      type: MemoryPressure
    - lastHeartbeatTime: "2019-03-06T19:13:05Z"
      lastTransitionTime: "2019-03-06T13:55:32Z"
      message: kubelet has no disk pressure
      reason: KubeletHasNoDiskPressure
      status: "False"
      type: DiskPressure
    - lastHeartbeatTime: "2019-03-06T19:13:05Z"
      lastTransitionTime: "2019-03-06T13:55:32Z"
      message: kubelet has sufficient PID available
      reason: KubeletHasSufficientPID
      status: "False"
      type: PIDPressure
    - lastHeartbeatTime: "2019-03-06T19:13:05Z"
      lastTransitionTime: "2019-03-06T13:56:02Z"
      message: kubelet is posting ready status. AppArmor enabled
      reason: KubeletReady
      status: "True"
      type: Ready
    daemonEndpoints:
      kubeletEndpoint:
        Port: 10250
    images:
    - names:
      - qualys/sensor:1.1.5
      sizeBytes: 351318875
    - names:
      - qualys/sensor:1.2.4
      - qualys/sensor:latest
      sizeBytes: 292121056
    - names:
      - our.docker.registry/entkubernetes/kube-apiserver-amd64@sha256:ad7ac6a924d88e7d4e1026233981ad0202812e6d9c5e30d37e689f82f9f4c2af
      - our.docker.registry/entkubernetes/kube-apiserver-amd64:v1.12.3
      sizeBytes: 193948168
    - names:
      - our.docker.registry/entkubernetes/kube-controller-manager-amd64@sha256:df1609649a677aea514c8a930a91287aa503646cade31209ce5735621191ec29
      - our.docker.registry/entkubernetes/kube-controller-manager-amd64:v1.12.3
      sizeBytes: 164127965
    - names:
      - our.docker.registry/entkubernetes/kube-proxy-amd64@sha256:5f2d707e25ebd80b091cf50b8e3a2ce2dead89cf625e00176e1164c4479f3e90
      - our.docker.registry/entkubernetes/kube-proxy-amd64:v1.12.3
      sizeBytes: 96469663
    - names:
      - our.docker.registry/calico/node@sha256:624ea7f603aa07cd6a1ede662a79554a3b7d38cf079347fb9be6881c0ba643ca
      - our.docker.registry/calico/node:v3.2.3
      sizeBytes: 75156626
    - names:
      - our.docker.registry/calico/cni@sha256:4c637d8021e7cf0b99a104e8383c4b1c26ecaa0f55b7391dae8138bdbb5277a2
      - our.docker.registry/calico/cni:v3.2.3
      sizeBytes: 75086078
    - names:
      - our.docker.registry/entkubernetes/kube-scheduler-amd64@sha256:40364c7850b323823aef64d753e5222363eb9e73ca650d0a71b7f208af5fd8c8
      - our.docker.registry/entkubernetes/kube-scheduler-amd64:v1.12.3
      sizeBytes: 58339009
    - names:
      - our.docker.registry/entkubernetes/fluent-bit@sha256:4db3ae2dca4eace5a933a890a3ad35d2282b9640b168ef914bd7a5b07d1236ad
      - our.docker.registry/entkubernetes/fluent-bit:1.0.4
      sizeBytes: 53039438
    - names:
      - our.docker.registry/entkubernetes/kube2iam@sha256:3fcf3bcd2a400f9d7300a2943bdeb2e130adf693963ecd8dd82de664a9f2967b
      - our.docker.registry/entkubernetes/kube2iam:0.10.0
      sizeBytes: 34025676
    - names:
      - our.docker.registry/entkubernetes/pause-amd64@sha256:fcaff905397ba63fd376d0c3019f1f1cb6e7506131389edbcb3d22719f1ae54d
      - our.docker.registry/entkubernetes/pause-amd64:3.1
      sizeBytes: 742472
    nodeInfo:
      architecture: amd64
      bootID: 1cf98c4f-937e-4ab6-9017-4811c715b960
      containerRuntimeVersion: docker://18.9.3
      kernelVersion: 4.4.0-1072-aws
      kubeProxyVersion: v1.12.3
      kubeletVersion: v1.12.3
      machineID: 8b9166bc791441aca27058bd8b01dd5e
      operatingSystem: linux
      osImage: Ubuntu 16.04.5 LTS
      systemUUID: EC28632E-E4BC-A66A-B8F3-256B2C4EE1C9
- apiVersion: v1
  kind: Node
  metadata:
    annotations:
      node.alpha.kubernetes.io/ttl: "0"
      projectcalico.org/IPv4Address: 10.205.235.159/22
      volumes.kubernetes.io/controller-managed-attach-detach: "true"
    creationTimestamp: "2019-03-06T13:54:00Z"
    labels:
      beta.kubernetes.io/arch: amd64
      beta.kubernetes.io/instance-type: c5.large
      beta.kubernetes.io/os: linux
      failure-domain.beta.kubernetes.io/region: us-east-1
      failure-domain.beta.kubernetes.io/zone: us-east-1a
      kubernetes.io/hostname: ip-10-205-235-159.ec2.internal
      node-role.kubernetes.io/etcd: "true"
    name: ip-10-205-235-159.ec2.internal
    resourceVersion: "70549"
    selfLink: /api/v1/nodes/ip-10-205-235-159.ec2.internal
    uid: 4b7ee691-4017-11e9-9764-0e5f588b4326
  spec:
    podCIDR: 192.168.129.0/24
    providerID: aws:///us-east-1a/i-053c2b94820cf9407
    taints:
    - effect: NoSchedule
      key: dedicated
      value: master
  status:
    addresses:
    - address: 10.205.235.159
      type: InternalIP
    - address: ip-10-205-235-159.dqa.ourdomain.com
      type: InternalDNS
    - address: ip-10-205-235-159.ec2.internal
      type: Hostname
    allocatable:
      attachable-volumes-aws-ebs: "25"
      cpu: "2"
      ephemeral-storage: "93619943269"
      hugepages-1Gi: "0"
      hugepages-2Mi: "0"
      memory: 3698628Ki
      pods: "110"
    capacity:
      attachable-volumes-aws-ebs: "25"
      cpu: "2"
      ephemeral-storage: 101584140Ki
      hugepages-1Gi: "0"
      hugepages-2Mi: "0"
      memory: 3801028Ki
      pods: "110"
    conditions:
    - lastHeartbeatTime: "2019-03-06T19:13:11Z"
      lastTransitionTime: "2019-03-06T13:54:00Z"
      message: kubelet has sufficient disk space available
      reason: KubeletHasSufficientDisk
      status: "False"
      type: OutOfDisk
    - lastHeartbeatTime: "2019-03-06T19:13:11Z"
      lastTransitionTime: "2019-03-06T13:54:00Z"
      message: kubelet has sufficient memory available
      reason: KubeletHasSufficientMemory
      status: "False"
      type: MemoryPressure
    - lastHeartbeatTime: "2019-03-06T19:13:11Z"
      lastTransitionTime: "2019-03-06T13:54:00Z"
      message: kubelet has no disk pressure
      reason: KubeletHasNoDiskPressure
      status: "False"
      type: DiskPressure
    - lastHeartbeatTime: "2019-03-06T19:13:11Z"
      lastTransitionTime: "2019-03-06T13:54:00Z"
      message: kubelet has sufficient PID available
      reason: KubeletHasSufficientPID
      status: "False"
      type: PIDPressure
    - lastHeartbeatTime: "2019-03-06T19:13:11Z"
      lastTransitionTime: "2019-03-06T13:56:00Z"
      message: kubelet is posting ready status. AppArmor enabled
      reason: KubeletReady
      status: "True"
      type: Ready
    daemonEndpoints:
      kubeletEndpoint:
        Port: 10250
    images:
    - names:
      - qualys/sensor:1.1.5
      sizeBytes: 351318875
    - names:
      - qualys/sensor:1.2.4
      - qualys/sensor:latest
      sizeBytes: 292121056
    - names:
      - our.docker.registry/entkubernetes/kube-proxy-amd64@sha256:5f2d707e25ebd80b091cf50b8e3a2ce2dead89cf625e00176e1164c4479f3e90
      - our.docker.registry/entkubernetes/kube-proxy-amd64:v1.12.3
      sizeBytes: 96469663
    - names:
      - our.docker.registry/calico/node@sha256:624ea7f603aa07cd6a1ede662a79554a3b7d38cf079347fb9be6881c0ba643ca
      - our.docker.registry/calico/node:v3.2.3
      sizeBytes: 75156626
    - names:
      - our.docker.registry/calico/cni@sha256:4c637d8021e7cf0b99a104e8383c4b1c26ecaa0f55b7391dae8138bdbb5277a2
      - our.docker.registry/calico/cni:v3.2.3
      sizeBytes: 75086078
    - names:
      - our.docker.registry/entkubernetes/fluent-bit@sha256:4db3ae2dca4eace5a933a890a3ad35d2282b9640b168ef914bd7a5b07d1236ad
      - our.docker.registry/entkubernetes/fluent-bit:1.0.4
      sizeBytes: 53039438
    - names:
      - our.docker.registry/entkubernetes/etcd@sha256:4bcd880f42a7f243530943c6c707d3419d80906d60fc3f6f77b9709ef3cc5e71
      - our.docker.registry/entkubernetes/etcd:v3.3.0
      sizeBytes: 39420828
    - names:
      - our.docker.registry/entkubernetes/kube2iam@sha256:3fcf3bcd2a400f9d7300a2943bdeb2e130adf693963ecd8dd82de664a9f2967b
      - our.docker.registry/entkubernetes/kube2iam:0.10.0
      sizeBytes: 34025676
    - names:
      - our.docker.registry/entkubernetes/pause-amd64@sha256:fcaff905397ba63fd376d0c3019f1f1cb6e7506131389edbcb3d22719f1ae54d
      - our.docker.registry/entkubernetes/pause-amd64:3.1
      sizeBytes: 742472
    nodeInfo:
      architecture: amd64
      bootID: e87df3c3-7da2-4794-8345-9323a7067d0a
      containerRuntimeVersion: docker://18.9.3
      kernelVersion: 4.4.0-1072-aws
      kubeProxyVersion: v1.12.3
      kubeletVersion: v1.12.3
      machineID: 8b9166bc791441aca27058bd8b01dd5e
      operatingSystem: linux
      osImage: Ubuntu 16.04.5 LTS
      systemUUID: EC20687E-AF97-DB39-3C27-2D3BB9E91D2A
- apiVersion: v1
  kind: Node
  metadata:
    annotations:
      node.alpha.kubernetes.io/ttl: "0"
      projectcalico.org/IPv4Address: 10.205.235.195/22
      volumes.kubernetes.io/controller-managed-attach-detach: "true"
    creationTimestamp: "2019-03-06T13:54:00Z"
    labels:
      beta.kubernetes.io/arch: amd64
      beta.kubernetes.io/instance-type: c5.large
      beta.kubernetes.io/os: linux
      failure-domain.beta.kubernetes.io/region: us-east-1
      failure-domain.beta.kubernetes.io/zone: us-east-1a
      kubernetes.io/hostname: ip-10-205-235-195.ec2.internal
      node-role.kubernetes.io/etcd: "true"
    name: ip-10-205-235-195.ec2.internal
    resourceVersion: "70522"
    selfLink: /api/v1/nodes/ip-10-205-235-195.ec2.internal
    uid: 4b57479a-4017-11e9-9764-0e5f588b4326
  spec:
    podCIDR: 192.168.133.0/24
    providerID: aws:///us-east-1a/i-0e36782b6b5f08957
    taints:
    - effect: NoSchedule
      key: dedicated
      value: master
  status:
    addresses:
    - address: 10.205.235.195
      type: InternalIP
    - address: ip-10-205-235-195.dqa.ourdomain.com
      type: InternalDNS
    - address: ip-10-205-235-195.ec2.internal
      type: Hostname
    allocatable:
      attachable-volumes-aws-ebs: "25"
      cpu: "2"
      ephemeral-storage: "93619943269"
      hugepages-1Gi: "0"
      hugepages-2Mi: "0"
      memory: 3698628Ki
      pods: "110"
    capacity:
      attachable-volumes-aws-ebs: "25"
      cpu: "2"
      ephemeral-storage: 101584140Ki
      hugepages-1Gi: "0"
      hugepages-2Mi: "0"
      memory: 3801028Ki
      pods: "110"
    conditions:
    - lastHeartbeatTime: "2019-03-06T19:13:03Z"
      lastTransitionTime: "2019-03-06T13:54:00Z"
      message: kubelet has sufficient disk space available
      reason: KubeletHasSufficientDisk
      status: "False"
      type: OutOfDisk
    - lastHeartbeatTime: "2019-03-06T19:13:03Z"
      lastTransitionTime: "2019-03-06T13:54:00Z"
      message: kubelet has sufficient memory available
      reason: KubeletHasSufficientMemory
      status: "False"
      type: MemoryPressure
    - lastHeartbeatTime: "2019-03-06T19:13:03Z"
      lastTransitionTime: "2019-03-06T13:54:00Z"
      message: kubelet has no disk pressure
      reason: KubeletHasNoDiskPressure
      status: "False"
      type: DiskPressure
    - lastHeartbeatTime: "2019-03-06T19:13:03Z"
      lastTransitionTime: "2019-03-06T13:54:00Z"
      message: kubelet has sufficient PID available
      reason: KubeletHasSufficientPID
      status: "False"
      type: PIDPressure
    - lastHeartbeatTime: "2019-03-06T19:13:03Z"
      lastTransitionTime: "2019-03-06T13:56:00Z"
      message: kubelet is posting ready status. AppArmor enabled
      reason: KubeletReady
      status: "True"
      type: Ready
    daemonEndpoints:
      kubeletEndpoint:
        Port: 10250
    images:
    - names:
      - qualys/sensor:1.1.5
      sizeBytes: 351318875
    - names:
      - qualys/sensor:1.2.4
      - qualys/sensor:latest
      sizeBytes: 292121056
    - names:
      - our.docker.registry/entkubernetes/kube-proxy-amd64@sha256:5f2d707e25ebd80b091cf50b8e3a2ce2dead89cf625e00176e1164c4479f3e90
      - our.docker.registry/entkubernetes/kube-proxy-amd64:v1.12.3
      sizeBytes: 96469663
    - names:
      - our.docker.registry/calico/node@sha256:624ea7f603aa07cd6a1ede662a79554a3b7d38cf079347fb9be6881c0ba643ca
      - our.docker.registry/calico/node:v3.2.3
      sizeBytes: 75156626
    - names:
      - our.docker.registry/calico/cni@sha256:4c637d8021e7cf0b99a104e8383c4b1c26ecaa0f55b7391dae8138bdbb5277a2
      - our.docker.registry/calico/cni:v3.2.3
      sizeBytes: 75086078
    - names:
      - our.docker.registry/entkubernetes/fluent-bit@sha256:4db3ae2dca4eace5a933a890a3ad35d2282b9640b168ef914bd7a5b07d1236ad
      - our.docker.registry/entkubernetes/fluent-bit:1.0.4
      sizeBytes: 53039438
    - names:
      - our.docker.registry/entkubernetes/k8s-dns-kube-dns-amd64@sha256:b99fc3eee2a9f052f7eb4cc00f15eb12fc405fa41019baa2d6b79847ae7284a8
      - our.docker.registry/entkubernetes/k8s-dns-kube-dns-amd64:1.14.10
      sizeBytes: 49549457
    - names:
      - our.docker.registry/entkubernetes/k8s-dns-sidecar-amd64@sha256:4f1ab957f87b94a5ec1edc26fae50da2175461f00afecf68940c4aa079bd08a4
      - our.docker.registry/entkubernetes/k8s-dns-sidecar-amd64:1.14.10
      sizeBytes: 41635309
    - names:
      - our.docker.registry/entkubernetes/k8s-dns-dnsmasq-nanny-amd64@sha256:bbb2a290a568125b3b996028958eb773f33b5b87a6b37bf38a28f8b62dddb3c8
      - our.docker.registry/entkubernetes/k8s-dns-dnsmasq-nanny-amd64:1.14.10
      sizeBytes: 40372149
    - names:
      - our.docker.registry/entkubernetes/etcd@sha256:4bcd880f42a7f243530943c6c707d3419d80906d60fc3f6f77b9709ef3cc5e71
      - our.docker.registry/entkubernetes/etcd:v3.3.0
      sizeBytes: 39420828
    - names:
      - our.docker.registry/entkubernetes/kube2iam@sha256:3fcf3bcd2a400f9d7300a2943bdeb2e130adf693963ecd8dd82de664a9f2967b
      - our.docker.registry/entkubernetes/kube2iam:0.10.0
      sizeBytes: 34025676
    - names:
      - our.docker.registry/entkubernetes/pause-amd64@sha256:fcaff905397ba63fd376d0c3019f1f1cb6e7506131389edbcb3d22719f1ae54d
      - our.docker.registry/entkubernetes/pause-amd64:3.1
      sizeBytes: 742472
    nodeInfo:
      architecture: amd64
      bootID: 0f8d0459-5692-4d3c-ae23-9f73eb38b4bc
      containerRuntimeVersion: docker://18.9.3
      kernelVersion: 4.4.0-1072-aws
      kubeProxyVersion: v1.12.3
      kubeletVersion: v1.12.3
      machineID: 8b9166bc791441aca27058bd8b01dd5e
      operatingSystem: linux
      osImage: Ubuntu 16.04.5 LTS
      systemUUID: EC25AA86-7C1F-75E1-1253-26C1A4677FE2
- apiVersion: v1
  kind: Node
  metadata:
    annotations:
      node.alpha.kubernetes.io/ttl: "0"
      projectcalico.org/IPv4Address: 10.205.236.92/22
      volumes.kubernetes.io/controller-managed-attach-detach: "true"
    creationTimestamp: "2019-03-06T13:55:13Z"
    labels:
      beta.kubernetes.io/arch: amd64
      beta.kubernetes.io/instance-type: m5.large
      beta.kubernetes.io/os: linux
      failure-domain.beta.kubernetes.io/region: us-east-1
      failure-domain.beta.kubernetes.io/zone: us-east-1c
      kubernetes.io/hostname: ip-10-205-236-92.ec2.internal
      node-role.kubernetes.io/worker: "true"
    name: ip-10-205-236-92.ec2.internal
    resourceVersion: "70542"
    selfLink: /api/v1/nodes/ip-10-205-236-92.ec2.internal
    uid: 7719300d-4017-11e9-9764-0e5f588b4326
  spec:
    podCIDR: 192.168.134.0/24
    providerID: aws:///us-east-1c/i-097fc1b6db2effe2d
  status:
    addresses:
    - address: 10.205.236.92
      type: InternalIP
    - address: ip-10-205-236-92.dqa.ourdomain.com
      type: InternalDNS
    - address: ip-10-205-236-92.ec2.internal
      type: Hostname
    allocatable:
      attachable-volumes-aws-ebs: "25"
      cpu: "2"
      ephemeral-storage: "93619943269"
      hugepages-1Gi: "0"
      hugepages-2Mi: "0"
      memory: 7770048Ki
      pods: "110"
    capacity:
      attachable-volumes-aws-ebs: "25"
      cpu: "2"
      ephemeral-storage: 101584140Ki
      hugepages-1Gi: "0"
      hugepages-2Mi: "0"
      memory: 7872448Ki
      pods: "110"
    conditions:
    - lastHeartbeatTime: "2019-03-06T19:13:10Z"
      lastTransitionTime: "2019-03-06T13:55:13Z"
      message: kubelet has sufficient disk space available
      reason: KubeletHasSufficientDisk
      status: "False"
      type: OutOfDisk
    - lastHeartbeatTime: "2019-03-06T19:13:10Z"
      lastTransitionTime: "2019-03-06T13:55:13Z"
      message: kubelet has sufficient memory available
      reason: KubeletHasSufficientMemory
      status: "False"
      type: MemoryPressure
    - lastHeartbeatTime: "2019-03-06T19:13:10Z"
      lastTransitionTime: "2019-03-06T13:55:13Z"
      message: kubelet has no disk pressure
      reason: KubeletHasNoDiskPressure
      status: "False"
      type: DiskPressure
    - lastHeartbeatTime: "2019-03-06T19:13:10Z"
      lastTransitionTime: "2019-03-06T13:55:13Z"
      message: kubelet has sufficient PID available
      reason: KubeletHasSufficientPID
      status: "False"
      type: PIDPressure
    - lastHeartbeatTime: "2019-03-06T19:13:10Z"
      lastTransitionTime: "2019-03-06T13:55:53Z"
      message: kubelet is posting ready status. AppArmor enabled
      reason: KubeletReady
      status: "True"
      type: Ready
    daemonEndpoints:
      kubeletEndpoint:
        Port: 10250
    images:
    - names:
      - our.docker.registry/co.elastic.docker/elasticsearch-oss@sha256:f59f7936d018e9d2329dfa0705b305e2c8596045e80039108d67218ae5a967bc
      - our.docker.registry/co.elastic.docker/elasticsearch-oss:6.6.0
      sizeBytes: 678455631
    - names:
      - our.docker.registry/heptio-images/kube-conformance@sha256:9ae818b51720e6fba16205c3b648d16ed9d92ac9b18245b0cd006df46cfaa931
      - our.docker.registry/heptio-images/kube-conformance:v1.12
      sizeBytes: 548682100
    - names:
      - our.docker.registry/co.elastic.docker/kibana-oss@sha256:e1fa3f513ce258b9b531233450c10c765d587e22b21b416ba71a651cfb2c6a1c
      - our.docker.registry/co.elastic.docker/kibana-oss:6.6.0
      sizeBytes: 459030535
    - names:
      - our.docker.registry/entkubernetes/datadog-agent@sha256:e04cc178565977c606ef8ef4bcd385d62ea800ba66d8c0be8014eb8e95f27f2e
      - our.docker.registry/entkubernetes/datadog-agent:6.9.0
      sizeBytes: 414298618
    - names:
      - qualys/sensor:1.1.5
      sizeBytes: 351318875
    - names:
      - qualys/sensor:1.2.4
      - qualys/sensor:latest
      sizeBytes: 292121056
    - names:
      - our.docker.registry/entkubernetes/grafana@sha256:cc4ab72a246606fcc777a762680812774c7f60fd0f86cd2010cf40531b1227c7
      - our.docker.registry/entkubernetes/grafana:5.4.3
      sizeBytes: 240411891
    - names:
      - our.docker.registry/entkubernetes/k8s-sidecar@sha256:19c302d1a313d1392c238c34589a1de3f764ecc70234aa55ad74d020f39e462c
      - our.docker.registry/entkubernetes/k8s-sidecar:0.0.6
      sizeBytes: 171264785
    - names:
      - our.docker.registry/entkubernetes/kube-proxy-amd64@sha256:5f2d707e25ebd80b091cf50b8e3a2ce2dead89cf625e00176e1164c4479f3e90
      - our.docker.registry/entkubernetes/kube-proxy-amd64:v1.12.3
      sizeBytes: 96469663
    - names:
      - our.docker.registry/entkubernetes/heapster@sha256:59fb34ffd902282d06fcc940a906df9787edf78651743f4c8c4abf8b3468c0e9
      - our.docker.registry/entkubernetes/heapster:v1.5.2
      sizeBytes: 75318342
    - names:
      - our.docker.registry/calico/node@sha256:624ea7f603aa07cd6a1ede662a79554a3b7d38cf079347fb9be6881c0ba643ca
      - our.docker.registry/calico/node:v3.2.3
      sizeBytes: 75156626
    - names:
      - our.docker.registry/calico/cni@sha256:4c637d8021e7cf0b99a104e8383c4b1c26ecaa0f55b7391dae8138bdbb5277a2
      - our.docker.registry/calico/cni:v3.2.3
      sizeBytes: 75086078
    - names:
      - our.docker.registry/calico/typha@sha256:39e64244e2d625c33db9700e997d978226060f72453d38f0c26f10fb8fd45dca
      - our.docker.registry/calico/typha:v3.2.3
      sizeBytes: 55287586
    - names:
      - our.docker.registry/entkubernetes/fluent-bit@sha256:4db3ae2dca4eace5a933a890a3ad35d2282b9640b168ef914bd7a5b07d1236ad
      - our.docker.registry/entkubernetes/fluent-bit:1.0.4
      sizeBytes: 53039438
    - names:
      - our.docker.registry/entkubernetes/node-operators@sha256:54fccfaf2e4d0a8787c828467084b239755cb60ed6b083d1a224839fef3f3888
      - our.docker.registry/entkubernetes/node-operators:0.1.76
      sizeBytes: 42878515
    - names:
      - our.docker.registry/entkubernetes/maintenance-controller@sha256:9962f6a23261465e1199f7810b25b85ac6e0af0cbb2bb274275518daae0cb8ef
      - our.docker.registry/entkubernetes/maintenance-controller:0.1.76
      sizeBytes: 39390336
    - names:
      - our.docker.registry/entkubernetes/addon-resizer@sha256:dcec9a5c2e20b8df19f3e9eeb87d9054a9e94e71479b935d5cfdbede9ce15895
      - our.docker.registry/entkubernetes/addon-resizer:1.7
      sizeBytes: 38983736
    - names:
      - our.docker.registry/entkubernetes/kube2iam@sha256:3fcf3bcd2a400f9d7300a2943bdeb2e130adf693963ecd8dd82de664a9f2967b
      - our.docker.registry/entkubernetes/kube2iam:0.10.0
      sizeBytes: 34025676
    - names:
      - our.docker.registry/heptio-images/sonobuoy@sha256:f9b879fca17b050f6fb8f1e36020ed32139cda2cde339b9dbe2c1a9a9ab1eaaa
      - our.docker.registry/heptio-images/sonobuoy:v0.13.0
      sizeBytes: 33410348
    - names:
      - our.docker.registry/entkubernetes/etcd-backup-controller@sha256:2816838951100b17f749afcb7ff5746b93000544b5b07b6d9d4705e52dd5c237
      - our.docker.registry/entkubernetes/etcd-backup-controller:0.1.76
      sizeBytes: 30994335
    - names:
      - our.docker.registry/entkubernetes/dex@sha256:eab6f44618ef75b1b116f15ade569aeebea25c35e0ccedc0872e75bc6f65f300
      - our.docker.registry/entkubernetes/dex:v2.10.0
      sizeBytes: 26079276
    - names:
      - our.docker.registry/prom-official/node-exporter@sha256:c390c8fea4cd362a28ad5070aedd6515aacdfdffd21de6db42ead05e332be5a9
      - our.docker.registry/prom-official/node-exporter:v0.17.0
      sizeBytes: 20982005
    - names:
      - our.docker.registry/entkubernetes/ca-helper@sha256:f18b4a7722991a42225da8e12aa78050554c253bd4e2fb74eb3953b458c36d49
      - our.docker.registry/entkubernetes/ca-helper:latest
      sizeBytes: 15503745
    - names:
      - our.docker.registry/entkubernetes/busybox@sha256:bbb143159af9eabdf45511fd5aab4fd2475d4c0e7fd4a5e154b98e838488e510
      - our.docker.registry/entkubernetes/busybox:1.30.0
      sizeBytes: 1199417
    - names:
      - our.docker.registry/public-cloud/busybox@sha256:29f5d56d12684887bdfa50dcd29fc31eea4aaf4ad3bec43daf19026a7ce69912
      - our.docker.registry/public-cloud/busybox:latest
      sizeBytes: 1093484
    - names:
      - our.docker.registry/entkubernetes/pause-amd64@sha256:fcaff905397ba63fd376d0c3019f1f1cb6e7506131389edbcb3d22719f1ae54d
      - our.docker.registry/entkubernetes/pause-amd64:3.1
      sizeBytes: 742472
    nodeInfo:
      architecture: amd64
      bootID: 3d668264-670f-4b4b-8b91-073393fa9c0e
      containerRuntimeVersion: docker://18.9.3
      kernelVersion: 4.4.0-1072-aws
      kubeProxyVersion: v1.12.3
      kubeletVersion: v1.12.3
      machineID: 8b9166bc791441aca27058bd8b01dd5e
      operatingSystem: linux
      osImage: Ubuntu 16.04.5 LTS
      systemUUID: EC2E5CCA-28EC-6C66-5DD8-22BC0331905D
    volumesAttached:
    - devicePath: /dev/xvdce
      name: kubernetes.io/aws-ebs/aws://us-east-1c/vol-0cdba481a2bdaa9e7
    - devicePath: /dev/xvdbj
      name: kubernetes.io/aws-ebs/aws://us-east-1c/vol-05b02862a034a14e2
    volumesInUse:
    - kubernetes.io/aws-ebs/aws://us-east-1c/vol-05b02862a034a14e2
    - kubernetes.io/aws-ebs/aws://us-east-1c/vol-0cdba481a2bdaa9e7
- apiVersion: v1
  kind: Node
  metadata:
    annotations:
      node.alpha.kubernetes.io/ttl: "0"
      projectcalico.org/IPv4Address: 10.205.237.119/22
      volumes.kubernetes.io/controller-managed-attach-detach: "true"
    creationTimestamp: "2019-03-06T13:55:18Z"
    labels:
      beta.kubernetes.io/arch: amd64
      beta.kubernetes.io/instance-type: m5.large
      beta.kubernetes.io/os: linux
      failure-domain.beta.kubernetes.io/region: us-east-1
      failure-domain.beta.kubernetes.io/zone: us-east-1c
      kubernetes.io/hostname: ip-10-205-237-119.ec2.internal
      node-role.kubernetes.io/worker: "true"
    name: ip-10-205-237-119.ec2.internal
    resourceVersion: "70529"
    selfLink: /api/v1/nodes/ip-10-205-237-119.ec2.internal
    uid: 79a265ae-4017-11e9-9764-0e5f588b4326
  spec:
    podCIDR: 192.168.137.0/24
    providerID: aws:///us-east-1c/i-0145e2569c64b5b6b
  status:
    addresses:
    - address: 10.205.237.119
      type: InternalIP
    - address: ip-10-205-237-119.dqa.ourdomain.com
      type: InternalDNS
    - address: ip-10-205-237-119.ec2.internal
      type: Hostname
    allocatable:
      attachable-volumes-aws-ebs: "25"
      cpu: "2"
      ephemeral-storage: "93619943269"
      hugepages-1Gi: "0"
      hugepages-2Mi: "0"
      memory: 7770048Ki
      pods: "110"
    capacity:
      attachable-volumes-aws-ebs: "25"
      cpu: "2"
      ephemeral-storage: 101584140Ki
      hugepages-1Gi: "0"
      hugepages-2Mi: "0"
      memory: 7872448Ki
      pods: "110"
    conditions:
    - lastHeartbeatTime: "2019-03-06T19:13:05Z"
      lastTransitionTime: "2019-03-06T13:55:18Z"
      message: kubelet has sufficient disk space available
      reason: KubeletHasSufficientDisk
      status: "False"
      type: OutOfDisk
    - lastHeartbeatTime: "2019-03-06T19:13:05Z"
      lastTransitionTime: "2019-03-06T13:55:18Z"
      message: kubelet has sufficient memory available
      reason: KubeletHasSufficientMemory
      status: "False"
      type: MemoryPressure
    - lastHeartbeatTime: "2019-03-06T19:13:05Z"
      lastTransitionTime: "2019-03-06T13:55:18Z"
      message: kubelet has no disk pressure
      reason: KubeletHasNoDiskPressure
      status: "False"
      type: DiskPressure
    - lastHeartbeatTime: "2019-03-06T19:13:05Z"
      lastTransitionTime: "2019-03-06T13:55:18Z"
      message: kubelet has sufficient PID available
      reason: KubeletHasSufficientPID
      status: "False"
      type: PIDPressure
    - lastHeartbeatTime: "2019-03-06T19:13:05Z"
      lastTransitionTime: "2019-03-06T13:56:08Z"
      message: kubelet is posting ready status. AppArmor enabled
      reason: KubeletReady
      status: "True"
      type: Ready
    daemonEndpoints:
      kubeletEndpoint:
        Port: 10250
    images:
    - names:
      - our.docker.registry/entkubernetes/datadog-agent@sha256:e04cc178565977c606ef8ef4bcd385d62ea800ba66d8c0be8014eb8e95f27f2e
      - our.docker.registry/entkubernetes/datadog-agent:6.9.0
      sizeBytes: 414298618
    - names:
      - our.docker.registry/entkubernetes/nginx-ingress-controller@sha256:63b48dd3ba635aa65409f7d6a319e293697810c1b9e885e6255bebf36a2506ab
      - our.docker.registry/entkubernetes/nginx-ingress-controller:0.16.2
      sizeBytes: 361632894
    - names:
      - qualys/sensor:1.1.5
      sizeBytes: 351318875
    - names:
      - qualys/sensor:1.2.4
      - qualys/sensor:latest
      sizeBytes: 292121056
    - names:
      - our.docker.registry/entkubernetes/kube-proxy-amd64@sha256:5f2d707e25ebd80b091cf50b8e3a2ce2dead89cf625e00176e1164c4479f3e90
      - our.docker.registry/entkubernetes/kube-proxy-amd64:v1.12.3
      sizeBytes: 96469663
    - names:
      - our.docker.registry/calico/node@sha256:624ea7f603aa07cd6a1ede662a79554a3b7d38cf079347fb9be6881c0ba643ca
      - our.docker.registry/calico/node:v3.2.3
      sizeBytes: 75156626
    - names:
      - our.docker.registry/calico/cni@sha256:4c637d8021e7cf0b99a104e8383c4b1c26ecaa0f55b7391dae8138bdbb5277a2
      - our.docker.registry/calico/cni:v3.2.3
      sizeBytes: 75086078
    - names:
      - our.docker.registry/entkubernetes/fluent-bit@sha256:4db3ae2dca4eace5a933a890a3ad35d2282b9640b168ef914bd7a5b07d1236ad
      - our.docker.registry/entkubernetes/fluent-bit:1.0.4
      sizeBytes: 53039438
    - names:
      - our.docker.registry/entkubernetes/node-operators@sha256:54fccfaf2e4d0a8787c828467084b239755cb60ed6b083d1a224839fef3f3888
      - our.docker.registry/entkubernetes/node-operators:0.1.76
      sizeBytes: 42878515
    - names:
      - our.docker.registry/entkubernetes/maintenance-controller@sha256:9962f6a23261465e1199f7810b25b85ac6e0af0cbb2bb274275518daae0cb8ef
      - our.docker.registry/entkubernetes/maintenance-controller:0.1.76
      sizeBytes: 39390336
    - names:
      - our.docker.registry/entkubernetes/kube2iam@sha256:3fcf3bcd2a400f9d7300a2943bdeb2e130adf693963ecd8dd82de664a9f2967b
      - our.docker.registry/entkubernetes/kube2iam:0.10.0
      sizeBytes: 34025676
    - names:
      - our.docker.registry/heptio-images/sonobuoy@sha256:f9b879fca17b050f6fb8f1e36020ed32139cda2cde339b9dbe2c1a9a9ab1eaaa
      - our.docker.registry/heptio-images/sonobuoy:v0.13.0
      sizeBytes: 33410348
    - names:
      - our.docker.registry/prom-official/node-exporter@sha256:c390c8fea4cd362a28ad5070aedd6515aacdfdffd21de6db42ead05e332be5a9
      - our.docker.registry/prom-official/node-exporter:v0.17.0
      sizeBytes: 20982005
    - names:
      - our.docker.registry/entkubernetes/keycloak-proxy@sha256:8d974badf5c836651344dbe65d2140c9579e67e712ada979a5370a7eb1f0270f
      - our.docker.registry/entkubernetes/keycloak-proxy:latest
      sizeBytes: 15359224
    - names:
      - our.docker.registry/entkubernetes/defaultbackend@sha256:5e635236018ac20db4a64f4b4a89b0ceea7867fea372984c01477b1522deea8b
      - our.docker.registry/entkubernetes/defaultbackend:1.3
      sizeBytes: 3618880
    - names:
      - our.docker.registry/entkubernetes/pause-amd64@sha256:fcaff905397ba63fd376d0c3019f1f1cb6e7506131389edbcb3d22719f1ae54d
      - our.docker.registry/entkubernetes/pause-amd64:3.1
      sizeBytes: 742472
    nodeInfo:
      architecture: amd64
      bootID: 96ed11f3-680f-40aa-b824-40c6237c8796
      containerRuntimeVersion: docker://18.9.3
      kernelVersion: 4.4.0-1072-aws
      kubeProxyVersion: v1.12.3
      kubeletVersion: v1.12.3
      machineID: 8b9166bc791441aca27058bd8b01dd5e
      operatingSystem: linux
      osImage: Ubuntu 16.04.5 LTS
      systemUUID: EC22CB8F-8D4B-3951-F7D1-DE58BFE7BDBC
- apiVersion: v1
  kind: Node
  metadata:
    annotations:
      node.alpha.kubernetes.io/ttl: "0"
      projectcalico.org/IPv4Address: 10.205.237.137/22
      volumes.kubernetes.io/controller-managed-attach-detach: "true"
    creationTimestamp: "2019-03-06T13:54:01Z"
    labels:
      beta.kubernetes.io/arch: amd64
      beta.kubernetes.io/instance-type: c5.large
      beta.kubernetes.io/os: linux
      failure-domain.beta.kubernetes.io/region: us-east-1
      failure-domain.beta.kubernetes.io/zone: us-east-1c
      kubernetes.io/hostname: ip-10-205-237-137.ec2.internal
      node-role.kubernetes.io/etcd: "true"
    name: ip-10-205-237-137.ec2.internal
    resourceVersion: "70548"
    selfLink: /api/v1/nodes/ip-10-205-237-137.ec2.internal
    uid: 4c41a1cd-4017-11e9-9764-0e5f588b4326
  spec:
    podCIDR: 192.168.131.0/24
    providerID: aws:///us-east-1c/i-0323fdcd9c9852a1a
    taints:
    - effect: NoSchedule
      key: dedicated
      value: master
  status:
    addresses:
    - address: 10.205.237.137
      type: InternalIP
    - address: ip-10-205-237-137.dqa.ourdomain.com
      type: InternalDNS
    - address: ip-10-205-237-137.ec2.internal
      type: Hostname
    allocatable:
      attachable-volumes-aws-ebs: "25"
      cpu: "2"
      ephemeral-storage: "93619943269"
      hugepages-1Gi: "0"
      hugepages-2Mi: "0"
      memory: 3698628Ki
      pods: "110"
    capacity:
      attachable-volumes-aws-ebs: "25"
      cpu: "2"
      ephemeral-storage: 101584140Ki
      hugepages-1Gi: "0"
      hugepages-2Mi: "0"
      memory: 3801028Ki
      pods: "110"
    conditions:
    - lastHeartbeatTime: "2019-03-06T19:13:11Z"
      lastTransitionTime: "2019-03-06T13:54:01Z"
      message: kubelet has sufficient disk space available
      reason: KubeletHasSufficientDisk
      status: "False"
      type: OutOfDisk
    - lastHeartbeatTime: "2019-03-06T19:13:11Z"
      lastTransitionTime: "2019-03-06T13:54:01Z"
      message: kubelet has sufficient memory available
      reason: KubeletHasSufficientMemory
      status: "False"
      type: MemoryPressure
    - lastHeartbeatTime: "2019-03-06T19:13:11Z"
      lastTransitionTime: "2019-03-06T13:54:01Z"
      message: kubelet has no disk pressure
      reason: KubeletHasNoDiskPressure
      status: "False"
      type: DiskPressure
    - lastHeartbeatTime: "2019-03-06T19:13:11Z"
      lastTransitionTime: "2019-03-06T13:54:01Z"
      message: kubelet has sufficient PID available
      reason: KubeletHasSufficientPID
      status: "False"
      type: PIDPressure
    - lastHeartbeatTime: "2019-03-06T19:13:11Z"
      lastTransitionTime: "2019-03-06T13:56:02Z"
      message: kubelet is posting ready status. AppArmor enabled
      reason: KubeletReady
      status: "True"
      type: Ready
    daemonEndpoints:
      kubeletEndpoint:
        Port: 10250
    images:
    - names:
      - qualys/sensor:1.1.5
      sizeBytes: 351318875
    - names:
      - qualys/sensor:1.2.4
      - qualys/sensor:latest
      sizeBytes: 292121056
    - names:
      - our.docker.registry/entkubernetes/kube-proxy-amd64@sha256:5f2d707e25ebd80b091cf50b8e3a2ce2dead89cf625e00176e1164c4479f3e90
      - our.docker.registry/entkubernetes/kube-proxy-amd64:v1.12.3
      sizeBytes: 96469663
    - names:
      - our.docker.registry/calico/node@sha256:624ea7f603aa07cd6a1ede662a79554a3b7d38cf079347fb9be6881c0ba643ca
      - our.docker.registry/calico/node:v3.2.3
      sizeBytes: 75156626
    - names:
      - our.docker.registry/calico/cni@sha256:4c637d8021e7cf0b99a104e8383c4b1c26ecaa0f55b7391dae8138bdbb5277a2
      - our.docker.registry/calico/cni:v3.2.3
      sizeBytes: 75086078
    - names:
      - our.docker.registry/entkubernetes/fluent-bit@sha256:4db3ae2dca4eace5a933a890a3ad35d2282b9640b168ef914bd7a5b07d1236ad
      - our.docker.registry/entkubernetes/fluent-bit:1.0.4
      sizeBytes: 53039438
    - names:
      - our.docker.registry/entkubernetes/etcd@sha256:4bcd880f42a7f243530943c6c707d3419d80906d60fc3f6f77b9709ef3cc5e71
      - our.docker.registry/entkubernetes/etcd:v3.3.0
      sizeBytes: 39420828
    - names:
      - our.docker.registry/entkubernetes/kube2iam@sha256:3fcf3bcd2a400f9d7300a2943bdeb2e130adf693963ecd8dd82de664a9f2967b
      - our.docker.registry/entkubernetes/kube2iam:0.10.0
      sizeBytes: 34025676
    - names:
      - our.docker.registry/entkubernetes/pause-amd64@sha256:fcaff905397ba63fd376d0c3019f1f1cb6e7506131389edbcb3d22719f1ae54d
      - our.docker.registry/entkubernetes/pause-amd64:3.1
      sizeBytes: 742472
    nodeInfo:
      architecture: amd64
      bootID: d662a9a1-be62-4639-a2c7-69e4aef259e4
      containerRuntimeVersion: docker://18.9.3
      kernelVersion: 4.4.0-1072-aws
      kubeProxyVersion: v1.12.3
      kubeletVersion: v1.12.3
      machineID: 8b9166bc791441aca27058bd8b01dd5e
      operatingSystem: linux
      osImage: Ubuntu 16.04.5 LTS
      systemUUID: EC2A5322-A217-383A-E3C6-30E20A6CBE13
- apiVersion: v1
  kind: Node
  metadata:
    annotations:
      node.alpha.kubernetes.io/ttl: "0"
      projectcalico.org/IPv4Address: 10.205.237.4/22
      volumes.kubernetes.io/controller-managed-attach-detach: "true"
    creationTimestamp: "2019-03-06T13:54:00Z"
    labels:
      beta.kubernetes.io/arch: amd64
      beta.kubernetes.io/instance-type: c5.large
      beta.kubernetes.io/os: linux
      failure-domain.beta.kubernetes.io/region: us-east-1
      failure-domain.beta.kubernetes.io/zone: us-east-1c
      kubernetes.io/hostname: ip-10-205-237-4.ec2.internal
      node-role.kubernetes.io/etcd: "true"
    name: ip-10-205-237-4.ec2.internal
    resourceVersion: "70539"
    selfLink: /api/v1/nodes/ip-10-205-237-4.ec2.internal
    uid: 4b62f2c8-4017-11e9-9764-0e5f588b4326
  spec:
    podCIDR: 192.168.128.0/24
    providerID: aws:///us-east-1c/i-003b07bd9ee5b8bff
    taints:
    - effect: NoSchedule
      key: dedicated
      value: master
  status:
    addresses:
    - address: 10.205.237.4
      type: InternalIP
    - address: ip-10-205-237-4.dqa.ourdomain.com
      type: InternalDNS
    - address: ip-10-205-237-4.ec2.internal
      type: Hostname
    allocatable:
      attachable-volumes-aws-ebs: "25"
      cpu: "2"
      ephemeral-storage: "93619943269"
      hugepages-1Gi: "0"
      hugepages-2Mi: "0"
      memory: 3698628Ki
      pods: "110"
    capacity:
      attachable-volumes-aws-ebs: "25"
      cpu: "2"
      ephemeral-storage: 101584140Ki
      hugepages-1Gi: "0"
      hugepages-2Mi: "0"
      memory: 3801028Ki
      pods: "110"
    conditions:
    - lastHeartbeatTime: "2019-03-06T19:13:09Z"
      lastTransitionTime: "2019-03-06T13:54:00Z"
      message: kubelet has sufficient disk space available
      reason: KubeletHasSufficientDisk
      status: "False"
      type: OutOfDisk
    - lastHeartbeatTime: "2019-03-06T19:13:09Z"
      lastTransitionTime: "2019-03-06T13:54:00Z"
      message: kubelet has sufficient memory available
      reason: KubeletHasSufficientMemory
      status: "False"
      type: MemoryPressure
    - lastHeartbeatTime: "2019-03-06T19:13:09Z"
      lastTransitionTime: "2019-03-06T13:54:00Z"
      message: kubelet has no disk pressure
      reason: KubeletHasNoDiskPressure
      status: "False"
      type: DiskPressure
    - lastHeartbeatTime: "2019-03-06T19:13:09Z"
      lastTransitionTime: "2019-03-06T13:54:00Z"
      message: kubelet has sufficient PID available
      reason: KubeletHasSufficientPID
      status: "False"
      type: PIDPressure
    - lastHeartbeatTime: "2019-03-06T19:13:09Z"
      lastTransitionTime: "2019-03-06T13:56:00Z"
      message: kubelet is posting ready status. AppArmor enabled
      reason: KubeletReady
      status: "True"
      type: Ready
    daemonEndpoints:
      kubeletEndpoint:
        Port: 10250
    images:
    - names:
      - qualys/sensor:1.1.5
      sizeBytes: 351318875
    - names:
      - qualys/sensor:1.2.4
      - qualys/sensor:latest
      sizeBytes: 292121056
    - names:
      - our.docker.registry/entkubernetes/kube-proxy-amd64@sha256:5f2d707e25ebd80b091cf50b8e3a2ce2dead89cf625e00176e1164c4479f3e90
      - our.docker.registry/entkubernetes/kube-proxy-amd64:v1.12.3
      sizeBytes: 96469663
    - names:
      - our.docker.registry/calico/node@sha256:624ea7f603aa07cd6a1ede662a79554a3b7d38cf079347fb9be6881c0ba643ca
      - our.docker.registry/calico/node:v3.2.3
      sizeBytes: 75156626
    - names:
      - our.docker.registry/calico/cni@sha256:4c637d8021e7cf0b99a104e8383c4b1c26ecaa0f55b7391dae8138bdbb5277a2
      - our.docker.registry/calico/cni:v3.2.3
      sizeBytes: 75086078
    - names:
      - our.docker.registry/entkubernetes/fluent-bit@sha256:4db3ae2dca4eace5a933a890a3ad35d2282b9640b168ef914bd7a5b07d1236ad
      - our.docker.registry/entkubernetes/fluent-bit:1.0.4
      sizeBytes: 53039438
    - names:
      - our.docker.registry/entkubernetes/etcd@sha256:4bcd880f42a7f243530943c6c707d3419d80906d60fc3f6f77b9709ef3cc5e71
      - our.docker.registry/entkubernetes/etcd:v3.3.0
      sizeBytes: 39420828
    - names:
      - our.docker.registry/entkubernetes/kube2iam@sha256:3fcf3bcd2a400f9d7300a2943bdeb2e130adf693963ecd8dd82de664a9f2967b
      - our.docker.registry/entkubernetes/kube2iam:0.10.0
      sizeBytes: 34025676
    - names:
      - our.docker.registry/entkubernetes/pause-amd64@sha256:fcaff905397ba63fd376d0c3019f1f1cb6e7506131389edbcb3d22719f1ae54d
      - our.docker.registry/entkubernetes/pause-amd64:3.1
      sizeBytes: 742472
    nodeInfo:
      architecture: amd64
      bootID: 9c27240a-ca5f-4f68-a63e-e115b2f66fc7
      containerRuntimeVersion: docker://18.9.3
      kernelVersion: 4.4.0-1072-aws
      kubeProxyVersion: v1.12.3
      kubeletVersion: v1.12.3
      machineID: 8b9166bc791441aca27058bd8b01dd5e
      operatingSystem: linux
      osImage: Ubuntu 16.04.5 LTS
      systemUUID: EC2C6BA8-7315-9447-6166-B4EDE11187F2
- apiVersion: v1
  kind: Node
  metadata:
    annotations:
      node.alpha.kubernetes.io/ttl: "0"
      projectcalico.org/IPv4Address: 10.205.239.197/22
      volumes.kubernetes.io/controller-managed-attach-detach: "true"
    creationTimestamp: "2019-03-06T13:55:18Z"
    labels:
      beta.kubernetes.io/arch: amd64
      beta.kubernetes.io/instance-type: m5.large
      beta.kubernetes.io/os: linux
      failure-domain.beta.kubernetes.io/region: us-east-1
      failure-domain.beta.kubernetes.io/zone: us-east-1c
      kubernetes.io/hostname: ip-10-205-239-197.ec2.internal
      node-role.kubernetes.io/worker: "true"
    name: ip-10-205-239-197.ec2.internal
    resourceVersion: "70530"
    selfLink: /api/v1/nodes/ip-10-205-239-197.ec2.internal
    uid: 79a77349-4017-11e9-9764-0e5f588b4326
  spec:
    podCIDR: 192.168.138.0/24
    providerID: aws:///us-east-1c/i-0c4e3168e44620159
  status:
    addresses:
    - address: 10.205.239.197
      type: InternalIP
    - address: ip-10-205-239-197.dqa.ourdomain.com
      type: InternalDNS
    - address: ip-10-205-239-197.ec2.internal
      type: Hostname
    allocatable:
      attachable-volumes-aws-ebs: "25"
      cpu: "2"
      ephemeral-storage: "93619943269"
      hugepages-1Gi: "0"
      hugepages-2Mi: "0"
      memory: 7770048Ki
      pods: "110"
    capacity:
      attachable-volumes-aws-ebs: "25"
      cpu: "2"
      ephemeral-storage: 101584140Ki
      hugepages-1Gi: "0"
      hugepages-2Mi: "0"
      memory: 7872448Ki
      pods: "110"
    conditions:
    - lastHeartbeatTime: "2019-03-06T19:13:05Z"
      lastTransitionTime: "2019-03-06T13:55:18Z"
      message: kubelet has sufficient disk space available
      reason: KubeletHasSufficientDisk
      status: "False"
      type: OutOfDisk
    - lastHeartbeatTime: "2019-03-06T19:13:05Z"
      lastTransitionTime: "2019-03-06T13:55:18Z"
      message: kubelet has sufficient memory available
      reason: KubeletHasSufficientMemory
      status: "False"
      type: MemoryPressure
    - lastHeartbeatTime: "2019-03-06T19:13:05Z"
      lastTransitionTime: "2019-03-06T13:55:18Z"
      message: kubelet has no disk pressure
      reason: KubeletHasNoDiskPressure
      status: "False"
      type: DiskPressure
    - lastHeartbeatTime: "2019-03-06T19:13:05Z"
      lastTransitionTime: "2019-03-06T13:55:18Z"
      message: kubelet has sufficient PID available
      reason: KubeletHasSufficientPID
      status: "False"
      type: PIDPressure
    - lastHeartbeatTime: "2019-03-06T19:13:05Z"
      lastTransitionTime: "2019-03-06T13:56:08Z"
      message: kubelet is posting ready status. AppArmor enabled
      reason: KubeletReady
      status: "True"
      type: Ready
    daemonEndpoints:
      kubeletEndpoint:
        Port: 10250
    images:
    - names:
      - our.docker.registry/co.elastic.docker/elasticsearch-oss@sha256:f59f7936d018e9d2329dfa0705b305e2c8596045e80039108d67218ae5a967bc
      - our.docker.registry/co.elastic.docker/elasticsearch-oss:6.6.0
      sizeBytes: 678455631
    - names:
      - our.docker.registry/entkubernetes/datadog-agent@sha256:e04cc178565977c606ef8ef4bcd385d62ea800ba66d8c0be8014eb8e95f27f2e
      - our.docker.registry/entkubernetes/datadog-agent:6.9.0
      sizeBytes: 414298618
    - names:
      - our.docker.registry/entkubernetes/nginx-ingress-controller@sha256:63b48dd3ba635aa65409f7d6a319e293697810c1b9e885e6255bebf36a2506ab
      - our.docker.registry/entkubernetes/nginx-ingress-controller:0.16.2
      sizeBytes: 361632894
    - names:
      - qualys/sensor:1.1.5
      sizeBytes: 351318875
    - names:
      - qualys/sensor:1.2.4
      - qualys/sensor:latest
      sizeBytes: 292121056
    - names:
      - our.docker.registry/entkubernetes/kube-proxy-amd64@sha256:5f2d707e25ebd80b091cf50b8e3a2ce2dead89cf625e00176e1164c4479f3e90
      - our.docker.registry/entkubernetes/kube-proxy-amd64:v1.12.3
      sizeBytes: 96469663
    - names:
      - our.docker.registry/calico/node@sha256:624ea7f603aa07cd6a1ede662a79554a3b7d38cf079347fb9be6881c0ba643ca
      - our.docker.registry/calico/node:v3.2.3
      sizeBytes: 75156626
    - names:
      - our.docker.registry/calico/cni@sha256:4c637d8021e7cf0b99a104e8383c4b1c26ecaa0f55b7391dae8138bdbb5277a2
      - our.docker.registry/calico/cni:v3.2.3
      sizeBytes: 75086078
    - names:
      - our.docker.registry/entkubernetes/fluent-bit@sha256:4db3ae2dca4eace5a933a890a3ad35d2282b9640b168ef914bd7a5b07d1236ad
      - our.docker.registry/entkubernetes/fluent-bit:1.0.4
      sizeBytes: 53039438
    - names:
      - our.docker.registry/entkubernetes/node-operators@sha256:54fccfaf2e4d0a8787c828467084b239755cb60ed6b083d1a224839fef3f3888
      - our.docker.registry/entkubernetes/node-operators:0.1.76
      sizeBytes: 42878515
    - names:
      - our.docker.registry/entkubernetes/maintenance-controller@sha256:9962f6a23261465e1199f7810b25b85ac6e0af0cbb2bb274275518daae0cb8ef
      - our.docker.registry/entkubernetes/maintenance-controller:0.1.76
      sizeBytes: 39390336
    - names:
      - our.docker.registry/entkubernetes/kube2iam@sha256:3fcf3bcd2a400f9d7300a2943bdeb2e130adf693963ecd8dd82de664a9f2967b
      - our.docker.registry/entkubernetes/kube2iam:0.10.0
      sizeBytes: 34025676
    - names:
      - our.docker.registry/prom-official/node-exporter@sha256:c390c8fea4cd362a28ad5070aedd6515aacdfdffd21de6db42ead05e332be5a9
      - our.docker.registry/prom-official/node-exporter:v0.17.0
      sizeBytes: 20982005
    - names:
      - our.docker.registry/public-cloud/busybox@sha256:29f5d56d12684887bdfa50dcd29fc31eea4aaf4ad3bec43daf19026a7ce69912
      - our.docker.registry/public-cloud/busybox:latest
      sizeBytes: 1093484
    - names:
      - our.docker.registry/entkubernetes/pause-amd64@sha256:fcaff905397ba63fd376d0c3019f1f1cb6e7506131389edbcb3d22719f1ae54d
      - our.docker.registry/entkubernetes/pause-amd64:3.1
      sizeBytes: 742472
    nodeInfo:
      architecture: amd64
      bootID: fccf3fc5-7c33-483d-b955-2353f1c04615
      containerRuntimeVersion: docker://18.9.3
      kernelVersion: 4.4.0-1072-aws
      kubeProxyVersion: v1.12.3
      kubeletVersion: v1.12.3
      machineID: 8b9166bc791441aca27058bd8b01dd5e
      operatingSystem: linux
      osImage: Ubuntu 16.04.5 LTS
      systemUUID: EC21EEBF-035A-6F86-2BD9-8407AA93BE5B
    volumesAttached:
    - devicePath: /dev/xvdbi
      name: kubernetes.io/aws-ebs/aws://us-east-1c/vol-069158f33662b6093
    - devicePath: /dev/xvdcd
      name: kubernetes.io/aws-ebs/aws://us-east-1c/vol-06f81a75b96494584
    volumesInUse:
    - kubernetes.io/aws-ebs/aws://us-east-1c/vol-069158f33662b6093
    - kubernetes.io/aws-ebs/aws://us-east-1c/vol-06f81a75b96494584
- apiVersion: v1
  kind: Node
  metadata:
    annotations:
      node.alpha.kubernetes.io/ttl: "0"
      projectcalico.org/IPv4Address: 10.205.239.52/22
      volumes.kubernetes.io/controller-managed-attach-detach: "true"
    creationTimestamp: "2019-03-06T13:54:05Z"
    labels:
      beta.kubernetes.io/arch: amd64
      beta.kubernetes.io/instance-type: m5.large
      beta.kubernetes.io/os: linux
      failure-domain.beta.kubernetes.io/region: us-east-1
      failure-domain.beta.kubernetes.io/zone: us-east-1c
      kubernetes.io/hostname: ip-10-205-239-52.ec2.internal
      node-role.kubernetes.io/master: "true"
    name: ip-10-205-239-52.ec2.internal
    resourceVersion: "70540"
    selfLink: /api/v1/nodes/ip-10-205-239-52.ec2.internal
    uid: 4e65f713-4017-11e9-9764-0e5f588b4326
  spec:
    podCIDR: 192.168.132.0/24
    providerID: aws:///us-east-1c/i-0e00818160c253a6a
    taints:
    - effect: NoSchedule
      key: dedicated
      value: master
  status:
    addresses:
    - address: 10.205.239.52
      type: InternalIP
    - address: ip-10-205-239-52.dqa.ourdomain.com
      type: InternalDNS
    - address: ip-10-205-239-52.ec2.internal
      type: Hostname
    allocatable:
      attachable-volumes-aws-ebs: "25"
      cpu: "2"
      ephemeral-storage: "93619943269"
      hugepages-1Gi: "0"
      hugepages-2Mi: "0"
      memory: 7770048Ki
      pods: "110"
    capacity:
      attachable-volumes-aws-ebs: "25"
      cpu: "2"
      ephemeral-storage: 101584140Ki
      hugepages-1Gi: "0"
      hugepages-2Mi: "0"
      memory: 7872448Ki
      pods: "110"
    conditions:
    - lastHeartbeatTime: "2019-03-06T19:13:09Z"
      lastTransitionTime: "2019-03-06T13:54:05Z"
      message: kubelet has sufficient disk space available
      reason: KubeletHasSufficientDisk
      status: "False"
      type: OutOfDisk
    - lastHeartbeatTime: "2019-03-06T19:13:09Z"
      lastTransitionTime: "2019-03-06T13:54:05Z"
      message: kubelet has sufficient memory available
      reason: KubeletHasSufficientMemory
      status: "False"
      type: MemoryPressure
    - lastHeartbeatTime: "2019-03-06T19:13:09Z"
      lastTransitionTime: "2019-03-06T13:54:05Z"
      message: kubelet has no disk pressure
      reason: KubeletHasNoDiskPressure
      status: "False"
      type: DiskPressure
    - lastHeartbeatTime: "2019-03-06T19:13:09Z"
      lastTransitionTime: "2019-03-06T13:54:05Z"
      message: kubelet has sufficient PID available
      reason: KubeletHasSufficientPID
      status: "False"
      type: PIDPressure
    - lastHeartbeatTime: "2019-03-06T19:13:09Z"
      lastTransitionTime: "2019-03-06T13:56:05Z"
      message: kubelet is posting ready status. AppArmor enabled
      reason: KubeletReady
      status: "True"
      type: Ready
    daemonEndpoints:
      kubeletEndpoint:
        Port: 10250
    images:
    - names:
      - qualys/sensor:1.1.5
      sizeBytes: 351318875
    - names:
      - qualys/sensor:1.2.4
      - qualys/sensor:latest
      sizeBytes: 292121056
    - names:
      - our.docker.registry/entkubernetes/kube-apiserver-amd64@sha256:ad7ac6a924d88e7d4e1026233981ad0202812e6d9c5e30d37e689f82f9f4c2af
      - our.docker.registry/entkubernetes/kube-apiserver-amd64:v1.12.3
      sizeBytes: 193948168
    - names:
      - our.docker.registry/entkubernetes/kube-controller-manager-amd64@sha256:df1609649a677aea514c8a930a91287aa503646cade31209ce5735621191ec29
      - our.docker.registry/entkubernetes/kube-controller-manager-amd64:v1.12.3
      sizeBytes: 164127965
    - names:
      - our.docker.registry/entkubernetes/kube-proxy-amd64@sha256:5f2d707e25ebd80b091cf50b8e3a2ce2dead89cf625e00176e1164c4479f3e90
      - our.docker.registry/entkubernetes/kube-proxy-amd64:v1.12.3
      sizeBytes: 96469663
    - names:
      - our.docker.registry/calico/node@sha256:624ea7f603aa07cd6a1ede662a79554a3b7d38cf079347fb9be6881c0ba643ca
      - our.docker.registry/calico/node:v3.2.3
      sizeBytes: 75156626
    - names:
      - our.docker.registry/calico/cni@sha256:4c637d8021e7cf0b99a104e8383c4b1c26ecaa0f55b7391dae8138bdbb5277a2
      - our.docker.registry/calico/cni:v3.2.3
      sizeBytes: 75086078
    - names:
      - our.docker.registry/entkubernetes/kube-scheduler-amd64@sha256:40364c7850b323823aef64d753e5222363eb9e73ca650d0a71b7f208af5fd8c8
      - our.docker.registry/entkubernetes/kube-scheduler-amd64:v1.12.3
      sizeBytes: 58339009
    - names:
      - our.docker.registry/entkubernetes/fluent-bit@sha256:4db3ae2dca4eace5a933a890a3ad35d2282b9640b168ef914bd7a5b07d1236ad
      - our.docker.registry/entkubernetes/fluent-bit:1.0.4
      sizeBytes: 53039438
    - names:
      - our.docker.registry/entkubernetes/kube2iam@sha256:3fcf3bcd2a400f9d7300a2943bdeb2e130adf693963ecd8dd82de664a9f2967b
      - our.docker.registry/entkubernetes/kube2iam:0.10.0
      sizeBytes: 34025676
    - names:
      - our.docker.registry/entkubernetes/pause-amd64@sha256:fcaff905397ba63fd376d0c3019f1f1cb6e7506131389edbcb3d22719f1ae54d
      - our.docker.registry/entkubernetes/pause-amd64:3.1
      sizeBytes: 742472
    nodeInfo:
      architecture: amd64
      bootID: 147aeb21-9260-41e1-a31f-a48604b1785d
      containerRuntimeVersion: docker://18.9.3
      kernelVersion: 4.4.0-1072-aws
      kubeProxyVersion: v1.12.3
      kubeletVersion: v1.12.3
      machineID: 8b9166bc791441aca27058bd8b01dd5e
      operatingSystem: linux
      osImage: Ubuntu 16.04.5 LTS
      systemUUID: EC296256-7C9D-C7DD-93FA-72396BEDEAFF
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

@johnSchnake
Copy link
Contributor

@andrewmyhre So the issue is that you have the etcd nodes which are not labled as masters yet have the noSchedule taint on them:

...
      node-role.kubernetes.io/etcd: "true"
...
    taints:
    - effect: NoSchedule
      key: dedicated
      value: master
...

So it does seem to be exactly the case #599 is trying to address. To get around this you'd have to either remove the taint or also label those nodes as masters (I'm not positive if there are any other consequences of that).

Let me know if I misread something or if my conclusion doesn't seem accurate to you. Otherwise I'll close this in the next day or so.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants