move etcd to openshift-etcd

[deads2k]

Moves etcd to openshift-etcd namespace. This is related to API freeze, where we commit to names and namespaces for the foreseeable future.

After this, operators make the switch permanently (many are bilingual right now), and we remove the old dns names from the cert.

10 successful CI installs so far

[deads2k]

@runcom is there any log on the bootstrap host (or better yet a directory) containing the content used to create the initial hash to make this easier to debug? If we had a directory, we'd be able to pull it out with install-gather.

[runcom]

@runcom is there any log on the bootstrap host (or better yet a directory) containing the content used to create the initial hash to make this easier to debug? If we had a directory, we'd be able to pull it out with install-gather.

directories in this file is what you need https://github.com/openshift/machine-config-operator/blob/master/manifests/bootstrap-pod-v2.yaml

basically /etc/mcc/boostrap, /etc/mcs/boostrap, /etc/ssl/mcs and /etc/mcs/kubeconfig

everything in those dirs is what the bootstrap serves so if you have the rendered in cluster MC and the one in bootstrap you can diff

Also, I believe that with #612, if, from now on, there will be a drift between current bootstrap and current in-cluster, one could simple $(diff /var/machine-config-daemon/currentconfig rendered-<pool>-<hash>) to find out what drifted

[deads2k]

Also, I believe that with #612, if, from now on, there will be a drift between current bootstrap and current in-cluster, one could simple $(diff /var/machine-config-daemon/currentconfig rendered-<pool>-<hash>) to find out what drifted

@runcom @sdodson That seems like it would be really valuable for install-gather.

[runcom]

@runcom @sdodson That seems like it would be really valuable for install-gather.

@deads2k what you quoted must be gathered on nodes, not bootstrap. The other dirs I've showed above are what's on the boostrap (and can be gathered with install-gather)

[deads2k]

@deads2k what you quoted must be gathered on nodes, not bootstrap. The other dirs I've showed above are what's on the boostrap (and can be gathered with install-gather)

@runcom install-gather punches through and collects from the nodes too.

[runcom]

@runcom install-gather punches through and collects from the nodes too.

oh ok cool, I'm sure the dirs from the bootstrap node would definitely contain what we need to debug when there's a skew tho. I'm not entirely sure that dir on the nodes contains anything if we have the drift right at boot. I think gathering /etc/mcc/boostrap, /etc/mcs/boostrap, /etc/ssl/mcs and /etc/mcs/kubeconfigfrom the bootstrap would definitely help as that's really what it's used/served at bootstrap. Diffing that with what's been generated in cluster would shed some light.

[deads2k]

oh ok cool, I'm sure the dirs from the bootstrap node would definitely contain what we need to debug when there's a skew tho. I'm not entirely sure that dir on the nodes contains anything if we have the drift right at boot. I think gathering /etc/mcc/boostrap, /etc/mcs/boostrap, /etc/ssl/mcs and /etc/mcs/kubeconfigfrom the bootstrap would definitely help as that's really what it's used/served at bootstrap. Diffing that with what's been generated in cluster would shed some light.

Thanks. It'll keep until Monday, but I have ended up in this spot many times and it's always been a guess and check mode of getting out of it. In this case, it must be something to do with etcd pod or certs that has namespace references and isn't in the files I touched here. They clearly come out correctly because everything directly using etcd is working fine.

[deads2k]

New app flake only

/retest

[deads2k]

Unauthorized

/retest

[deads2k]

network flake I've seen in other pulls

            "message": "(combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_installer-1-ip-10-0-153-95.ec2.internal_openshift-kube-controller-manager_ad847397-65ba-11e9-8292-129bab5f859e_0(28f64991ec9ef53ca109bd7e19880dfb19867338856c95cd7a4696a0dd0feddc): netplugin failed but error parsing its diagnostic message \"\": unexpected end of JSON input",

/retest

[deads2k]

green and conflicts. rebasing

3 green CI installs so far.

[deads2k]

https://bugzilla.redhat.com/show_bug.cgi?id=1702104

/retest

[deads2k]

another install success

/test e2e-aws

[sdodson]

infra timeout
/test e2e-aws

[deads2k]

/test e2e-aws-op

[kikisdeliveryservice]

@deads2k is there a BZ/open issue somewhere for this change?

[deads2k]

@deads2k is there a BZ/open issue somewhere for this change?

It's related to API freeze and committing to names and namespaces "forever".

/hold

waiting for more e2e results.

[kikisdeliveryservice]

@deads2k great! can we add that to the commit message body?? 😃

[deads2k]

@deads2k great! can we add that to the commit message body??

@kikisdeliveryservice It's you again! :) Sure, after these tests complete.

[deads2k]

Now with updated commit message.

@runcom ptal. I think you lgtm'd via slack before. I plan to release the hold for merge early tomorrow AM.

[deads2k]

/hold cancel
/assign @runcom

tag/merge at will in your tomorrow (wednesday) AM.

[runcom]

/lgtm

[openshift-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: deads2k, runcom

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [runcom]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[runcom]

/hold

[deads2k]

Service catalog pull is ready and lgtm'd, notifications made, got the ok to merge tonight. Let's go!

/hold cancel