-
Notifications
You must be signed in to change notification settings - Fork 1.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Bug 1748452: data/bootstrap: Replace openshift.sh with cluster-bootstrap #1381
Bug 1748452: data/bootstrap: Replace openshift.sh with cluster-bootstrap #1381
Conversation
/hold We probably want this in 4.1, but if I change the base to point at |
/test e2e-openstack |
f0dcefe
to
d607896
Compare
@wking: This pull request references Bugzilla bug 1748452, which is invalid:
Comment In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
/bugzilla refresh |
@wking: This pull request references Bugzilla bug 1748452, which is valid. The bug has been moved to the POST state. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
the apiserver team that owns the cluster-bootstrap has made it very clear in the past that it should only be used for bootstrapping the k8s control plane, therefore we have the openshift.service that keeps the separation of concerns, Therefore I don't agree with using the bootkube.service's cluster-bootstrap to push these objects that are not required to boot the k8s control plane. |
e2e-aws failed with:
in
I'm not clear yet on why they'd care about them pushing these manifests vs. us pushing these in parallel. Maybe it's the unstructured casting issue I'm hitting :p. |
This may have been what was giving us errors like: Sep 10 19:41:16 ip-10-0-3-146 bootkube.sh[1605]: "99_openshift-cluster-api_master-user-data-secret.yaml": unable to convert asset "99_openshift-cluster-api_master-user-data-secret.yaml" to unstructed Sep 10 19:41:16 ip-10-0-3-146 bootkube.sh[1605]: "99_openshift-cluster-api_worker-user-data-secret.yaml": unable to convert asset "99_openshift-cluster-api_worker-user-data-secret.yaml" to unstructed after failing this cast to *unstructured.Unstructured [2]. I don't have a theory for why a leading empty line would matter, but it was the only thing that stuck at me when eyeballing the user-data manifests. [1]: openshift#1381 (comment) [2]: https://github.com/openshift/library-go/blob/7d4acc018c610623ee1413b1e7aebe2ac675d35f/pkg/assets/create/creater.go#L264
Ah, looks like 9e4b0d4 didn't help. Still getting:
from the e2e-aws log bundle. |
This may have been what was giving us errors like: Sep 10 19:41:16 ip-10-0-3-146 bootkube.sh[1605]: "99_openshift-cluster-api_master-user-data-secret.yaml": unable to convert asset "99_openshift-cluster-api_master-user-data-secret.yaml" to unstructed Sep 10 19:41:16 ip-10-0-3-146 bootkube.sh[1605]: "99_openshift-cluster-api_worker-user-data-secret.yaml": unable to convert asset "99_openshift-cluster-api_worker-user-data-secret.yaml" to unstructed after failing this cast to *unstructured.Unstructured [2]. I don't have a theory for why the unstructured cast fails for Lists, but that's what I see in local testing: $ cat test.go package main import ( "io/ioutil" "log" "os" "github.com/ghodss/yaml" "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" "k8s.io/apimachinery/pkg/runtime" ) func main() { file, err := os.Open("test.yaml") if err != nil { log.Fatal(err) } manifestBytes, err := ioutil.ReadAll(file) if err != nil { log.Fatal(err) } manifestJSON, err := yaml.YAMLToJSON(manifestBytes) if err != nil { log.Fatal(err) } manifestObj, err := runtime.Decode(unstructured.UnstructuredJSONScheme, manifestJSON) if err != nil { log.Fatal(err) } log.Print(manifestObj) log.Print("") manifestUnstructured, ok := manifestObj.(*unstructured.Unstructured) if !ok { log.Fatal(manifestUnstructured) } log.Print(manifestUnstructured) } $ cat <<EOF >test.yaml > apiVersion: v1 > kind: Secret > metadata: > name: master-user-data > namespace: openshift-machine-api > type: Opaque > data: > disableTemplating: dHJ1ZQo= > userData: dHJ1ZQo= > EOF $ go run test.go 2019/09/12 10:32:07 &{map[kind:Secret metadata:map[name:master-user-data namespace:openshift-machine-api] type:Opaque apiVersion:v1 data:map[userData:dHJ1ZQo= disableTemplating:dHJ1ZQo=]]} 2019/09/12 10:32:07 2019/09/12 10:32:07 &{map[apiVersion:v1 data:map[disableTemplating:dHJ1ZQo= userData:dHJ1ZQo=] kind:Secret metadata:map[name:master-user-data namespace:openshift-machine-api] type:Opaque]} $ cat <<EOF >test.yaml > kind: List > apiVersion: v1 > metadata: > resourceVersion: "" > selfLink: "" > items: > - apiVersion: v1 > kind: Secret > metadata: > name: master-user-data > namespace: openshift-machine-api > type: Opaque > data: > disableTemplating: dHJ1ZQo= > userData: dHJ1ZQo= > EOF $ go run test.go 2019/09/12 10:33:20 &{map[kind:List metadata:map[resourceVersion: selfLink:] apiVersion:v1] [{map[metadata:map[name:master-user-data namespace:openshift-machine-api] type:Opaque apiVersion:v1 data:map[disableTemplating:dHJ1ZQo= userData:dHJ1ZQo=] kind:Secret]}]} 2019/09/12 10:33:20 2019/09/12 10:33:20 <nil> exit status 1 [1]: openshift#1381 (comment) [2]: https://github.com/openshift/library-go/blob/7d4acc018c610623ee1413b1e7aebe2ac675d35f/pkg/assets/create/creater.go#L264
9e4b0d4
to
f374118
Compare
With this commit, I take advantage of openshift/cluster-bootstrap@fc5e0941 (start: wire the library-go dynamic client create, 2019-02-05, openshift/cluster-bootstrap#14) to replace our previous openshift.sh (with a minor change to the manifest directory). I'm currently using a cp in bootkube.sh to shift those manifests into the generic directory; I plan on consolidating Openshift into Manifests in pkg/asset/manifests in follow-up work. This change is especially important since the pivot to loopback kubeconfigs in openshift.sh: 82d81d9 (data/data/bootstrap: use loopback kubeconfig for API access, 2019-07-24, openshift#2086), because once cluster-bootstrap (launched from bootkube.sh) decides it's done it tears down the bootstrap control plane. Without the bootstrap control plane, further attempts by openshift.sh to push manifests via the loopback kubeconfig fail [1]. We could roll reporting into bootkube.sh as well (dropping progress.service), but Abhinav wanted to keep it separate [2]. [1]: https://bugzilla.redhat.com/show_bug.cgi?id=1748452 [2]: openshift#1381 (comment)
This may have been what was giving us errors like: Sep 10 19:41:16 ip-10-0-3-146 bootkube.sh[1605]: "99_openshift-cluster-api_master-user-data-secret.yaml": unable to convert asset "99_openshift-cluster-api_master-user-data-secret.yaml" to unstructed Sep 10 19:41:16 ip-10-0-3-146 bootkube.sh[1605]: "99_openshift-cluster-api_worker-user-data-secret.yaml": unable to convert asset "99_openshift-cluster-api_worker-user-data-secret.yaml" to unstructed after failing this cast to *unstructured.Unstructured [2]. I don't have a theory for why the unstructured cast fails for Lists, but that's what I see in local testing: $ cat test.go package main import ( "io/ioutil" "log" "os" "github.com/ghodss/yaml" "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" "k8s.io/apimachinery/pkg/runtime" ) func main() { file, err := os.Open("test.yaml") if err != nil { log.Fatal(err) } manifestBytes, err := ioutil.ReadAll(file) if err != nil { log.Fatal(err) } manifestJSON, err := yaml.YAMLToJSON(manifestBytes) if err != nil { log.Fatal(err) } manifestObj, err := runtime.Decode(unstructured.UnstructuredJSONScheme, manifestJSON) if err != nil { log.Fatal(err) } log.Print(manifestObj) log.Print("") manifestUnstructured, ok := manifestObj.(*unstructured.Unstructured) if !ok { log.Fatal(manifestUnstructured) } log.Print(manifestUnstructured) } $ cat <<EOF >test.yaml > apiVersion: v1 > kind: Secret > metadata: > name: master-user-data > namespace: openshift-machine-api > type: Opaque > data: > disableTemplating: dHJ1ZQo= > userData: dHJ1ZQo= > EOF $ go run test.go 2019/09/12 10:32:07 &{map[kind:Secret metadata:map[name:master-user-data namespace:openshift-machine-api] type:Opaque apiVersion:v1 data:map[userData:dHJ1ZQo= disableTemplating:dHJ1ZQo=]]} 2019/09/12 10:32:07 2019/09/12 10:32:07 &{map[apiVersion:v1 data:map[disableTemplating:dHJ1ZQo= userData:dHJ1ZQo=] kind:Secret metadata:map[name:master-user-data namespace:openshift-machine-api] type:Opaque]} $ cat <<EOF >test.yaml > kind: List > apiVersion: v1 > metadata: > resourceVersion: "" > selfLink: "" > items: > - apiVersion: v1 > kind: Secret > metadata: > name: master-user-data > namespace: openshift-machine-api > type: Opaque > data: > disableTemplating: dHJ1ZQo= > userData: dHJ1ZQo= > EOF $ go run test.go 2019/09/12 10:33:20 &{map[kind:List metadata:map[resourceVersion: selfLink:] apiVersion:v1] [{map[metadata:map[name:master-user-data namespace:openshift-machine-api] type:Opaque apiVersion:v1 data:map[disableTemplating:dHJ1ZQo= userData:dHJ1ZQo=] kind:Secret]}]} 2019/09/12 10:33:20 2019/09/12 10:33:20 <nil> exit status 1 [1]: openshift#1381 (comment) [2]: https://github.com/openshift/library-go/blob/7d4acc018c610623ee1413b1e7aebe2ac675d35f/pkg/assets/create/creater.go#L264
13e7a4d
to
b982886
Compare
All green :). |
Actually, looks like we may need to trigger these manually after a force-push? /test e2e-gcp |
All green :). Another round for luck: /test e2e-aws |
Azure failed with:
We hit that flake sometimes (although not usually on Azure). Doesn't seem like a setup-time issue, but just in case, we'll go around again: /test e2e-azure |
All green again :) |
/test all /hold cancel |
/retest |
@abhinavdahiya, green enough? |
/lgtm |
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: abhinavdahiya, wking The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
/retest Please review the full test history for this PR and help us cut down flakes. |
1 similar comment
/retest Please review the full test history for this PR and help us cut down flakes. |
@wking: All pull requests linked via external trackers have merged. Bugzilla bug 1748452 has been moved to the MODIFIED state. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
…ntrol-plane teardown Since 82d81d9 (data/data/bootstrap: use loopback kubeconfig for API access, 2019-07-24, openshift#2086), we've been pushing the OpenShift-specific components via a loopback kubeconfig and the bootstrap control plane. Since 108a45b (data/bootstrap: Replace openshift.sh with cluster-bootstrap, 2019-01-24, landed 2019-09-30, openshift#1381), that push has always happened before the bootstrap control plane shuts down. I've changed "into" to "via" because the data passes through either control plane on its way to rest in the shared etcd cluster. I've dropped "then" from the final step because none of the other steps said "then". I think ordering is clear enough from our use of an ordered list ;).
With this commit, I take advantage of openshift/cluster-bootstrap@fc5e0941 (start: wire the library-go dynamic client create, 2019-02-05, openshift/cluster-bootstrap#14) to replace our previous openshift.sh (with a minor change to the manifest directory). I'm currently using a cp in bootkube.sh to shift those manifests into the generic directory; I plan on consolidating Openshift into Manifests in pkg/asset/manifests in follow-up work. This change is especially important since the pivot to loopback kubeconfigs in openshift.sh: 82d81d9 (data/data/bootstrap: use loopback kubeconfig for API access, 2019-07-24, openshift#2086), because once cluster-bootstrap (launched from bootkube.sh) decides it's done it tears down the bootstrap control plane. Without the bootstrap control plane, further attempts by openshift.sh to push manifests via the loopback kubeconfig fail [1]. We could roll reporting into bootkube.sh as well (dropping progress.service), but Abhinav wanted to keep it separate [2]. [1]: https://bugzilla.redhat.com/show_bug.cgi?id=1748452 [2]: openshift#1381 (comment)
This may have been what was giving us errors like: Sep 10 19:41:16 ip-10-0-3-146 bootkube.sh[1605]: "99_openshift-cluster-api_master-user-data-secret.yaml": unable to convert asset "99_openshift-cluster-api_master-user-data-secret.yaml" to unstructed Sep 10 19:41:16 ip-10-0-3-146 bootkube.sh[1605]: "99_openshift-cluster-api_worker-user-data-secret.yaml": unable to convert asset "99_openshift-cluster-api_worker-user-data-secret.yaml" to unstructed after failing this cast to *unstructured.Unstructured [2]. I don't have a theory for why the unstructured cast fails for Lists, but that's what I see in local testing: $ cat test.go package main import ( "io/ioutil" "log" "os" "github.com/ghodss/yaml" "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" "k8s.io/apimachinery/pkg/runtime" ) func main() { file, err := os.Open("test.yaml") if err != nil { log.Fatal(err) } manifestBytes, err := ioutil.ReadAll(file) if err != nil { log.Fatal(err) } manifestJSON, err := yaml.YAMLToJSON(manifestBytes) if err != nil { log.Fatal(err) } manifestObj, err := runtime.Decode(unstructured.UnstructuredJSONScheme, manifestJSON) if err != nil { log.Fatal(err) } log.Print(manifestObj) log.Print("") manifestUnstructured, ok := manifestObj.(*unstructured.Unstructured) if !ok { log.Fatal(manifestUnstructured) } log.Print(manifestUnstructured) } $ cat <<EOF >test.yaml > apiVersion: v1 > kind: Secret > metadata: > name: master-user-data > namespace: openshift-machine-api > type: Opaque > data: > disableTemplating: dHJ1ZQo= > userData: dHJ1ZQo= > EOF $ go run test.go 2019/09/12 10:32:07 &{map[kind:Secret metadata:map[name:master-user-data namespace:openshift-machine-api] type:Opaque apiVersion:v1 data:map[userData:dHJ1ZQo= disableTemplating:dHJ1ZQo=]]} 2019/09/12 10:32:07 2019/09/12 10:32:07 &{map[apiVersion:v1 data:map[disableTemplating:dHJ1ZQo= userData:dHJ1ZQo=] kind:Secret metadata:map[name:master-user-data namespace:openshift-machine-api] type:Opaque]} $ cat <<EOF >test.yaml > kind: List > apiVersion: v1 > metadata: > resourceVersion: "" > selfLink: "" > items: > - apiVersion: v1 > kind: Secret > metadata: > name: master-user-data > namespace: openshift-machine-api > type: Opaque > data: > disableTemplating: dHJ1ZQo= > userData: dHJ1ZQo= > EOF $ go run test.go 2019/09/12 10:33:20 &{map[kind:List metadata:map[resourceVersion: selfLink:] apiVersion:v1] [{map[metadata:map[name:master-user-data namespace:openshift-machine-api] type:Opaque apiVersion:v1 data:map[disableTemplating:dHJ1ZQo= userData:dHJ1ZQo=] kind:Secret]}]} 2019/09/12 10:33:20 2019/09/12 10:33:20 <nil> exit status 1 [1]: openshift#1381 (comment) [2]: https://github.com/openshift/library-go/blob/7d4acc018c610623ee1413b1e7aebe2ac675d35f/pkg/assets/create/creater.go#L264
…ntrol-plane teardown Since 82d81d9 (data/data/bootstrap: use loopback kubeconfig for API access, 2019-07-24, openshift#2086), we've been pushing the OpenShift-specific components via a loopback kubeconfig and the bootstrap control plane. Since 108a45b (data/bootstrap: Replace openshift.sh with cluster-bootstrap, 2019-01-24, landed 2019-09-30, openshift#1381), that push has always happened before the bootstrap control plane shuts down. I've changed "into" to "via" because the data passes through either control plane on its way to rest in the shared etcd cluster. I've dropped "then" from the final step because none of the other steps said "then". I think ordering is clear enough from our use of an ordered list ;).
With this commit, I take advantage of openshift/cluster-bootstrap@fc5e0941 (start: wire the library-go dynamic client create, 2019-02-05, openshift/cluster-bootstrap#14) to replace our previous openshift.sh (with a minor change to the manifest directory). I'm currently using a cp in bootkube.sh to shift those manifests into the generic directory; I plan on consolidating Openshift into Manifests in pkg/asset/manifests in follow-up work. This change is especially important since the pivot to loopback kubeconfigs in openshift.sh: 82d81d9 (data/data/bootstrap: use loopback kubeconfig for API access, 2019-07-24, openshift#2086), because once cluster-bootstrap (launched from bootkube.sh) decides it's done it tears down the bootstrap control plane. Without the bootstrap control plane, further attempts by openshift.sh to push manifests via the loopback kubeconfig fail [1]. We could roll reporting into bootkube.sh as well (dropping progress.service), but Abhinav wanted to keep it separate [2]. [1]: https://bugzilla.redhat.com/show_bug.cgi?id=1748452 [2]: openshift#1381 (comment)
This may have been what was giving us errors like: Sep 10 19:41:16 ip-10-0-3-146 bootkube.sh[1605]: "99_openshift-cluster-api_master-user-data-secret.yaml": unable to convert asset "99_openshift-cluster-api_master-user-data-secret.yaml" to unstructed Sep 10 19:41:16 ip-10-0-3-146 bootkube.sh[1605]: "99_openshift-cluster-api_worker-user-data-secret.yaml": unable to convert asset "99_openshift-cluster-api_worker-user-data-secret.yaml" to unstructed after failing this cast to *unstructured.Unstructured [2]. I don't have a theory for why the unstructured cast fails for Lists, but that's what I see in local testing: $ cat test.go package main import ( "io/ioutil" "log" "os" "github.com/ghodss/yaml" "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" "k8s.io/apimachinery/pkg/runtime" ) func main() { file, err := os.Open("test.yaml") if err != nil { log.Fatal(err) } manifestBytes, err := ioutil.ReadAll(file) if err != nil { log.Fatal(err) } manifestJSON, err := yaml.YAMLToJSON(manifestBytes) if err != nil { log.Fatal(err) } manifestObj, err := runtime.Decode(unstructured.UnstructuredJSONScheme, manifestJSON) if err != nil { log.Fatal(err) } log.Print(manifestObj) log.Print("") manifestUnstructured, ok := manifestObj.(*unstructured.Unstructured) if !ok { log.Fatal(manifestUnstructured) } log.Print(manifestUnstructured) } $ cat <<EOF >test.yaml > apiVersion: v1 > kind: Secret > metadata: > name: master-user-data > namespace: openshift-machine-api > type: Opaque > data: > disableTemplating: dHJ1ZQo= > userData: dHJ1ZQo= > EOF $ go run test.go 2019/09/12 10:32:07 &{map[kind:Secret metadata:map[name:master-user-data namespace:openshift-machine-api] type:Opaque apiVersion:v1 data:map[userData:dHJ1ZQo= disableTemplating:dHJ1ZQo=]]} 2019/09/12 10:32:07 2019/09/12 10:32:07 &{map[apiVersion:v1 data:map[disableTemplating:dHJ1ZQo= userData:dHJ1ZQo=] kind:Secret metadata:map[name:master-user-data namespace:openshift-machine-api] type:Opaque]} $ cat <<EOF >test.yaml > kind: List > apiVersion: v1 > metadata: > resourceVersion: "" > selfLink: "" > items: > - apiVersion: v1 > kind: Secret > metadata: > name: master-user-data > namespace: openshift-machine-api > type: Opaque > data: > disableTemplating: dHJ1ZQo= > userData: dHJ1ZQo= > EOF $ go run test.go 2019/09/12 10:33:20 &{map[kind:List metadata:map[resourceVersion: selfLink:] apiVersion:v1] [{map[metadata:map[name:master-user-data namespace:openshift-machine-api] type:Opaque apiVersion:v1 data:map[disableTemplating:dHJ1ZQo= userData:dHJ1ZQo=] kind:Secret]}]} 2019/09/12 10:33:20 2019/09/12 10:33:20 <nil> exit status 1 [1]: openshift#1381 (comment) [2]: https://github.com/openshift/library-go/blob/7d4acc018c610623ee1413b1e7aebe2ac675d35f/pkg/assets/create/creater.go#L264
…ntrol-plane teardown Since 82d81d9 (data/data/bootstrap: use loopback kubeconfig for API access, 2019-07-24, openshift#2086), we've been pushing the OpenShift-specific components via a loopback kubeconfig and the bootstrap control plane. Since 108a45b (data/bootstrap: Replace openshift.sh with cluster-bootstrap, 2019-01-24, landed 2019-09-30, openshift#1381), that push has always happened before the bootstrap control plane shuts down. I've changed "into" to "via" because the data passes through either control plane on its way to rest in the shared etcd cluster. I've dropped "then" from the final step because none of the other steps said "then". I think ordering is clear enough from our use of an ordered list ;).
With this commit, I take advantage of openshift/cluster-bootstrap@fc5e0941 (openshift/cluster-bootstrap#14) to replace our previous openshift.sh (with a minor change to the manifest directory). I'm currently using a cp in bootkube.sh to shift those manifests into the generic directory; I plan on consolidating
Openshift
intoManifests
inpkg/asset/manifests
in follow-up work.This spins off part of my earlier #1147 (hopefully the less-controversial part ;).