Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bug 1748452: data/bootstrap: Replace openshift.sh with cluster-bootstrap #1381

Merged
merged 2 commits into from
Oct 1, 2019

Conversation

wking
Copy link
Member

@wking wking commented Mar 7, 2019

With this commit, I take advantage of openshift/cluster-bootstrap@fc5e0941 (openshift/cluster-bootstrap#14) to replace our previous openshift.sh (with a minor change to the manifest directory). I'm currently using a cp in bootkube.sh to shift those manifests into the generic directory; I plan on consolidating Openshift into Manifests in pkg/asset/manifests in follow-up work.

This spins off part of my earlier #1147 (hopefully the less-controversial part ;).

@openshift-ci-robot openshift-ci-robot added approved Indicates a PR has been approved by an approver from all required OWNERS files. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Mar 7, 2019
@wking wking added the version/4.1 Tracking changes that should end up in 4.1 release label Mar 7, 2019
@wking
Copy link
Member Author

wking commented Mar 7, 2019

/hold

We probably want this in 4.1, but if I change the base to point at master-4.1 the diff will be huge ;). I'm just going to put a hold on this while it's under review, and if we get to the point where we want to merge it I'll file a separate PR to fast-forward master-4.1 and then rebase this one on top.

@openshift-ci-robot openshift-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Mar 7, 2019
@trown
Copy link

trown commented Apr 16, 2019

/test e2e-openstack

@openshift-ci-robot openshift-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Apr 24, 2019
@openshift-ci-robot openshift-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Sep 10, 2019
@wking
Copy link
Member Author

wking commented Sep 10, 2019

Rebased with f0dcefe -> d607896. And since the pivot to loopback kubeconfigs in #2086, having a separate openshift.sh pushing in parallel (and possibly after) cluster-bootstrap is now a racy bug.

/retitle Bug 1748452: data/bootstrap: Replace openshift.sh with cluster-bootstrap

@openshift-ci-robot openshift-ci-robot changed the title data/bootstrap: Replace openshift.sh with cluster-bootstrap Bug 1748452: data/bootstrap: Replace openshift.sh with cluster-bootstrap Sep 10, 2019
@openshift-ci-robot
Copy link
Contributor

@wking: This pull request references Bugzilla bug 1748452, which is invalid:

  • expected the bug to target the "4.2.0" release, but it targets "4.3.0" instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

Bug 1748452: data/bootstrap: Replace openshift.sh with cluster-bootstrap

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-ci-robot openshift-ci-robot added the bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. label Sep 10, 2019
@wking
Copy link
Member Author

wking commented Sep 10, 2019

/bugzilla refresh

@openshift-ci-robot openshift-ci-robot added bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. and removed bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. labels Sep 10, 2019
@openshift-ci-robot
Copy link
Contributor

@wking: This pull request references Bugzilla bug 1748452, which is valid. The bug has been moved to the POST state.

In response to this:

/bugzilla refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@abhinavdahiya
Copy link
Contributor

the apiserver team that owns the cluster-bootstrap has made it very clear in the past that it should only be used for bootstrapping the k8s control plane, therefore we have the openshift.service that keeps the separation of concerns, Therefore I don't agree with using the bootkube.service's cluster-bootstrap to push these objects that are not required to boot the k8s control plane.

@wking
Copy link
Member Author

wking commented Sep 11, 2019

e2e-aws failed with:

Sep 10 19:41:16 ip-10-0-3-146 bootkube.sh[1605]: "99_openshift-cluster-api_master-user-data-secret.yaml": unable to convert asset "99_openshift-cluster-api_master-user-data-secret.yaml" to unstructed
Sep 10 19:41:16 ip-10-0-3-146 bootkube.sh[1605]: "99_openshift-cluster-api_worker-user-data-secret.yaml": unable to convert asset "99_openshift-cluster-api_worker-user-data-secret.yaml" to unstructed

in ./bundle/bootstrap/journals/bootkube.log in the gathered logs. I'm not clear on why yet.

the apiserver team that owns the cluster-bootstrap has made it very clear in the past that it should only be used for bootstrapping the k8s control plane

I'm not clear yet on why they'd care about them pushing these manifests vs. us pushing these in parallel. Maybe it's the unstructured casting issue I'm hitting :p.

wking added a commit to wking/openshift-installer that referenced this pull request Sep 11, 2019
This may have been what was giving us errors like:

  Sep 10 19:41:16 ip-10-0-3-146 bootkube.sh[1605]: "99_openshift-cluster-api_master-user-data-secret.yaml": unable to convert asset "99_openshift-cluster-api_master-user-data-secret.yaml" to unstructed
  Sep 10 19:41:16 ip-10-0-3-146 bootkube.sh[1605]: "99_openshift-cluster-api_worker-user-data-secret.yaml": unable to convert asset "99_openshift-cluster-api_worker-user-data-secret.yaml" to unstructed

after failing this cast to *unstructured.Unstructured [2].  I don't
have a theory for why a leading empty line would matter, but it was
the only thing that stuck at me when eyeballing the user-data
manifests.

[1]: openshift#1381 (comment)
[2]: https://github.com/openshift/library-go/blob/7d4acc018c610623ee1413b1e7aebe2ac675d35f/pkg/assets/create/creater.go#L264
@wking
Copy link
Member Author

wking commented Sep 11, 2019

Ah, looks like 9e4b0d4 didn't help. Still getting:

Sep 11 19:54:03 ip-10-0-5-50 bootkube.sh[1606]: "99_openshift-cluster-api_master-user-data-secret.yaml": unable to convert asset "99_openshift-cluster-api_master-user-data-secret.yaml" to unstructed
Sep 11 19:54:03 ip-10-0-5-50 bootkube.sh[1606]: "99_openshift-cluster-api_worker-user-data-secret.yaml": unable to convert asset "99_openshift-cluster-api_worker-user-data-secret.yaml" to unstructed

from the e2e-aws log bundle.

wking added a commit to wking/openshift-installer that referenced this pull request Sep 12, 2019
This may have been what was giving us errors like:

  Sep 10 19:41:16 ip-10-0-3-146 bootkube.sh[1605]: "99_openshift-cluster-api_master-user-data-secret.yaml": unable to convert asset "99_openshift-cluster-api_master-user-data-secret.yaml" to unstructed
  Sep 10 19:41:16 ip-10-0-3-146 bootkube.sh[1605]: "99_openshift-cluster-api_worker-user-data-secret.yaml": unable to convert asset "99_openshift-cluster-api_worker-user-data-secret.yaml" to unstructed

after failing this cast to *unstructured.Unstructured [2].  I don't
have a theory for why the unstructured cast fails for Lists, but
that's what I see in local testing:

  $ cat test.go
  package main

  import (
    "io/ioutil"
    "log"
    "os"

    "github.com/ghodss/yaml"
    "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"
    "k8s.io/apimachinery/pkg/runtime"
  )

  func main() {
    file, err := os.Open("test.yaml")
    if err != nil {
      log.Fatal(err)
    }

    manifestBytes, err := ioutil.ReadAll(file)
    if err != nil {
      log.Fatal(err)
    }

    manifestJSON, err := yaml.YAMLToJSON(manifestBytes)
    if err != nil {
      log.Fatal(err)
    }

    manifestObj, err := runtime.Decode(unstructured.UnstructuredJSONScheme, manifestJSON)
    if err != nil {
      log.Fatal(err)
    }
    log.Print(manifestObj)
    log.Print("")

    manifestUnstructured, ok := manifestObj.(*unstructured.Unstructured)
    if !ok {
      log.Fatal(manifestUnstructured)
    }

    log.Print(manifestUnstructured)
  }
  $ cat <<EOF >test.yaml
  > apiVersion: v1
  > kind: Secret
  > metadata:
  >   name: master-user-data
  >   namespace: openshift-machine-api
  > type: Opaque
  > data:
  >   disableTemplating: dHJ1ZQo=
  >   userData: dHJ1ZQo=
  > EOF
  $ go run test.go
  2019/09/12 10:32:07 &{map[kind:Secret metadata:map[name:master-user-data namespace:openshift-machine-api] type:Opaque apiVersion:v1 data:map[userData:dHJ1ZQo= disableTemplating:dHJ1ZQo=]]}
  2019/09/12 10:32:07
  2019/09/12 10:32:07 &{map[apiVersion:v1 data:map[disableTemplating:dHJ1ZQo= userData:dHJ1ZQo=] kind:Secret metadata:map[name:master-user-data namespace:openshift-machine-api] type:Opaque]}
  $ cat <<EOF >test.yaml
  > kind: List
  > apiVersion: v1
  > metadata:
  >   resourceVersion: ""
  >   selfLink: ""
  > items:
  > - apiVersion: v1
  >   kind: Secret
  >   metadata:
  >     name: master-user-data
  >     namespace: openshift-machine-api
  >   type: Opaque
  >   data:
  >     disableTemplating: dHJ1ZQo=
  >     userData: dHJ1ZQo=
  > EOF
  $ go run test.go
  2019/09/12 10:33:20 &{map[kind:List metadata:map[resourceVersion: selfLink:] apiVersion:v1] [{map[metadata:map[name:master-user-data namespace:openshift-machine-api] type:Opaque apiVersion:v1 data:map[disableTemplating:dHJ1ZQo= userData:dHJ1ZQo=] kind:Secret]}]}
  2019/09/12 10:33:20
  2019/09/12 10:33:20 <nil>
  exit status 1

[1]: openshift#1381 (comment)
[2]: https://github.com/openshift/library-go/blob/7d4acc018c610623ee1413b1e7aebe2ac675d35f/pkg/assets/create/creater.go#L264
@openshift-ci-robot openshift-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Sep 12, 2019
With this commit, I take advantage of
openshift/cluster-bootstrap@fc5e0941 (start: wire the library-go
dynamic client create, 2019-02-05, openshift/cluster-bootstrap#14) to
replace our previous openshift.sh (with a minor change to the manifest
directory).  I'm currently using a cp in bootkube.sh to shift those
manifests into the generic directory; I plan on consolidating
Openshift into Manifests in pkg/asset/manifests in follow-up work.

This change is especially important since the pivot to loopback
kubeconfigs in openshift.sh: 82d81d9 (data/data/bootstrap: use
loopback kubeconfig for API access, 2019-07-24, openshift#2086), because once
cluster-bootstrap (launched from bootkube.sh) decides it's done it
tears down the bootstrap control plane.  Without the bootstrap control
plane, further attempts by openshift.sh to push manifests via the
loopback kubeconfig fail [1].

We could roll reporting into bootkube.sh as well (dropping
progress.service), but Abhinav wanted to keep it separate [2].

[1]: https://bugzilla.redhat.com/show_bug.cgi?id=1748452
[2]: openshift#1381 (comment)
This may have been what was giving us errors like:

  Sep 10 19:41:16 ip-10-0-3-146 bootkube.sh[1605]: "99_openshift-cluster-api_master-user-data-secret.yaml": unable to convert asset "99_openshift-cluster-api_master-user-data-secret.yaml" to unstructed
  Sep 10 19:41:16 ip-10-0-3-146 bootkube.sh[1605]: "99_openshift-cluster-api_worker-user-data-secret.yaml": unable to convert asset "99_openshift-cluster-api_worker-user-data-secret.yaml" to unstructed

after failing this cast to *unstructured.Unstructured [2].  I don't
have a theory for why the unstructured cast fails for Lists, but
that's what I see in local testing:

  $ cat test.go
  package main

  import (
    "io/ioutil"
    "log"
    "os"

    "github.com/ghodss/yaml"
    "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"
    "k8s.io/apimachinery/pkg/runtime"
  )

  func main() {
    file, err := os.Open("test.yaml")
    if err != nil {
      log.Fatal(err)
    }

    manifestBytes, err := ioutil.ReadAll(file)
    if err != nil {
      log.Fatal(err)
    }

    manifestJSON, err := yaml.YAMLToJSON(manifestBytes)
    if err != nil {
      log.Fatal(err)
    }

    manifestObj, err := runtime.Decode(unstructured.UnstructuredJSONScheme, manifestJSON)
    if err != nil {
      log.Fatal(err)
    }
    log.Print(manifestObj)
    log.Print("")

    manifestUnstructured, ok := manifestObj.(*unstructured.Unstructured)
    if !ok {
      log.Fatal(manifestUnstructured)
    }

    log.Print(manifestUnstructured)
  }
  $ cat <<EOF >test.yaml
  > apiVersion: v1
  > kind: Secret
  > metadata:
  >   name: master-user-data
  >   namespace: openshift-machine-api
  > type: Opaque
  > data:
  >   disableTemplating: dHJ1ZQo=
  >   userData: dHJ1ZQo=
  > EOF
  $ go run test.go
  2019/09/12 10:32:07 &{map[kind:Secret metadata:map[name:master-user-data namespace:openshift-machine-api] type:Opaque apiVersion:v1 data:map[userData:dHJ1ZQo= disableTemplating:dHJ1ZQo=]]}
  2019/09/12 10:32:07
  2019/09/12 10:32:07 &{map[apiVersion:v1 data:map[disableTemplating:dHJ1ZQo= userData:dHJ1ZQo=] kind:Secret metadata:map[name:master-user-data namespace:openshift-machine-api] type:Opaque]}
  $ cat <<EOF >test.yaml
  > kind: List
  > apiVersion: v1
  > metadata:
  >   resourceVersion: ""
  >   selfLink: ""
  > items:
  > - apiVersion: v1
  >   kind: Secret
  >   metadata:
  >     name: master-user-data
  >     namespace: openshift-machine-api
  >   type: Opaque
  >   data:
  >     disableTemplating: dHJ1ZQo=
  >     userData: dHJ1ZQo=
  > EOF
  $ go run test.go
  2019/09/12 10:33:20 &{map[kind:List metadata:map[resourceVersion: selfLink:] apiVersion:v1] [{map[metadata:map[name:master-user-data namespace:openshift-machine-api] type:Opaque apiVersion:v1 data:map[disableTemplating:dHJ1ZQo= userData:dHJ1ZQo=] kind:Secret]}]}
  2019/09/12 10:33:20
  2019/09/12 10:33:20 <nil>
  exit status 1

[1]: openshift#1381 (comment)
[2]: https://github.com/openshift/library-go/blob/7d4acc018c610623ee1413b1e7aebe2ac675d35f/pkg/assets/create/creater.go#L264
@wking
Copy link
Member Author

wking commented Sep 17, 2019

Ah, fixed another part of progress.service restoration with 13e7a4d -> b982886.

@wking
Copy link
Member Author

wking commented Sep 17, 2019

All green :).

@wking
Copy link
Member Author

wking commented Sep 17, 2019

Actually, looks like we may need to trigger these manually after a force-push?

/test e2e-gcp
/test e2e-azure

@wking
Copy link
Member Author

wking commented Sep 18, 2019

All green :). Another round for luck:

/test e2e-aws
/test e2e-azure
/test e2e-gcp

@wking
Copy link
Member Author

wking commented Sep 18, 2019

Azure failed with:

Sep 18 03:15:12.412: INFO: Waiting for the OAuth server route to be ready: EOF
...
fail [github.com/openshift/origin/test/extended/oauth/expiration.go:30]: Unexpected error:
    <*errors.errorString | 0xc000273390>: {
        s: "timed out waiting for the condition",
    }
    timed out waiting for the condition
occurred
...
failed: (1m39s) 2019-09-18T03:15:44 "[Feature:OAuthServer] [Token Expiration] Using a OAuth client with a non-default token max age to generate tokens that do not expire works as expected when using a code authorization flow [Suite:openshift/conformance/parallel]"

We hit that flake sometimes (although not usually on Azure). Doesn't seem like a setup-time issue, but just in case, we'll go around again:

/test e2e-azure

@wking
Copy link
Member Author

wking commented Sep 18, 2019

All green again :)

@abhinavdahiya
Copy link
Contributor

/test all

/hold cancel

@openshift-ci-robot openshift-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Sep 24, 2019
@wking
Copy link
Member Author

wking commented Sep 26, 2019

/retest

@wking
Copy link
Member Author

wking commented Sep 26, 2019

@abhinavdahiya, green enough?

@abhinavdahiya
Copy link
Contributor

/lgtm

@openshift-ci-robot openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Sep 30, 2019
@openshift-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: abhinavdahiya, wking

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:
  • OWNERS [abhinavdahiya,wking]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

1 similar comment
@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-merge-robot openshift-merge-robot merged commit 4243b7f into openshift:master Oct 1, 2019
@openshift-ci-robot
Copy link
Contributor

@wking: All pull requests linked via external trackers have merged. Bugzilla bug 1748452 has been moved to the MODIFIED state.

In response to this:

Bug 1748452: data/bootstrap: Replace openshift.sh with cluster-bootstrap

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@wking wking deleted the drop-openshift.sh branch October 1, 2019 03:01
wking added a commit to wking/openshift-installer that referenced this pull request Oct 8, 2019
…ntrol-plane teardown

Since 82d81d9 (data/data/bootstrap: use loopback kubeconfig for API
access, 2019-07-24, openshift#2086), we've been pushing the OpenShift-specific
components via a loopback kubeconfig and the bootstrap control plane.
Since 108a45b (data/bootstrap: Replace openshift.sh with
cluster-bootstrap, 2019-01-24, landed 2019-09-30, openshift#1381), that push
has always happened before the bootstrap control plane shuts down.

I've changed "into" to "via" because the data passes through either
control plane on its way to rest in the shared etcd cluster.

I've dropped "then" from the final step because none of the other
steps said "then".  I think ordering is clear enough from our use of
an ordered list ;).
alaypatel07 pushed a commit to alaypatel07/installer that referenced this pull request Nov 13, 2019
With this commit, I take advantage of
openshift/cluster-bootstrap@fc5e0941 (start: wire the library-go
dynamic client create, 2019-02-05, openshift/cluster-bootstrap#14) to
replace our previous openshift.sh (with a minor change to the manifest
directory).  I'm currently using a cp in bootkube.sh to shift those
manifests into the generic directory; I plan on consolidating
Openshift into Manifests in pkg/asset/manifests in follow-up work.

This change is especially important since the pivot to loopback
kubeconfigs in openshift.sh: 82d81d9 (data/data/bootstrap: use
loopback kubeconfig for API access, 2019-07-24, openshift#2086), because once
cluster-bootstrap (launched from bootkube.sh) decides it's done it
tears down the bootstrap control plane.  Without the bootstrap control
plane, further attempts by openshift.sh to push manifests via the
loopback kubeconfig fail [1].

We could roll reporting into bootkube.sh as well (dropping
progress.service), but Abhinav wanted to keep it separate [2].

[1]: https://bugzilla.redhat.com/show_bug.cgi?id=1748452
[2]: openshift#1381 (comment)
alaypatel07 pushed a commit to alaypatel07/installer that referenced this pull request Nov 13, 2019
This may have been what was giving us errors like:

  Sep 10 19:41:16 ip-10-0-3-146 bootkube.sh[1605]: "99_openshift-cluster-api_master-user-data-secret.yaml": unable to convert asset "99_openshift-cluster-api_master-user-data-secret.yaml" to unstructed
  Sep 10 19:41:16 ip-10-0-3-146 bootkube.sh[1605]: "99_openshift-cluster-api_worker-user-data-secret.yaml": unable to convert asset "99_openshift-cluster-api_worker-user-data-secret.yaml" to unstructed

after failing this cast to *unstructured.Unstructured [2].  I don't
have a theory for why the unstructured cast fails for Lists, but
that's what I see in local testing:

  $ cat test.go
  package main

  import (
    "io/ioutil"
    "log"
    "os"

    "github.com/ghodss/yaml"
    "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"
    "k8s.io/apimachinery/pkg/runtime"
  )

  func main() {
    file, err := os.Open("test.yaml")
    if err != nil {
      log.Fatal(err)
    }

    manifestBytes, err := ioutil.ReadAll(file)
    if err != nil {
      log.Fatal(err)
    }

    manifestJSON, err := yaml.YAMLToJSON(manifestBytes)
    if err != nil {
      log.Fatal(err)
    }

    manifestObj, err := runtime.Decode(unstructured.UnstructuredJSONScheme, manifestJSON)
    if err != nil {
      log.Fatal(err)
    }
    log.Print(manifestObj)
    log.Print("")

    manifestUnstructured, ok := manifestObj.(*unstructured.Unstructured)
    if !ok {
      log.Fatal(manifestUnstructured)
    }

    log.Print(manifestUnstructured)
  }
  $ cat <<EOF >test.yaml
  > apiVersion: v1
  > kind: Secret
  > metadata:
  >   name: master-user-data
  >   namespace: openshift-machine-api
  > type: Opaque
  > data:
  >   disableTemplating: dHJ1ZQo=
  >   userData: dHJ1ZQo=
  > EOF
  $ go run test.go
  2019/09/12 10:32:07 &{map[kind:Secret metadata:map[name:master-user-data namespace:openshift-machine-api] type:Opaque apiVersion:v1 data:map[userData:dHJ1ZQo= disableTemplating:dHJ1ZQo=]]}
  2019/09/12 10:32:07
  2019/09/12 10:32:07 &{map[apiVersion:v1 data:map[disableTemplating:dHJ1ZQo= userData:dHJ1ZQo=] kind:Secret metadata:map[name:master-user-data namespace:openshift-machine-api] type:Opaque]}
  $ cat <<EOF >test.yaml
  > kind: List
  > apiVersion: v1
  > metadata:
  >   resourceVersion: ""
  >   selfLink: ""
  > items:
  > - apiVersion: v1
  >   kind: Secret
  >   metadata:
  >     name: master-user-data
  >     namespace: openshift-machine-api
  >   type: Opaque
  >   data:
  >     disableTemplating: dHJ1ZQo=
  >     userData: dHJ1ZQo=
  > EOF
  $ go run test.go
  2019/09/12 10:33:20 &{map[kind:List metadata:map[resourceVersion: selfLink:] apiVersion:v1] [{map[metadata:map[name:master-user-data namespace:openshift-machine-api] type:Opaque apiVersion:v1 data:map[disableTemplating:dHJ1ZQo= userData:dHJ1ZQo=] kind:Secret]}]}
  2019/09/12 10:33:20
  2019/09/12 10:33:20 <nil>
  exit status 1

[1]: openshift#1381 (comment)
[2]: https://github.com/openshift/library-go/blob/7d4acc018c610623ee1413b1e7aebe2ac675d35f/pkg/assets/create/creater.go#L264
alaypatel07 pushed a commit to alaypatel07/installer that referenced this pull request Nov 13, 2019
…ntrol-plane teardown

Since 82d81d9 (data/data/bootstrap: use loopback kubeconfig for API
access, 2019-07-24, openshift#2086), we've been pushing the OpenShift-specific
components via a loopback kubeconfig and the bootstrap control plane.
Since 108a45b (data/bootstrap: Replace openshift.sh with
cluster-bootstrap, 2019-01-24, landed 2019-09-30, openshift#1381), that push
has always happened before the bootstrap control plane shuts down.

I've changed "into" to "via" because the data passes through either
control plane on its way to rest in the shared etcd cluster.

I've dropped "then" from the final step because none of the other
steps said "then".  I think ordering is clear enough from our use of
an ordered list ;).
jhixson74 pushed a commit to jhixson74/installer that referenced this pull request Dec 6, 2019
With this commit, I take advantage of
openshift/cluster-bootstrap@fc5e0941 (start: wire the library-go
dynamic client create, 2019-02-05, openshift/cluster-bootstrap#14) to
replace our previous openshift.sh (with a minor change to the manifest
directory).  I'm currently using a cp in bootkube.sh to shift those
manifests into the generic directory; I plan on consolidating
Openshift into Manifests in pkg/asset/manifests in follow-up work.

This change is especially important since the pivot to loopback
kubeconfigs in openshift.sh: 82d81d9 (data/data/bootstrap: use
loopback kubeconfig for API access, 2019-07-24, openshift#2086), because once
cluster-bootstrap (launched from bootkube.sh) decides it's done it
tears down the bootstrap control plane.  Without the bootstrap control
plane, further attempts by openshift.sh to push manifests via the
loopback kubeconfig fail [1].

We could roll reporting into bootkube.sh as well (dropping
progress.service), but Abhinav wanted to keep it separate [2].

[1]: https://bugzilla.redhat.com/show_bug.cgi?id=1748452
[2]: openshift#1381 (comment)
jhixson74 pushed a commit to jhixson74/installer that referenced this pull request Dec 6, 2019
This may have been what was giving us errors like:

  Sep 10 19:41:16 ip-10-0-3-146 bootkube.sh[1605]: "99_openshift-cluster-api_master-user-data-secret.yaml": unable to convert asset "99_openshift-cluster-api_master-user-data-secret.yaml" to unstructed
  Sep 10 19:41:16 ip-10-0-3-146 bootkube.sh[1605]: "99_openshift-cluster-api_worker-user-data-secret.yaml": unable to convert asset "99_openshift-cluster-api_worker-user-data-secret.yaml" to unstructed

after failing this cast to *unstructured.Unstructured [2].  I don't
have a theory for why the unstructured cast fails for Lists, but
that's what I see in local testing:

  $ cat test.go
  package main

  import (
    "io/ioutil"
    "log"
    "os"

    "github.com/ghodss/yaml"
    "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"
    "k8s.io/apimachinery/pkg/runtime"
  )

  func main() {
    file, err := os.Open("test.yaml")
    if err != nil {
      log.Fatal(err)
    }

    manifestBytes, err := ioutil.ReadAll(file)
    if err != nil {
      log.Fatal(err)
    }

    manifestJSON, err := yaml.YAMLToJSON(manifestBytes)
    if err != nil {
      log.Fatal(err)
    }

    manifestObj, err := runtime.Decode(unstructured.UnstructuredJSONScheme, manifestJSON)
    if err != nil {
      log.Fatal(err)
    }
    log.Print(manifestObj)
    log.Print("")

    manifestUnstructured, ok := manifestObj.(*unstructured.Unstructured)
    if !ok {
      log.Fatal(manifestUnstructured)
    }

    log.Print(manifestUnstructured)
  }
  $ cat <<EOF >test.yaml
  > apiVersion: v1
  > kind: Secret
  > metadata:
  >   name: master-user-data
  >   namespace: openshift-machine-api
  > type: Opaque
  > data:
  >   disableTemplating: dHJ1ZQo=
  >   userData: dHJ1ZQo=
  > EOF
  $ go run test.go
  2019/09/12 10:32:07 &{map[kind:Secret metadata:map[name:master-user-data namespace:openshift-machine-api] type:Opaque apiVersion:v1 data:map[userData:dHJ1ZQo= disableTemplating:dHJ1ZQo=]]}
  2019/09/12 10:32:07
  2019/09/12 10:32:07 &{map[apiVersion:v1 data:map[disableTemplating:dHJ1ZQo= userData:dHJ1ZQo=] kind:Secret metadata:map[name:master-user-data namespace:openshift-machine-api] type:Opaque]}
  $ cat <<EOF >test.yaml
  > kind: List
  > apiVersion: v1
  > metadata:
  >   resourceVersion: ""
  >   selfLink: ""
  > items:
  > - apiVersion: v1
  >   kind: Secret
  >   metadata:
  >     name: master-user-data
  >     namespace: openshift-machine-api
  >   type: Opaque
  >   data:
  >     disableTemplating: dHJ1ZQo=
  >     userData: dHJ1ZQo=
  > EOF
  $ go run test.go
  2019/09/12 10:33:20 &{map[kind:List metadata:map[resourceVersion: selfLink:] apiVersion:v1] [{map[metadata:map[name:master-user-data namespace:openshift-machine-api] type:Opaque apiVersion:v1 data:map[disableTemplating:dHJ1ZQo= userData:dHJ1ZQo=] kind:Secret]}]}
  2019/09/12 10:33:20
  2019/09/12 10:33:20 <nil>
  exit status 1

[1]: openshift#1381 (comment)
[2]: https://github.com/openshift/library-go/blob/7d4acc018c610623ee1413b1e7aebe2ac675d35f/pkg/assets/create/creater.go#L264
jhixson74 pushed a commit to jhixson74/installer that referenced this pull request Dec 6, 2019
…ntrol-plane teardown

Since 82d81d9 (data/data/bootstrap: use loopback kubeconfig for API
access, 2019-07-24, openshift#2086), we've been pushing the OpenShift-specific
components via a loopback kubeconfig and the bootstrap control plane.
Since 108a45b (data/bootstrap: Replace openshift.sh with
cluster-bootstrap, 2019-01-24, landed 2019-09-30, openshift#1381), that push
has always happened before the bootstrap control plane shuts down.

I've changed "into" to "via" because the data passes through either
control plane on its way to rest in the shared etcd cluster.

I've dropped "then" from the final step because none of the other
steps said "then".  I think ordering is clear enough from our use of
an ordered list ;).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. lgtm Indicates that a PR is ready to be merged. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants