Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Bug 1743873: data/bootstrap: extract ironic images from release image #2234

Merged
merged 1 commit into from
Aug 28, 2019
Merged

Bug 1743873: data/bootstrap: extract ironic images from release image #2234

merged 1 commit into from
Aug 28, 2019

Conversation

stbenjam
Copy link
Member

@stbenjam stbenjam commented Aug 19, 2019 •
edited
Loading

startironic.sh now uses the ironic images specified in the release. In
order to do that, this change pulls out the code from bootkube to
get the images to it's own script, and adds a service that downloads the
release image so it only happens once.

fixes #2090

@openshift-ci-robot openshift-ci-robot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Aug 19, 2019
@stbenjam
Copy link
Member Author

/label platform/baremetal

/assign @hardys

@stbenjam
Copy link
Member Author

/label platform/baremetal

@openshift-ci-robot openshift-ci-robot added the platform/baremetal IPI bare metal hosts platform label Aug 19, 2019
@stbenjam
Copy link
Member Author

stbenjam commented Aug 19, 2019 •
edited
Loading

This uses the correct images now:

From the release image:

  ironic-ipa-downloader                         sha256:1bfd9417e5b03ca611c09c805b925914d72441bf6ed7fd6f78f0a3edbdd425ea
  ironic-rhcos-downloader                       sha256:3ea6994ad12c057cfb2bcc4c5d228155f236e58ba9edf1294be6d14d1c72b169

podman ps:

0a034420fc82  quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3ea6994ad12c057cfb2bcc4c5d228155f236e58ba9edf1294be6d14d1c72b169  /usr/local/bin/ge...  20 seconds ago      Up 20 seconds ago                     coreos-downloader
b81b932a7ee0  quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1bfd9417e5b03ca611c09c805b925914d72441bf6ed7fd6f78f0a3edbdd425ea  /usr/local/bin/ge...  30 seconds ago      Exited (0) 24 seconds ago             ipa-downloader

stbenjam
stbenjam commented Aug 19, 2019
View reviewed changes
data/data/bootstrap/baremetal/files/usr/local/bin/startironic.sh.template
@@ -87,7 +86,6 @@ podman wait -i 1000 ipa-downloader
podman wait -i 1000 coreos-downloader
while ! curl --fail http://localhost/images/rhcos-ootpa-latest.qcow2.md5sum ; do sleep 1; done
while ! curl --fail --head http://localhost/images/ironic-python-agent.initramfs ; do sleep 1; done
while ! curl --fail --head http://localhost/images/ironic-python-agent.tar.headers ; do sleep 1; done
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The rhel based IPA image doesn't produce this file, so the move to openshift images made this wait forever.

@metal3ci
Copy link

Build ABORTED, see build http://10.8.144.11:8080/job/dev-tools/1066/

@metal3ci
Copy link

Build SUCCESS, see build http://10.8.144.11:8080/job/dev-tools/1067/

@hardys
Copy link
Contributor

hardys commented Aug 20, 2019

/lgtm

@openshift-ci-robot openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Aug 20, 2019
@stbenjam
Copy link
Member Author

/retitle Bug 1743873: data/data/bootstrap: extract ironic images from release image

@openshift-ci-robot openshift-ci-robot changed the title data/data/bootstrap: extract ironic images from release image Bug 1743873: data/data/bootstrap: extract ironic images from release image Aug 20, 2019
@openshift-ci-robot
Copy link
Contributor

@stbenjam: This pull request references Bugzilla bug 1743873, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

Bug 1743873: data/data/bootstrap: extract ironic images from release image

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-ci-robot openshift-ci-robot added the bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. label Aug 20, 2019
abhinavdahiya
abhinavdahiya reviewed Aug 20, 2019
View reviewed changes
data/data/bootstrap/files/usr/local/bin/release-image.sh.template Outdated
#!/usr/bin/env bash
# release-image.sh is a helper library for interacting with a release image.

if ! podman inspect {{.ReleaseImage}} &>/dev/null; then
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If multiple services run this concurrently? what is the behavior?

Copy link
Member

@wking wking Aug 20, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

currently podman pulls multiple copies in parallel. containers/image#611 is long-running work on locks to make parallel pulls more efficient, but who knows if/when that will land.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like flock is available on RHCOS. I can write something off that, unless you have a preference for a different way

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was more interested making sure 2 runs of podman simultaneously wouldn't cause some corruption.. performance wasn't a concern..

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think it causes any corruption issues, but if more than a couple things start sourcing this it could get ugly performance-wise. Is there any objection to using something like flock? It's not very portable, though, but it is on RHCOS. People also do hacky things with mkdir as it's guaranteed to be atomic.

(
    flock 9
    if ! release=$( podman inspect {{.ReleaseImage}} --format '{{"{{"}} index .RepoDigests 0 {{"}}"}}' ) || [[ -z "${release}" ]]; then
        echo "Warning: Could not resolve release image to pull by digest" 2>&1
        release="{{.ReleaseImage}}"
    fi
) 9>/var/lock/release-image.lock

Copy link
Member

@wking wking Aug 21, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm fine locking somehow in shell. I'm also fine serializing these two services via some systemd magic. And because this is metal-only, I'm fine not locking at all too ;). Ahbinav may be more opinionated.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You could have a systemd service to pull the release image and have bootkube and ironic depend on it

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, @markmc, I think that is a nice approach. I've updated this PR.

@wking wking changed the title Bug 1743873: data/data/bootstrap: extract ironic images from release image Bug 1743873: data/bootstrap: extract ironic images from release image Aug 20, 2019
@openshift-ci-robot openshift-ci-robot removed the lgtm Indicates that a PR is ready to be merged. label Aug 21, 2019
@stbenjam stbenjam mentioned this pull request Aug 21, 2019
Merged
@openshift-ci-robot
Copy link
Contributor

@stbenjam: This pull request references Bugzilla bug 1743873, which is valid.

In response to this:

Bug 1743873: data/bootstrap: extract ironic images from release image

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@metal3ci
Copy link

Build FAILURE, see build http://10.8.144.11:8080/job/dev-tools/1078/

@metal3ci
Copy link

Build FAILURE, see build http://10.8.144.11:8080/job/dev-tools/1101/

@stbenjam
data/bootstrap: extract ironic images from release image
d1ec85a 
startironic.sh now uses the ironic images specified in the release. In
order to do that, this change pulls out the code from bootkube to
get the images to it's own script, and adds a service that downloads the
release image so it only happens once.
@metal3ci
Copy link

Build FAILURE, see build http://10.8.144.11:8080/job/dev-tools/1102/

@abhinavdahiya
Copy link
Contributor

/lgtm

@openshift-ci-robot openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Aug 27, 2019
@openshift-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: abhinavdahiya, hardys, stbenjam

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

4 similar comments
@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-ci-robot
Copy link
Contributor

openshift-ci-robot commented Aug 27, 2019 •
edited
Loading

@stbenjam: The following tests failed, say /retest to rerun them all:

Test name Commit Details Rerun command
ci/prow/e2e-libvirt d1ec85a link /test e2e-libvirt
ci/prow/e2e-openstack d1ec85a link /test e2e-openstack

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@openshift-merge-robot openshift-merge-robot merged commit 1d4549d into openshift:master Aug 28, 2019
@openshift-ci-robot
Copy link
Contributor

@stbenjam: All pull requests linked via external trackers have merged. Bugzilla bug 1743873 has been moved to the MODIFIED state.

In response to this:

Bug 1743873: data/bootstrap: extract ironic images from release image

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@stbenjam stbenjam deleted the ironic branch August 28, 2019 01:17
This was referenced Aug 28, 2019
Closed
Closed
wking added a commit to wking/openshift-installer that referenced this pull request Jan 6, 2020
@wking
data/bootstrap/files/etc/motd: Mention release-image.service
a419c02 
Catching up with d1ec85a (data/bootstrap: extract ironic images
from release image, 2019-08-19, openshift#2234).
@wking wking mentioned this pull request Jan 6, 2020
Merged
wking added a commit to wking/openshift-installer that referenced this pull request Jan 6, 2020
@wking
data/bootstrap/files/etc/motd: Mention release-image.service
f15a406 
Catching up with d1ec85a (data/bootstrap: extract ironic images
from release image, 2019-08-19, openshift#2234).
wking added a commit to wking/openshift-installer that referenced this pull request Jan 11, 2020
@wking
data/bootstrap/files/etc/motd: Mention release-image.service
8626b5e 
Catching up with d1ec85a (data/bootstrap: extract ironic images
from release image, 2019-08-19, openshift#2234).
openshift-cherrypick-robot pushed a commit to openshift-cherrypick-robot/installer that referenced this pull request Feb 19, 2020
@wking
data/bootstrap/files/etc/motd: Mention release-image.service
8b4d277 
Catching up with d1ec85a (data/bootstrap: extract ironic images
from release image, 2019-08-19, openshift#2234).
@wking wking mentioned this pull request May 19, 2020
Closed
wking added a commit to wking/openshift-installer that referenced this pull request May 19, 2020
@wking
data/data/bootstrap/files/usr/local/bin/release-image-download: Drop …
0c7ab90 
…--quiet

We've had the option since 9dfdc3c (bootkube.sh: suppress podman
output when pulling images, 2019-02-21, openshift#1293).  But since d1ec85a
(data/bootstrap: extract ironic images from release image, 2019-08-19, openshift#2234)
that image pool has happened in a separate unit so output will be in
journals/release-image.log and not cluttering up
journals/bootkube.log.  So get noisy again, to get information like:

  $ podman pull quay.io/openshift-release-dev/ocp-release-nightly@sha256:542a73a1c3a7ce5e8fa89ea3cd8a9c346dec1f0d437cf7e40bcb773683770be1
  Trying to pull quay.io/openshift-release-dev/ocp-release-nightly@sha256:542a73a1c3a7ce5e8fa89ea3cd8a9c346dec1f0d437cf7e40bcb773683770be1...Failed
  error pulling image "quay.io/openshift-release-dev/ocp-release-nightly@sha256:542a73a1c3a7ce5e8fa89ea3cd8a9c346dec1f0d437cf7e40bcb773683770be1": unable to pull quay.io/openshift-release-dev/ocp-release-nightly@sha256:542a73a1c3a7ce5e8fa89ea3cd8a9c346dec1f0d437cf7e40bcb773683770be1: unable to pull image: Error determining manifest MIME type for docker://quay.io/openshift-release-dev/ocp-release-nightly@sha256:542a73a1c3a7ce5e8fa89ea3cd8a9c346dec1f0d437cf7e40bcb773683770be1: unexpected http code: 500 (Internal Server Error), URL: https://quay.io/v2/auth?scope=repository%3Aopenshift-release-dev%2Focp-release-nightly%3Apull&service=quay.io

instead of silence like:

  -- Logs begin at Tue 2020-05-19 18:47:10 UTC, end at Tue 2020-05-19 19:12:00 UTC. --
  May 19 18:47:40 ip-10-0-29-43 systemd[1]: Starting Download the OpenShift Release Image...
  May 19 18:47:40 ip-10-0-29-43 release-image-download.sh[1747]: Pulling quay.io/openshift-release-dev/ocp-release-nightly@sha256:542a73a1c3a7ce5e8fa89ea3cd8a9c346dec1f0d437cf7e40bcb773683770be1...
  May 19 18:47:41 ip-10-0-29-43 podman[1749]: 2020-05-19 18:47:41.135323376 +0000 UTC m=+0.318008863 system refresh
wking added a commit to wking/openshift-installer that referenced this pull request May 19, 2020
@wking
data/data/bootstrap/files/usr/local/bin/release-image-download: Drop …
af9a65b 
…--quiet

We've had the option since 9dfdc3c (bootkube.sh: suppress podman
output when pulling images, 2019-02-21, openshift#1293).  But since d1ec85a
(data/bootstrap: extract ironic images from release image, 2019-08-19, openshift#2234)
that image pool has happened in a separate unit, so output will be in
journals/release-image.log and not cluttering up
journals/bootkube.log.  So get noisy again, to get information like:

  $ podman pull quay.io/openshift-release-dev/ocp-release-nightly@sha256:542a73a1c3a7ce5e8fa89ea3cd8a9c346dec1f0d437cf7e40bcb773683770be1
  Trying to pull quay.io/openshift-release-dev/ocp-release-nightly@sha256:542a73a1c3a7ce5e8fa89ea3cd8a9c346dec1f0d437cf7e40bcb773683770be1...Failed
  error pulling image "quay.io/openshift-release-dev/ocp-release-nightly@sha256:542a73a1c3a7ce5e8fa89ea3cd8a9c346dec1f0d437cf7e40bcb773683770be1": unable to pull quay.io/openshift-release-dev/ocp-release-nightly@sha256:542a73a1c3a7ce5e8fa89ea3cd8a9c346dec1f0d437cf7e40bcb773683770be1: unable to pull image: Error determining manifest MIME type for docker://quay.io/openshift-release-dev/ocp-release-nightly@sha256:542a73a1c3a7ce5e8fa89ea3cd8a9c346dec1f0d437cf7e40bcb773683770be1: unexpected http code: 500 (Internal Server Error), URL: https://quay.io/v2/auth?scope=repository%3Aopenshift-release-dev%2Focp-release-nightly%3Apull&service=quay.io

instead of silence like:

  -- Logs begin at Tue 2020-05-19 18:47:10 UTC, end at Tue 2020-05-19 19:12:00 UTC. --
  May 19 18:47:40 ip-10-0-29-43 systemd[1]: Starting Download the OpenShift Release Image...
  May 19 18:47:40 ip-10-0-29-43 release-image-download.sh[1747]: Pulling quay.io/openshift-release-dev/ocp-release-nightly@sha256:542a73a1c3a7ce5e8fa89ea3cd8a9c346dec1f0d437cf7e40bcb773683770be1...
  May 19 18:47:41 ip-10-0-29-43 podman[1749]: 2020-05-19 18:47:41.135323376 +0000 UTC m=+0.318008863 system refresh
wking added a commit to wking/openshift-installer that referenced this pull request May 19, 2020
@wking
data/data/bootstrap/files/usr/local/bin/release-image-download: Drop …
0da1087 
…--quiet

We've had the option since 9dfdc3c (bootkube.sh: suppress podman
output when pulling images, 2019-02-21, openshift#1293).  But since d1ec85a
(data/bootstrap: extract ironic images from release image, 2019-08-19, openshift#2234)
that image pull has happened in a separate unit, so output will be in
journals/release-image.log and not cluttering up
journals/bootkube.log.  So get noisy again, to get information like:

  $ podman pull quay.io/openshift-release-dev/ocp-release-nightly@sha256:542a73a1c3a7ce5e8fa89ea3cd8a9c346dec1f0d437cf7e40bcb773683770be1
  Trying to pull quay.io/openshift-release-dev/ocp-release-nightly@sha256:542a73a1c3a7ce5e8fa89ea3cd8a9c346dec1f0d437cf7e40bcb773683770be1...Failed
  error pulling image "quay.io/openshift-release-dev/ocp-release-nightly@sha256:542a73a1c3a7ce5e8fa89ea3cd8a9c346dec1f0d437cf7e40bcb773683770be1": unable to pull quay.io/openshift-release-dev/ocp-release-nightly@sha256:542a73a1c3a7ce5e8fa89ea3cd8a9c346dec1f0d437cf7e40bcb773683770be1: unable to pull image: Error determining manifest MIME type for docker://quay.io/openshift-release-dev/ocp-release-nightly@sha256:542a73a1c3a7ce5e8fa89ea3cd8a9c346dec1f0d437cf7e40bcb773683770be1: unexpected http code: 500 (Internal Server Error), URL: https://quay.io/v2/auth?scope=repository%3Aopenshift-release-dev%2Focp-release-nightly%3Apull&service=quay.io

instead of silence like:

  -- Logs begin at Tue 2020-05-19 18:47:10 UTC, end at Tue 2020-05-19 19:12:00 UTC. --
  May 19 18:47:40 ip-10-0-29-43 systemd[1]: Starting Download the OpenShift Release Image...
  May 19 18:47:40 ip-10-0-29-43 release-image-download.sh[1747]: Pulling quay.io/openshift-release-dev/ocp-release-nightly@sha256:542a73a1c3a7ce5e8fa89ea3cd8a9c346dec1f0d437cf7e40bcb773683770be1...
  May 19 18:47:41 ip-10-0-29-43 podman[1749]: 2020-05-19 18:47:41.135323376 +0000 UTC m=+0.318008863 system refresh
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@abhinavdahiya abhinavdahiya abhinavdahiya left review comments

@wking wking wking left review comments

@markmc markmc markmc left review comments

@jcpowermac jcpowermac Awaiting requested review from jcpowermac

@jhixson74 jhixson74 Awaiting requested review from jhixson74

Assignees

@hardys hardys

@abhinavdahiya abhinavdahiya

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. lgtm Indicates that a PR is ready to be merged. platform/baremetal IPI bare metal hosts platform size/M Denotes a PR that changes 30-99 lines, ignoring generated files.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

baremetal: Migrate Ironic services to upstream images
10 participants
@stbenjam @metal3ci @hardys @openshift-ci-robot @dantrainor @abhinavdahiya @openshift-bot @markmc @wking @openshift-merge-robot