Feature 2160 add images subcommand #2303

curtbushko · 2021-06-09T18:08:58Z

This PR addresses Feature 2160 - ability to include additional images in the kindest/node image and hopefully addresses the concerns in the discussion.

The PR:

adds a subcommand to the build target kind build add-image IMAGE [IMAGES...] in order to keep the command separate from kind build node-image
pulls images locally before they are added into the resulting image to speed up future builds
tars those images temporarily (similar to what kind load does)
starts up the base image and use critl to copy/stream the images into it
saves the image using docker save

Extras:

follows the same patterns as node image when it comes to code layout and args
has defaults for the base image used (kindest/node:latest) and the resulting image (kindest/custom-node:latest)
can handle adding multiple images at the same time

An example of the command being used:

>> ./bin/kind build add-image --base-image=docker.io/kindest/node:v1.19.11 --image=myrepo/custom-node:latest alpine:3.7
Starting to add images to base image
Creating build container based on docker.io/kindest/node:v1.19.11
Building in kind-build-1623261338-1249758035
Saving images into tar file at /private/var/folders/81/myw19h2108xdx31prlv3qsd00000gn/T/images-tar466634204/images.tar
Importing images into build container kind-build-1623261338-1249758035
Saving new image myrepo/custom-node:latest
sha256:bc17c0f40f826574adb8aa886c29404b02c5e1fd89f2ff8d58f7cb55f59dce1e
Add image build completed.

>> cat <<EOF | kind create cluster --config=-
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  image: myrepo/custom-node:latest
EOF

Creating cluster "kind" ...
 ✓ Ensuring node image (myrepo/custom-node:latest) 🖼
 ✓ Preparing nodes 📦
 ✓ Writing configuration 📜
 ✓ Starting control-plane 🕹️
 ✓ Installing CNI 🔌
 ✓ Installing StorageClass 💾
Set kubectl context to "kind-kind"
You can now use your cluster with:

kubectl cluster-info --context kind-kind

Have a question, bug, or feature request? Let us know! https://kind.sigs.k8s.io/#community 🙂

>> docker exec -it kind-control-plane crictl images
IMAGE                                      TAG                  IMAGE ID            SIZE
docker.io/kindest/kindnetd                 v20210326-1e038dc5   6de166512aa22       54MB
docker.io/library/alpine                   3.7                  51af6a90e1cee       4.47MB
docker.io/rancher/local-path-provisioner   v0.0.14              e422121c9c5f9       13.4MB
k8s.gcr.io/build-image/debian-base         v2.1.0               c7c6c86897b63       21.1MB
k8s.gcr.io/coredns                         1.7.0                bfe3a36ebd252       14MB
k8s.gcr.io/etcd                            3.4.13-0             0369cf4303ffd       86.7MB
k8s.gcr.io/kube-apiserver                  v1.19.11             2fb26b7ebd23a       120MB
k8s.gcr.io/kube-controller-manager         v1.19.11             1af1564e1890d       112MB
k8s.gcr.io/kube-proxy                      v1.19.11             d2c2a8b355f56       120MB
k8s.gcr.io/kube-scheduler                  v1.19.11             120d339f7d67a       47.7MB
k8s.gcr.io/pause                           3.5                  ed210e3e4a5ba       301kB

Please let me know if there is something you would like to see changed or fixed.

…ommand

k8s-ci-robot · 2021-06-09T18:09:06Z

Welcome @curtbushko!

It looks like this is your first PR to kubernetes-sigs/kind 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes-sigs/kind has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

k8s-ci-robot · 2021-06-09T18:09:06Z

Hi @curtbushko. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

BenTheElder · 2021-06-09T18:14:48Z

We may need to consider --arch support similar to kind build node-image. Unfortunately when doing that ...

pulls images locally before they are added into the resulting image to speed up future builds

That will not work once this supports multi-arch, perhaps best to head that off now. (see: #2176)

BenTheElder · 2021-06-09T18:14:59Z

/ok-to-test

curtbushko · 2021-06-09T19:14:34Z

/retest

curtbushko · 2021-06-09T21:11:31Z

We may need to consider --arch support similar to kind build node-image. Unfortunately when doing that ...

pulls images locally before they are added into the resulting image to speed up future builds

That will not work once this supports multi-arch, perhaps best to head that off now. (see: #2176)

Is there a way I could use docker pull image:tag@sha and parse out the manifest through docker manifest inspect and still use the local docker cache?

What I am aiming for is being being able to build our product images locally and build a node image for all developers to use. That gets difficult if I have to pull images from a private registry while building.

BenTheElder · 2021-06-09T21:43:38Z

Is there a way I could use docker pull image:tag@sha and parse out the manifest through docker manifest inspect and still use the local docker cache?

The problem I found is that if you need to ask docker to pull an image for a platform other than the host and then save it for loading you run into issues (e.g. try pulling the image for each platform you're building for and loading it, it will just save the host platform one, there's no way to save --platform), so the only solution I found was to pull for the platform directly to the node container we're going to commit.

I think there's more on this in the linked PR? But without a solution for that multi-arch support is untenable, and multi-arch support is very much needed now.

BenTheElder · 2021-06-09T21:44:17Z

Also AFAIK there is no way to ask docker for layers locally (versus images by name), the only way docker exposes layers is through pushing to registries.

curtbushko · 2021-06-11T04:03:27Z

Thanks for the explanation and insight. I did go through your PR to see how you did it and I even took a dive into docker engine to see how it was doing save. It's too bad that they didn't add --platform to save even though they added it to other commands.

I found something very interesting! It looks like save does save the correct local image but it changes the manifest as it is writing it.

I pulled an odd platform version and built my custom image. I am on a mac.

>> docker pull --platform=s390x alpine:3.6

>> docker run alpine:3.6
WARNING: The requested image's platform (linux/s390x) does not match the detected host platform (linux/amd64) and no specific platform was requested
qemu-s390x: warning: 'msa5-base' requires 'kimd-sha-512'.
qemu-s390x: warning: 'msa5-base' requires 'klmd-sha-512'.

>> ./bin/kind build add-image alpine:3.6
Starting to add images to base image
Creating build container based on docker.io/kindest/node:latest
Building in kind-build-1623283038-727782895
Saving images into tar file at /private/var/folders/81/myw19h2108xdx31prlv3qsd00000gn/T/images-tar588984380/images.tar
Importing images into build container kind-build-1623283038-727782895
Saving new image kindest/custom-node:latest
sha256:a362f1ce00e277e34acb9dc2423039f168cd76c6e48678725278d56bbee6de20
Add image build completed.

>> cat <<EOF | kind create cluster --config=-
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  image: myrepo/custom-node:latest
EOF

Creating cluster "kind" ...
....

Inspect the images locally and in the custom node. The rootfs layers match.

>> docker inspect alpine:3.6 |grep -A5 'RootFS'
        "RootFS": {
            "Type": "layers",
            "Layers": [
                "sha256:f129ff7d977bdbcb6eb0ec9e2e24c6cbe68e60e46d9620e79e09e73ab2e545e5"
            ]
        },

>> docker exec -it kind-control-plane crictl inspecti docker.io/library/alpine:3.6 |grep -A5 'rootfs'
      "rootfs": {
        "type": "layers",
        "diff_ids": [
          "sha256:f129ff7d977bdbcb6eb0ec9e2e24c6cbe68e60e46d9620e79e09e73ab2e545e5"
        ]
      },

Even the creation dates are the same!

>> docker inspect alpine:3.6 |grep 'Created'
        "Created": "2019-03-08T03:35:59.716135296Z",

>> docker exec -it kind-control-plane crictl inspecti docker.io/library/alpine:3.6 | grep 'created'
        "created": "2019-03-08T03:35:59.716135296Z",

>> docker pull --platform=amd64 alpine:3.6 && docker inspect alpine:3.6 |grep 'Created'
3.6: Pulling from library/alpine
Digest: sha256:66790a2b79e1ea3e1dabac43990c54aca5d1ddf268d9a5a0285e4167c8b24475
Status: Image is up to date for alpine:3.6
docker.io/library/alpine:3.6
        "Created": "2019-03-07T22:20:00.563496859Z",

Bonus: If I run an alpine pod in my custom node image, the logs give me:

> kubectl logs -f alpine
qemu-s390x: warning: 'msa5-base' requires 'kimd-sha-512'.
qemu-s390x: warning: 'msa5-base' requires 'klmd-sha-512'.
qemu-s390x: warning: 'msa5-base' requires 'kimd-sha-512'.
qemu-s390x: warning: 'msa5-base' requires 'klmd-sha-512'

Just for fun, I pulled a mips version of redis, built it into my node and tried running it

kubeclt logs -f redis
qemu-mips64el: unable to find CPU model 'max'

Digging into this was quite fun!

aojea · 2021-06-23T09:19:58Z

@BenTheElder does it makes sense to duplicate the code for the docker commands pkg/build/nodeimage/internal/container/docker ?

pkg/build/addimage/buildcontext.go

pkg/build/addimage/defaults.go

curtbushko · 2021-06-23T15:48:53Z

@BenTheElder does it makes sense to duplicate the code for the docker commands pkg/build/nodeimage/internal/container/docker ?

In general I would say no but should it also go to a shared location?

BenTheElder · 2021-06-23T18:22:35Z

@BenTheElder does it makes sense to duplicate the code for the docker commands pkg/build/nodeimage/internal/container/docker ?

eh, I prescribe to WET: Write Everything Thrice. Once or twice is not enough usage to design a good abstraction, DRY it out once you hit at least 3 times. but more importantly the code in any internal/ is not meant to be a publicly importable API (We did not put enough work into making it a good stable API, they are just implementation details of the good APIs), so it will still have to be private, and so far we have made the rule that pkg/cmd/ is doable using only it's own internal details or the public APIs, the CLI does not import any shared internals so we know the CLI commands are vetting the public APIs.

raphink · 2022-04-11T06:39:01Z

Is there any news on this PR? This would be extremely useful!

batistein · 2022-07-23T10:00:18Z

Any progress on this PR? This is extremly useful and is already used as a fork in multiple projekt for round about a year. No issues so far, so I don't see any reason to defer a merge. Clean-up tasks could be done in a follow-up PR!

BenTheElder

I've personally been overwhelmed earlier this year, and then the past few weeks on vacation.

Will try to be more responsive here.

BenTheElder · 2022-07-26T17:55:22Z

pkg/build/addimage/internal/container/docker/archive.go

@@ -0,0 +1,167 @@
+/*


In this case we do not need to duplicate this entire package, we can just move it to be pkg/build/internal/container/docker instead of the existing pkg/build/nodeimage/internal/container/docker, and not duplicate any of it. It will be available to pkg/build/foo and pkg/build/bar then.

if we are going to duplicate it, this command should only need a small subset of it, not the whole package.

pkg/build/addimage/build.go

BenTheElder · 2022-07-26T17:57:28Z

pkg/build/addimage/buildcontext.go

+	}
+	c.logger.V(0).Info("Building in " + containerID)
+
+	// For kubernetes v1.15+ (actually 1.16 alpha versions) we may need to


we should not be doing this for arbitrary images we're adding, this is a quirk of component images from Kubernetes's own build process.

BenTheElder · 2022-07-26T18:01:47Z

pkg/build/addimage/buildcontext.go

+
+	// create build container
+	c.logger.V(0).Info("Creating build container based on " + c.baseImage)
+	// pull images to local docker


we shouldn't do this anymore, it's too problematic for multi-arch, as discussed in previous review comments.

we pull straight to containerd in the current version of the node image build

this is also a UX issue that is not thought out here, we should probably have --pull vs no --pull.

without --pull we should use the local docker stored image or tarball, which makes a predictable experience for locally built images

with --pull + --arch we can support multi-arch builds

--arch without --pull should possibly be prohibited or throw a warning, unless the inputs are tarballs not image names.

supporting tarballs would make it easier to deal with architectures locally without a registry

anhpngt · 2022-08-08T12:52:13Z

pkg/build/addimage/internal/container/docker/archive.go

+		for i, tag := range entry.RepoTags {
+			parts := strings.Split(tag, ":")
+			if len(parts) > 2 {
+				return nil, fmt.Errorf("invalid repotag: %s", entry)


This check would prevent images stored in a local registry I believe. For example:

localhost:5001/asia.gcr.io/my-gcp-project/my-docker-image:my-tag localhost:5001/postgres:14.4

not sure if it would be an intended behavior

paichinger · 2022-10-12T08:08:16Z

This is a great feature. I just successfully tested it on my machine, great job!
I'm using the fork currently to build custom node images containing a bunch of images I need to have in place in order to run a big application at my job. This saves me a lot of time and traffic.
Great job, I really hope this makes it into the main repo soon!
Cheers!

raphink · 2023-01-16T11:12:36Z

Is there anything I can do to help with this feature?

stmcginnis · 2023-01-16T16:12:33Z

Looks like there are some comments to be addressed yet. Commits should also be squashed.

…ommand

k8s-ci-robot · 2023-08-03T01:05:19Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: curtbushko
Once this PR has been reviewed and has the lgtm label, please ask for approval from aojea. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot · 2023-08-03T01:05:19Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: curtbushko
Once this PR has been reviewed and has the lgtm label, please ask for approval from aojea. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot · 2023-08-03T01:09:42Z

@curtbushko: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-kind-verify	`eab6fdb`	link	true	`/test pull-kind-verify`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

curtbushko · 2024-10-10T15:45:28Z

Closing in favour of the more recent #3634

BenTheElder · 2024-10-11T04:09:58Z

Thanks @curtbushko, I think we're close on that one, followed up on the discussion there again.

curtbushko added 5 commits June 2, 2021 09:18

first pass, adding images works

91118b2

Merge branch 'kubernetes-sigs:main' into feature-2160-add-images-subc…

85b810b

…ommand

Merge branch 'kubernetes-sigs:main' into feature-2160-add-images-subc…

0c74bd3

…ommand

Merge branch 'kubernetes-sigs:main' into feature-2160-add-images-subc…

01ba997

…ommand

change defaults for the images

3a0ed9a

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Jun 9, 2021

k8s-ci-robot requested review from amwat and neolit123 June 9, 2021 18:09

k8s-ci-robot added needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Jun 9, 2021

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Jun 9, 2021

remove unused const that make verify found

f3542c5

BenTheElder assigned munnerz, aojea and amwat Jun 23, 2021

aojea reviewed Jun 23, 2021

View reviewed changes

pkg/build/addimage/buildcontext.go Outdated Show resolved Hide resolved

aojea reviewed Jun 23, 2021

View reviewed changes

pkg/build/addimage/defaults.go Outdated Show resolved Hide resolved

updates based on comments, use the same default node image as buildnode

2adbcd2

BenTheElder mentioned this pull request Jun 24, 2021

add additional-images flag to build command #2159

Closed

nitishm mentioned this pull request Jun 24, 2021

How to build a node image with preloaded docker images #2063

Closed

BenTheElder reviewed Jul 26, 2022

View reviewed changes

anhpngt reviewed Aug 8, 2022

View reviewed changes

Merge branch 'kubernetes-sigs:main' into feature-2160-add-images-subc…

eab6fdb

…ommand

BenTheElder mentioned this pull request Aug 14, 2023

Custom Node Image with Preinstalled Deployments #3329

Closed

stmcginnis mentioned this pull request May 31, 2024

Add ability to include images in custom node build #3634

Open

curtbushko closed this Oct 10, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature 2160 add images subcommand #2303

Feature 2160 add images subcommand #2303

curtbushko commented Jun 9, 2021

k8s-ci-robot commented Jun 9, 2021

k8s-ci-robot commented Jun 9, 2021

BenTheElder commented Jun 9, 2021

BenTheElder commented Jun 9, 2021

curtbushko commented Jun 9, 2021

curtbushko commented Jun 9, 2021 •

edited

Loading

BenTheElder commented Jun 9, 2021

BenTheElder commented Jun 9, 2021 •

edited

Loading

curtbushko commented Jun 11, 2021

aojea commented Jun 23, 2021

curtbushko commented Jun 23, 2021 •

edited

Loading

BenTheElder commented Jun 23, 2021 •

edited

Loading

raphink commented Apr 11, 2022

batistein commented Jul 23, 2022

BenTheElder left a comment

BenTheElder Jul 26, 2022

BenTheElder Jul 26, 2022

BenTheElder Jul 26, 2022

BenTheElder Jul 26, 2022

BenTheElder Jul 26, 2022

anhpngt Aug 8, 2022 •

edited

Loading

anhpngt Aug 8, 2022

paichinger commented Oct 12, 2022

raphink commented Jan 16, 2023

stmcginnis commented Jan 16, 2023

k8s-ci-robot commented Aug 3, 2023

k8s-ci-robot commented Aug 3, 2023

k8s-ci-robot commented Aug 3, 2023 •

edited

Loading

curtbushko commented Oct 10, 2024

BenTheElder commented Oct 11, 2024

Feature 2160 add images subcommand #2303

Feature 2160 add images subcommand #2303

Conversation

curtbushko commented Jun 9, 2021

k8s-ci-robot commented Jun 9, 2021

k8s-ci-robot commented Jun 9, 2021

BenTheElder commented Jun 9, 2021

BenTheElder commented Jun 9, 2021

curtbushko commented Jun 9, 2021

curtbushko commented Jun 9, 2021 • edited Loading

BenTheElder commented Jun 9, 2021

BenTheElder commented Jun 9, 2021 • edited Loading

curtbushko commented Jun 11, 2021

aojea commented Jun 23, 2021

curtbushko commented Jun 23, 2021 • edited Loading

BenTheElder commented Jun 23, 2021 • edited Loading

raphink commented Apr 11, 2022

batistein commented Jul 23, 2022

BenTheElder left a comment

Choose a reason for hiding this comment

BenTheElder Jul 26, 2022

Choose a reason for hiding this comment

BenTheElder Jul 26, 2022

Choose a reason for hiding this comment

BenTheElder Jul 26, 2022

Choose a reason for hiding this comment

BenTheElder Jul 26, 2022

Choose a reason for hiding this comment

BenTheElder Jul 26, 2022

Choose a reason for hiding this comment

anhpngt Aug 8, 2022 • edited Loading

Choose a reason for hiding this comment

anhpngt Aug 8, 2022

Choose a reason for hiding this comment

paichinger commented Oct 12, 2022

raphink commented Jan 16, 2023

stmcginnis commented Jan 16, 2023

k8s-ci-robot commented Aug 3, 2023

k8s-ci-robot commented Aug 3, 2023

k8s-ci-robot commented Aug 3, 2023 • edited Loading

curtbushko commented Oct 10, 2024

BenTheElder commented Oct 11, 2024

curtbushko commented Jun 9, 2021 •

edited

Loading

BenTheElder commented Jun 9, 2021 •

edited

Loading

curtbushko commented Jun 23, 2021 •

edited

Loading

BenTheElder commented Jun 23, 2021 •

edited

Loading

anhpngt Aug 8, 2022 •

edited

Loading

k8s-ci-robot commented Aug 3, 2023 •

edited

Loading