Multi-platform ko #38

jonjohnsonjr · 2019-06-17T21:07:15Z

This is a sketch of what I had in mind for #6

For this PR, I've set the default base image for this repo to ubuntu so that you can just fetch this branch and run ko publish ./cmd/ko to test out the change. It should build a different binary for every platform and publish a manifest list instead of just a manifest.

$ ko publish ./cmd/ko
2019/06/17 14:04:04 Using base index.docker.io/library/ubuntu:latest for github.com/google/ko/cmd/ko
2019/06/17 14:04:04 No matching credentials were found, falling back on anonymous
2019/06/17 14:04:06 Building github.com/google/ko/cmd/ko
2019/06/17 14:04:12 Building github.com/google/ko/cmd/ko
2019/06/17 14:04:15 Building github.com/google/ko/cmd/ko
2019/06/17 14:04:19 Building github.com/google/ko/cmd/ko
2019/06/17 14:04:23 Building github.com/google/ko/cmd/ko
2019/06/17 14:04:27 Building github.com/google/ko/cmd/ko
2019/06/17 14:04:30 Publishing gcr.io/jonjohnson-test/ko/ko-099ba5bcefdead87f92606265fb99ac0:latest
2019/06/17 14:04:32 existing manifest: sha256:e66f9706d017b28a5f63fd2b1a9395a43852bb8691c2a3d789bf6187ed2a83ef
2019/06/17 14:04:32 existing manifest: sha256:6ae8959c22069d43bbaa848496237a653f4644244f638d11aa05dee555e2d7d9
2019/06/17 14:04:32 existing manifest: sha256:d835632ac512ee32b2489d44c15ecb03a139785b2e3d475a07670288fc95242e
2019/06/17 14:04:32 existing manifest: sha256:ce8f3571502ac3b5c86f1d527c712abd4cad953c97f24ba9b28a6505e502f1ad
2019/06/17 14:04:32 existing manifest: sha256:5c60f74c26ec1726f06e6c35ca1a3aee0c567d37001143ec70e9cc6f5b1a763b
2019/06/17 14:04:33 existing manifest: sha256:c1940d8d12f448cf17212cf76e012d8b8480e0b033fac5ff6068402a8f0f590f
2019/06/17 14:04:34 gcr.io/jonjohnson-test/ko/ko-099ba5bcefdead87f92606265fb99ac0:latest: digest: sha256:96ac9a314510e380cf3dd3aae3abc51dad577eecf53ebf6691f0c4d665d90621 size: 1411
2019/06/17 14:04:34 Published gcr.io/jonjohnson-test/ko/ko-099ba5bcefdead87f92606265fb99ac0@sha256:96ac9a314510e380cf3dd3aae3abc51dad577eecf53ebf6691f0c4d665d90621
gcr.io/jonjohnson-test/ko/ko-099ba5bcefdead87f92606265fb99ac0@sha256:96ac9a314510e380cf3dd3aae3abc51dad577eecf53ebf6691f0c4d665d90621

$ crane manifest gcr.io/jonjohnson-test/ko/ko-099ba5bcefdead87f92606265fb99ac0@sha256:96ac9a314510e380cf3dd3aae3abc51dad577eecf53ebf6691f0c4d665d90621 | jq .
{
  "schemaVersion": 2,
  "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
  "manifests": [
    {
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "size": 1073,
      "digest": "sha256:e66f9706d017b28a5f63fd2b1a9395a43852bb8691c2a3d789bf6187ed2a83ef",
      "platform": {
        "architecture": "amd64",
        "os": "linux"
      }
    },
    {
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "size": 1073,
      "digest": "sha256:6ae8959c22069d43bbaa848496237a653f4644244f638d11aa05dee555e2d7d9",
      "platform": {
        "architecture": "arm",
        "os": "linux",
        "variant": "v7"
      }
    },
    {
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "size": 1073,
      "digest": "sha256:d835632ac512ee32b2489d44c15ecb03a139785b2e3d475a07670288fc95242e",
      "platform": {
        "architecture": "arm64",
        "os": "linux",
        "variant": "v8"
      }
    },
    {
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "size": 1073,
      "digest": "sha256:ce8f3571502ac3b5c86f1d527c712abd4cad953c97f24ba9b28a6505e502f1ad",
      "platform": {
        "architecture": "386",
        "os": "linux"
      }
    },
    {
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "size": 1073,
      "digest": "sha256:5c60f74c26ec1726f06e6c35ca1a3aee0c567d37001143ec70e9cc6f5b1a763b",
      "platform": {
        "architecture": "ppc64le",
        "os": "linux"
      }
    },
    {
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "size": 1073,
      "digest": "sha256:c1940d8d12f448cf17212cf76e012d8b8480e0b033fac5ff6068402a8f0f590f",
      "platform": {
        "architecture": "s390x",
        "os": "linux"
      }
    }
  ]
}

I haven't actually tested pulling/running this on different platforms (I don't have any), so YMMV, but publishing works 👍

I don't actually like this very much because I've made every package deal with a new abstraction (I've called it Steve so that we don't ever merge this) that can represent either base image or a base manifest list. A better approach would probably be to make everything deal with a manifest list to simplify things.

When pulling base image:

if it's a manifest list, pass that along as the base
if it's an image, wrap it as a singleton manifest list

When publishing the artifacts:

if it's a singleton manifest list, unwrap it and publish just the image
otherwise, publish it as a manifest list

This isn't perfect (what if the user actually does want to publish a singleton manifest list?), so maybe it's not the best idea. Open to suggestions :)

jonjohnsonjr · 2019-06-24T20:22:52Z

google/go-containerregistry#474 would make my proposed implementation easier.

jonjohnsonjr · 2019-06-24T20:23:38Z

pkg/build/gobuild.go

+ return nil, fmt.Errorf("oops")
+}
+
+type index struct {


All of this could go away with google/go-containerregistry#474

vdemeester · 2019-06-25T14:54:00Z

I don't actually like this very much because I've made every package deal with a new abstraction (I've called it Steve so that we don't ever merge this) that can represent either base image or a base manifest list. A better approach would probably be to make everything deal with a manifest list to simplify things.

When pulling base image:
* if it's a manifest list, pass that along as the base

* if it's an image, wrap it as a singleton manifest list
When publishing the artifacts:
* if it's a singleton manifest list, unwrap it and publish just the image

* otherwise, publish it as a manifest list
This isn't perfect (what if the user actually does want to publish a singleton manifest list?), so maybe it's not the best idea. Open to suggestions :)

Could there be a flag or environment variable to customize this behavior ? (like it does what you describe by default but the user could force to build a singleton manifest list)

jonjohnsonjr · 2019-06-25T17:14:39Z

Could there be a flag or environment variable to customize this behavior ? (like it does what you describe by default but the user could force to build a singleton manifest list)

Hm.. that gives me an idea. We could add a --platform flag to select which platform to use from a manifest list, and have some sentinel value (all?) to indicate we publish a manifest list. This would make it possible to publish a singleton manifest list, but also allow targeting a specific platform.

vdemeester · 2019-06-25T17:17:17Z

Hm.. that gives me an idea. We could add a --platform flag to select which platform to use from a manifest list, and have some sentinel value (all?) to indicate we publish a manifest list. This would make it possible to publish a singleton manifest list, but also allow targeting a specific platform.

I like that idea 😻

I don't actually like this very much because I've made every package deal with a new abstraction (I've called it Steve so that we don't ever merge this) that can represent either base image or a base manifest list. A better approach would probably be to make everything deal with a manifest list to simplify things.

on the Steve package (:joy_cat:), I would also think a better approach would be to make everything deal with manifest list :angel:

vincent-pli · 2019-07-02T02:35:13Z

@jonjohnsonjr
I think when user run ko build, in most cases he does not expect to all platform, even he pick up a multi-arch image as base image.
Could we use the GOARCH to tell ko which arch he expect to. based on that, ko could check if the base image support that arch and raise error if not support.

If GPARCH not set, ko could release a all-platform image.

jonjohnsonjr · 2019-07-02T16:42:10Z

I think when user run ko build, in most cases he does not expect to all platform, even he pick up a multi-arch image as base image.

Agreed, my first thought was to just default to amd64/linux (since MacOS -> Linux Kubernetes is so common), and it's the default in docker distribution.

Using GOARCH/GOOS also makes sense.

Another idea I just had was to try to derive the os/arch based on the cluster we're deploying to:

$ kubectl get nodes -ojsonpath="{.items[0].status.nodeInfo.architecture}/{.items[0].status.nodeInfo.operatingSystem}"
amd64/linux

Though, that would add a little bit of latency to everything...

Some combination of those things is probably going to be the best user experience, e.g.:

If the base image is a single image, use that image's platform.
If the --platform flag is set, use that (all == build every platform).
If GOOS and GOARCH environment variables are set, use them.
If we can determine the platform via kubectl get nodes, use that (might not make sense for ko publish or ko resolve, but definitely ko apply).
Else, use amd64/linux.

vincent-pli · 2019-07-02T23:59:22Z

That's great, make sense, expect to see the new implements

lysannef · 2019-07-04T09:42:02Z

Hi @jonjohnsonjr , @junawaneshivani and I have been working on testing the changes in this PR on ppc64le platform.
However we have been facing an issue with the go build of this repo
We have tried the build in a Kubernetes environment as well.

Error encountered:

# github.com/jonjohnsonjr/ko/cmd/ko
cmd/ko/main.go:33:26: cannot use cmds (type *"github.com/jonjohnsonjr/ko/vendor/github.com/spf13/cobra".Command) as type *"github.com/google/ko/vendor/github.com/spf13/cobra".Command in argument to commands.AddKubeCommands

Are we missing any step?

vincent-pli · 2019-07-05T02:18:53Z

@lysannef
You should change the path of project to github.com/google/ko

lysannef · 2019-07-05T11:00:48Z

@vincent-pli These are the steps I took:

git clone https://github.com/google/ko.git
cd ko
git fetch origin +refs/pull/38/merge   ( so as to reflect the changes made by @jonjohnsonjr )
git checkout FETCH_HEAD
cd cmd/ko
go build

trying a go build in this repository, still throws the same error.

@lysannef
You should change the path of project to github.com/google/ko

Could you please elaborate on the same. We are trying to test the changes made in @jonjohnsonjr 's repository at the multi-platform-wip branch. How would changing to the original ko repo work?

vincent-pli · 2019-07-05T14:11:22Z

So what's the path of your ko project?

lysannef · 2019-07-08T05:12:05Z

So what's the path of your ko project?

@vincent-pli The path we are testing: github.com/jonjohnsonjr/ko/cmd/ko
Changing the path to the google/ko repository with path : github.com/ko/cmd/ko and tried building it which resulted in the same error.

Using the path suggested by you : github.com/google/ko
the build proceeds however none of jonjohnsonjr's changes are reflecting and the log is as follows

2019/07/08 00:47:27 Building github.com/google/ko/cmd/ko
2019/07/08 00:47:35 Using base index.docker.io/library/ubuntu:latest for github.com/google/ko/cmd/ko
2019/07/08 00:47:35 No matching credentials were found, falling back on anonymous
2019/07/08 00:47:39 Loading ko.local/ko-099ba5bcefdead87f92606265fb99ac0:333f19bd74774f8eccc0277a79375c80c6f810abb7dfda2496bdaabc089af622
2019/07/08 00:47:47 Loaded ko.local/ko-099ba5bcefdead87f92606265fb99ac0:333f19bd74774f8eccc0277a79375c80c6f810abb7dfda2496bdaabc089af622
2019/07/08 00:47:47 Adding tag latest
2019/07/08 00:47:48 Added tag latest
ko.local/ko-099ba5bcefdead87f92606265fb99ac0:333f19bd74774f8eccc0277a79375c80c6f810abb7dfda2496bdaabc089af622

vincent-pli · 2019-07-08T06:22:01Z

I'm little confuse, so I just try it on ppc64le

Git clone git@github.com:jonjohnsonjr/ko.git to GOPATH as path: /root/go/src/github.com/google/ko
Run go build in root/go/src/github.com/google/ko/cmd/ko, then get binary ko, mv it to /root/go/bin
run ko publish ./cmd/ko in root/go/src/github.com/google/ko

It works, and get correct result.
BTW, are you a ibmer?

lysannef · 2019-07-08T06:51:17Z

Git clone git@github.com:jonjohnsonjr/ko.git to GOPATH as path: /root/go/src/github.com/google/ko
Run go build in root/go/src/github.com/google/ko/cmd/ko, then get binary ko, mv it to /root/go/bin
run ko publish ./cmd/ko in root/go/src/github.com/google/ko

I followed these steps and this is the result I get:

$ ko publish ./cmd/ko
2019/07/08 02:42:23 Building github.com/google/ko/cmd/ko
2019/07/08 02:42:29 Using base gcr.io/distroless/static:latest for github.com/google/ko/cmd/ko
2019/07/08 02:42:29 No matching credentials were found, falling back on anonymous
2019/07/08 02:42:35 Loading ko.local/ko-099ba5bcefdead87f92606265fb99ac0:9d2e16d79eb082396f5374b765ee5b6c62f2485d13bac726fe6335aad39f2a11
2019/07/08 02:42:40 Loaded ko.local/ko-099ba5bcefdead87f92606265fb99ac0:9d2e16d79eb082396f5374b765ee5b6c62f2485d13bac726fe6335aad39f2a11
2019/07/08 02:42:40 Adding tag latest
2019/07/08 02:42:40 Added tag latest

Based on the ouput given above I dont see the additional images being created , like

2019/06/17 14:04:34 gcr.io/jonjohnson-test/ko/ko-099ba5bcefdead87f92606265fb99ac0:latest: digest: sha256:96ac9a314510e380cf3dd3aae3abc51dad577eecf53ebf6691f0c4d665d90621 size: 1411
2019/06/17 14:04:34 Published gcr.io/jonjohnson-test/ko/ko-099ba5bcefdead87f92606265fb99ac0@sha256:96ac9a314510e380cf3dd3aae3abc51dad577eecf53ebf6691f0c4d665d90621
gcr.io/jonjohnson-test/ko/ko-099ba5bcefdead87f92606265fb99ac0@sha256:96ac9a314510e380cf3dd3aae3abc51dad577eecf53ebf6691f0c4d665d90621

BTW, are you a ibmer?

Yes I am an IBMer

vincent-pli · 2019-07-08T07:18:23Z

@lysannef you can find me by ST: Peng QL Li 😄

lysannef · 2019-08-02T08:37:06Z

@jonjohnsonjr We have validated the working of these changes on ppc64le architecture. The architecture specific images are generated , and the manifest is as below.

 docker manifest inspect localhost:5000/knative/ko-099ba5bcefdead87f92606265fb99ac0@sha256:9fe6837835e5fe4f19bd037f2d85bc350a32a9904e088fdc74d6c46fb627b56a --insecure
{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
   "manifests": [
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 1235,
         "digest": "sha256:15271f24993d960daeea17c856be3f5cc89acdae902e6a5f6634c9c1d94c4ff9",
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 1235,
         "digest": "sha256:5092ff253eb247d1dfaba556f575bcbd69f2371505a2e459b6593055841f8f0a",
         "platform": {
            "architecture": "arm",
            "os": "linux",
            "variant": "v7"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 1235,
         "digest": "sha256:fd7eb6a9c0ab9494e49204bd51090939d23d754e32e9f1f1810e482646372353",
         "platform": {
            "architecture": "arm64",
            "os": "linux",
            "variant": "v8"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 1235,
         "digest": "sha256:8b5f22a0386daecb29fc2a1705eba24800adf788a06cb953d2e698a23eba9e50",
         "platform": {
            "architecture": "386",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 1235,
         "digest": "sha256:2de4326fd4c4c15b2e08ef785b7225dcf895a1640147aa25dfd2f353cf9a6072",
         "platform": {
            "architecture": "ppc64le",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 1235,
         "digest": "sha256:b8398cec700268886cb888081b45d7aeacef8a726bd28c33658190ff8734ed78",
         "platform": {
            "architecture": "s390x",
            "os": "linux"
         }
      }
   ]
}

Have there been any plans about up-streaming these changes and around when would that be expected?

jonjohnsonjr · 2019-09-14T16:33:07Z

Pinging back here for a quick update. I just landed google/go-containerregistry#520 which I believe is everything I need upstream to implement this "for real". I'll try to revisit this PR in the next couple weeks.

@mattmoor had a nice idea that the default flag value for platforms should depend on which command we're invoking:

For ko apply, we're almost certainly deploying to a single platform, which is probably amd64/linux, so we should default to --platform=amd64/linux. (Possibly use kubectl to determine the target cluster's platform as well)
For ko resolve, we're probably producing a release, so we could default to --platform=all.
For ko publish, we don't know where it's going, I'm tempted to just default to --platform=amd64/linux for now.

junawaneshivani · 2019-10-09T05:12:33Z

Hi @jonjohnsonjr, any further progress on this PR?

jonjohnsonjr · 2019-10-10T22:01:31Z

any further progress on this PR?

Just pushed something that I'm pretty happy with. I'm not 100% certain what we want to do with the --platform flag, but this seems pretty good.

junawaneshivani · 2020-02-07T06:46:26Z

@jonjohnsonjr I see there have been few commits since I last checked.. Does the --platform flag produce images for the right architecture now ? (given we have used a multi-arch base image )

barthy1 · 2020-07-10T11:47:27Z

Hi @jonjohnsonjr
this PR is very important for tekton s390x and ppc64le enablement - tektoncd/pipeline#856
Do you see how we can help you from Z/Power side with your work in this direction?

seth-priya · 2020-07-23T10:04:51Z

@jonjohnsonjr just wanted to check is there is any work remaining in terms of validating these changes with which we @junawaneshivani and myself on the Power side can help with? or any other blockers that need to be worked through? thanks in advance!

pkg/build/build.go

pkg/build/gobuild.go

pkg/build/recorder.go

pkg/commands/config.go

pkg/commands/options/build.go

pkg/publish/default_test.go

pkg/publish/default.go

mattmoor · 2020-09-24T21:18:16Z

Generally, this seems like a good first chunk to land, but we should open issues for "smarter" platform modes.

Honestly, looking through, I'm pretty happy with how simple this was. Most of the diff seemed like changing signatures, which is pretty cool.

I'd like to get an early form of this in, land #162 and then cut 0.6 🤩

mattmoor · 2020-09-24T21:27:49Z

Oh, one other thing I've been thinking is that when the base image is an image index, it would be good for use to log what platforms we see and build for logged to stderr.

jonjohnsonjr · 2020-09-24T22:28:10Z

Oh, one other thing I've been thinking is that when the base image is an image index, it would be good for use to log what platforms we see and build for logged to stderr.

I did this really lazily by just logging architecture every time in our Building ... log, LMK if this is too lazy.

mattmoor · 2020-09-24T22:28:32Z

I saw, seemed fine.

vdemeester · 2020-09-25T07:16:09Z

🎉

jonjohnsonjr mentioned this pull request Jun 17, 2019

Add support for other architectures tektoncd/pipeline#856

Open

13 tasks

junawaneshivani mentioned this pull request Jun 24, 2019

Issues in building knative on Power PC, ppc64le architecture knative/build#514

Open

jonjohnsonjr commented Jun 24, 2019

View reviewed changes

lysannef mentioned this pull request Aug 22, 2019

Use GOOS/GOARCH from base image #77

Merged

jonjohnsonjr force-pushed the multi-platform-wip branch from da7c5e9 to 2a797ec Compare October 10, 2019 22:00

jonjohnsonjr marked this pull request as ready for review October 10, 2019 22:02

jonjohnsonjr assigned imjasonh Oct 10, 2019

jonjohnsonjr force-pushed the multi-platform-wip branch from 62a7be5 to 532aa42 Compare December 6, 2019 19:35

jonjohnsonjr mentioned this pull request Dec 6, 2019

Allow mutation of image or index media types google/go-containerregistry#624

Merged

jonjohnsonjr force-pushed the multi-platform-wip branch from 532aa42 to 3ea73f0 Compare December 9, 2019 18:45

barthy1 mentioned this pull request Jul 28, 2020

Run Tekton tests and builds for other architectures tektoncd/plumbing#495

Open

jonjohnsonjr force-pushed the multi-platform-wip branch 2 times, most recently from 026422b to 03a8e82 Compare August 12, 2020 16:22

vdemeester mentioned this pull request Aug 21, 2020

[Question] Where can I get s390x and ppc64le build tektoncd/pipeline#3122

Closed