TEP-0019: Add other architecture support

[barthy1]

This TEP proposes the way to add other hardware architectures support in Tekton.
It doesn't cover any specific architecture and can be used as reference for arch specific TEPs.

This TEP proposes main concept how Tekton tests and builds can be done for non-amd64 architecture
and provides samples of the Tekton tasks to run the build and e2e tests for Tekton pipeline.
It suggests to use native hardware for arch specific tasks and amd64 Tekton cluster (similar to dogfooding or itself)
to play master role and manage all the steps.

[tekton-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: vdemeester

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• teps/OWNERS [vdemeester]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[imjasonh]

cc @mattmoor

[imjasonh]

With recently added support for building multi-arch images in ko, and multi-arch support for distroless (which we use as a base for most of our images), we can soon start to build multi-arch images using ko resolve --platform=all, from the normal dogfooding cluster we use today.

There will need to be some work to make the git-init steps, and some others, multi-arch-capable, but until then at least Pipelines that don't use the git resource should be supported just fine.

With those aside, once we get Tekton's release onto --platform=all, supporting new arch boils down to adding support for that arch to gcr.io/distroless/static, which is just a few static files in a container.

[vdemeester]

One thing on --platform=all, what if we don't want to support all platform ? (aka, I think it does get the list of platform from the base image used)

[imjasonh]

Is there a reason you don't want support for all possible platforms?

If so, we _could) consider modifying ko based on our needs to allow us to specify more complex --platform flag values. Today, it supports --platform=amd64 (or any single arch), or --platform=all (build for all platforms supported by the base image). If we wanted --platform=amd64,arm64,s390x or --platform=all,-amd64 we could certainly discuss that use case with the ko maintainers. But I'd like to make that request with a well-formed reason for why we don't want to build for all available platforms.

[mattmoor]

Yeah, all was added as a starting point. There are a few more "fun" models that @jonjohnsonjr and I were discussing, but decided this was a nice and simple starting point. Flags is obviously one of the easiest ways to go.

I really wanted to do something smart around intersecting the base images of the collection of import referenced being resolved, but @jonjohnsonjr pointed out that users might want more heterogeneity than this allows (e.g. if I'm targeting a mixed cluster and my node selectors express things my base images do not).

At the end of the day, you have control of the base images you use, so --platform=all is a pretty powerful (albeit complex) escape hatch for folks to use here.

[vdemeester]

Is there a reason you don't want support for all possible platforms?

Yes, man power 🙃, hardware availability, platform knowledge. We will need to support those, fix bugs, reproduce problems, … this is.. some work 🙃. Currently, gcr.io/distroless/… supports amd64 and arm64 right ? so we would support only those (which is fine). What if I want to support more ? Using another base (ubuntu) that supports more is fine, but then, maybe ubuntu also support arm or something I don't have the power to support as of today (hardware availability to test things, etc…).

As long as it supports single arch (--platform=amd64), I am fine, we can setup our release pipeline to loop over the arch we want to support. --platform=amd64,arm64,s390x would be nice to have, but is not something required.

At the end of the day, you have control of the base images you use, so --platform=all is a pretty powerful (albeit complex) escape hatch for folks to use here.

Yes and no 🙃 We can choose the base image, and in that sense, we are in control of the base image. If we choose to use distroless, then, we don't control the supported arch anymore (at least not with all 🙃). --platform=myarch is good enough for me anyway.

My only worry is, what happens if I use --platform=s390x and the base image doesn't support it ? 😝 We may need to maintain our own set of base image (maybe in plumbing) if at some point we want to support arch that are not available in the current base images used 😝

[vdemeester]

btw, I am just thinking outloud, I am really happy ko supports multi-platform, and the default behavior of --platform=all seems good to me too 😛 😉

[barthy1]

I was insanely happy when leant that multi-arch support PR was merged to ko. And I agree that it makes things much easier.
At the same time I agree with @vdemeester that --platform=all can be too powerful as tests still should be running on native hardware and "support for all possible platforms" for me means that native hardware should be available for these platforms, which is not true at this moment.

[barthy1]

Technical details :) I verified that ko resolve --platform=all, using alpine base image in .ko.yaml, builds installable/runnable versions of pipeline, triggers, operator, dashboard for amd64 and s390x (with multi-arch images)

[imjasonh]

I understand the concern about the support burden of having Tekton's images available in a plethora of architectures. If it helps, we can document that Tekton only officially supports amd64 and s390x, for example, even though other architectures might be available.

Having this distinction could be helpful, since it means users who might be interested in testing and eventually committing to support arm64 could play with those out of the box (or, for some othernew-arch, assuming distroless supports new-arch), without having to build their own images.

[mattmoor]

cc @duglin @afrittoli

IBM had raised a desire to have support for Power and Z in the context of Knative, and a willingness (I think) by IBM to contribute resources for CI. I thought Tekton might be in a similar boat.

For arm64, I'm going to try to use an EKS Graviton cluster for as much of my development as it will allow.

---

Totally ack the concerns, though. To Jason's point, perhaps it would be enough to distinguish between "supported" and "best effort"? You could even produce distinct bundles with ko resolve --platform={supported} and ko resolve --platform=all.

Feedback on how to control this is certainly welcome and desired 🙏

[barthy1]

@vdemeester @imjasonh @mattmoor thank you for the feedback for the first version of this TEP. Following your input I updated it, specifically adding ko usage for the builds

[dlorenc]

This looks ready to merge in the proposed state.

/lgtm