Base ECK docker image on distroless instead of UBI by default #5580

naemono · 2022-04-13T18:00:14Z

This will update the default Dockerfile for ECK operator to use a scratch image instead of UBI. An additional UBI dockerfile has been added, which is leveraged in CI tooling to build/push an additional image to both:

docker.elastic.co
docker.io

So, a total of 4 images (not including different platforms) will be built when building a new release (non-SNAPSHOT)

docker.elastic.co/elastic/eck-operator:x.y.z
docker.io/elastic/eck-operator:x.y.z
docker.elastic.co/elastic/eck-operator:x.y.z-ubi
docker.io/elastic/eck-operator:x.y.z-ubi

This will be paired with an additional PR for the internal redhat tooling leveraged during release.

Reason for Draft:

Additional testing is being done now to ensure:

These new scratch images work in non-Openshift environment.
The UBI images work in Openshift environments.

Update makefile to add new section for building ubi image. Make spacing consistent in makefile include '-ubi' suffix in image* when building CSV files for redhat. Ensure UBI build references the ubi dockerfile

naemono · 2022-04-13T19:39:40Z

The additional ubi image build adds about 30 seconds to the full release.

### Start ###
13:46:52  # The following UBI build should already have the binary cached, and should
.......
13:47:14  #32 pushing layers
13:47:18  #32 pushing layers 3.3s done
13:47:18  #32 pushing manifest for docker.elastic.co/eck/eck-operator:2.3.0-rc0-ubi@sha256:7dd8d566fdcb6f0a20ad7b02f79de25131592be9b0fe85c1c3ea994e820b9157
13:47:20  #32 pushing manifest for docker.elastic.co/eck/eck-operator:2.3.0-rc0-ubi@sha256:7dd8d566fdcb6f0a20ad7b02f79de25131592be9b0fe85c1c3ea994e820b9157 1.9s done
13:47:22  #32 pushing layers 2.0s done
13:47:22  #32 pushing manifest for docker.io/elastic/eck-operator:2.3.0-rc0-ubi@sha256:7dd8d566fdcb6f0a20ad7b02f79de25131592be9b0fe85c1c3ea994e820b9157
13:47:22  #32 pushing manifest for docker.io/elastic/eck-operator:2.3.0-rc0-ubi@sha256:7dd8d566fdcb6f0a20ad7b02f79de25131592be9b0fe85c1c3ea994e820b9157 0.7s done
13:47:22  #32 DONE 11.0s
### end ###

pebrc · 2022-04-14T16:44:08Z

I think we should adopt the same naming convention as Elasticsearch and the rest of the Elastic Stack: $REPO-ubi8:$VERSION. I know there are good arguments for putting it on the version as metadata as defined in semver, but just for consistency across Elastic's offerings we should follow the established pattern.

pebrc · 2022-04-14T17:10:12Z

Dockerfile


 # Copy the operator binary into a lighter image
-FROM registry.access.redhat.com/ubi8/ubi-minimal:8.5
+FROM scratch


Should this be distroless instead? We used to use that in the past. AFAIK one of the advantages is that it comes with CA certificates and supports tmp (not sure if we need the latter)

I can move this to distroless. I had run through all e2e tests using scratch, and -ubi builds and hadn't hit a single failure.

I am trying to think of scenarios where the operator actually makes HTTP requests that would require root CA certs. The only one I can think of now is the external secrets feature we added recently. The operator makes a request to the linked Elastic Stack applications to determine their version, which would require those certs. We don't test this setup in e2e tests (we only approximate it with another Elasticsearch cluster in the same k8s cluster). @thbkrkr may have more insights here.

But I guess distroless puts us on the safe side wrt that.

Yes, we are in a better place using distroless for the reasons you outlined.

naemono · 2022-04-20T19:15:17Z

After fighting with config/samples, and having my own data in there causing havoc in e2e tests, this has finally been verified, and ready for 👀

naemono · 2022-04-20T19:16:18Z

run/e2e-tests

naemono · 2022-04-21T13:09:57Z

run/e2e-tests

naemono · 2022-04-21T17:43:40Z

run/e2e-tests

…'s already non-root.

naemono · 2022-04-21T23:40:01Z

run/e2e-tests

pebrc

Almost good to go. We already have some UBI logic in the generator for the OLM versions of ECK. I would tie the image onto that so that we use it only for the certified operator.

hack/operatorhub/templates/csv.tpl

test/e2e/samples_test.go

pebrc · 2022-04-28T08:02:44Z

Dockerfile

@@ -39,16 +39,11 @@ LABEL name="Elastic Cloud on Kubernetes" \
      description="Elastic Cloud on Kubernetes automates the deployment, provisioning, management, and orchestration of Elasticsearch, Kibana, APM Server, Beats, and Enterprise Search on Kubernetes" \
      io.k8s.description="Elastic Cloud on Kubernetes automates the deployment, provisioning, management, and orchestration of Elasticsearch, Kibana, APM Server, Beats, and Enterprise Search on Kubernetes"


Sorry I missed this initially. These label choices where motivated by the need to override existing labels in the UBI base image. I think now that we are moving back to distroless there is no need for these often redundant (description and io.k8s.description and summary, io.k8s.display-name and name) labels.

I wonder if we should use OCI standardised labels on this docker image instead (for the UBI one we have to stick with the current set of labels to keep the overrides)
https://github.com/opencontainers/image-spec/blob/main/annotations.md

This is a good point. I've removed the overrides, and added OCI annotations as described in link, and grouped the k8s labels together alphabetically. lmk if I've covered them all.

Also verified that the golang:1.x image we base things from does not have any preexisting labels:

"Labels": {}

Dockerfile

Dockerfile.ubi

Co-authored-by: Thibault Richard <thbkrkr@users.noreply.github.com>

thbkrkr

LGTM

Makefile

hack/operatorhub/templates/csv.tpl

Co-authored-by: Thibault Richard <thbkrkr@users.noreply.github.com>

Dockerfile

Co-authored-by: Peter Brachwitz <peter.brachwitz@gmail.com>

…c#5580) * Update default dockerfile to add scratch. Update makefile to add new section for building ubi image. Make spacing consistent in makefile include '-ubi' suffix in image* when building CSV files for redhat. Ensure UBI build references the ubi dockerfile * move from scratch to distroless * Update image name to include ubi suffix, not tag * Move template to have suffix on image name, not tag * Add without integration check to apmserver sample test builders * Add newline to generate new git hash * Attempt to use different machine type * Remove user specification in dockerfile for distroless:nonroot, as it's already non-root. * revert change to machine type * Only use ubi image when certified-operator. * removing WithoutIntegrationCheck() from this PR * use UbiOnly instead of string comparison. * Add OCI annotations to dockerfile, and remove legacy labels. * Update Dockerfile label to correct base name Co-authored-by: Thibault Richard <thbkrkr@users.noreply.github.com> * Update Dockerfile.ubi for updated golang version Co-authored-by: Thibault Richard <thbkrkr@users.noreply.github.com> * Convert to ubi8 in Makefile Co-authored-by: Thibault Richard <thbkrkr@users.noreply.github.com> * Convert to ubi8 in hack/operatorhub/templates/csv.tpl Co-authored-by: Thibault Richard <thbkrkr@users.noreply.github.com> * Convert to ubi8 again in hack/operatorhub/templates/csv.tpl Co-authored-by: Thibault Richard <thbkrkr@users.noreply.github.com> * Convert to ubi8 once more in hack/operatorhub/templates/csv.tpl Co-authored-by: Thibault Richard <thbkrkr@users.noreply.github.com> * Update license Label Co-authored-by: Peter Brachwitz <peter.brachwitz@gmail.com> Co-authored-by: Thibault Richard <thbkrkr@users.noreply.github.com> Co-authored-by: Peter Brachwitz <peter.brachwitz@gmail.com>

naemono added the :ci Things related to Continuous Integration, automation and releases label Apr 13, 2022

botelastic bot added the triage label Apr 13, 2022

Update default dockerfile to add scratch.

f0a3009

Update makefile to add new section for building ubi image. Make spacing consistent in makefile include '-ubi' suffix in image* when building CSV files for redhat. Ensure UBI build references the ubi dockerfile

naemono force-pushed the 4561-replace-ubi-with-scratch branch from d945e60 to f0a3009 Compare April 13, 2022 18:01

thbkrkr added the >enhancement Enhancement of existing functionality label Apr 14, 2022

botelastic bot removed the triage label Apr 14, 2022

pebrc reviewed Apr 14, 2022

View reviewed changes

naemono added 5 commits April 14, 2022 14:08

move from scratch to distroless

d804c9e

Update image name to include ubi suffix, not tag

a71f6e9

Move template to have suffix on image name, not tag

a2d8d99

Add without integration check to apmserver sample test builders

7f7c12d

Add newline to generate new git hash

ee1d87b

naemono marked this pull request as ready for review April 20, 2022 19:14

thbkrkr added the v2.3.0 label Apr 21, 2022

Attempt to use different machine type

78a8ad9

Remove user specification in dockerfile for distroless:nonroot, as it…

714ba8b

…'s already non-root.

revert change to machine type

c1fd560

pebrc reviewed Apr 27, 2022

View reviewed changes

hack/operatorhub/templates/csv.tpl Outdated Show resolved Hide resolved

test/e2e/samples_test.go Outdated Show resolved Hide resolved

pebrc changed the title ~~4561 replace ubi with scratch~~ Base ECK docker image on distroless instead of UBI by default Apr 27, 2022

naemono added 3 commits April 27, 2022 10:40

Only use ubi image when certified-operator.

1dcd8fa

removing WithoutIntegrationCheck() from this PR

ff4778b

use UbiOnly instead of string comparison.

450e264

pebrc reviewed Apr 28, 2022

View reviewed changes

Add OCI annotations to dockerfile, and remove legacy labels.

7771839

thbkrkr reviewed Apr 28, 2022

View reviewed changes

Dockerfile Outdated Show resolved Hide resolved

Dockerfile.ubi Outdated Show resolved Hide resolved

naemono and others added 3 commits April 28, 2022 11:27

Update Dockerfile label to correct base name

04722ad

Co-authored-by: Thibault Richard <thbkrkr@users.noreply.github.com>

Update Dockerfile.ubi for updated golang version

04c3370

Co-authored-by: Thibault Richard <thbkrkr@users.noreply.github.com>

Merge branch 'main' into 4561-replace-ubi-with-scratch

4638e24

thbkrkr approved these changes Apr 28, 2022

View reviewed changes

Makefile Outdated Show resolved Hide resolved

hack/operatorhub/templates/csv.tpl Outdated Show resolved Hide resolved

hack/operatorhub/templates/csv.tpl Outdated Show resolved Hide resolved

hack/operatorhub/templates/csv.tpl Outdated Show resolved Hide resolved

naemono and others added 4 commits April 28, 2022 15:37

Convert to ubi8 in Makefile

cfc347f

Co-authored-by: Thibault Richard <thbkrkr@users.noreply.github.com>

Convert to ubi8 in hack/operatorhub/templates/csv.tpl

b351a3d

Co-authored-by: Thibault Richard <thbkrkr@users.noreply.github.com>

Convert to ubi8 again in hack/operatorhub/templates/csv.tpl

62adade

Co-authored-by: Thibault Richard <thbkrkr@users.noreply.github.com>

Convert to ubi8 once more in hack/operatorhub/templates/csv.tpl

717951d

Co-authored-by: Thibault Richard <thbkrkr@users.noreply.github.com>

pebrc reviewed Apr 29, 2022

View reviewed changes

Dockerfile Outdated Show resolved Hide resolved

Update license Label

9c5f277

Co-authored-by: Peter Brachwitz <peter.brachwitz@gmail.com>

pebrc approved these changes Apr 29, 2022

View reviewed changes

naemono merged commit 56d8edc into elastic:main Apr 29, 2022

thbkrkr removed the :ci Things related to Continuous Integration, automation and releases label May 30, 2022

thbkrkr mentioned this pull request Jun 14, 2022

Fix ubi8 suffix in Makefile #5772

Merged

thbkrkr mentioned this pull request Aug 31, 2022

Restore operator binary installation in slash #5983

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Base ECK docker image on distroless instead of UBI by default #5580

Base ECK docker image on distroless instead of UBI by default #5580

naemono commented Apr 13, 2022

naemono commented Apr 13, 2022

pebrc commented Apr 14, 2022 •

edited

Loading

pebrc Apr 14, 2022

naemono Apr 14, 2022

pebrc Apr 14, 2022 •

edited

Loading

thbkrkr Apr 28, 2022

naemono commented Apr 20, 2022

naemono commented Apr 20, 2022

naemono commented Apr 21, 2022

naemono commented Apr 21, 2022

naemono commented Apr 21, 2022

pebrc left a comment

pebrc Apr 28, 2022

naemono Apr 28, 2022 •

edited

Loading

thbkrkr left a comment

		@@ -39,16 +39,11 @@ LABEL name="Elastic Cloud on Kubernetes" \
		description="Elastic Cloud on Kubernetes automates the deployment, provisioning, management, and orchestration of Elasticsearch, Kibana, APM Server, Beats, and Enterprise Search on Kubernetes" \
		io.k8s.description="Elastic Cloud on Kubernetes automates the deployment, provisioning, management, and orchestration of Elasticsearch, Kibana, APM Server, Beats, and Enterprise Search on Kubernetes"

Base ECK docker image on distroless instead of UBI by default #5580

Base ECK docker image on distroless instead of UBI by default #5580

Conversation

naemono commented Apr 13, 2022

naemono commented Apr 13, 2022

pebrc commented Apr 14, 2022 • edited Loading

pebrc Apr 14, 2022

Choose a reason for hiding this comment

naemono Apr 14, 2022

Choose a reason for hiding this comment

pebrc Apr 14, 2022 • edited Loading

Choose a reason for hiding this comment

thbkrkr Apr 28, 2022

Choose a reason for hiding this comment

naemono commented Apr 20, 2022

naemono commented Apr 20, 2022

naemono commented Apr 21, 2022

naemono commented Apr 21, 2022

naemono commented Apr 21, 2022

pebrc left a comment

Choose a reason for hiding this comment

pebrc Apr 28, 2022

Choose a reason for hiding this comment

naemono Apr 28, 2022 • edited Loading

Choose a reason for hiding this comment

thbkrkr left a comment

Choose a reason for hiding this comment

pebrc commented Apr 14, 2022 •

edited

Loading

pebrc Apr 14, 2022 •

edited

Loading

naemono Apr 28, 2022 •

edited

Loading