Add vtadmin binary to release and Docker images #9405

jakon89 · 2021-12-16T21:58:35Z

Signed-off-by: Adam Jedro adamjedro@gmail.com

Description

This adds vtadmin binary (server) to the release package and Docker images. I would like to run vtadmin for qa/prod clusters but unless binary is in the official package I am not allowed to do it.

Related Issue(s)

Checklist

Should this PR be backported?
Tests were added or are not required
Documentation was added or is not required

Deployment Notes

Signed-off-by: Adam Jedro <adamjedro@gmail.com>

doeg · 2021-12-16T22:05:26Z

This looks good to me, and I appreciate you taking the time to do this! 💖

That said, I'm definitely not the person to sign off on go releases -- @ajm188 @deepthi @rohit-nayak-ps would one of you mind signing off on this?

deepthi

For all docker image changes we like to see a before and after size to make sure it is not getting too bloated. vtadmin will be a core component so we eventually need to get it in regardless of size, but it will be nice to see how much bigger the image gets.

deepthi · 2021-12-17T00:11:46Z

docker/k8s/Dockerfile

@@ -50,6 +50,7 @@ COPY --from=base /vt/bin/vtgate /vt/bin/
 COPY --from=base /vt/bin/vttablet /vt/bin/
 COPY --from=base /vt/bin/vtworker /vt/bin/
 COPY --from=base /vt/bin/vtbackup /vt/bin/
+COPY --from=base /vt/bin/vtadmin /vt/bin/


Are you using the k8s image? The standard image is vitess/lite. You are already adding vtadmin to the make install target, that gets it into all the lite images. This change should not be required.

I publish all the individual component images from the k8s image, so IMO this should stay. There is probably an additional step required to copy out the files needed for the web. This step COPY --from=base $VTROOT/web /vt/web/ is what copies them for the existing UI, so I'd expect to see something similar for vtadmin.

Once this gets merged and we come up with an appropriate Dockerfile, I'll add a new directory and vtadmin image repo on Docker Hub that approximates the existing vtctld. https://github.com/vitessio/vitess/blob/main/docker/k8s/vtctld/Dockerfile

Side note: there's an open issue for consolidating images, but for now we still need to support people using lite and the component images. #7433

@derekperkins
I have added /web/vtadmin to the lite image.

btw there is a difference between vtadmin and orchestrator web files: orchestrator stores js/css files in final form in the repo, however, for vtadmin we copy the whole frontend project that needs to be built by npm run build - only then we get the final js/css files.

This is not a problem since in my project I use vitess/lite image as a base builder and then copy binaries/web files to final images (vtadmin-web, vtorc, vtgate, vtadmin-server).
Example of how I do it for vtorc:

ARG BASE_REGISTRY=gcr.io ARG BASE_IMAGE=distroless/base ARG BASE_TAG=nonroot FROM vitess/lite:v12.0.1 AS builder FROM ${BASE_REGISTRY}/${BASE_IMAGE}:${BASE_TAG} COPY --from=builder vt/bin/vtorc /vtorc COPY --from=builder vt/web/orchestrator /web/orchestrator USER 65532 ENTRYPOINT ["/vtorc"] HEALTHCHECK NONE

but IMO we should include the final build (js/css etc) somehow.

I see three potential ways of fixing it:

we copy raw web/vtadmin to docker/lite and it's up to the user of docker/lite to build the frontend. That adds 10M to the docker/lite size. This is what is done in that PR for now - https://github.com/vitessio/vitess/pull/9405/files#diff-bdcbcdaa92c26cad86d024c8d1061d14c36e20f67c5c8e577c17e3bbb196c607R54

we run npm run build as a part of docker/lite build and copy build/ dir into final docker/lite. That adds additional 13M to the size of docker/lite (just checked)

we start building separate vtadmin-web image:

use node Docker image as a builder

copy all vtadmin web files

run npm install

copy over generated js/css files to the final image (and serve them by nginx or another webserver)

cc @doeg ^

I will be happy to send a PR if we decide on the final form.

@jakon89 wow, thanks so much for the thorough notes! I really really appreciate it.

Between the three, I don't have a strong opinion. I'm happy to go with whatever is most ergonomic for Vitess operators! 🤔 The third option is the closest to what we do at Slack, though that doesn't mean it's the best for everyone. @jakon89 @derekperkins do either you have a preference?

P.S. There is definitely work to be done to make the front-end build smaller; the TypeScript generated from the protobufs is particularly huge.

#3 is my preference, that for now I would put in /docker/k8s/vtadmin. You mentioned vtadmin-web, are you building separate images for the server and the web ui?

It's probably also worth building into lite per #2, but I'll leave that decision to @deepthi.

If we follow the same pattern as we do for Golang binaries, we should do #2. I'm fine with that. Not sure if you want to do that as a separate PR, or include it in this one. If this PR is complete, it LGTM.

@deepthi let's merge that, I will implement building vtadmin-ui in new PR. Thanks

jakon89 · 2021-12-17T10:36:09Z

Hey @deepthi

The binary alone is 52M on linux:

vitess@5432da5b3fcb:/vt/bin$ ls -lh | grep vtadmin
-rwxr-xr-x 1 vitess vitess  52M Dec 16 20:14 vtadmin

and web files take 10M:

adam@local vtadmin % pwd
/Users/adam/dev/vitess/web/vtadmin
adam@local vtadmin % du -hs
 10M	.

Docker image before:

vitess/lite                                             latest         84a8cde24a65   About a minute ago   1.13GB

and after:

REPOSITORY                                              TAG            IMAGE ID       CREATED          SIZE
vitess/lite                                             latest         8ea1f7d96be8   32 seconds ago   1.2GB

Signed-off-by: Adam Jedro <adamjedro@gmail.com>

deepthi · 2022-01-04T22:24:22Z

Not sure why all the CI actions got canceled. I re-ran a unit test and a cluster test before merging.

L3o-pold · 2022-04-06T10:09:32Z

docker/lite/Dockerfile.mysql57

@@ -51,6 +51,7 @@ ENV PATH $VTROOT/bin:$PATH
 # Copy artifacts from builder layer.
 COPY --from=builder --chown=vitess:vitess /vt/install /vt
 COPY --from=builder --chown=vitess:vitess /vt/src/vitess.io/vitess/web/orchestrator /vt/web/orchestrator
+COPY --from=builder --chown=vitess:vitess /vt/src/vitess.io/vitess/web/vtadmin /vt/web/vtadmin


Why this part was not included in mysql8?

deepthi · 2022-06-26T15:11:29Z

@jakon89 how are you using this? see discussion in #10543

jakon89 · 2022-06-27T12:57:41Z

@deepthi we are building separate images vtadmin-ui and vtadmin-api out of vitess-lite docker image. AFAIK #10543 introduced a single image that can be used to run any of the app (frontend or backend) so most probably we are going to migrate to it once vitess starts publishing a new image to the official docker repo.

Add vtadmin binaries to release and Docker images

141ed42

Signed-off-by: Adam Jedro <adamjedro@gmail.com>

jakon89 requested review from derekperkins and dkhenry as code owners December 16, 2021 21:58

jakon89 changed the title ~~Add vtadmin binaries to release and Docker images~~ Add vtadmin binary to release and Docker images Dec 16, 2021

deepthi reviewed Dec 17, 2021

View reviewed changes

deepthi added Component: Build/CI release notes Type: CI/Build labels Dec 17, 2021

add vtadmin frontend to vitess/lite image

05a04c0

Signed-off-by: Adam Jedro <adamjedro@gmail.com>

deepthi approved these changes Dec 23, 2021

View reviewed changes

deepthi merged commit adef6b9 into vitessio:main Jan 4, 2022

jakon89 deleted the add-vtadmin-binary-to-release branch January 5, 2022 09:14

L3o-pold reviewed Apr 6, 2022

View reviewed changes

deepthi mentioned this pull request Jun 25, 2022

add vtadmin docker image #10543

Merged

3 tasks

deepthi mentioned this pull request Jun 26, 2022

add vtadmin web files to all lite images #10581

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add vtadmin binary to release and Docker images #9405

Add vtadmin binary to release and Docker images #9405

jakon89 commented Dec 16, 2021 •

edited

Loading

doeg commented Dec 16, 2021

deepthi left a comment

deepthi Dec 17, 2021 •

edited

Loading

derekperkins Dec 17, 2021

derekperkins Dec 17, 2021

jakon89 Dec 17, 2021

jakon89 Dec 17, 2021 •

edited

Loading

doeg Dec 17, 2021

derekperkins Dec 22, 2021

deepthi Dec 23, 2021 •

edited

Loading

jakon89 Dec 24, 2021

jakon89 commented Dec 17, 2021 •

edited

Loading

deepthi commented Jan 4, 2022

L3o-pold Apr 6, 2022

deepthi commented Jun 26, 2022

jakon89 commented Jun 27, 2022

Add vtadmin binary to release and Docker images #9405

Add vtadmin binary to release and Docker images #9405

Conversation

jakon89 commented Dec 16, 2021 • edited Loading

Description

Related Issue(s)

Checklist

Deployment Notes

doeg commented Dec 16, 2021

deepthi left a comment

Choose a reason for hiding this comment

deepthi Dec 17, 2021 • edited Loading

Choose a reason for hiding this comment

derekperkins Dec 17, 2021

Choose a reason for hiding this comment

derekperkins Dec 17, 2021

Choose a reason for hiding this comment

jakon89 Dec 17, 2021

Choose a reason for hiding this comment

jakon89 Dec 17, 2021 • edited Loading

Choose a reason for hiding this comment

doeg Dec 17, 2021

Choose a reason for hiding this comment

derekperkins Dec 22, 2021

Choose a reason for hiding this comment

deepthi Dec 23, 2021 • edited Loading

Choose a reason for hiding this comment

jakon89 Dec 24, 2021

Choose a reason for hiding this comment

jakon89 commented Dec 17, 2021 • edited Loading

deepthi commented Jan 4, 2022

L3o-pold Apr 6, 2022

Choose a reason for hiding this comment

deepthi commented Jun 26, 2022

jakon89 commented Jun 27, 2022

jakon89 commented Dec 16, 2021 •

edited

Loading

deepthi Dec 17, 2021 •

edited

Loading

jakon89 Dec 17, 2021 •

edited

Loading

deepthi Dec 23, 2021 •

edited

Loading

jakon89 commented Dec 17, 2021 •

edited

Loading