Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add cargo to the Docker image #543

Closed
josevalim opened this issue Sep 4, 2021 · 22 comments
Closed

Add cargo to the Docker image #543

josevalim opened this issue Sep 4, 2021 · 22 comments
Labels
chore Technical changes wontfix This will not be worked on

Comments

@josevalim
Copy link
Contributor

I think it will be an increasingly common use case, so we should probably add it. How much does it add to the image? 🤔

@sobolevn
Copy link
Contributor

sobolevn commented Sep 4, 2021

I would be happy to contribute this (with the size comparison).

@josevalim
Copy link
Contributor Author

@sobolevn feel free to give it a try and let us know then! The Dockerfile is in the repo. :) Thanks!

@sobolevn
Copy link
Contributor

sobolevn commented Sep 4, 2021

Before adding rust-related stuff

Total size: 98.6MB

The most interesting part: RUN /bin/sh -c apk add --no-cache openssl ncurses-libs git # buildkit 13.1MB

All layers:

IMAGE                                                                     CREATED          CREATED BY                                                                                                                                                                               SIZE      COMMENT
sha256:1f47f6cf35ce112e246e120ce18a1eb7ecf521413d1a0c2302f80a40ba7b7af4   7 minutes ago    CMD ["/app/bin/livebook" "start"]                                                                                                                                                        0B        buildkit.dockerfile.v0
<missing>                                                                 7 minutes ago    RUN /bin/sh -c find /app -executable -type f -exec chmod +x {} + # buildkit                                                                                                              12.2kB    buildkit.dockerfile.v0
<missing>                                                                 7 minutes ago    COPY /app/_build/prod/rel/livebook /app # buildkit                                                                                                                                       19MB      buildkit.dockerfile.v0
<missing>                                                                 10 minutes ago   ENV LIVEBOOK_IP=0.0.0.0                                                                                                                                                                  0B        buildkit.dockerfile.v0
<missing>                                                                 10 minutes ago   RUN /bin/sh -c mix local.hex --force &&     mix local.rebar --force # buildkit                                                                                                           1.95MB    buildkit.dockerfile.v0
<missing>                                                                 10 minutes ago   RUN /bin/sh -c mkdir $HOME && chmod 777 $HOME # buildkit                                                                                                                                 0B        buildkit.dockerfile.v0
<missing>                                                                 10 minutes ago   ENV HOME=/home/livebook                                                                                                                                                                  0B        buildkit.dockerfile.v0
<missing>                                                                 10 minutes ago   WORKDIR /data                                                                                                                                                                            0B        buildkit.dockerfile.v0
<missing>                                                                 10 minutes ago   RUN /bin/sh -c apk add --no-cache     openssl ncurses-libs     git # buildkit                                                                                                            13.1MB    buildkit.dockerfile.v0
<missing>                                                                 3 months ago     /bin/sh -c #(nop) COPY dir:d55f0d9d25201138a937fe196e7c1dbceaaab82468e05364e3d1710d4f5275e3 in /usr/local                                                                                6.64MB    
<missing>                                                                 3 months ago     /bin/sh -c #(nop) COPY dir:63a7482e7941b2cc36a87255f70dc225f1136f7cd3a9a8a00fd16cc7a67a06b7 in /usr/local                                                                                48MB      
<missing>                                                                 3 months ago     |1 ERLANG=24.0 /bin/sh -c apk add --update --no-cache   libstdc++   ncurses   $(if [ "${ERLANG:0:1}" = "1" ]; then echo "libressl"; else echo "openssl"; fi)   unixodbc   lksctp-tools   4.41MB    
<missing>                                                                 3 months ago     /bin/sh -c #(nop)  ARG ERLANG                                                                                                                                                            0B        
<missing>                                                                 5 months ago     /bin/sh -c #(nop)  CMD ["/bin/sh"]                                                                                                                                                       0B        
<missing>                                                                 5 months ago     /bin/sh -c #(nop) ADD file:6b081cabb4b256ee07587d249c4989b5b679375529542b81550a65b6f19f274e in /                                                                                         5.61MB 

With apk add cargo

Total size: 495MB

The most interesting part: RUN /bin/sh -c apk add --no-cache openssl ncurses-libs git cargo # buildkit 409MB

All layers:

IMAGE                                                                     CREATED         CREATED BY                                                                                                                                                                               SIZE      COMMENT
sha256:ed16c7940889783ee81ae4ff44c3b7b489fdc235292896731665eabe6449832d   2 minutes ago   CMD ["/app/bin/livebook" "start"]                                                                                                                                                        0B        buildkit.dockerfile.v0
<missing>                                                                 2 minutes ago   RUN /bin/sh -c find /app -executable -type f -exec chmod +x {} + # buildkit                                                                                                              12.2kB    buildkit.dockerfile.v0
<missing>                                                                 2 minutes ago   COPY /app/_build/prod/rel/livebook /app # buildkit                                                                                                                                       19MB      buildkit.dockerfile.v0
<missing>                                                                 2 minutes ago   ENV LIVEBOOK_IP=0.0.0.0                                                                                                                                                                  0B        buildkit.dockerfile.v0
<missing>                                                                 2 minutes ago   RUN /bin/sh -c mix local.hex --force &&     mix local.rebar --force # buildkit                                                                                                           1.95MB    buildkit.dockerfile.v0
<missing>                                                                 2 minutes ago   RUN /bin/sh -c mkdir $HOME && chmod 777 $HOME # buildkit                                                                                                                                 0B        buildkit.dockerfile.v0
<missing>                                                                 2 minutes ago   ENV HOME=/home/livebook                                                                                                                                                                  0B        buildkit.dockerfile.v0
<missing>                                                                 2 minutes ago   WORKDIR /data                                                                                                                                                                            0B        buildkit.dockerfile.v0
<missing>                                                                 2 minutes ago   RUN /bin/sh -c apk add --no-cache     openssl ncurses-libs     git     cargo # buildkit                                                                                                  409MB     buildkit.dockerfile.v0
<missing>                                                                 3 months ago    /bin/sh -c #(nop) COPY dir:d55f0d9d25201138a937fe196e7c1dbceaaab82468e05364e3d1710d4f5275e3 in /usr/local                                                                                6.64MB    
<missing>                                                                 3 months ago    /bin/sh -c #(nop) COPY dir:63a7482e7941b2cc36a87255f70dc225f1136f7cd3a9a8a00fd16cc7a67a06b7 in /usr/local                                                                                48MB      
<missing>                                                                 3 months ago    |1 ERLANG=24.0 /bin/sh -c apk add --update --no-cache   libstdc++   ncurses   $(if [ "${ERLANG:0:1}" = "1" ]; then echo "libressl"; else echo "openssl"; fi)   unixodbc   lksctp-tools   4.41MB    
<missing>                                                                 3 months ago    /bin/sh -c #(nop)  ARG ERLANG                                                                                                                                                            0B        
<missing>                                                                 5 months ago    /bin/sh -c #(nop)  CMD ["/bin/sh"]                                                                                                                                                       0B        
<missing>                                                                 5 months ago    /bin/sh -c #(nop) ADD file:6b081cabb4b256ee07587d249c4989b5b679375529542b81550a65b6f19f274e in /                                                                                         5.61MB  

Problems:

Using rust as the second step in our build

Total size: 1.87GB

Layers:

IMAGE                                                                     CREATED          CREATED BY                                                                                                                                                                                     SIZE      COMMENT
sha256:0d136274e2f8d296bb590027e42263191d3f0cb2bf45f6d060f6e16d38807e99   50 minutes ago   CMD ["/app/bin/livebook" "start"]                                                                                                                                                              0B        buildkit.dockerfile.v0
<missing>                                                                 50 minutes ago   RUN /bin/sh -c find /app -executable -type f -exec chmod +x {} + # buildkit                                                                                                                    12.2kB    buildkit.dockerfile.v0
<missing>                                                                 50 minutes ago   COPY /app/_build/prod/rel/livebook /app # buildkit                                                                                                                                             19MB      buildkit.dockerfile.v0
<missing>                                                                 50 minutes ago   ENV LIVEBOOK_IP=0.0.0.0                                                                                                                                                                        0B        buildkit.dockerfile.v0
<missing>                                                                 50 minutes ago   RUN /bin/sh -c mix local.hex --force &&     mix local.rebar --force # buildkit                                                                                                                 1.95MB    buildkit.dockerfile.v0
<missing>                                                                 51 minutes ago   RUN /bin/sh -c mkdir $HOME && chmod 777 $HOME &&     chmod -R a+w $RUSTUP_HOME $CARGO_HOME # buildkit                                                                                          905MB     buildkit.dockerfile.v0
<missing>                                                                 51 minutes ago   ENV HOME=/home/livebook                                                                                                                                                                        0B        buildkit.dockerfile.v0
<missing>                                                                 51 minutes ago   WORKDIR /data                                                                                                                                                                                  0B        buildkit.dockerfile.v0
<missing>                                                                 51 minutes ago   COPY /usr/local/cargo /usr/local/cargo # buildkit                                                                                                                                              14.7MB    buildkit.dockerfile.v0
<missing>                                                                 51 minutes ago   COPY /usr/local/rustup /usr/local/rustup # buildkit                                                                                                                                            729MB     buildkit.dockerfile.v0
<missing>                                                                 51 minutes ago   RUN /bin/sh -c apk add --no-cache     openssl ncurses-libs     git gcc musl-dev # buildkit                                                                                                     136MB     buildkit.dockerfile.v0
<missing>                                                                 51 minutes ago   ENV RUSTUP_HOME=/usr/local/rustup CARGO_HOME=/usr/local/cargo PATH=/usr/local/cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin RUSTFLAGS=-C target-feature=-crt-static   0B        buildkit.dockerfile.v0
<missing>                                                                 3 months ago     /bin/sh -c #(nop) COPY dir:d55f0d9d25201138a937fe196e7c1dbceaaab82468e05364e3d1710d4f5275e3 in /usr/local                                                                                      6.64MB    
<missing>                                                                 3 months ago     /bin/sh -c #(nop) COPY dir:63a7482e7941b2cc36a87255f70dc225f1136f7cd3a9a8a00fd16cc7a67a06b7 in /usr/local                                                                                      48MB      
<missing>                                                                 3 months ago     |1 ERLANG=24.0 /bin/sh -c apk add --update --no-cache   libstdc++   ncurses   $(if [ "${ERLANG:0:1}" = "1" ]; then echo "libressl"; else echo "openssl"; fi)   unixodbc   lksctp-tools         4.41MB    
<missing>                                                                 3 months ago     /bin/sh -c #(nop)  ARG ERLANG                                                                                                                                                                  0B        
<missing>                                                                 5 months ago     /bin/sh -c #(nop)  CMD ["/bin/sh"]                                                                                                                                                             0B        
<missing>                                                                 5 months ago     /bin/sh -c #(nop) ADD file:6b081cabb4b256ee07587d249c4989b5b679375529542b81550a65b6f19f274e in /                                                                                               5.61MB 

We need to add extra musl-dev and gcc, because without them several rust's libs do not compile.
We are also using RUSTFLAGS='-C target-feature=-crt-static' hack from rust-lang/cargo#7154
Without it, it was failing with the target x86_64-unknown-linux-musl does not support these crate types

The bad thing about this image is that it is really big. The good thing that it works!

Снимок экрана 2021-09-04 в 22 04 53

But, installation time is really slow. It took almost 3000 seconds to install a single package.
I suspect that the reason is in RUSTFLAGS='-C target-feature=-crt-static', but nothing else worked.

Is it possible to switch to debian-slim? It will hopefully solve:

  • musl problems
  • RUSTFLAGS and installation time
  • It might probably be even smaller than alpine with all the heavy dependecies we have

@jonatanklosko
Copy link
Member

Thanks for looking into this!

With apk add cargo

We could update the base elixir images, so they will be alpine 3.14, which should give us more recent rust/cargo versions.

98.6MB -> 495MB is still a very significant increase though so I'm not sure, we deferred on adding the C build dependencies that bumped the image to just 280MB. @josevalim wdyt?

@sobolevn
Copy link
Contributor

sobolevn commented Sep 4, 2021

We could update the base elixir images, so they will be alpine 3.14, which should give us more recent rust/cargo versions.

In this case there will be still a lot of problems with musl / libc during compilation.

@josevalim
Copy link
Contributor Author

By including Rust, does it mean we automatically get the C toolchain too (or most of it)?

@josevalim
Copy link
Contributor Author

Is it possible to switch to debian-slim? It will hopefully solve:

Can you try building a version with debian-slim or ubuntu-slim and let us know if it is any better?

@sobolevn
Copy link
Contributor

sobolevn commented Sep 4, 2021

By including Rust, does it mean we automatically get the C toolchain too (or most of it)?

Not directly, but by adding musl-dev, which is required to build libc crate on alpine. https://crates.io/crates/libc

Can you try building a version with debian-slim or ubuntu-slim and let us know if it is any better?

Yes, will do tomorrow.

@thbar
Copy link

thbar commented Sep 4, 2021

In this case there will be still a lot of problems with musl / libc during compilation.

I can report I have experienced that recently (see for instance rust-lang/cargo#7563). It appears to be a bit of a rabbit hole, and ultimately I decided to move one of our images to Ubuntu for that reason.

By including Rust, does it mean we automatically get the C toolchain too (or most of it)?

I was about to open an issue to report that exqlite was not installable in the LiveBook Docker image due to "make" and "gcc" missing. This means that ecto_sqlite3 cannot be used. I had to add build-base (in the case of Alpine) for this.

I wonder if adding Rust would also add the tooling required to support SQLite datasets, something which is quite common in data exploration.

@cigrainger
Copy link

cigrainger commented Sep 5, 2021

What about a tag with Explorer preinstalled or with the precompiled binary? (or just a tag with Rust and C toolchain support, so we get the best of all worlds).

@josevalim
Copy link
Contributor Author

We probably want to precompile stuff but it is still in the open if we want to do that at the Hex.pm level or at the Docker level. The issue with doing it in Docker is that if there is a new Explorer version, then the cache won't be helpful? In any case, I think we should at least move to Ubuntu so people can build their images on top of hours with Rust?

@sobolevn
Copy link
Contributor

sobolevn commented Sep 5, 2021

This is already better than alpine timings (3x):
Снимок экрана 2021-09-05 в 17 01 49

But, still pretty bad 🙂
In 1000 seconds I can write the whole project in Python 😆

New debian Image

Based on hexpm/elixir:1.12.3-erlang-24.0.6-debian-buster-20210326
Total size: 1.51GB

Layers:

IMAGE                                                                     CREATED          CREATED BY                                                                                                                                                                                                                                                                                                                            SIZE      COMMENT
sha256:1c8f05f277d06404fc314e7805ec999befa1be23eae96b87d4937fbe721152d8   26 minutes ago   CMD ["/app/bin/livebook" "start"]                                                                                                                                                                                                                                                                                                     0B        buildkit.dockerfile.v0
<missing>                                                                 26 minutes ago   RUN /bin/sh -c find /app -executable -type f -exec chmod +x {} + # buildkit                                                                                                                                                                                                                                                           12.2kB    buildkit.dockerfile.v0
<missing>                                                                 26 minutes ago   COPY /app/_build/prod/rel/livebook /app # buildkit                                                                                                                                                                                                                                                                                    19MB      buildkit.dockerfile.v0
<missing>                                                                 26 minutes ago   ENV LIVEBOOK_IP=0.0.0.0                                                                                                                                                                                                                                                                                                               0B        buildkit.dockerfile.v0
<missing>                                                                 26 minutes ago   RUN /bin/sh -c mix local.hex --force &&     mix local.rebar --force # buildkit                                                                                                                                                                                                                                                        1.95MB    buildkit.dockerfile.v0
<missing>                                                                 26 minutes ago   RUN /bin/sh -c mkdir $HOME && chmod 777 $HOME &&     chmod -R a+w $RUSTUP_HOME $CARGO_HOME # buildkit                                                                                                                                                                                                                                 584MB     buildkit.dockerfile.v0
<missing>                                                                 26 minutes ago   ENV HOME=/home/livebook                                                                                                                                                                                                                                                                                                               0B        buildkit.dockerfile.v0
<missing>                                                                 26 minutes ago   WORKDIR /data                                                                                                                                                                                                                                                                                                                         0B        buildkit.dockerfile.v0
<missing>                                                                 26 minutes ago   COPY /usr/local/cargo /usr/local/cargo # buildkit                                                                                                                                                                                                                                                                                     15.7MB    buildkit.dockerfile.v0
<missing>                                                                 26 minutes ago   COPY /usr/local/rustup /usr/local/rustup # buildkit                                                                                                                                                                                                                                                                                   396MB     buildkit.dockerfile.v0
<missing>                                                                 26 minutes ago   RUN /bin/sh -c apt-get update && apt-get upgrade -y &&     apt-get install --no-install-recommends -y         build-essential ca-certificates libncurses5-dev         git &&     apt-get purge -y --auto-remove -o APT::AutoRemove::RecommendsImportant=false &&     apt-get clean -y &&     rm -rf /var/lib/apt/lists/* # buildkit   288MB     buildkit.dockerfile.v0
<missing>                                                                 26 minutes ago   ENV RUSTUP_HOME=/usr/local/rustup CARGO_HOME=/usr/local/cargo PATH=/usr/local/cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin                                                                                                                                                                                  0B        buildkit.dockerfile.v0
<missing>                                                                 5 hours ago      /bin/sh -c #(nop) COPY dir:a90c0dd8981f88851c523d2fcf156a70fe61b1388fed8579f0077e11ed1c62ff in /usr/local                                                                                                                                                                                                                             6.65MB    
<missing>                                                                 2 days ago       /bin/sh -c #(nop)  ENV LANG=C.UTF-8                                                                                                                                                                                                                                                                                                   0B        
<missing>                                                                 2 days ago       /bin/sh -c #(nop) COPY dir:13f2fa0f7a364384fe6038e3eb82b6202a2ad70e0bf67a325976add497009095 in /usr/local                                                                                                                                                                                                                             64.4MB    
<missing>                                                                 2 days ago       /bin/sh -c apt-get update &&   apt-get -y --no-install-recommends install     libodbc1     libssl1.1     libsctp1                                                                                                                                                                                                                     24.6MB    
<missing>                                                                 5 months ago     /bin/sh -c #(nop)  CMD ["bash"]                                                                                                                                                                                                                                                                                                       0B        
<missing>                                                                 5 months ago     /bin/sh -c #(nop) ADD file:89234bb2f86c7eb890235a48904d1c9898a8d287a525c4fe5698d4a04cdd8f12 in /                                                                                                                                                                                                                                      114MB

We probably want to precompile stuff but it is still in the open if we want to do that at the Hex.pm level or at the Docker level

Python solves this problem by

  1. conda: https://conda.io that has all libs pre-compiled
  2. lots of official images from jupyter https://hub.docker.com/u/jupyter
  3. lots of custom images from the community https://jupyter-docker-stacks.readthedocs.io/en/latest/index.html

@josevalim
Copy link
Contributor Author

@sobolevn is it faster because it is not using musl? What would happen if we use alpine and then use rustup to install rust for us? Would that use the libc toolchain instead or it is going to use whatever is on the OS instead?

But yeah, the fact it still takes 1000s means we definitely need to precompile stuff.

@sobolevn
Copy link
Contributor

sobolevn commented Sep 5, 2021

is it faster because it is not using musl?

Yes, I think so. I'm not a musl / libc expert, but I've just noticed that things on Alpine compile slower than on Debian.
Internet has this info: https://andygrove.io/2020/05/why-musl-extremely-slow/

What would happen if we use alpine and then use rustup to install rust for us? Would that use the libc toolchain instead or it is going to use whatever is on the OS instead?

It looks like rustup is indipendent from libc (both crate and system-dep). But, I might be wrong here.

@josevalim
Copy link
Contributor Author

@sobolevn I see, thanks! In the article above, the image was still below <100MB. Maybe we can achieve similar low numbers here too if we have debian slim images?

@josevalim
Copy link
Contributor Author

I guess an alternative is also to install lib in the alpine image... but not sure how much of a good idea that is.

@sobolevn
Copy link
Contributor

sobolevn commented Sep 5, 2021

@josevalim sure, slim version will be much smaller! At the moment elixir:slim is 283MB
https://github.com/erlef/docker-elixir/blob/2b5da5abf10f3026d01b6dda3c509f4f09f60063/1.12/slim/Dockerfile

@josevalim
Copy link
Contributor Author

josevalim commented Sep 6, 2021

Requested slim images here: hexpm/bob#98

@sobolevn btw, there is another option here, which is to provide a rustup package in Elixir that fetches and downloads the requested precompiled Rust. We would need to make it play well Rustler but sounds very doable. One benefit of said package is that we can also control the Rust version instead of depending on a system one.

This was referenced Sep 18, 2021
@josevalim josevalim added the chore Technical changes label Sep 28, 2021
@josevalim
Copy link
Contributor Author

We are currently exploring shipping a precompiled version of Explorer where rustc nor c compilers will be required. So for now, consider this issue as "stale". If for some reason those efforts fail, I will ping back here.

@josevalim josevalim added invalid wontfix This will not be worked on and removed invalid labels Oct 3, 2021
@dimitarvp
Copy link

Sorry to barge in. Just wanted to share a tidbit that shaved off ~500MB off of our Docker builder image (that we only extract an Elixir release from in the end but still).

If you end up downloading rustup and installing Rust through that in your Dockerfile, do make sure to use the --profile-minimal CLI option like so:

RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- --default-toolchain stable --profile minimal --target x86_64-unknown-linux-gnu -y

This way or running curl (namely --tlsv1.2) also saved me headaches on systems where SSL/TLS system libraries weren't properly updated.

--default-toolchain stable and --target x86_64-unknown-linux-gnu are optional but I always prefer to be explicit. YMMV.

Details here: https://rust-lang.github.io/rustup/concepts/profiles.html


Bonus points: the Debian/Ubuntu packages that made sure we can use Rustler fine are: ca-certificates curl build-essential pkg-config openssl libssl-dev

@josevalim
Copy link
Contributor Author

Thank you for sharing! As per this issue, we just shipped precompilation for html5ever and we are contributing the mechanisms to Rustler soon, so we should be seeing more precompiled packages from now on.

@dimitarvp
Copy link

Really curious about rustler contributions from the Elixir core team, will keep an eye out. Thanks for letting us know! ❤️

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
chore Technical changes wontfix This will not be worked on
Projects
None yet
Development

No branches or pull requests

6 participants