Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Migrate to Alpine Linux for smaller images #41

Closed
achimnol opened this issue Mar 3, 2017 · 6 comments
Closed

Migrate to Alpine Linux for smaller images #41

achimnol opened this issue Mar 3, 2017 · 6 comments
Assignees
@achimnol
Labels
enhancement

Comments

@achimnol
Copy link
Member

achimnol commented Mar 3, 2017 •
edited
Loading

My initial tests show that it's viable to migrate to Alpine Linux for our kernel images.

  • sorna-jail works well if compiled inside Alpine Linux. (Binaries compiled in Ubuntu has glibc-specific symbol references: __fprintf_chk, __vfprintf_chk)
  • The image size of a Python 3.6 base kernel (without numpy/scipy stuffs but including jail support) is 112 MB only! (previously it was ~600 MB)

The core ideas to reduce image size are:

  • Alpine Linux tiself. The base image is less than 8 MB, just containing a busybox, the package manager apk, and a few utililties such as scanelf. libc is replaced with musl transparently.
  • Remove build-only depenencies at the end of a single RUN command so that each image layer has small footprints.
  • The package manager (apk) in Alpine Linux provides a concept of "virtual" package installation space, so we can easily purge a set of packages. Also, most Alpine Linux pacakges are made to be independent with minimal cross-dependencies.

Challenges remainig:

  • Migrate CUDA kernels to base our Alpine images. I think this is theorectically possible, but there may be unexpected glibc-specific dependency in CUDA binaries.
@achimnol achimnol self-assigned this Mar 3, 2017
achimnol added a commit that referenced this issue Mar 3, 2017
@achimnol
refs #41: Initial work to make base images
c2c97fb
achimnol added a commit that referenced this issue Mar 3, 2017
@achimnol
refs #41: Rewrite python3 kernel using Alpine Linux (1.4 GB -> 610 MB)
d5105d2 
 * It is feature-parity with previous Ubuntu-based kernel, but its size
   is reduced from 1.4 GB to 610 MB.

 * To utilizie docker's build caches during development & testing,
   I used the new `docker build --squash` option so that my local
   docker daemon preserves intermediate layers for caching but
   squashes all layers into a single layer after successful builds.

   The original image is about 800 MB and the squashed one is 610 MB.
   This difference happens because the original image has intermediate
   layers which contains deleted files in later layers.
achimnol added a commit that referenced this issue Mar 3, 2017
@achimnol
refs #41: Rewrite nodejs6 kernel using Alpine Linux (568 MB -> 62 MB)
2103fd3 
 * Compiling NodeJS from scratch takes long time (due to icu and v8)...
   Maybe improved using rocker?
achimnol added a commit that referenced this issue Mar 4, 2017
@achimnol
refs #41, #1: Update Python kernel to use packaged OpenBLAS/LAPACK
f6b6630 
 * This reduces the image build time and Dockerfile complexity.
achimnol added a commit that referenced this issue Mar 4, 2017
@achimnol
refs #41, #5: Update NodeJS kernel to use compiled packages
cda936b 
 * Just noticed that pkgs.alpinelinux.org has latest nodejs package
   updates on its front page....

 * This reduce the image build time A LOT.

 * The image size is further reduced from 62 MB to 49 MB.
achimnol added a commit that referenced this issue Mar 5, 2017
@achimnol
refs #41, #11: Rewrite Julia to use new base images (1.58 GB -> 1.17 GB)
7956a7b 
 * Removed Winston, Gadfly packages as they do not work well with new Plots
   wrapper package and complicates installation due to dependencies to Cairo.
achimnol added a commit that referenced this issue Mar 5, 2017
@achimnol
refs #41, #11: Add Julia full source tarball.
0b00d61 
 * Downloading this from our build machine takes tooooo long time.
achimnol added a commit that referenced this issue Mar 5, 2017
@achimnol
refs #41, #1: Update python2 kernel using Alpine Linux (1.35 GB -> 60…
5d38d0c 
…0 MB)
achimnol added a commit that referenced this issue Mar 6, 2017
@achimnol
refs #41: Postpone migration of TensorFlow to Alpine Linux
e95f2a0 
 * JDK 8, Python 3.6-dev packages are readily available, but
   Google's Bazel seems not working on Alpine yet.

   ref) bazelbuild/bazel#1492

   Let's come back after some time here.
achimnol added a commit that referenced this issue Mar 6, 2017
@achimnol
refs #41, #22: Rewrite lua5 kernel using Alpine Linux (408 MB -> 166 MB)
d945415 
 * Also upgrade from Lua 5.2 to Lua 5.3
achimnol added a commit that referenced this issue Mar 6, 2017
@achimnol
refs #41, #4: Rewrite php7 kernel using Alpine Linux (571 MB -> 48 MB)
c2bb131
achimnol added a commit that referenced this issue Mar 6, 2017
@achimnol
refs #41, #4: Remove unused file
0be6e02
achimnol added a commit that referenced this issue Mar 6, 2017
@achimnol
refs #41, #22: Add missing deletion of .build-deps (166 MB -> 14MB)
1ce532a
achimnol added a commit that referenced this issue Mar 6, 2017
@achimnol
refs #41, #19: Rewrite git kernel using Alpine Linux (1.1 GB -> 372 MB)
00f9461 
 * Alpine's bash requires setgid/setuid for some reason.
   (Setting USER environment variable did not the trick..)

 * In the future, we need Jail policy to be statically parsed from
   image-specific configuration files instead of built into the binary.
@achimnol
Copy link
Member Author

Another requirement is to use multi-staged docker builds.
This is a perfect match for building jail binaries separately from the main kernel images.

adrysn added a commit that referenced this issue Jun 2, 2017
@adrysn
refs #41: update - base image to refer to alpine:3.6
892007a
@adrysn
Copy link
Member

adrysn commented Jun 2, 2017

base, base-python2-minimal, base-python3-minimal, c, cpp, go, java, php7, python2, python3 images based on alpine 3.6 is pushed to docker hub.

@adrysn
Copy link
Member

adrysn commented Jun 2, 2017

git, lua5 images based on alpine 3.6 is pushed to docker hub.

@achimnol
Copy link
Member Author

achimnol commented Jun 2, 2017

For python2/python3 kernels, have you squashed the final images?
If not, please add --squash option when building the image.
I see too much size increase in some new images...

@adrysn
Copy link
Member

adrysn commented Jun 2, 2017

Aha, there is an option for flattening intermediate layers. I'll re-create images with squash option.

adrysn added a commit that referenced this issue Jun 23, 2017
@adrysn
refs #41: change - add execute flag to compiled file before run
d1804f3 
For some unknown reason, compiled executable binary has no execute flage when jail is wrapped around. For bypass temporarily, just chmod the executable to 755.
adrysn added a commit that referenced this issue Jun 30, 2017
@adrysn
refs #41: add - base_run.py to base image
3c384b6
achimnol added a commit that referenced this issue Aug 24, 2017
@achimnol
refs #41: Update Alpine base images (upgraded to Python 3.6.2)
3864acc
achimnol added a commit that referenced this issue Aug 26, 2017
@achimnol
refs #41, #49, #54: Rewrite build chain to exploit multi-staged builds
a3e2256 
 * All wheel packages that take long time to build are now stored inside
   "lablup/kernel-base-pythonX-wheels" images

   - Unless you change the version/list of dependent wheels, the whole
     build process will take much much LESS time!

 * The base image is updated to use latest jail and runtime UID matching
   entrypoint.
achimnol added a commit that referenced this issue Aug 26, 2017
@achimnol
refs #41: Apply squash build
72b52c7
achimnol added a commit that referenced this issue Sep 1, 2017
@achimnol
refs #41: (WIP) Succeeded to run CUDA on Alpine Linux
73c2da7 
 * Upgraded the nvidia driver of indominus to 375.26 from 367.48

 * However, cudaGetDeviceCount() API fails which is called from Python ctypes
   with error code 35 (insufficient driver version).
achimnol added a commit that referenced this issue Sep 1, 2017
@achimnol
refs #41: Update cuda-alpine image to have C++ runtime
4295a2b
achimnol added a commit that referenced this issue Sep 4, 2017
@achimnol
refs #1, #41: Add minideb version of Python 3 (base) kernels
a3a5a23
achimnol added a commit that referenced this issue Sep 5, 2017
@achimnol
refs #41, #1, #51: Rebuild images using Debian Jessie
fae02e0 
 * CUDA (or properietary closed source binary package) is the all source of evil....
achimnol added a commit that referenced this issue Sep 5, 2017
@achimnol
refs #41: Fix missing gosu by backporting it in base Dockerfiles
8c75f39 
 * Also collect the same Python-based runner command to minimal
   Python base images.

 * Remove unused build-and-deploy.sh script and add a new build.sh
   script.
@achimnol
Copy link
Member Author

achimnol commented Sep 7, 2017

Closing: we now use Alpine Linux and Bitnami's minifieid Debian Linux as the base container images.
(minideb is larger than alpine by 20% to 50% on average, but not a few folds like before!)

@achimnol achimnol closed this as completed Sep 7, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@achimnol achimnol

Labels
enhancement
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@achimnol @adrysn