Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add initial version of centos 5 with gcc 5.2 #20

Merged
merged 40 commits into from
Mar 29, 2016
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
40 commits
Select commit Hold shift + click to select a range
dad5eb3
add initial version of Continuum Holy Build Box
msarahan Jan 14, 2016
26c92ab
replace hbb with custom GCC 5.2 build
msarahan Feb 6, 2016
918acef
add maintainer to Dockerfile
msarahan Feb 6, 2016
81d3d95
fix some cleanup code
msarahan Feb 7, 2016
53a4db1
add patch program from yum
msarahan Feb 7, 2016
a90a8f6
split up GCC build part and miniconda install
msarahan Feb 8, 2016
21f7490
document aliases and run instructions a bit
msarahan Feb 8, 2016
5879d68
add multilib support for builder image
Feb 15, 2016
ac6f3ff
fix initial output when entering builder image
msarahan Feb 15, 2016
215081f
move prefix to /usr/local; add cc link to gcc
Feb 24, 2016
d628755
add sudo support, fix bzip2 executable availability
Feb 24, 2016
f328273
work on creating non-root user at container creation, rather than ima…
msarahan Feb 24, 2016
bfd7c41
unifying work on user creation
msarahan Feb 24, 2016
e262e87
add startup text, fix private key import on Linux native systems
msarahan Feb 25, 2016
cb03294
remove binutils from centos, make our binutils be on PATH
msarahan Feb 25, 2016
4efea85
add note on startup about compatibility & print default C++ ABI
msarahan Feb 25, 2016
bb36c08
add no strict host checking for github and bremen to avoid prompt
msarahan Mar 3, 2016
c94c806
update gmp->6.1.0; separate abi 4 and 5 gcc installs
msarahan Mar 4, 2016
29da18d
Merge branch 'add_hbb_64' of github.com:msarahan/docker-images into a…
msarahan Mar 4, 2016
a2fa7df
pass through docker variables in start script; add scripts for differ…
msarahan Mar 4, 2016
91c6a6c
Merge branch 'add_hbb_64' of github.com:msarahan/docker-images into a…
msarahan Mar 4, 2016
ba6bc5a
clean up some text output on container startup
msarahan Mar 4, 2016
e3fd421
break gcc builder image into scripts
msarahan Mar 14, 2016
761db7c
rename docker-build to conda_builder_linux. restructure build folder.
msarahan Mar 14, 2016
bbea4e8
fix conversion to .sh in centos5_gcc52_base Dockerfile
msarahan Mar 14, 2016
b77c875
correctly pass command from start scripts into containers
msarahan Mar 14, 2016
e1d4e68
update conda_builder_linux readme
msarahan Mar 15, 2016
3733ef0
update conda_builder_linux readme (formatting)
msarahan Mar 15, 2016
f1ac3c3
flesh out centos5_gcc52_base README
msarahan Mar 15, 2016
13f2b14
space-saving measures in base image. add lto support.
msarahan Mar 21, 2016
e7a74a8
document start script examples; set up CXXFLAGS for ABI selection
msarahan Mar 21, 2016
0dfa776
change GCC base image usage docs
msarahan Mar 22, 2016
b93384a
gcc image: clean up kernel, change x86 target naming
msarahan Mar 28, 2016
7641713
filter uname output to trick build scripts into using the multilib co…
msarahan Mar 28, 2016
12e09f1
switch over to use images hosted at continuumio
msarahan Mar 28, 2016
0aaad30
add 32-bit run scripts, adjust readme
msarahan Mar 28, 2016
bf60334
fix broken anaconda setup
msarahan Mar 28, 2016
9d5d83b
update readme for dockerhub location; add yum install for Qt build
msarahan Mar 28, 2016
ab5ebbd
move libstc++ symlinking to centos base
msarahan Mar 29, 2016
b899084
fix USER passthrough. remove less packages so qt builds. Fix libstdc…
msarahan Mar 29, 2016
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions centos5_gcc5_base/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
FROM centos:5.11
MAINTAINER Michael Sarahan <msarahan@continuum.io>

WORKDIR /build_scripts
COPY install_yum_deps.sh build_gcc.sh yum_cleanup.sh /build_scripts/
RUN bash install_yum_deps.sh && \
bash build_gcc.sh && \
bash yum_cleanup.sh && \
rm -rf /build_scripts
# replace ancient system libstdc++, plug ours in.
# This is a workaround for some recipes that do not respect LIBRARY_PATH.
RUN ln -sf /usr/local/lib64/libstdc++.so.6.0.21 /usr/lib64/libstdc++.so.6
RUN ln -sf /usr/local/lib/libstdc++.so.6.0.21 /usr/lib/libstdc++.so.6
RUN ln -sf /usr/local/lib64/libgcc_s.so.1 /usr/lib64/libgcc_s.so.1
RUN ln -sf /usr/local/lib/libgcc_s.so.1 /usr/lib/libgcc_s.so.1
14 changes: 14 additions & 0 deletions centos5_gcc5_base/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
# centos5_gcc52_base

Docker image (CentOS 5.11) with GCC 5.2, Binutils 2.26.

Libraries built with this docker container should be compatible with most Linux
distributions. Additionally, GCC5.2 allows compilation of C++11 and C++14 code.

This image is extremely limited. It is meant as a base layer for other tools
that utilize the compiler.

GCC in this image is compiled to output code with the C++11 ABI. If you want
the C++98 (GCC4) ABI, you need to set ```CXXFLAGS="${CXXFLAGS} -D_GLIBCXX_USE_CXX11_ABI=0"```

Docker Hub location: https://hub.docker.com/r/continuumio/centos5_gcc5_base/
69 changes: 69 additions & 0 deletions centos5_gcc5_base/build_gcc.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,69 @@
# Stop at any error, show all commands
set -ex

GCC_VER=5.2.0
GMP_VER=6.1.0
MPC_VER=1.0.3
MPFR_VER=3.1.4
ISL_VER=0.14

# build gcc 5.2.0 (http://en.librehat.com/blog/build-gcc-5-dot-2-on-rhel-6/)
urls="http://mirrors-usa.go-parts.com/gcc/releases/gcc-${GCC_VER}/gcc-${GCC_VER}.tar.bz2 \
http://gnu.askapache.com/gmp/gmp-${GMP_VER}.tar.bz2 \
ftp://ftp.gnu.org/gnu/mpc/mpc-${MPC_VER}.tar.gz \
http://www.mpfr.org/mpfr-current/mpfr-${MPFR_VER}.tar.bz2 \
ftp://gcc.gnu.org/pub/gcc/infrastructure/isl-${ISL_VER}.tar.bz2"
for url in $urls; do
curl $url -LO ;
done
wait
mkdir /gcc-build && tar xjf gcc-${GCC_VER}.tar.bz2 -C /gcc-build --strip-components=1
mkdir /gcc-build/gmp && tar xjf gmp-${GMP_VER}.tar.bz2 -C /gcc-build/gmp --strip-components=1
mkdir /gcc-build/mpc && tar xf mpc-${MPC_VER}.tar.gz -C /gcc-build/mpc --strip-components=1
mkdir /gcc-build/mpfr && tar xjf mpfr-${MPFR_VER}.tar.bz2 -C /gcc-build/mpfr --strip-components=1
mkdir /gcc-build/isl && tar xjf isl-${ISL_VER}.tar.bz2 -C /gcc-build/isl --strip-components=1
rm -rf *tar\.*
mkdir /gcc-build/work
cd /gcc-build/work
curl -LO http://ftp.gnu.org/gnu/binutils/binutils-2.26.tar.bz2
tar xf binutils-2.26.tar.bz2
cd /gcc-build/work/binutils-2.26
scl enable devtoolset-2 './configure --prefix=/usr/local \
--enable-plugin \
--with-sysroot=/ \
--enable-targets=x86_64-redhat-linux-gnu,i686-redhat-linux-gnu \
&& make -j$(getconf _NPROCESSORS_ONLN) && make install'
cd /gcc-build/work

scl enable devtoolset-2 '/gcc-build/configure \
--build x86_64-redhat-linux-gnu \
--enable-__cxa_atexit \
--enable-bootstrap \
--enable-checking=release \
--enable-clocale=gnu \
--enable-languages=c,c++,fortran,lto \
--enable-libstdcxx-threads \
--enable-libstdcxx-time \
--enable-linker-build-id \
--enable-multilib \
--enable-plugin \
--enable-shared \
--enable-threads=posix \
--disable-gnu-unique-object \
--disable-libada \
--disable-libgcj \
--disable-libquadmath-support \
--disable-libstdcxx-pch \
--disable-libunwind-exceptions \
--host x86_64-redhat-linux-gnu \
--prefix=/usr/local \
--target=x86_64-redhat-linux-gnu \
--with-ld=/usr/local/bin/ld \
--with-linker-hash-style=gnu \
--with-tune=generic'
scl enable devtoolset-2 'make -j$(getconf _NPROCESSORS_ONLN) bootstrap-lean'
scl enable devtoolset-2 'make install-strip'
ln -sf /usr/local/bin/gcc /usr/local/bin/cc
rm -rf /gcc-build
rm -rf /usr/local/gcc_tmp

5 changes: 5 additions & 0 deletions centos5_gcc5_base/install_yum_deps.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
yum install -y curl.x86_64 bzip2.x86_64 yum-utils glibc-devel patch \
unzip bison yasm file make libtool.x86_64 pkgconfig.x86_64
curl http://linuxsoft.cern.ch/cern/devtoolset/slc5-devtoolset.repo -o /etc/yum.repos.d/slc5-devtoolset.repo
rpm --import http://ftp.mirrorservice.org/sites/ftp.scientificlinux.org/linux/scientific/51/i386/RPM-GPG-KEYs/RPM-GPG-KEY-cern
yum install -y centos-release-SCL devtoolset-2
10 changes: 10 additions & 0 deletions centos5_gcc5_base/yum_cleanup.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
yum remove -y devtoolset-2* curl binutils kernel
rm -rf /opt/rh
rm /etc/yum.repos.d/slc5-devtoolset.repo
yum -y remove wireless-tools gtk2 libX11 hicolor-icon-theme \
avahi freetype bitstream-vera-fonts > /dev/null 2>&1
yum clean all

# print out all installed packages, and then again by size
yum list installed
rpm -qa --queryformat '%10{size} - %-25{name} \t %{version}\n' | sort -n
31 changes: 31 additions & 0 deletions conda_builder_linux/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
from continuumio/centos5_gcc5_base:5.11-5.2-0
MAINTAINER Michael Sarahan <msarahan@continuum.io>

WORKDIR /build_scripts
COPY build/yum_install_syslibs.sh \
build/install_miniconda.sh \
build/yum_cleanup.sh \
/build_scripts/

RUN bash yum_install_syslibs.sh && \
bash install_miniconda.sh && \
bash yum_cleanup.sh && \
rm -rf /build_scripts

# Yum may have messed up symlinks to libstdc++. Make sure they are still ours.
RUN ln -sf /usr/local/lib64/libstdc++.so.6.0.21 /usr/lib64/libstdc++.so.6
RUN ln -sf /usr/local/lib/libstdc++.so.6.0.21 /usr/lib/libstdc++.so.6
RUN ln -sf /usr/local/lib64/libgcc_s.so.1 /usr/lib64/libgcc_s.so.1
RUN ln -sf /usr/local/lib/libgcc_s.so.1 /usr/lib/libgcc_s.so.1

RUN useradd -m --uid 1000 -G wheel dev
RUN echo '%wheel ALL=(ALL) NOPASSWD:ALL' >> /etc/sudoers

# this is where we'll mount shares from the user
RUN mkdir -p /opt/share
ADD build/internal_startup.sh /opt/share/internal_startup.sh
ADD build/alias_32bit.sh /opt/share/alias_32bit.sh
RUN chmod -R 777 /opt

WORKDIR /home/dev
USER dev
119 changes: 119 additions & 0 deletions conda_builder_linux/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,119 @@
# conda_builder_linux

Docker container (CentOS 5.11) with GCC 5.2, Binutils 2.26, and a bootstrapped
[miniconda](http://conda.pydata.org/miniconda.html) installed and ready to use.

Libraries built with this docker container should be compatible with most Linux
distributions. Additionally, GCC5.2 allows compilation of C++11 and C++14 code.

Usage
-----

A helper script for running is provided that mounts your .ssh folder (for
passwordless SSH) and ~/.gitconfig file in the container. To run this, download
the docker_wrapper.sh script and either start_cpp*.sh script, and run the start
script from your Docker terminal. On Windws and Mac with Docker Toolbox, this is
your Docker Quickstart terminal.

Once that starts, it will display some system information:

```
Welcome to the conda-builder image, brought to you by Continuum Analytics.

Binaries produced with this image should be compatible with any Linux OS that is
at least CentOS 5 or newer (Glibc lower bound), and anything that uses G++ 5.2
or older (libstdc++ upper bound)

GCC is: gcc (GCC) 5.2.0
Default C++ ABI: 4 (C++98)
GLIBC is: glibc 2.5
Native arch is x64. To build for 32-bit, set CONDA_FORCE_32BIT=1,
or outside of conda-build, CFLAGS="-m32"
ld/binutils is: GNU ld (GNU Binutils) 2.26.20160125

The dev user (currently signed in) has passwordless sudo access.
miniconda (2.7) is installed at /opt/miniconda.
git is also available.
Your .gitconfig has been imported.
Your ssh private key has been imported for passwordless ssh.

Helpful aliases:
clone_recipes: clones the conda/conda-recipes repo from Github
clone_anaconda: clones the continuumIO/anaconda (private) repo from Github
anaconda_setup: clones anaconda repo and sets up continuum internal build
system.
```

The docker image is made to be used either interactively, or by passing
commands, so that the image is effectively just a run environment. Any arguments
you pass in are parsed first to pull out ```docker run``` arguments, then any
unrecognized parts are fed into the container and executed using ```eval```. If
you did not supply any command for the start script to run directly, you'll be
dropped at an interactive prompt.

Interactive use
===============

Starting the image in interactive mode looks like:

start_cpp98.sh

For mounting volumes or setting environment variables, you can pass the -v or -e arguments:

# mount your local recipes folder in the container, avoid a clone in the container.
start_cpp98.sh -v ~/code/my_recipes:/home/dev/recipes

# build in 32-bit (compiler is 64-bit native, but with multilib can build 32-bit packages)
start_cpp98_32.sh

The interactive prompt looks like:

[dev@da2f4a3e941b ~]$

Here you have quick access to git, conda, and conda-build, all installed and on
PATH. Additionally, a few aliases are set up by default:

- **clone_recipes**: clone the conda recipes repository from
https://github.com/conda/conda-recipes
- **clone_anaconda**: clone the conda recipes repository from
https://github.com/continuumIO/anaconda (This is a Continuum private repo.
Checking this out requires your ssh keys to be set up properly.)
- **anaconda_setup**: clone the conda recipes repository from
https://github.com/continuumIO/anaconda and run ```python setup.py
develop``` to install the build system's commands

Scripted use
============

Starting the image to run a command looks like:

start_cpp98.sh my_build_script.sh && echo "weeee"

Again, you can pass flags to the docker run command through the start_*.sh script:

# mount your local recipes folder in the container, avoid a clone in the container.
start_cpp98.sh -v ~/code/my_recipes:/home/dev/recipes my_build_script.sh

# build in 32-bit (compiler is 64-bit native, but with multilib can build 32-bit packages)
start_cpp98_32.sh my_build_script.sh

Docker Hub location
===================

https://hub.docker.com/r/continuumio/conda_builder_linux/

NOTES
-----

There are two installations of GCC in /usr/local. These have been compiled
each with a different C++ ABI setting (GCC4 compatible, and GCC5 compatible.)

- Neither of these are on PATH by default - the startup scripts add one or
the other.
- Libraries compiled with one are generally incompatible with consumers
compiled with another.
- Continuum is using GCC4 compatibility mode for now, with a switch
anticipated at some point. This will be well-announced.
- Your choice here does NOT limit where you software will run: if you
compile everything yourself with only one ABI, you should be fine on any
platform. It is only when things are mixed that they break.
7 changes: 7 additions & 0 deletions conda_builder_linux/build/install_miniconda.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
curl http://repo.continuum.io/miniconda/Miniconda2-latest-Linux-x86_64.sh -o Miniconda.sh
/bin/bash Miniconda.sh -b -p /opt/miniconda
rm Miniconda.sh
/opt/miniconda/bin/conda config --set show_channel_urls True
/opt/miniconda/bin/conda update --yes --all
/opt/miniconda/bin/conda install --yes git conda-build curl anaconda-client
/opt/miniconda/bin/conda clean --tarballs --packages
93 changes: 93 additions & 0 deletions conda_builder_linux/build/internal_startup.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,93 @@
#!/bin/bash

if [[ -z "${ABI}" ]]; then
echo "WARNING: No ABI default set. Falling back to compatibility mode with GCC 4."
export ABI=4
fi

# Setup home environment

export PATH=/usr/local/bin:/opt/miniconda/bin:$PATH
export LD_LIBRARY_PATH=/usr/local/lib64:/usr/local/lib:/opt/miniconda/lib:$LD_LIBRARY_PATH
export LIBRARY_PATH=/usr/local/lib64:/usr/local/lib:/opt/miniconda/lib:$LIBRARY_PATH
export INCLUDE=/opt/miniconda/include:$INCLUDE
export CXXFLAGS="${CXXFLAGS} -Wabi=2"

if [ $ABI -lt 5 ]; then
export CXXFLAGS="${CXXFLAGS} -D_GLIBCXX_USE_CXX11_ABI=0"
else
export CXXFLAGS="${CXXFLAGS} -D_GLIBCXX_USE_CXX11_ABI=1"
fi

echo "alias clone_recipes='git clone https://github.com/conda/conda-recipes'" >> ~/.bashrc
# Continuum internal build system (private repo, requires on-site or VPN, may require Docker VM restart if network settings change)
echo "alias clone_anaconda='git clone git@github.com:continuumIO/anaconda'" >> ~/.bashrc
echo "alias anaconda_setup='clone_anaconda && cd anaconda && python setup.py develop && cd .. && mkdir aroot'" >> ~/.bashrc

if [[ "$ARCH" -eq "64" || -z "$ARCH" ]]; then
ARCH_DOC=$'To build 32-bit code, set the ARCH environment
variable to 32. (-e \"ARCH=32\" docker argument)'
else
ARCH_DOC=$'Pretending to be i686 (using multilib gcc) and
filtering uname to output i686.'
export CFLAGS="${CFLAGS} -m32"
export CXXFLAGS="${CXXFLAGS} -m32"
export CONDA_FORCE_32BIT=1
# this will get picked up by subshells
sudo mv /bin/uname /bin/uname_x64
sudo ln -s /opt/share/alias_32bit.sh /bin/uname
sudo chmod +x /bin/uname
fi

echo
echo "Welcome to the conda-builder image, brought to you by Continuum Analytics."
echo
echo "Binaries produced with this image should be compatible with any Linux OS"
echo "that is at least CentOS 5 or newer (Glibc lower bound), and anything "
echo "that uses G++ 5.2 or older (libstdc++ upper bound)"
echo
echo " GCC is: $(gcc --version | head -1)"
echo " Default C++ ABI: ${ABI} (C++$([ "${ABI}" == "4" ] && echo "98" || echo "11"))"
echo " GLIBC is: $(getconf GNU_LIBC_VERSION)"
echo " ld/binutils is: $(ld --version | head -1)"
echo
echo " Native arch is x86_64. ${ARCH_DOC}"
echo
echo " The dev user (currently signed in) has passwordless sudo access."
echo " miniconda (2.7) is installed at /opt/miniconda."
echo " git is also available."

if [ -f "/home/dev/.gitconfig" ]; then
echo " Your .gitconfig has been imported."
fi

# jumping through hoops for file ownership - we can't have the owner be
# the native linux owner, and we also can't have permissions too wide open,
# or ssh complains.
if [ -f /id_rsa ]; then
mkdir -p .ssh
sudo cp /id_rsa .ssh/
chmod 700 .ssh
sudo chown dev: .ssh/id_rsa
sudo chmod 600 .ssh/id_rsa
echo -e "Host github.com\n\tStrictHostKeyChecking no\n" >> ~/.ssh/config
echo -e "Host bremen\n\tStrictHostKeyChecking no\n" >> ~/.ssh/config
echo " Your ssh private key has been imported for passwordless ssh."
fi

echo

echo "Helpful aliases:"
echo " clone_recipes: clones the conda/conda-recipes repo from Github"
echo " clone_anaconda: clones the continuumIO/anaconda (private) repo from Github"
echo " anaconda_setup: clones anaconda repo and sets up continuum internal build system."

echo

if [[ $# < 1 ]]; then
# interactive session
bash
else
# Run whatever the user wants to pass in
eval "$@"
fi
6 changes: 6 additions & 0 deletions conda_builder_linux/build/yum_cleanup.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
yum -y erase wireless-tools avahi curl > /dev/null 2>&1
yum -y clean all

# print out all installed packages, and then again by size
yum list installed
rpm -qa --queryformat '%10{size} - %-25{name} \t %{version}\n' | sort -n
4 changes: 4 additions & 0 deletions conda_builder_linux/build/yum_install_syslibs.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
yum install -y glib2-devel libX11-devel libXext-devel libXrender-devel \
mesa-libGL-devel libICE-devel libSM-devel ncurses-devel \
openssh-clients.x86_64 patch.x86_64 dbus-devel gtk2-devel \
chrpath.x86_64 nano bzip2 xz sudo curl
Loading