Unable to build using Docker on SemaphoreCI

[ariya]

The Linux static binary can be built using Docker (#13822). As reported by @Vanuan, in some cases it doesn't work.

[ariya]

@Vanuan Please provide some more details to reproduce your problem.

[Vanuan]

Ok. So, let's start from the original issue.

icu and openssl (#12948)

In Qt5 there were new dependencies introduced: icu and openssl libraries.
We wanted to have a single Phantomjs binary for all linux distributions, and different distributions have different versions of icu and openssl, so there's no other way than to compile those libraries into phantomjs to do that.
We successfully found a way to link phantomjs with static versions of icu and openssl. The trick is to remove -dev packages of those, install static version using source packages and add some build.py options.

glibc (GNU C Library) (#13822)

One problem still remains. If we build phantomjs on systems where newer version of glibc is installed, we can't use this binary on systems where older version of glibc is installed. It seems that there's no other solution than to build phantomjs on older systems. (however, there is another option I'm currently exploring: http://stackoverflow.com/questions/2856438/how-can-i-link-to-a-specific-glibc-version)

SemaphoreCI

SemaphoreCI provides only Ubuntu 14.04 as a build machine. It has glibc 2.19 installed.
So when we link to that glibc version we can't run that binary in CentOS 6 (which has glibc 2.13).
Using Docker with debian:wheezy was proposed as a solution (glibc 2.13).
SemaphoreCI does provide Ubuntu 14.04 with Docker preinstalled. But either building inside docker is slower, or SemaphoreCI limits performance on machines with docker, the problem is that total time needed to build phantomjs inside docker is more than an hour. SemaphoreCI limits the build time to one hour.

[ariya]

The trick is to remove -dev packages of those, install static version using source packages and add some build.py options.

This is not exactly what I experienced. I discovered that there is no need to remove the -dev packages at all. Those development libraries are simply ignored if PhantomJS build script is correctly told to use the static ICU and OpenSSL instead.

[ariya]

It seems that there's no other solution than to build phantomjs on older systems.

Related to this is what I found out in #13822 (comment). If PhantomJS binary was built on Ubuntu 14.04, it will not run on CentOS 6 and even Debian wheezy. Hence, the build needs to happen on wheezy so that it's usable in any more modern system.

[Vanuan]

This is not exactly what I experienced

I used your script in Ubuntu 14.04 and it worked only after I removed *-dev packages.

[Vanuan]

If PhantomJS binary was built on Ubuntu 14.04, it will not run on CentOS 6 and even Debian wheezy

Exactly, and I explained why (different versions of glibc)

[Vanuan]

Maybe ariya/erik image doesn't have *-dev packages.

[ariya]

SemaphoreCI provides only Ubuntu 14.04 as a build machine

This is the problem. That docker-build.sh is designed for wheezy, see the README:

docker run -v $PWD:/src debian:wheezy /src/deploy/docker-build.sh

[ariya]

Maybe ariya/erik image doesn't have *-dev packages.

That image is obsoleted, it's not necessary anymore since it's better to use the official wheezy image.

[Vanuan]

That image is obsoleted, it's not necessary anymore since it's better to use the official wheezy image.

Base wheezy image doesn't have *-dev packages neither, so there's nothing to remove.
I'm pretty sure if you install *-dev packages, you'll hit the problem with icu and openssl

[Vanuan]

This is the problem. That docker-build.sh is designed for wheezy, see the README:

I understand that. I even know why we need it (glibc problem).

[ariya]

I understand that. I even know why we need it (glibc problem).

So why don't you try running that docker command (probably need sudo) rather than executing docker-build.sh directly (which is bound to fail)?

[Vanuan]

So why don't you try running that docker command

If you read my message carefully, you'll understand I've tried that:

SemaphoreCI does provide Ubuntu 14.04 with Docker preinstalled. But either building inside docker is slower, or SemaphoreCI limits performance on machines with docker, the problem is that total time needed to build phantomjs inside docker is more than an hour. SemaphoreCI limits the build time to one hour.

[Vanuan]

Here's an output of lscpu on SemaphoreCI machine with docker:

Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                1
On-line CPU(s) list:   0
Thread(s) per core:    1
Core(s) per socket:    1
Socket(s):             1
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 42
Stepping:              1
CPU MHz:               3491.914
BogoMIPS:              6983.82
Virtualization:        VT-x
Hypervisor vendor:     KVM
Virtualization type:   full
L1d cache:             32K
L1i cache:             32K
L2 cache:              4096K
NUMA node0 CPU(s):     0

[ariya]

If you read my message carefully, you'll understand I've tried that:

I understand that. I assume you've done that with the chosen platform "Ubuntu 14.04 LTS v1512 (beta with Docker support)". It is the usual running-docker-inside-docker. It can work, but this is definitely bound to whatever resource limitation (memory, CPU, etc) on the "outer" container.

However, what I proposed was running it with the usual platform "Ubuntu 14.04 LTS v1512" (with 8-core CPU) where you install Docker yourself (apt-get and all that stuff). Note that this can only work if SemaphoreCI build machine permits installing Docker yourself.

[Vanuan]

Ok, trying to install docker on "Ubuntu 14.04 LTS v1512"

[ariya]

CPU(s): 1

No wonder the build is going really slow.

Comparing the memory (cat /proc/meminfo) also gives the following. 4 GB is definitely still sufficient for the final linking. However it confirms that resource restrictions (CPU, memory) are applied to each running container.
Inside the container:

MemTotal:        4048428 kB
MemFree:         1303584 kB
Buffers:          430488 kB

On the build machine (outside the container):

MemTotal:       32808080 kB
MemFree:          573080 kB
Buffers:         6260296 kB

I don't think there's anything from PhantomJS side that we can do something about. You probably want to bring up with the SemaphoreCI team, e.g. allowing a container to use more CPU cores.

[Vanuan]

It looks like Docker doesn't support that kernel version/configuration:
https://semaphoreci.com/vanuan/phantomjs/branches/master/builds/72

ERRO[0000] 'overlay' not found as a supported filesystem on this host. Please ensure kernel is new enough and has overlay support loaded. 
WARN[0000] Running modprobe bridge br_netfilter failed with message: modprobe: WARNING: Module br_netfilter not found.
, error: exit status 1 
INFO[0000] Firewalld running: false                     
INFO[0000] Default bridge (docker0) is assigned with an IP address 172.17.0.1/16. Daemon option --bip can be used to set a preferred IP address 
WARN[0000] Your kernel does not support cgroup memory limit: mountpoint for memory not found 
WARN[0000] mountpoint for cpu not found                 
WARN[0000] mountpoint for blkio not found               
WARN[0000] mountpoint for cpuset not found              
FATA[0000] Error starting daemon: Devices cgroup isn't mounted 

http://stackoverflow.com/questions/32002882/error-starting-docker-daemon-on-ubuntu-14-04-devices-cgroup-isnt-mounted

[Vanuan]

Maybe I can create virtualbox inside semaphore's virtual environment? Not sure if it will help running time.

[ariya]

• Is there anything remaining that we can do on PhantomJS side? It seems that it is now left to SemaphoreCI to either support the setup or not.
• @Vanuan May I know what was your original intention to try this setup (static Linux on SemaphoreCI)?

[Vanuan]

@ariya I need phantomjs binary on CentOS 6. That centos machine doesn't have enough power to build phantomjs there. And I don't have enough resources on my local environment too.

[ariya]

@Vanuan In that case, the easiest path is to use VirtualBox on your beefy laptop/destop, install CentOS 6, build it there, and extract the binary.

[Vanuan]

Response from Semaphore team:

Yes, sadly there are some restrictions in place which prevent installing Docker in the regular platform. I will let you know if there's a possibility for increasing the core count in the Docker enabled platform.

[ariya]

This issue will be closed since we can't follow it up any further. It is a limitation on SemaphoreCI.

Note that we still plan to use it for checking pull request. SemaphoreCI is perfectly capable of running a regular build, i.e. not inside Docker (#13850).