Do not create squashed tar in-memory, store it on disk

[goldmann]

When the build is executed on a host that has small amount of RAM and/or the image we squash is big enough - sometimes the tool can eat whole memory and fail with a MemoryError.

[hjc]

Can confirm that I would love this feature. We have a lot of OSX devs, so they're forced to use Vagrant to do builds. We give our VBox machines a nice 2GB of memory, but even with that much and all services stopped, I still get out of memory errors when run from a virtual machine.

Is there any advice you could give me for trying to tackle this? If so, I might try it myself since I'm equipped to reproduce the bug.

[goldmann]

In fact this is now implemented as part of this commit: cd2198d But there is one problem with this - we do read the whole tar into memory either case to send it later to the docker client. If this could be improved - I would close this issue.

And yes - feel free to take a look at it - we're talking about this line: https://github.com/goldmann/docker-squash/blob/1.0.0rc3/docker_squash/image.py#L269

[hjc]

Awesome! I'll try to play with that a bit this weekend!

[goldmann]

I think I nailed it, see #79 :)

[hjc]

Hah! So simple!

I was gonna break into it tonight, but it looks like that could do it. I'll spool up my Vagrant tomorrow, run the script and let you know.

Thanks!

[goldmann]

Cool, let me know. My tests say it works great ;)

[hjc]

=/. What are your tests? Because mine aren't passing =/.

Here's my log of running everything:

vagrant@dev:/opt/dev$ free -h
             total       used       free     shared    buffers     cached
Mem:          2.0G       941M       1.0G       1.4M       4.5M        48M
-/+ buffers/cache:       888M       1.1G
Swap:           0B         0B         0B
vagrant@dev:/opt/dev$ sudo pip install -U https://github.com/goldmann/docker-squash/archive/master.zip
Collecting https://github.com/goldmann/docker-squash/archive/master.zip
  Downloading https://github.com/goldmann/docker-squash/archive/master.zip
Requirement already up-to-date: docker-py in /usr/local/lib/python2.7/dist-packages (from docker-squash==1.0.0rc3)
Requirement already up-to-date: six in /usr/local/lib/python2.7/dist-packages (from docker-squash==1.0.0rc3)
Requirement already up-to-date: requests>=2.5.2 in /usr/local/lib/python2.7/dist-packages (from docker-py->docker-squash==1.0.0rc3)
Requirement already up-to-date: websocket-client>=0.32.0 in /usr/local/lib/python2.7/dist-packages (from docker-py->docker-squash==1.0.0rc3)
Requirement already up-to-date: backports.ssl-match-hostname in /usr/local/lib/python2.7/dist-packages (from websocket-client>=0.32.0->docker-py->docker-squash==1.0.0rc3)
Installing collected packages: docker-squash
  Found existing installation: docker-squash 1.0.0rc3
    Uninstalling docker-squash-1.0.0rc3:
      Successfully uninstalled docker-squash-1.0.0rc3
  Running setup.py install for docker-squash ... done
Successfully installed docker-squash-1.0.0rc3
vagrant@dev:/opt/dev$ docker-squash  --version
1.0.0rc3
vagrant@dev:/opt/dev$ docker-squash -f stackbrew/ubuntu:trusty -t squashed:dev 262cd4a7eb36
2016-04-06 02:08:04,111 root         INFO     docker-squash version 1.0.0rc3, Docker 20f81dd, API 1.22...
2016-04-06 02:08:04,113 root         INFO     Using v2 image format
2016-04-06 02:08:04,211 root         INFO     Old image has 30 layers
2016-04-06 02:08:04,215 root         INFO     Checking if squashing is necessary...
2016-04-06 02:08:04,216 root         INFO     Attempting to squash last 26 layers...
2016-04-06 02:08:04,216 root         INFO     Saving image sha256:262cd4a7eb36b3b39e0809c159dfe1ad61dabb84641197b5e33627b10bdaa09f to /tmp/docker-squash-rAyYdC/old/image.tar file...
Killed
vagrant@dev:/opt/dev$ sudo tail -n10 /var/log/syslog
Apr  6 02:13:17 dev kernel: [384377.461478] [23894] 65534 23894    14172     2697      31        0             0 gunicorn
Apr  6 02:13:17 dev kernel: [384377.461480] [23903] 65534 23903    46653     8533      57        0             0 gunicorn
Apr  6 02:13:17 dev kernel: [384377.461482] [23974] 65534 23974     1110       25       8        0             0 entrypoint_work
Apr  6 02:13:17 dev kernel: [384377.461484] [23981] 65534 23981    28942     9538      59        0             0 celery
Apr  6 02:13:17 dev kernel: [384377.461487] [24031] 65534 24031    49851    10389      68        0             0 celery
Apr  6 02:13:17 dev kernel: [384377.461489] [24930]  1000 24930   296771   270470     557        0             0 docker-squash
Apr  6 02:13:17 dev kernel: [384377.461491] [25183]     0 25183   105556     7381      92        0             0 salt-minion
Apr  6 02:13:17 dev kernel: [384377.461493] Out of memory: Kill process 24930 (docker-squash) score 528 or sacrifice child
Apr  6 02:13:17 dev kernel: [384377.462055] Killed process 24930 (docker-squash) total-vm:1187084kB, anon-rss:1081880kB, file-rss:0kB
Apr  6 02:17:02 dev CRON[25333]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)

Info on the Docker image:

vagrant@dev:/opt/dev$ docker images | grep 262cd
<none>                                 <none>              262cd4a7eb36        6 days ago          2.726 GB
vagrant@dev:/opt/dev$ docker history 262cd
IMAGE               CREATED             CREATED BY                                      SIZE                COMMENT
262cd4a7eb36        6 days ago          /bin/sh -c #(nop) USER [nobody]                 0 B                 
fa4d3f9f8f4f        6 days ago          /bin/sh -c rm /var/log/*log /var/log/apt/* /v   0 B                 
3aea7b33798a        6 days ago          /bin/sh -c rm -rf /opt/dev/no   0 B                 
81a1259ec30c        6 days ago          /bin/sh -c chmod +x ./node_modules/.bin/gulp    7.82 MB             
a5e2b96d7efe        6 days ago          /bin/sh -c npm install                          38.4 kB             
e38aeab68111        6 days ago          /bin/sh -c #(nop) WORKDIR /opt/dev   0 B                 
14cd96fe729f        6 days ago          /bin/sh -c #(nop) ADD dir:bf7fc3f7d378dccd579   1.038 GB            
cf93f93934f4        6 days ago          /bin/sh -c pip install -r /requirements.txt     143.8 MB            
c3a6fc2c35a6        6 days ago          /bin/sh -c #(nop) ADD file:44e5689a7f908f5ce6   681 B               
8cbe36dda0e9        6 days ago          /bin/sh -c pip install gunicorn ipython ipdb    8.506 MB            
713785a98b1b        6 days ago          /bin/sh -c apt-get install -y --fix-missing p   74.9 MB             
f8b57e4a4499        6 days ago          /bin/sh -c apt-get remove -y curl apt-transpo   764.9 kB            
14f99c885899        6 days ago          /bin/sh -c mv /root/.cabal/bin/pandoc /usr/bi   57.03 MB            
1bcf787813ef        6 days ago          /bin/sh -c cabal install alex happy pandoc --   452.3 MB            
deb9689bfc56        6 days ago          /bin/sh -c cabal update                         247.6 MB            
feaa16cebd1e        6 days ago          /bin/sh -c apt-get install -y build-essential   472 MB              
e7061a33608c        2 weeks ago         /bin/sh -c apt-get update                       21.88 MB            
db803be12735        2 weeks ago         /bin/sh -c curl -s https://deb.nodesource.com   26.88 kB            
cdfd8ab9494d        2 weeks ago         /bin/sh -c apt-get install -y curl apt-transp   11.86 MB            
44913e268815        2 weeks ago         /bin/sh -c echo 'deb-src https://deb.nodesour   108 B               
c503f5b98bba        2 weeks ago         /bin/sh -c echo 'deb https://deb.nodesource.c   52 B                
1b5e34352872        2 weeks ago         /bin/sh -c echo "deb http://us-east-1.ec2.arc   78 B                
df2153f193dc        2 weeks ago         /bin/sh -c echo "deb http://us-east-1.ec2.arc   70 B                
6fa1f863beca        2 weeks ago         /bin/sh -c #(nop) ENV DEBIAN_FRONTEND=noninte   0 B                 
bd2ed2500328        2 weeks ago         /bin/sh -c locale-gen --purge en_US.UTF-8       1.621 MB            
13f32d767c72        2 weeks ago         /bin/sh -c echo -e 'LANG="en_US.UTF-8"\nLANGU   43 B                
3876b81b5a81        11 weeks ago        /bin/sh -c #(nop) CMD ["/bin/bash"]             0 B                 
<missing>           11 weeks ago        /bin/sh -c sed -i 's/^#\s*\(deb.*universe\)$/   1.895 kB            
<missing>           11 weeks ago        /bin/sh -c echo '#!/bin/sh' > /usr/sbin/polic   194.5 kB            
<missing>           11 weeks ago        /bin/sh -c #(nop) ADD file:7ce20ce3daa6af21db   187.7 MB 
vagrant@dev:/opt/dev$ docker --version
Docker version 1.10.3, build 20f81dd

What did I mess up @goldmann? Or were you not expecting this to work with only 1GB of memory? If it's the latter, I'll probably still poke around sending the Docker daemon the tar in chunks.

[goldmann]

Well, this is expected :) Please note the title of the issue: "Do not create squashed tar in-memory, store it on disk". This issue is about creating the final, squashed image on the disk and then upload it to Docker daemon. Your failure is in a different place - when the squash tool fetches the tar image from the daemon. This isn't solved yet, but I do plan to fix it too.

I created #81 to track this.