Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

MemoryError when squashing large images #132

Closed
johnccchang opened this issue Nov 29, 2016 · 1 comment
Closed

MemoryError when squashing large images #132

johnccchang opened this issue Nov 29, 2016 · 1 comment
Assignees
@goldmann
Labels
type/enhancement
Milestone
1.0.5

Comments

@johnccchang
Copy link

johnccchang commented Nov 29, 2016 •
edited by goldmann
Loading

I am testing squash process with different size of images and it seems hitting memory bound and throwing MemoryError at the end of process when trying with the image of 12.88 GB. Is there a way to get around this? Thanks.

[ec2-user@ip-172-31-6-240 ~]$ docker images
REPOSITORY                   TAG                 IMAGE ID            CREATED             SIZE
persist                      v5                  2a23c4d09356        14 hours ago        18.27 GB
persist                      v4                  192868276bff        15 hours ago        15.57 GB
persist                      v3                  ce47a7db4686        15 hours ago        12.88 GB
persist                      v2                  897fd55a78a5        15 hours ago        10.18 GB
persist                      v1                  5dccae780e66        16 hours ago        7.487 GB
persist                      v0                  e41b9809d08d        17 hours ago        4.792 GB
jupyter/all-spark-notebook   latest              96139b6593c5        5 weeks ago         4.79 GB

[ec2-user@ip-172-31-6-240 ~]$ time docker-squash -v -f 4 -d -t squashed:t3 ce47a7db4686
2016-11-29 17:00:14,159 root         DEBUG    Running version 1.1.0rc1.dev
2016-11-29 17:00:14,160 root         DEBUG    Preparing Docker client...
2016-11-29 17:00:14,160 docker.auth.auth DEBUG    Trying paths: ['/home/ec2-user/.docker/config.json', '/home/ec2-user/.dockercfg']
2016-11-29 17:00:14,160 docker.auth.auth DEBUG    Found file at path: /home/ec2-user/.docker/config.json
2016-11-29 17:00:14,160 docker.auth.auth DEBUG    Found 'auths' section
2016-11-29 17:00:14,161 docker.auth.auth DEBUG    Found entry (registry=u'https://751568579428.dkr.ecr.us-east-1.amazonaws.com', username=u'AWS')
2016-11-29 17:00:14,169 requests.packages.urllib3.connectionpool DEBUG    http://localhost:None "GET /version HTTP/1.1" 200 164
2016-11-29 17:00:14,171 requests.packages.urllib3.connectionpool DEBUG    http://localhost:None "GET /v1.23/_ping HTTP/1.1" 200 2
2016-11-29 17:00:14,172 root         DEBUG    Docker client ready
2016-11-29 17:00:14,173 requests.packages.urllib3.connectionpool DEBUG    http://localhost:None "GET /v1.23/version HTTP/1.1" 200 164
2016-11-29 17:00:14,174 root         INFO     docker-squash version 1.1.0rc1.dev, Docker b9f10c9/1.11.2, API 1.23...
2016-11-29 17:00:14,174 root         INFO     Using v2 image format
2016-11-29 17:00:14,177 root         DEBUG    Using /tmp/docker-squash-0LAkKs as the temporary directory
2016-11-29 17:00:14,181 requests.packages.urllib3.connectionpool DEBUG    http://localhost:None "GET /v1.23/images/ce47a7db4686/json HTTP/1.1" 200 None
2016-11-29 17:00:14,204 requests.packages.urllib3.connectionpool DEBUG    http://localhost:None "GET /v1.23/images/sha256:ce47a7db46860708d2600f00b4ad7416a55ac18e4f4200e771f3dabcf14bc9c4/history HTTP/1.1" 200 None
2016-11-29 17:00:14,206 root         INFO     Old image has 78 layers
2016-11-29 17:00:14,206 root         DEBUG    Old layers: [u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'sha256:96139b6593c594acf44d9d55418444e2c56577c640e1f79d96003fe0cad4e3bd', u'sha256:e41b9809d08d0f6f2a77aeef4fb07e582656e0170ba003c8caa7ef71c35e1623', u'sha256:5dccae780e66956c61d4955c094f22d7b1b410445ec7313d9ba3294fe71e9ecd', u'sha256:897fd55a78a5a6599c921c90875c70f1a8ac443cf01c69e6bbcee425722a8962', u'sha256:ce47a7db46860708d2600f00b4ad7416a55ac18e4f4200e771f3dabcf14bc9c4']
2016-11-29 17:00:14,206 root         DEBUG    We detected number of layers as the argument to squash
2016-11-29 17:00:14,206 root         INFO     Checking if squashing is necessary...
2016-11-29 17:00:14,206 root         INFO     Attempting to squash last 4 layers...
2016-11-29 17:00:14,207 root         DEBUG    Layers to squash: [u'sha256:e41b9809d08d0f6f2a77aeef4fb07e582656e0170ba003c8caa7ef71c35e1623', u'sha256:5dccae780e66956c61d4955c094f22d7b1b410445ec7313d9ba3294fe71e9ecd', u'sha256:897fd55a78a5a6599c921c90875c70f1a8ac443cf01c69e6bbcee425722a8962', u'sha256:ce47a7db46860708d2600f00b4ad7416a55ac18e4f4200e771f3dabcf14bc9c4']
2016-11-29 17:00:14,207 root         DEBUG    Layers to move: [u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'<missing>', u'sha256:96139b6593c594acf44d9d55418444e2c56577c640e1f79d96003fe0cad4e3bd']
2016-11-29 17:00:14,207 root         INFO     Saving image sha256:ce47a7db46860708d2600f00b4ad7416a55ac18e4f4200e771f3dabcf14bc9c4 to /tmp/docker-squash-0LAkKs/old/image.tar file...
2016-11-29 17:00:14,207 root         DEBUG    Try #1...
2016-11-29 17:07:37,481 requests.packages.urllib3.connectionpool DEBUG    http://localhost:None "GET /v1.23/images/sha256:ce47a7db46860708d2600f00b4ad7416a55ac18e4f4200e771f3dabcf14bc9c4/get HTTP/1.1" 200 None
2016-11-29 17:11:03,023 root         INFO     Image saved!
2016-11-29 17:11:03,023 root         INFO     Unpacking /tmp/docker-squash-0LAkKs/old/image.tar tar file to /tmp/docker-squash-0LAkKs/old directory
2016-11-29 17:14:39,306 root         INFO     Archive unpacked!
2016-11-29 17:14:39,306 root         DEBUG    Removing exported tar (/tmp/docker-squash-0LAkKs/old/image.tar)...
2016-11-29 17:14:39,820 root         INFO     Squashing image 'ce47a7db4686'...
2016-11-29 17:14:39,821 root         DEBUG    Reading '/tmp/docker-squash-0LAkKs/old/manifest.json' JSON file...
2016-11-29 17:14:39,821 root         DEBUG    Reading '/tmp/docker-squash-0LAkKs/old/ce47a7db46860708d2600f00b4ad7416a55ac18e4f4200e771f3dabcf14bc9c4.json' JSON file...
2016-11-29 17:14:39,824 root         INFO     Starting squashing...
2016-11-29 17:14:39,824 root         DEBUG    Generating list of files in layer 'c94ee45783f53a4124d06ef85229530ce653f2cdcf4ee72991c6d76fe57e6125'...
2016-11-29 17:14:40,447 root         DEBUG    Done, found 8248 files
2016-11-29 17:14:40,448 root         DEBUG    Generating list of files in layer '866d119d4baf42fcf4bc875fc6971320d7a634bbe65f83f2496bc2057b854a3d'...
2016-11-29 17:14:40,462 root         DEBUG    Done, found 0 files
2016-11-29 17:14:40,462 root         DEBUG    Generating list of files in layer '843a75caceb0cb9aa7ecdfda1b501c2895cfd71059e2baf92243a5edd0b6805b'...
2016-11-29 17:14:40,463 root         DEBUG    Done, found 0 files
2016-11-29 17:14:40,463 root         DEBUG    Generating list of files in layer 'e617553346bd75cf4591dd5266eb591fb7541a8b5599a953b39b70a942f76915'...
2016-11-29 17:14:40,464 root         DEBUG    Done, found 0 files
2016-11-29 17:14:40,464 root         DEBUG    Generating list of files in layer '5a34f9f2508461bd54d3298dfafb534263cb4781f66a9cd1a838c1db052ce213'...
2016-11-29 17:14:40,464 root         DEBUG    Done, found 0 files
2016-11-29 17:14:40,465 root         DEBUG    Generating list of files in layer 'd96a733457a250bf3426d278c1bb990a4b308f0b44b4aca5108d1961844f5468'...
2016-11-29 17:14:40,932 root         DEBUG    Done, found 6085 files
2016-11-29 17:14:40,932 root         DEBUG    Generating list of files in layer '0b32fb1fc42511de9391dfb2474903072c3ef04448b34e5e9624f69966ad3872'...
2016-11-29 17:14:40,944 root         DEBUG    Done, found 6 files
2016-11-29 17:14:40,944 root         DEBUG    Generating list of files in layer '799e2eee7fe9381840881b58250640ac418ad5d0301661e3351dcd2087e25519'...
2016-11-29 17:14:40,947 root         DEBUG    Done, found 4 files
2016-11-29 17:14:40,947 root         DEBUG    Generating list of files in layer 'ff986be97029749ed12dbcf1545944b7247ea4e5f2d5b001be1af5ad36a82c6b'...
2016-11-29 17:14:40,949 root         DEBUG    Done, found 20 files
2016-11-29 17:14:40,949 root         DEBUG    Generating list of files in layer 'd8221fc6efefb9b56f05f3eb9b29dbc3609cfbed57643192f4180025fbce98e4'...
2016-11-29 17:14:40,949 root         DEBUG    Done, found 5 files
2016-11-29 17:14:40,949 root         DEBUG    Generating list of files in layer '5d1df7afc8e7a764770e18c2ee9f98a9bb27b3c103c9cb81ac9d6e0ced547b5a'...
2016-11-29 17:14:42,613 root         DEBUG    Done, found 11261 files
2016-11-29 17:14:42,613 root         DEBUG    Generating list of files in layer '3b0085d2682ce7cec545828b1af90f9bd45eaa35c25ce5550ea8d9c3cf8e90c3'...
2016-11-29 17:14:42,628 root         DEBUG    Done, found 4 files
2016-11-29 17:14:42,628 root         DEBUG    Generating list of files in layer '394e4726135b465af6fea55895624b80663360726ca38d5f777fc51363c3caed'...
2016-11-29 17:14:44,156 root         DEBUG    Done, found 11177 files
2016-11-29 17:14:44,156 root         DEBUG    Generating list of files in layer 'eec762d834bc9395c1160a17080f1ec0f916ba02aa2ee895c47ce6c45e4d7bfe'...
2016-11-29 17:14:44,250 root         DEBUG    Done, found 1035 files
2016-11-29 17:14:44,250 root         DEBUG    Generating list of files in layer '668ed0d91e9a370320a8ae49366aa94dd69596dec135a453944415deaee21e6b'...
2016-11-29 17:14:44,258 root         DEBUG    Done, found 4 files
2016-11-29 17:14:44,258 root         DEBUG    Generating list of files in layer 'ca2463159cbfcfff82dd94e012ab21279646a94c8b54149baa81b459a9000c59'...
2016-11-29 17:14:44,258 root         DEBUG    Done, found 4 files
2016-11-29 17:14:44,259 root         DEBUG    Generating list of files in layer 'a9e59aa498291d8a86ba952336d6a81139d09e868432ffd8f87699a3c133014c'...
2016-11-29 17:14:44,260 root         DEBUG    Done, found 4 files
2016-11-29 17:14:44,260 root         DEBUG    Generating list of files in layer '4da907be85d69a57ca26f9c7e5aba66f3e8ca369f3df8a7e4ddf6ba39c57bac4'...
2016-11-29 17:14:44,261 root         DEBUG    Done, found 4 files
2016-11-29 17:14:44,261 root         DEBUG    Generating list of files in layer 'e45fc9392d0f193f2f10ea7d93679ad9eef850578b997121ba51ccf531576d13'...
2016-11-29 17:14:44,262 root         DEBUG    Done, found 4 files
2016-11-29 17:14:44,262 root         DEBUG    Generating list of files in layer 'c3ccd0153718b9288436def8ef600af4e9ebcd1131603bd6edf36e253036ba42'...
2016-11-29 17:14:50,495 root         DEBUG    Done, found 75576 files
2016-11-29 17:14:50,495 root         DEBUG    Generating list of files in layer 'b31ff4846be1bf2cba99e62146d226a1e09f81ca3e1d80468e5d4f71822160fc'...
2016-11-29 17:14:50,708 root         DEBUG    Done, found 887 files
2016-11-29 17:14:50,708 root         DEBUG    Generating list of files in layer '83bf5bb5ec0d784ef2f0404f3b4bb51b9db6f16b15f579ee19e1646d672eeb44'...
2016-11-29 17:14:56,617 root         DEBUG    Done, found 33714 files
2016-11-29 17:14:56,618 root         DEBUG    Generating list of files in layer 'cf34ec8501e69fee06d40186f3bc5fa72644b5e05a3ed28fc325f81e17dc2cdd'...
2016-11-29 17:14:56,661 root         DEBUG    Done, found 12 files
2016-11-29 17:14:56,661 root         DEBUG    Generating list of files in layer 'c897933fc1e3caf8d66ca032454b14ebf89157d83e87f8f7831443130ce53317'...
2016-11-29 17:15:02,217 root         DEBUG    Done, found 66386 files
2016-11-29 17:15:02,217 root         DEBUG    Generating list of files in layer 'd369420963582dfb97cb51b1e6fcbe891cd7b2feb88f1f13de3a23abc82fd2c3'...
2016-11-29 17:15:02,298 root         DEBUG    Done, found 5 files
2016-11-29 17:15:02,298 root         DEBUG    Generating list of files in layer '68d6b96507988b830b5cfc9b30fe0cdd41d4d70e17d36ca125bb2346461b6565'...
2016-11-29 17:15:02,320 root         DEBUG    Done, found 66 files
2016-11-29 17:15:02,320 root         DEBUG    Generating list of files in layer 'd5b081aa028acd0fb949947c9a554bd5cadb822a2edf649b61a1ec3e794bf1ac'...
2016-11-29 17:15:02,321 root         DEBUG    Done, found 5 files
2016-11-29 17:15:02,321 root         DEBUG    Generating list of files in layer '4aaea2fb68f65796bf60fb6703a3b61dccb406abb52510e5c43696efe6647a08'...
2016-11-29 17:15:02,323 root         DEBUG    Done, found 6 files
2016-11-29 17:15:02,323 root         DEBUG    Generating list of files in layer '4c73d4c551416d320a5fb19e4498a956b3d388960f0bfec9927f5a6820cdd7aa'...
2016-11-29 17:15:02,325 root         DEBUG    Done, found 11 files
2016-11-29 17:15:02,325 root         DEBUG    Generating list of files in layer '023c40805d1ed0da143e59d827ba8d750574d0c3bf683748c4e2d81cd711c60a'...
2016-11-29 17:15:02,458 root         DEBUG    Done, found 876 files
2016-11-29 17:15:02,459 root         DEBUG    Generating list of files in layer 'd7f87579ac2d5993b74b64187fd0af28a64edc9e2432c5174640163b00049d3c'...
2016-11-29 17:15:02,547 root         DEBUG    Done, found 937 files
2016-11-29 17:15:02,547 root         DEBUG    Generating list of files in layer 'c3713a240d71d43fe5711ee1a7e16637de906e511ef1a543d47d259175c50c8d'...
2016-11-29 17:15:02,549 root         DEBUG    Done, found 3 files
2016-11-29 17:15:02,549 root         DEBUG    Generating list of files in layer 'ab599fd8392e3369cd1fffc4e73d940e1ed1a4fcca367e354681cfa8636a96ab'...
2016-11-29 17:15:02,656 root         DEBUG    Done, found 506 files
2016-11-29 17:15:02,656 root         DEBUG    Generating list of files in layer '8abe560041907e6868cbddf8569f636895a90b2de5735183999a1656528b6360'...
2016-11-29 17:15:02,678 root         DEBUG    Done, found 90 files
2016-11-29 17:15:02,678 root         DEBUG    Generating list of files in layer '6bf9eb6629287998a8e853a2abd27e1178651dc3b32174c10bc96720dde23d3a'...
2016-11-29 17:15:05,274 root         DEBUG    Done, found 16090 files
2016-11-29 17:15:05,274 root         DEBUG    Generating list of files in layer '30ce8111643063d1d6208355cb992ffe3f8d2e9c61f3989ac075f6e2aab9e709'...
2016-11-29 17:15:05,304 root         DEBUG    Done, found 40 files
2016-11-29 17:15:05,304 root         DEBUG    Generating list of files in layer '81e2df327b398463dd410df04429ad594f14f4394c5d1a9432174efbb850045f'...
2016-11-29 17:15:05,313 root         DEBUG    Done, found 45 files
2016-11-29 17:15:05,313 root         INFO     Squashing file '/tmp/docker-squash-0LAkKs/old/a4ffdf3f64e1dc0dac83141b6a26e40b2263aa98fad213f3f2d7d90e9dbbcfaa/layer.tar'...
2016-11-29 17:15:17,950 root         DEBUG    Searching for marker files in '/tmp/docker-squash-0LAkKs/old/a4ffdf3f64e1dc0dac83141b6a26e40b2263aa98fad213f3f2d7d90e9dbbcfaa/layer.tar' archive...
2016-11-29 17:15:17,956 root         DEBUG    Done, found 0 files
2016-11-29 17:15:17,956 root         DEBUG    Searching for symbolic links in '/tmp/docker-squash-0LAkKs/old/a4ffdf3f64e1dc0dac83141b6a26e40b2263aa98fad213f3f2d7d90e9dbbcfaa/layer.tar' archive...
2016-11-29 17:15:17,973 root         DEBUG    Done, found 0 files
2016-11-29 17:16:04,643 root         INFO     Squashing file '/tmp/docker-squash-0LAkKs/old/5df2a72cdd3920f4e85a2cbf330644c280c62ce3cce9daa164dfdc8ad8af867d/layer.tar'...
2016-11-29 17:16:24,220 root         DEBUG    Searching for marker files in '/tmp/docker-squash-0LAkKs/old/5df2a72cdd3920f4e85a2cbf330644c280c62ce3cce9daa164dfdc8ad8af867d/layer.tar' archive...
2016-11-29 17:16:24,231 root         DEBUG    Done, found 0 files
2016-11-29 17:16:24,231 root         DEBUG    Searching for symbolic links in '/tmp/docker-squash-0LAkKs/old/5df2a72cdd3920f4e85a2cbf330644c280c62ce3cce9daa164dfdc8ad8af867d/layer.tar' archive...
2016-11-29 17:16:24,243 root         DEBUG    Done, found 0 files
2016-11-29 17:16:24,248 root         DEBUG    Skipping '/home' file because it's older than file already added to the archive
2016-11-29 17:16:24,249 root         DEBUG    Skipping '/home/jovyan' file because it's older than file already added to the archive
2016-11-29 17:16:24,249 root         DEBUG    Skipping '/home/jovyan/.local' file because it's older than file already added to the archive
2016-11-29 17:16:24,249 root         DEBUG    Skipping '/home/jovyan/.local/share' file because it's older than file already added to the archive
2016-11-29 17:16:24,249 root         DEBUG    Skipping '/home/jovyan/.local/share/jupyter' file because it's older than file already added to the archive
2016-11-29 17:16:24,249 root         DEBUG    Skipping '/home/jovyan/.local/share/jupyter/runtime' file because it's older than file already added to the archive
2016-11-29 17:16:24,249 root         DEBUG    Skipping '/home/jovyan/.local/share/jupyter/runtime/nbserver-6.json' file because it's older than file already added to the archive
2016-11-29 17:16:24,249 root         DEBUG    Skipping '/home/jovyan/work' file because it's older than file already added to the archive
2016-11-29 17:17:46,122 root         INFO     Squashing file '/tmp/docker-squash-0LAkKs/old/6e2dd91c247e4c32194883c39b0adccce0efc8673265dbc9fbfbdc07a9f08003/layer.tar'...
2016-11-29 17:17:58,874 root         DEBUG    Searching for marker files in '/tmp/docker-squash-0LAkKs/old/6e2dd91c247e4c32194883c39b0adccce0efc8673265dbc9fbfbdc07a9f08003/layer.tar' archive...
2016-11-29 17:17:58,880 root         DEBUG    Done, found 0 files
2016-11-29 17:17:58,880 root         DEBUG    Searching for symbolic links in '/tmp/docker-squash-0LAkKs/old/6e2dd91c247e4c32194883c39b0adccce0efc8673265dbc9fbfbdc07a9f08003/layer.tar' archive...
2016-11-29 17:17:58,897 root         DEBUG    Done, found 0 files
2016-11-29 17:17:58,897 root         DEBUG    Skipping '/home' file because it's older than file already added to the archive
2016-11-29 17:17:58,897 root         DEBUG    Skipping '/home/jovyan' file because it's older than file already added to the archive
2016-11-29 17:17:58,908 root         DEBUG    Skipping '/home/jovyan/.local' file because it's older than file already added to the archive
2016-11-29 17:17:58,908 root         DEBUG    Skipping '/home/jovyan/.local/share' file because it's older than file already added to the archive
2016-11-29 17:17:58,909 root         DEBUG    Skipping '/home/jovyan/.local/share/jupyter' file because it's older than file already added to the archive
2016-11-29 17:17:58,912 root         DEBUG    Skipping '/home/jovyan/.local/share/jupyter/runtime' file because it's older than file already added to the archive
2016-11-29 17:17:58,914 root         DEBUG    Skipping '/home/jovyan/.local/share/jupyter/runtime/nbserver-6.json' file because it's older than file already added to the archive
2016-11-29 17:17:58,914 root         DEBUG    Skipping '/home/jovyan/work' file because it's older than file already added to the archive
2016-11-29 17:19:52,276 root         INFO     Squashing file '/tmp/docker-squash-0LAkKs/old/24fcc424934366fd55ef1e47bd453690fa56aee355d98fc981b7582ff3d928aa/layer.tar'...
2016-11-29 17:19:52,329 root         DEBUG    Searching for marker files in '/tmp/docker-squash-0LAkKs/old/24fcc424934366fd55ef1e47bd453690fa56aee355d98fc981b7582ff3d928aa/layer.tar' archive...
2016-11-29 17:19:52,330 root         DEBUG    Done, found 0 files
2016-11-29 17:19:52,330 root         DEBUG    Searching for symbolic links in '/tmp/docker-squash-0LAkKs/old/24fcc424934366fd55ef1e47bd453690fa56aee355d98fc981b7582ff3d928aa/layer.tar' archive...
2016-11-29 17:19:52,330 root         DEBUG    Done, found 0 files
2016-11-29 17:19:52,331 root         DEBUG    Skipping '/home' file because it's older than file already added to the archive
2016-11-29 17:19:52,331 root         DEBUG    Skipping '/home/jovyan' file because it's older than file already added to the archive
2016-11-29 17:19:52,333 root         DEBUG    Skipping '/home/jovyan/.ipython' file because it's older than file already added to the archive
2016-11-29 17:19:52,340 root         DEBUG    Skipping '/home/jovyan/.ipython/profile_default' file because it's older than file already added to the archive
2016-11-29 17:19:52,348 root         DEBUG    Skipping '/home/jovyan/.ipython/profile_default/history.sqlite' file because it's older than file already added to the archive
2016-11-29 17:19:52,362 root         DEBUG    Skipping '/home/jovyan/.local' file because it's older than file already added to the archive
2016-11-29 17:19:52,363 root         DEBUG    Skipping '/home/jovyan/.local/share' file because it's older than file already added to the archive
2016-11-29 17:19:52,363 root         DEBUG    Skipping '/home/jovyan/.local/share/jupyter' file because it's older than file already added to the archive
2016-11-29 17:19:52,364 root         DEBUG    Skipping '/home/jovyan/.local/share/jupyter/nbsignatures.db' file because it's older than file already added to the archive
2016-11-29 17:19:52,368 root         DEBUG    Skipping '/home/jovyan/.local/share/jupyter/runtime' file because it's older than file already added to the archive
2016-11-29 17:19:52,377 root         DEBUG    Skipping '/home/jovyan/work' file because it's older than file already added to the archive
2016-11-29 17:19:52,940 root         INFO     Squashing finished!
2016-11-29 17:21:14,612 root         ERROR    
Traceback (most recent call last):
  File "/home/ec2-user/.local/lib/python2.7/site-packages/docker_squash/cli.py", line 64, in run
    from_layer=args.from_layer, tag=args.tag, output_path=args.output_path, tmp_dir=args.tmp_dir, development=args.development, cleanup=args.cleanup).run()
  File "/home/ec2-user/.local/lib/python2.7/site-packages/docker_squash/squash.py", line 59, in run
    return self.squash(image)
  File "/home/ec2-user/.local/lib/python2.7/site-packages/docker_squash/squash.py", line 82, in squash
    new_image_id = image.squash()
  File "/home/ec2-user/.local/lib/python2.7/site-packages/docker_squash/image.py", line 72, in squash
    ret = self._squash()
  File "/home/ec2-user/.local/lib/python2.7/site-packages/docker_squash/v2_image.py", line 42, in _squash
    self.diff_ids = self._generate_diff_ids()
  File "/home/ec2-user/.local/lib/python2.7/site-packages/docker_squash/v2_image.py", line 197, in _generate_diff_ids
    diff_ids.append(hashlib.sha256(f.read()).hexdigest())
MemoryError
2016-11-29 17:21:14,616 root         ERROR    Execution failed, consult logs above. If you think this is our fault, please file an issue: https://github.com/goldmann/docker-squash/issues, thanks!

real	21m0.901s
user	3m45.832s
sys	0m50.380s
@goldmann
Copy link
Owner

I do see some space for improvement here, I'll take a look ;) Thanks for the report!

@goldmann goldmann self-assigned this Dec 1, 2016
@goldmann goldmann added this to the 1.0.5 milestone Dec 1, 2016
goldmann added a commit that referenced this issue Dec 6, 2016
@goldmann
Hash computation optimizations
ef3c6ea 
Instead of reading whole layer into memory, read it in 10MB
chunks.

Fixes #132
@goldmann goldmann mentioned this issue Dec 6, 2016
Merged
goldmann added a commit that referenced this issue Dec 6, 2016
@goldmann
Merge pull request #137 from goldmann/gh-132-hash-improvements
fd1f0a8 
Hash computation optimizations
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@goldmann goldmann

Labels
type/enhancement
Projects
None yet
Milestone
1.0.5
Development

No branches or pull requests
2 participants
@goldmann @johnccchang