v1.2.1 appears to have introduced a destination path bug

[salimepoint]

When our CI build squashes with v1.2.0 all works fine. When we update to v1.2.1, docker-squash consistently produces the following error:

2024-09-04 04:19:47,488 squash.py:54         INFO  docker-squash version 1.2.1, Docker 20.10.22, API 1.41...
2024-09-04 04:19:47,489 squash.py:96         INFO  Using v2 image format
2024-09-04 04:19:47,496 image.py:179        INFO  Old image has 37 layers
2024-09-04 04:19:47,496 image.py:211        INFO  Checking if squashing is necessary...
2024-09-04 04:19:47,496 image.py:223        INFO  Attempting to squash last 19 layers...
2024-09-04 04:19:47,496 image.py:371        INFO  Saving image sha256:22e24d09ec86cd1d82aa51144a2968c1b71d701a3c89dff4aa78494001376a97 to /tmp/docker-squash-8g0lt41d/old directory...
/usr/lib64/python3.9/tarfile.py:2239: RuntimeWarning: The default behavior of tarfile extraction has been changed to disallow common exploits (including CVE-2007-4559). By default, absolute/parent paths are disallowed and some mode bits are cleared. See https://access.redhat.com/articles/7004769 for more details.
  warnings.warn(
2024-09-04 04:20:10,179 image.py:406        INFO  Image saved!
2024-09-04 04:20:10,180 image.py:232        INFO  Squashing image 'repository/image-name:latest'...
2024-09-04 04:20:10,181 image.py:749        INFO  Starting squashing for /tmp/docker-squash-8g0lt41d/new/squashed/layer.tar...
2024-09-04 04:20:11,806 image.py:774        INFO  Squashing file '/tmp/docker-squash-8g0lt41d/old/6de5a3cc55bf67a0a9912bb8e8b830d312945fddf669e62fb3fa51b209d644f3/layer.tar'...
2024-09-04 04:20:11,807 image.py:774        INFO  Squashing file '/tmp/docker-squash-8g0lt41d/old/650281fa92917ce5cc58443055a098ad491e3d7d0873cf6f6417497893eda8e4/layer.tar'...
2024-09-04 04:20:11,852 image.py:774        INFO  Squashing file '/tmp/docker-squash-8g0lt41d/old/fdfe5f9717b65f1a1fa917364a390e3e2c9cff126d9d21812e1eed0d616dc31d/layer.tar'...
2024-09-04 04:20:11,916 image.py:774        INFO  Squashing file '/tmp/docker-squash-8g0lt41d/old/2d542cc66a9ce04cb5c3cee12c9906e6e6543c6effe0d92c63bdd0652ceb0d9b/layer.tar'...
2024-09-04 04:20:11,918 image.py:774        INFO  Squashing file '/tmp/docker-squash-8g0lt41d/old/b50ec3f8cadd012be4a54dd2046ffd81ab043097f72f7e80951cca6a06b2bd11/layer.tar'...
2024-09-04 04:20:11,921 image.py:774        INFO  Squashing file '/tmp/docker-squash-8g0lt41d/old/7bb9a0215c5d110cb703651916f655c3b5a3d5bae9c8fa24250f110aa934274f/layer.tar'...
2024-09-04 04:20:11,924 image.py:774        INFO  Squashing file '/tmp/docker-squash-8g0lt41d/old/49877f5b395a0f14f45d7731b9baea92bcaf178cba8b2f028d389e051b458f2a/layer.tar'...
2024-09-04 04:20:12,280 image.py:774        INFO  Squashing file '/tmp/docker-squash-8g0lt41d/old/ca9ef9fe254730e2c5d7dd71bb0bd77590702db2dd9ff1c607b9a9206b0e429f/layer.tar'...
2024-09-04 04:20:12,281 image.py:774        INFO  Squashing file '/tmp/docker-squash-8g0lt41d/old/3ccfad99e583270b9138adb6f283e106234c50530335a0db3fe0f20bad7d04e6/layer.tar'...
2024-09-04 04:20:12,287 image.py:774        INFO  Squashing file '/tmp/docker-squash-8g0lt41d/old/de5196ec3a9b895974e9ee0bd55c55f632e53dfa5697e5c74092826e85c8bf66/layer.tar'...
2024-09-04 04:20:12,507 image.py:774        INFO  Squashing file '/tmp/docker-squash-8g0lt41d/old/4ebcb9421dccd2b9f1e7f3d3fd320fef88b46ae9ee2fc0e83b7c2b1d8ef52290/layer.tar'...
2024-09-04 04:20:12,511 image.py:774        INFO  Squashing file '/tmp/docker-squash-8g0lt41d/old/f8f29280761153e60c3687610045be2c25bb5bab69621551b49ebf82ad265cec/layer.tar'...
2024-09-04 04:20:12,579 image.py:774        INFO  Squashing file '/tmp/docker-squash-8g0lt41d/old/fa7e760796c7d0374914924c0d8a2ccf82ea80f01952e9bbcc966ce8f2a5d26c/layer.tar'...
2024-09-04 04:20:12,811 image.py:774        INFO  Squashing file '/tmp/docker-squash-8g0lt41d/old/6f0f8d4f29d24f50d9ec225271a327d8bde27662657e4a3876b817c129c79305/layer.tar'...
2024-09-04 04:20:16,933 image.py:922        INFO  Squashing finished!
2024-09-04 04:20:18,351 cli.py:142        ERROR Destination path '/tmp/docker-squash-8g0lt41d/new/layer.tar' already exists
2024-09-04 04:20:18,352 cli.py:144        ERROR Execution failed, consult logs above. If you think this is our fault, please file an issue: https://github.com/goldmann/docker-squash/issues, thanks!

For now I've fixed our CI build on 1.2.0 but obviously would prefer to be on the latest and greatest :-) Any suggestions of things to try (or a v1.2.2 release 😁) most appreciated..

Thanks in advance,
Steve.

[Permafacture]

Weird, I just got this error on 1.2.0

[salimepoint]

Weird, I just got this error on 1.2.0

Oh well - there goes the idea it was related to the 1.2.1 changes :-( Regardless, @goldmann if you find time to investigate I'm more than happy to add some debug code or other changes into my local docker squash to help analyse this (given I have a consistently reproducible case). Thanks again for the product.

[goldmann]

Hm, do you have maybe an image that could be used as a reproducer? And invocation.

[ypapouin]

Same issue with 1.2.1 in our CI

1.2.0 :

2024-07-30 12:24:21,470 squash.py:54         INFO  docker-squash version 1.2.0, Docker 20.10.17, API 1.41...
2024-07-30 12:24:21,471 squash.py:96         INFO  Using v2 image format
2024-07-30 12:24:21,504 image.py:179        INFO  Old image has 80 layers
2024-07-30 12:24:21,504 image.py:211        INFO  Checking if squashing is necessary...
2024-07-30 12:24:21,505 image.py:223        INFO  Attempting to squash last 40 layers...
2024-07-30 12:24:21,505 image.py:371        INFO  Saving image sha256:8b45486c4ec677aea26cb451c4190d51cba8ea9246debb9b48210d099f14a052 to /tmp/docker-squash-km5vdnxv/old directory...
2024-07-30 12:27:13,018 image.py:406        INFO  Image saved!
2024-07-30 12:27:13,020 image.py:232        INFO  Squashing image 'docker-registry.decgroupe.com/erp/odoo-images:14.0-latest'...
2024-07-30 12:27:13,021 image.py:749        INFO  Starting squashing for /tmp/docker-squash-km5vdnxv/new/squashed/layer.tar...
2024-07-30 12:27:18,138 image.py:774        INFO  Squashing file '/tmp/docker-squash-km5vdnxv/old/07442e6dd4964d56fd09ca08cb0c771caf8a07c080ef58f01588eb5627cf2127/layer.tar'...
2024-07-30 12:35:01,468 image.py:774        INFO  Squashing file '/tmp/docker-squash-km5vdnxv/old/4a420af6e2e83623038060da57f8d8a13888155e6663ef340b37ff127a6cfebf/layer.tar'...
2024-07-30 12:35:01,484 image.py:774        INFO  Squashing file '/tmp/docker-squash-km5vdnxv/old/11a34b31c4aa7fcfca5a4b5ed55e2208e5f7b89817413cf7a7d45a8d47d20eb2/layer.tar'...
2024-07-30 12:35:01,497 image.py:774        INFO  Squashing file '/tmp/docker-squash-km5vdnxv/old/d38aa457d52d345c9ca3ef08c096418c64f34eaa36cc65e30a9ef4b8c793b193/layer.tar'...
2024-07-30 12:35:03,850 image.py:922        INFO  Squashing finished!
2024-07-30 12:35:31,506 image.py:244        INFO  Original image size: 2918.90 MB
2024-07-30 12:35:31,506 image.py:245        INFO  Squashed image size: 2930.61 MB
2024-07-30 12:35:31,506 image.py:248        INFO  If the squashed image is larger than original it means that there were no meaningful files to squash and it just added metadata. Are you sure you specified correct parameters?
2024-07-30 12:35:31,506 squash.py:136        INFO  New squashed image ID is 01f55fa52616c3a908868cd8a90e1d4ff86cff273d70d8ca0ac50d1f1984ec33
2024-07-30 12:38:05,275 image.py:280        INFO  Image registered in Docker daemon as docker-registry.decgroupe.com/erp/odoo-images:14.0-latest
2024-07-30 12:38:05,684 squash.py:156        INFO  Done

1.2.1 :

2024-09-26 11:35:12,673 squash.py:54         INFO  docker-squash version 1.2.1, Docker 20.10.17, API 1.41...
2024-09-26 11:35:12,674 squash.py:96         INFO  Using v2 image format
2024-09-26 11:35:12,704 image.py:179        INFO  Old image has 80 layers
2024-09-26 11:35:12,705 image.py:211        INFO  Checking if squashing is necessary...
2024-09-26 11:35:12,705 image.py:223        INFO  Attempting to squash last 40 layers...
2024-09-26 11:35:12,705 image.py:371        INFO  Saving image sha256:30b69e1be5ddeb4758e2b34669df0b1a883a394815aaa4a83b799a4a119c2da4 to /tmp/docker-squash-6umegzem/old directory...
2024-09-26 11:36:24,879 image.py:406        INFO  Image saved!
2024-09-26 11:36:24,881 image.py:232        INFO  Squashing image 'docker-registry.decgroupe.com/erp/odoo-images:14.0-latest'...
2024-09-26 11:36:24,882 image.py:749        INFO  Starting squashing for /tmp/docker-squash-6umegzem/new/squashed/layer.tar...
2024-09-26 11:36:29,597 image.py:774        INFO  Squashing file '/tmp/docker-squash-6umegzem/old/67f0433fa8fcabffd67304081a436893f3f433f8914fa2cab906629534f1b80e/layer.tar'...
2024-09-26 11:45:11,694 image.py:774        INFO  Squashing file '/tmp/docker-squash-6umegzem/old/529f3d23704d3464a3a969a8449646869faa947270573dfdd97143c511fd6775/layer.tar'...
2024-09-26 11:45:11,710 image.py:774        INFO  Squashing file '/tmp/docker-squash-6umegzem/old/76941ee15cf17b835033c241918eecd8cd0bc8c6385f221738f7e3cb2b6d8323/layer.tar'...
2024-09-26 11:45:11,725 image.py:774        INFO  Squashing file '/tmp/docker-squash-6umegzem/old/cd50e3b35f74f2e1a6baad18c01a4a374a00aa53dc44378f8907a00c780046df/layer.tar'...
2024-09-26 11:45:14,625 image.py:922        INFO  Squashing finished!
2024-09-26 11:45:26,870 cli.py:142        ERROR Destination path '/tmp/docker-squash-6umegzem/new/layer.tar' already exists
2024-09-26 11:45:26,870 cli.py:144        ERROR Execution failed, consult logs above. If you think this is our fault, please file an issue: https://github.com/goldmann/docker-squash/issues, thanks!

[rnc]

@ypapouin Are you able to provide a reproducer ?

[ypapouin]

No I'm not able to, but looking on code changes 0030222, It looks like oci_format is now always True, so the culprit should be there:

  if self.oci_format:
      old_layer_path = self.old_image_manifest["Config"]
  else:
      if self.layer_paths_to_squash[0]:
          old_layer_path = self.layer_paths_to_squash[0]
      else:
          old_layer_path = layer_path_id
      old_layer_path = os.path.join(old_layer_path, "json")

[salimepoint]

Hm, do you have maybe an image that could be used as a reproducer? And invocation.

Sorry @goldmann - didn't see this come through. Unfortunately I can't provide a reproducer either but I'm happy to run tests, add debug, etc.. (although appreciate it's not the the easiest (or desirable) path to debug an issue)..

[cgf120]

What am I supposed to offer you? A Image or an environment or something?

[cgf120]

os:
Linux 85045 5.15.0-118-generic #128-Ubuntu SMP Fri Jul 5 09:28:59 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

my cmd :

docker-squash -f 2 -t aaaaa:v1 test:v1.0

image:

run docker instpect ,and get layers

"RootFS": {
"Type": "layers",
"Layers": [
"sha256:e0a9f5911802534ba097660206feabeb0247a81e409029167b30e2e1f2803b57",
"sha256:47654eeadbc543f1dd44ebb41c6ca0954b9f1813efabdcd46b0e7f17ac4e9fd1",
"sha256:efe2b79b53de08e199a2ce107d83adc0cbdff94f605e03f5ef51f3df3ae31cfd",
"sha256:46d54736d31f4bfeb749544e22e8611dba11128bdb2cebb820a3b452b50e7d52",
"sha256:809d3bb9c80fb3d31d4c061ba0b38ba4e83b6329e33c2cb2bbf27251a8e527c6",
"sha256:1ffbbe19418f9726ebb365371ef96637c68f9d25af3f76a4f01f760e87fb31be",
"sha256:1b30dd39de2709bc00beebf392ea1f931d5f037d08e0f675a50855385cddad93",
"sha256:89fd878224a017aead1d4c83d5c62749307aa6f1a9e22087144ef339c3df8c1e",
"sha256:2b4acda8678c6f7ae0075eabb2c0c21078b80671800996ad2cf730f15fa0ef26",
"sha256:5fb5b17c5835e3e44f7c48dfc15f29504febad945f26dcfa11470d0070b166e4",
"sha256:b7fa3c6c5beea49f65d51f65f06a4a8af32b48f128c02bea3822e5afaf4ebf23",
"sha256:9b39c4e5264e7c4b3762a1f1d06bb00db03e2e21893ea703e8ad6e8f2e0f5fba",
"sha256:3e0968ffdd5dfeca6e6942ead4c15b072c7ba07f1b70f1e8c74e6c196f3761b0",
"sha256:e7ecf7e01dc755c81bbf4118af25595bc4676853dcf7234dded901b3170af734",
"sha256:cbc5c993246b2e527864e904e35bcdf4e2b0968b8ed87d7dbfd90788f3dfd760",
"sha256:e33f2cba5a03a0dc7bdfa36ebbaba7026c60b42e5a1400fb7949196f18323afa",
"sha256:6125ee6bdd7a9cc50cb2ee65789cac9287f45e7cd9c059e5114bc52b0b1da9ad",
"sha256:5fa07b59b35dbfb051ebbe33d4ff0369a1abc4e0771b3ab2cfa66ade98705f16",
"sha256:0d60e1ff6f92040f5c4b3a18f22b55a779ae506662d41cad7da042cde06948d0",
"sha256:26614f4df8a4dfee6ba12e9f76ffa13fa72dac43203e18b9b9bca8c7b9da7362",
"sha256:72227655fbb1d939fbc301a2d299675fd6ce7f0d586eb16b0aa9d2bb50b8d632"
]
},

run docker history test:v1.0 get
MAGE CREATED CREATED BY SIZE COMMENT
dee1f8adc8f2 7 days ago bash /root/start.sh 85.1MB
2 weeks ago bash /root/start.sh 58.5MB
3 weeks ago bash /root/start.sh 58.4MB
3 weeks ago bash /root/start.sh 338MB
4 weeks ago /bin/sh -c #(nop) CMD ["bash" "/root/start.… 0B
4 weeks ago /bin/bash 385MB
4 weeks ago /bin/bash 11.2GB
5 weeks ago RUN /bin/sh -c pip install --no-cache-dir to… 5.25GB buildkit.dockerfile.v0
5 weeks ago RUN |1 PATH=/root/miniconda3/bin:/root/minic… 889MB buildkit.dockerfile.v0
5 weeks ago ARG PATH=/root/miniconda3/bin:/root/minicond… 0B buildkit.dockerfile.v0
5 weeks ago ENV DEBIAN_FRONTEND=noninteractive TZ=Asia/S… 0B buildkit.dockerfile.v0
5 weeks ago RUN /bin/sh -c ln -snf /usr/share/zoneinfo/$… 1.78GB buildkit.dockerfile.v0
5 weeks ago ENV DEBIAN_FRONTEND=noninteractive TZ=Asia/S… 0B buildkit.dockerfile.v0
6 months ago RUN |1 TARGETARCH=amd64 /bin/sh -c apt-get u… 1.02GB buildkit.dockerfile.v0
6 months ago LABEL com.nvidia.cudnn.version=9.1.0.70-1 0B buildkit.dockerfile.v0
6 months ago LABEL maintainer=NVIDIA CORPORATION <cudatoo… 0B buildkit.dockerfile.v0
6 months ago ARG TARGETARCH 0B buildkit.dockerfile.v0
6 months ago ENV NV_CUDNN_PACKAGE_DEV=libcudnn9-dev-cuda-… 0B buildkit.dockerfile.v0
6 months ago ENV NV_CUDNN_PACKAGE=libcudnn9-cuda-12=9.1.0… 0B buildkit.dockerfile.v0
6 months ago ENV NV_CUDNN_PACKAGE_NAME=libcudnn9-cuda-12 0B buildkit.dockerfile.v0
6 months ago ENV NV_CUDNN_VERSION=9.1.0.70-1 0B buildkit.dockerfile.v0
6 months ago ENV LIBRARY_PATH=/usr/local/cuda/lib64/stubs 0B buildkit.dockerfile.v0
6 months ago RUN |1 TARGETARCH=amd64 /bin/sh -c apt-mark … 390kB buildkit.dockerfile.v0
6 months ago RUN |1 TARGETARCH=amd64 /bin/sh -c apt-get u… 4.98GB buildkit.dockerfile.v0
6 months ago LABEL maintainer=NVIDIA CORPORATION <cudatoo… 0B buildkit.dockerfile.v0
6 months ago ARG TARGETARCH 0B buildkit.dockerfile.v0
6 months ago ENV NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.21.… 0B buildkit.dockerfile.v0
6 months ago ENV NCCL_VERSION=2.21.5-1 0B buildkit.dockerfile.v0
6 months ago ENV NV_LIBNCCL_DEV_PACKAGE_VERSION=2.21.5-1 0B buildkit.dockerfile.v0
6 months ago ENV NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 0B buildkit.dockerfile.v0
6 months ago ENV NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-4=1… 0B buildkit.dockerfile.v0
6 months ago ENV NV_NVPROF_VERSION=12.4.127-1 0B buildkit.dockerfile.v0
6 months ago ENV NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-… 0B buildkit.dockerfile.v0
6 months ago ENV NV_CUDA_NSIGHT_COMPUTE_VERSION=12.4.1-1 0B buildkit.dockerfile.v0
6 months ago ENV NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-1… 0B buildkit.dockerfile.v0
6 months ago ENV NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-… 0B buildkit.dockerfile.v0
6 months ago ENV NV_LIBCUBLAS_DEV_VERSION=12.4.5.8-1 0B buildkit.dockerfile.v0
6 months ago ENV NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-4=12… 0B buildkit.dockerfile.v0
6 months ago ENV NV_LIBNPP_DEV_VERSION=12.2.5.30-1 0B buildkit.dockerfile.v0
6 months ago ENV NV_LIBCUSPARSE_DEV_VERSION=12.3.1.170-1 0B buildkit.dockerfile.v0
6 months ago ENV NV_NVML_DEV_VERSION=12.4.127-1 0B buildkit.dockerfile.v0
6 months ago ENV NV_CUDA_CUDART_DEV_VERSION=12.4.127-1 0B buildkit.dockerfile.v0
6 months ago ENV NV_CUDA_LIB_VERSION=12.4.1-1 0B buildkit.dockerfile.v0
6 months ago ENTRYPOINT ["/opt/nvidia/nvidia_entrypoint.s… 0B buildkit.dockerfile.v0
6 months ago ENV NVIDIA_PRODUCT_NAME=CUDA 0B buildkit.dockerfile.v0
6 months ago COPY nvidia_entrypoint.sh /opt/nvidia/ # bui… 2.53kB buildkit.dockerfile.v0
6 months ago COPY entrypoint.d/ /opt/nvidia/entrypoint.d/… 3.06kB buildkit.dockerfile.v0
6 months ago RUN |1 TARGETARCH=amd64 /bin/sh -c apt-mark … 263kB buildkit.dockerfile.v0
6 months ago RUN |1 TARGETARCH=amd64 /bin/sh -c apt-get u… 2.05GB buildkit.dockerfile.v0
6 months ago LABEL maintainer=NVIDIA CORPORATION <cudatoo… 0B buildkit.dockerfile.v0
6 months ago ARG TARGETARCH 0B buildkit.dockerfile.v0
6 months ago ENV NV_LIBNCCL_PACKAGE=libnccl2=2.21.5-1+cud… 0B buildkit.dockerfile.v0
6 months ago ENV NCCL_VERSION=2.21.5-1 0B buildkit.dockerfile.v0
6 months ago ENV NV_LIBNCCL_PACKAGE_VERSION=2.21.5-1 0B buildkit.dockerfile.v0
6 months ago ENV NV_LIBNCCL_PACKAGE_NAME=libnccl2 0B buildkit.dockerfile.v0
6 months ago ENV NV_LIBCUBLAS_PACKAGE=libcublas-12-4=12.4… 0B buildkit.dockerfile.v0
6 months ago ENV NV_LIBCUBLAS_VERSION=12.4.5.8-1 0B buildkit.dockerfile.v0
6 months ago ENV NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-4 0B buildkit.dockerfile.v0
6 months ago ENV NV_LIBCUSPARSE_VERSION=12.3.1.170-1 0B buildkit.dockerfile.v0
6 months ago ENV NV_LIBNPP_PACKAGE=libnpp-12-4=12.2.5.30-1 0B buildkit.dockerfile.v0
6 months ago ENV NV_LIBNPP_VERSION=12.2.5.30-1 0B buildkit.dockerfile.v0
6 months ago ENV NV_NVTX_VERSION=12.4.127-1 0B buildkit.dockerfile.v0
6 months ago ENV NV_CUDA_LIB_VERSION=12.4.1-1 0B buildkit.dockerfile.v0
6 months ago ENV NVIDIA_DRIVER_CAPABILITIES=compute,utili… 0B buildkit.dockerfile.v0
6 months ago ENV NVIDIA_VISIBLE_DEVICES=all 0B buildkit.dockerfile.v0
6 months ago COPY NGC-DL-CONTAINER-LICENSE / # buildkit 17.3kB buildkit.dockerfile.v0
6 months ago ENV LD_LIBRARY_PATH=/usr/local/nvidia/lib:/u… 0B buildkit.dockerfile.v0
6 months ago ENV PATH=/usr/local/nvidia/bin:/usr/local/cu… 0B buildkit.dockerfile.v0
6 months ago RUN |1 TARGETARCH=amd64 /bin/sh -c echo "/us… 46B buildkit.dockerfile.v0
6 months ago RUN |1 TARGETARCH=amd64 /bin/sh -c apt-get u… 156MB buildkit.dockerfile.v0
6 months ago ENV CUDA_VERSION=12.4.1 0B buildkit.dockerfile.v0
6 months ago RUN |1 TARGETARCH=amd64 /bin/sh -c apt-get u… 10.6MB buildkit.dockerfile.v0
6 months ago LABEL maintainer=NVIDIA CORPORATION <cudatoo… 0B buildkit.dockerfile.v0
6 months ago ARG TARGETARCH 0B buildkit.dockerfile.v0
6 months ago ENV NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-4 0B buildkit.dockerfile.v0
6 months ago ENV NV_CUDA_CUDART_VERSION=12.4.127-1 0B buildkit.dockerfile.v0
6 months ago ENV NVIDIA_REQUIRE_CUDA=cuda>=12.4 brand=tes… 0B buildkit.dockerfile.v0
6 months ago ENV NVARCH=x86_64 0B buildkit.dockerfile.v0
6 months ago /bin/sh -c #(nop) CMD ["/bin/bash"] 0B
6 months ago /bin/sh -c #(nop) ADD file:3bd10da0673e2e72c… 77.9MB
6 months ago /bin/sh -c #(nop) LABEL org.opencontainers.… 0B
6 months ago /bin/sh -c #(nop) LABEL org.opencontainers.… 0B
6 months ago /bin/sh -c #(nop) ARG LAUNCHPAD_BUILD_ARCH 0B
6 months ago /bin/sh -c #(nop) ARG RELEASE 0B

[cgf120]

my image will multiple commit containers saved as new image, This is the only thing that could possibly be different from normal image

[cgf120]

@goldmann

[RyDoRe]

Got the same issue as well...The last layers got added as docker commits and trying to sqash them all together is resulting in the same output. Squashing all layers work. Looks like the issue comes up, when there are layers to move. The layers in my case are mostly ''.

For me the output before the crash looks like this:

DEBUG Cleaning up /tmp/docker-squash-_8o1dz8j temporary directory 
ERROR Destination path '/tmp/docker-squash-_8o1dz8j/new/layer.tar' already exists
Traceback (most recent call last):
  File "/home/user/.local/lib/python3.8/site-packages/docker_squash/cli.py", line 122, in run
    squash.Squash(
  File "/home/user/.local/lib/python3.8/site-packages/docker_squash/squash.py", line 99, in run
    return self.squash(image)
  File "/home/user/.local/lib/python3.8/site-packages/docker_squash/squash.py", line 134, in squash
    new_image_id = image.squash()
  File "/home/user/.local/lib/python3.8/site-packages/docker_squash/image.py", line 84, in squash
    ret = self._squash()
  File "/home/user/.local/lib/python3.8/site-packages/docker_squash/v2_image.py", line 103, in _squash
    self._move_layers(
  File "/home/user/.local/lib/python3.8/site-packages/docker_squash/image.py", line 513, in _move_layers
    shutil.move(os.path.join(src, layer_id), dest)
  File "/usr/lib/python3.8/shutil.py", line 789, in move
    raise Error("Destination path '%s' already exists" % real_dst)
shutil.Error: Destination path '/tmp/docker-squash-_8o1dz8j/new/layer.tar' already exists

[rnc]

@cgf120 @RyDoRe Are either of you able to provide a Dockerfile to reproduce the issue?
@RyDoRe Have you only seen this with 1.2.1 and not 1.2.0 ?

[RyDoRe]

Only tested that with 1.2.1, but I can try 1.2.0 as well.
The Image I'm having issues with I unfortunately can't provide. but I can take a look, if i can create a Test Image this works with

[rnc]

@RyDoRe Can you provide the command line you were using please?

[RyDoRe]

docker-squash -v -f 4 -t image:squashed-tag image:that_I_squashed

Was quite straight forward. I have 42 layers in total. It crashes with trying to squash only the upper4 layers and succeeds with all layers

[RyDoRe]

The way I reproduced this with a test image:
I used e.g. the mariadb:latest image, ran a container of that and created e.g. a folder, commited those changes to the image and redid that so I had two additional layer, than I tried to run the the squash command to squash the 2 upper layers and got the same error.

[goldmann]

I just released 1.2.2.