[SR-11760] SwiftPM builds inside Docker do not re-use existing build artifacts #4651

swift-ci · 2019-11-11T13:28:31Z


Previous ID	SR-11760
Radar	rdar://problem/73985925
Original Reporter	MrMage (JIRA User)
Type	Bug

Attachment: Download

Environment

Swift 4.2 all the way till Swift 5.2.4.

Additional Detail from JIRA


Votes	19
Component/s	Package Manager
Labels	Bug
Assignee	None
Priority	Medium

md5: 6c4efa086896ce15ed52fa119adc9b54

Issue Description:

See https://forums.swift.org/t/incremental-builds-with-swift-package-manager-and-linux-docker/26538 for a full description. @aciidb0mb3r has asked to file a Jira ticket about this issue.

My suspicion is that SwiftPM for some reason considers build artifacts in lower Docker layers stale and thinks that it needs to rebuild them.

beccadax · 2019-11-11T23:01:11Z

@aciidb0mb3r

swift-ci · 2019-11-13T23:39:01Z

Comment by Eric Stern (JIRA)

As a small (or maybe not so small) addition - support for something along the lines of `swift build --dependencies-only` would be hugely beneficial inside of Dockerized builds once this is resolved, in order to maximize the effect of layer caching.

It's quite clumsy and counterintuitive to make "dummy files" (as shown in the linked discussion) just to overwrite them with actual source code later.

swift-ci · 2019-11-14T08:37:33Z

Comment by Daniel Alm (JIRA)

Fully agree with Firehed (JIRA User). Creating the dummy files is only a workaround; having an option to build only the dependencies would be a much better solution in the long term.

swift-ci · 2019-11-19T11:21:17Z

Comment by Daniel Alm (JIRA)

I have investigated a bit more and found the following:

This seems to affect only non-Swift dependencies, e.g. C and C++ ones. This is still significant, however, as SwiftNIO packages a full copy BoringSSL, which takes long to compile.
The output of stat indeed changes when run in a different Docker layer; in particular, fractional timestamp values are removed. Example stat output from the layer the compilation has been run in:

  File: .build/debug/CNIOBoringSSL.build/module.modulemap
  Size: 163         Blocks: 8          IO Block: 4096   regular file
Device: 801h/2049d  Inode: 426531      Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2019-11-19 11:13:29.897272000 +0000
Modify: 2019-11-19 11:13:29.517272000 +0000
Change: 2019-11-19 11:13:29.517272000 +0000
 Birth: -
  File: .build/debug/CNIOBoringSSL.build/crypto/crypto.c.d
  Size: 4546        Blocks: 16         IO Block: 4096   regular file
Device: 801h/2049d  Inode: 427653      Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2019-11-19 11:13:48.345814000 +0000
Modify: 2019-11-19 11:13:48.335814000 +0000
Change: 2019-11-19 11:13:48.335814000 +0000
 Birth: -
  File: .build/debug/CNIOBoringSSL.build/crypto/crypto.c.o
  Size: 8664        Blocks: 24         IO Block: 4096   regular file
Device: 801h/2049d  Inode: 427635      Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2019-11-19 11:13:48.285814000 +0000
Modify: 2019-11-19 11:13:48.335814000 +0000
Change: 2019-11-19 11:13:48.335814000 +0000
 Birth: -

And from a "later" layer:

  File: .build/debug/CNIOBoringSSL.build/module.modulemap
  Size: 163         Blocks: 8          IO Block: 4096   regular file
Device: 801h/2049d  Inode: 2930655     Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2019-11-19 11:13:29.000000000 +0000
Modify: 2019-11-19 11:13:29.000000000 +0000
Change: 2019-11-19 11:14:10.058600000 +0000
 Birth: -
  File: .build/debug/CNIOBoringSSL.build/crypto/crypto.c.d
  Size: 4546        Blocks: 16         IO Block: 4096   regular file
Device: 801h/2049d  Inode: 2930545     Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2019-11-19 11:13:48.000000000 +0000
Modify: 2019-11-19 11:13:48.000000000 +0000
Change: 2019-11-19 11:14:09.948600000 +0000
 Birth: -
  File: .build/debug/CNIOBoringSSL.build/crypto/crypto.c.o
  Size: 8664        Blocks: 24         IO Block: 4096   regular file
Device: 801h/2049d  Inode: 2930546     Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2019-11-19 11:13:48.000000000 +0000
Modify: 2019-11-19 11:13:48.000000000 +0000
Change: 2019-11-19 11:14:09.948600000 +0000
 Birth: -

Unfortunately, even packaging the build archive in a POSIX tar which is supposed to fully preserves timestamps yields different timestamps, as well as causing a rebuild of all non-Swift dependencies:

  File: .build/debug/CNIOBoringSSL.build/module.modulemap
  Size: 163         Blocks: 8          IO Block: 4096   regular file
Device: 801h/2049d  Inode: 428027      Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2019-11-19 11:14:15.888600000 +0000
Modify: 2019-11-19 11:13:29.517272000 +0000
Change: 2019-11-19 11:14:15.888600000 +0000
 Birth: -
  File: .build/debug/CNIOBoringSSL.build/crypto/crypto.c.d
  Size: 4546        Blocks: 16         IO Block: 4096   regular file
Device: 801h/2049d  Inode: 696234      Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2019-11-19 11:14:15.958600000 +0000
Modify: 2019-11-19 11:13:48.335814000 +0000
Change: 2019-11-19 11:14:15.958600000 +0000
 Birth: -
  File: .build/debug/CNIOBoringSSL.build/crypto/crypto.c.o
  Size: 8664        Blocks: 24         IO Block: 4096   regular file
Device: 801h/2049d  Inode: 697210      Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2019-11-19 11:14:16.008600000 +0000
Modify: 2019-11-19 11:13:48.335814000 +0000
Change: 2019-11-19 11:14:16.008600000 +0000
 Birth: -

I have uploaded a full build log to https://gist.github.com/MrMage/943f0de9009ee183ae66591eafad6c7a and a sample project to https://github.com/MrMage/docker-caching-test. The Dockerfile is a bit messy with all these experiments, but I think you can see what I had tried to somehow force Swift to use the build cache.

@aciidb0mb3r I would really appreciate if you could take a look. Thanks!

aciidgh · 2019-11-19T16:29:53Z

Thanks for the investigation! If the timestamp is changing then I suspect we'll have to use file hashing to determine if the manifest changed.

swift-ci · 2019-11-20T08:14:23Z

Comment by Daniel Alm (JIRA)

@aciidb0mb3r another option might be to use only the integer part of the timestamp, dropping the fractional part.

swift-ci · 2019-12-12T15:14:20Z

Comment by Pierpaolo Frasa (JIRA)

Is there any update on this issue?

swift-ci · 2020-02-02T06:07:24Z

Comment by Will Lisac (JIRA)

Something like `swift build --dependencies-only` would be a great addition.

I'm also using the dummy files technique and it becomes increasingly tedious as the number of targets grows.

swift-ci · 2020-02-25T13:23:14Z

Comment by Bridger Maxwell (JIRA)

Would it be appropriate to round the timestamp here in the Swift Package Manager source?

https://github.com/apple/swift-package-manager/blob/7f35752db78977ff5e2ca3c38cc5427091647698/TSC/Sources/TSCBasic/FileInfo.swift#L55

swift-ci · 2020-06-29T09:15:34Z

Comment by Daniel Alm (JIRA)

This might help, provided it also gains support for Linux: #2774

drewcrawford · 2020-08-19T01:59:38Z

I think one cause of detecting artifacts as "stale" is that llbuild looks at the inode number. There is an open PR to change this algorithm swiftlang/swift-llbuild#641

I believe that separately, there is a way to adjust llbuild behavior by supplying a key to the right location. According to the documentation

> A file-system field may be supplied that toggles how the build system handles file system use. The default mode uses detailed stat information for detecting file changes. The device-agnostic mode will ignore device and inode values.

swift-ci · 2020-11-15T21:59:55Z

Comment by Bridger Maxwell (JIRA)

The blog post for Swift 5.3 mentions better caching https://swift.org/blog/swift-5-3-released :

Faster incremental build times by avoiding duplicated compiler work across source files, and more accurately identifying code that has not changed from the previous build

I can't find any other information about this change. Any chance it relates to this bug? Lately I'm desperate for something to speed up my Swift builds.

swift-ci · 2020-11-22T03:45:35Z

Comment by Daniel R (JIRA)

+1 on something like `swift build --dependencies-only`

I'm also creating dummy files as a workaround.

weissi · 2021-02-03T18:01:48Z

@swift-ci create

weissi · 2021-02-12T13:40:58Z

My colleague Euan Harris just made me aware of a new (experimental) Docker feature in BuildKit which can alleviate this problem. If you create your Dockerfile like so

# syntax = docker/dockerfile:1.2
FROM 5.3-focal AS builder
ADD . /code/
WORKDIR /code
RUN --mount=type=cache,target=/code/.build swift build -c release && cp /code/.build/release/my-binary /app

FROM swift:5.3-focal-slim AS runner
COPY --from=builder /app /app
ENTRYPOINT /app

then it will rebuild fast. You will need to use docker buildx build . and not docker build .. (Partial) docs about BuildKit https://docs.docker.com/buildx/working-with-buildx/ and https://docs.docker.com/develop/develop-images/build_enhancements/ .

As an explainer, the --mount=type=cache,target=/code/.build will create a cache volume that is persisted to /code/.build. It keeps the inode numbers as well as the timestamps intact and therefore SwiftPM will not rebuild unnecessarily.

Long story short, if you can, use BuildKit.

If for whatever reason you cannot use BuildKit, you can contact me directly, I have a hack that lets you use cached builds with plain old stuff (no BuildKit).

swift-ci · 2021-06-20T16:55:05Z

Comment by ML (JIRA)

I ran into the same problem with aws codebuild which uses a buildspec.yml for the build commands. @weissi would you be so kind and share your "hack" as buildkit is not suitable in codebuild(as far as I know).

Thanks!

swift-ci · 2022-02-18T21:19:57Z

Comment by spk (JIRA)

@weissi using CircleCI here. Could you please share your "hack" ?

Thanks!

weissi · 2022-02-21T11:20:12Z

spk (JIRA User) Sure, I used this C file that I compiled into a dynamic library (.so) and then loaded using LD_PRELOAD

weissi · 2022-02-21T11:20:58Z

Oh, with "this file" I mean the now attached override-stat.c. This lies to the SwiftPM and just returns a chopped of mtime (just the seconds, not the nano seconds) and makes the inode number a function of the sip-hash'd file name instead of a more or less random number. That makes SwiftPM trust that the files from Docker indeed haven't changed (despite the fact that both their inode numbers and mtime (missing nanoseconds)) have changed.

Again, the right solution is to use this docker feature:

RUN --mount=type=cache,target=/code/.build swift build -c release

which instructs docker to mount a volume instead of copying in the files. That'll make everything work without hacks.

swift-ci · 2022-02-21T15:19:21Z

Comment by spk (JIRA)

Thanks so much! 🙂

Terky · 2022-07-02T21:46:24Z

@weissi, it looks like the hack you provided in the original JIRA ticket is gone. Could you please attach that file again?

Also is there any chance this issue will be resolved without making a switch to the BuildKit?

weissi · 2022-07-04T10:57:06Z

@Terky there you go. You'd need to compile override-stat.c as a shared library and then LD_PRELOAD it with every swift build command you run...

But really, you should switch to BuildKit with cache layers. BuildKit is now enabled by default in docker build, any particular reason you can't adopt that?

override-stat.tar.gz

danielBreitlauch · 2022-09-22T09:50:46Z

BuildKit with cache mount is still no full solution:
Github Actions spawn a new node with every run. The BuildKit cache folder will be deleted. It is possible to hack/cache that folder and restoring it on the new runner. But then the files have new Inodes.

IMHO there is no way around detecting code changes without Inode checks.

weissi · 2022-09-22T14:58:57Z

IMHO there is no way around detecting code changes without Inode checks.

Yes, alas, I think this is true at the moment. I couldn't immediately find a way to turn off the inode number comparisons (https://github.com/apple/swift-llbuild/blob/b7b4c5eaa798708d6233ca07908a7cab75620040/products/llbuildSwift/BuildValue.swift#L59). @dmbryson any thoughts?

weissi · 2022-09-22T14:59:31Z

@tomerd do you think you would consider a --ignore-inode-number-changes or similar option to swift build which just leaves out the inode numbers?

neonichu · 2022-09-22T16:46:41Z

We have rdar://96403757 (SwiftPM - consider setting file-system: device-agnostic in build manifest) which may resolve this? I'm not sure we even need an option, I think passing device-agnostic in the llbuild manifest might be a reasonable thing to do always.

doozMen · 2023-08-29T08:56:26Z

I still have this issue. Is there an approach to build it. Is the option to ignore the inode numbers added? Could somebody with more knowledge then me create some documentation on recommended setup of a swiftpm build for CI? Or does the latter exist already?

sets file-system as device-agnostic in build manifest ### Motivation: This is the default mode in XCBuild, and allows incremental builds to work across reboots on APFS. It would also likely address #4651. ### Modifications: sets file-system as device-agnostic in build manifest ### Result: Restarting your system (or inode changes) doesn't invoke unnecessary builds Related to rdar://96403757.

Cherry-pick of #7052. ### Motivation: This is the default mode in XCBuild, and allows incremental builds to work across reboots on APFS. It would also likely address #4651. ### Modifications: Set `file-system` as `device-agnostic` in the generated build manifest ### Result: Restarting your system (or inode changes) doesn't invoke unnecessary builds Related to rdar://96403757. Co-authored-by: Kishore Pusukuri <kishoreguptaos@gmail.com>

MaxDesiatov · 2024-02-19T14:47:57Z

@doozMen is this issue reproducible for you with latest 5.10 development snapshots?

doozMen · 2024-02-21T08:45:28Z

@doozMen is this issue reproducible for you with latest 5.10 development snapshots?

Hmm sorry I had a look at your fix in #7189 and that seams like the valid thing to do to solve the issue. But installing a development snapshot to verify this seams like something I do not have the time for right now. I would love to test it with 5.10 do.

MaxDesiatov · 2024-02-21T09:44:17Z

The PR linked should address the issue in 5.10 and main development snapshots. Feel free to reopen if that's not the case.

therickys93 · 2024-03-12T07:35:45Z

Goodmorning to everybody,
maybe this is not related to the same issue, if so please ignore it.
I have a simple project using Vapor and I configured the cache in gitlab runner in order to preserve the build between different stages. I use swift 5.10 release and it caches correctly the package download, but it recompiles again all the dependencies.
In the discussion that you were doing here, is it something that you already considered or is this new to you?
Thanks

fred-sch · 2024-08-25T16:57:36Z

We are also experiencing problems with C dependencies not being properly cached from a previous build when running builds or tests in gitlab-ci for a simple Vapor app.

I set up a simple test project with pipeline here. You can see the output of a job where the build should have been cached entirely here. When producing release builds these build times naturally increase even a bit more as you can see here.

Is there any recommended way to deal with this or anything spm can do about it?

maerki · 2024-08-25T19:18:30Z

@fred-sch have you tried the cache mentioned above using RUN —mount=type=cache,target=/code/.build swift build -c release and docker buildx build?

So far, I have not found a configuration that properly reuses cached dependency build results on GitHub CI, even though it works with local docker builds.

fred-sch · 2024-08-26T07:20:15Z

@maerki, I can confirm that RUN —mount=type=cache,target=/code/.build swift build -c release mitigates the problem.

Unfortunately, it would complicate our pipelines a bit to execute everything in dind in gitlab-ci. Particularly, I don't think unit tests should be executed during a docker build stage. Since I have not found a really satisfying solution to make something like swift test --enable-code-coverage reuse the .build cache, I was wondering if there potentially is a more general fix on the SPM side?

swift-ci transferred this issue from apple/swift-issues Apr 25, 2022

shahmishal transferred this issue from swiftlang/swift May 4, 2022

tomerd removed the Package Manager label May 14, 2022

vzsg mentioned this issue Oct 25, 2022

Buliding for debug... vapor/vapor#2902

Closed

pusukuri mentioned this issue Nov 3, 2023

Set file-system as device-agnostic in build manifest #7052

Merged

MaxDesiatov mentioned this issue Dec 12, 2023

[5.10] Set file-system as device-agnostic in build manifest #7189

Merged

MaxDesiatov closed this as completed Feb 21, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SR-11760] SwiftPM builds inside Docker do not re-use existing build artifacts #4651

[SR-11760] SwiftPM builds inside Docker do not re-use existing build artifacts #4651

swift-ci commented Nov 11, 2019

beccadax commented Nov 11, 2019

swift-ci commented Nov 13, 2019

swift-ci commented Nov 14, 2019

swift-ci commented Nov 19, 2019

aciidgh commented Nov 19, 2019

swift-ci commented Nov 20, 2019

swift-ci commented Dec 12, 2019

swift-ci commented Feb 2, 2020

swift-ci commented Feb 25, 2020

swift-ci commented Jun 29, 2020

drewcrawford commented Aug 19, 2020

swift-ci commented Nov 15, 2020

swift-ci commented Nov 22, 2020

weissi commented Feb 3, 2021

weissi commented Feb 12, 2021

swift-ci commented Jun 20, 2021

swift-ci commented Feb 18, 2022

weissi commented Feb 21, 2022

weissi commented Feb 21, 2022

swift-ci commented Feb 21, 2022

Terky commented Jul 2, 2022

weissi commented Jul 4, 2022

danielBreitlauch commented Sep 22, 2022

weissi commented Sep 22, 2022

weissi commented Sep 22, 2022

neonichu commented Sep 22, 2022

doozMen commented Aug 29, 2023

MaxDesiatov commented Feb 19, 2024

doozMen commented Feb 21, 2024

MaxDesiatov commented Feb 21, 2024

therickys93 commented Mar 12, 2024

fred-sch commented Aug 25, 2024

maerki commented Aug 25, 2024

fred-sch commented Aug 26, 2024

[SR-11760] SwiftPM builds inside Docker do not re-use existing build artifacts #4651

[SR-11760] SwiftPM builds inside Docker do not re-use existing build artifacts #4651

Comments

swift-ci commented Nov 11, 2019

beccadax commented Nov 11, 2019

swift-ci commented Nov 13, 2019

swift-ci commented Nov 14, 2019

swift-ci commented Nov 19, 2019

aciidgh commented Nov 19, 2019

swift-ci commented Nov 20, 2019

swift-ci commented Dec 12, 2019

swift-ci commented Feb 2, 2020

swift-ci commented Feb 25, 2020

swift-ci commented Jun 29, 2020

drewcrawford commented Aug 19, 2020

swift-ci commented Nov 15, 2020

swift-ci commented Nov 22, 2020

weissi commented Feb 3, 2021

weissi commented Feb 12, 2021

swift-ci commented Jun 20, 2021

swift-ci commented Feb 18, 2022

weissi commented Feb 21, 2022

weissi commented Feb 21, 2022

swift-ci commented Feb 21, 2022

Terky commented Jul 2, 2022

weissi commented Jul 4, 2022

danielBreitlauch commented Sep 22, 2022

weissi commented Sep 22, 2022

weissi commented Sep 22, 2022

neonichu commented Sep 22, 2022

doozMen commented Aug 29, 2023

MaxDesiatov commented Feb 19, 2024

doozMen commented Feb 21, 2024

MaxDesiatov commented Feb 21, 2024

therickys93 commented Mar 12, 2024

fred-sch commented Aug 25, 2024

maerki commented Aug 25, 2024

fred-sch commented Aug 26, 2024