Execution root symlink forest includes all files in a package instead of only ones needed #19581

erikkerber · 2023-09-21T15:46:15Z

Description of the bug:

When running a build, the symlink forest will include all files from a package beyond files actually needed, used, or referenced by a build.

This leads to various quality-of-life, and reproducibility issues.

An example of the output base from the open source rules_xcodeproj looks like this when a target in the root package is built:

Which category does this issue belong to?

No response

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

Clone rules_xcodeproj
bazel build tools:xcodeproj
cd into bazel-output-base/execroot/_main/
- Notice the execroot has symlinked two packages used in that build
cd back to the workspace root
bazel build buildifier.check
- Notice the execroot has symlnked all files in the root, including BUILD files and .git directories

Which operating system are you running Bazel on?

macOS 14.0

What is the output of `bazel info release`?

release 6.4.0rc1

If `bazel info release` returns `development version` or `(@non-git)`, tell us how you built Bazel.

No response

What's the output of `git remote get-url origin; git rev-parse master; git rev-parse HEAD` ?

No response

Is this a regression? If yes, please try to identify the Bazel commit where the bug was introduced.

No

Have you found anything relevant by searching the web?

No response

Any other information, logs, or outputs that you want to share?

This has existed for a while as a nuisance (if you traverse bazel-out), it almost always tricks tools into thinking it's a Git directory, but became a more acute problem in the iOS community with Xcode 15 as the IDE performance is extremely degraded trying to parse what Git views as an extremely large Git diff in the output base.

bazelbuild.slack.com discussion

The text was updated successfully, but these errors were encountered:

erikkerber · 2023-11-10T03:03:55Z

This appears to reproduce in all scenarios in Bazel 7 (tested on rc3).

In other words, both bazel build tools:xcodeproj and bazel build buildifier.check in the repro steps above result in all files and directories in the repository being symlinked.

joeleba · 2023-11-10T09:08:10Z

This is unfortunately a known limitation of skymeld (#14057), which is on by default in Bazel 7. In this mode, we create a symlink to each entry under the source root.

If this breaks you, please include --noexperimental_merged_skyframe_analysis_execution in your blazerc to disable skymeld.

If I understand correctly, the main nuisance here is with the .git file also being symlinked to from the execroot, right?

brentleyjones · 2023-11-10T14:03:42Z

If I understand correctly, the main nuisance here is with the .git file also being symlinked to from the execroot, right?

Yes. If minimally that wasn't symlinked, we would be fine. One step more would to also not symlink anything in .bazelignore. Ideally only files that need to be symlinked are symlinked.

brentleyjones · 2023-11-10T14:05:58Z

If this breaks you, please include --noexperimental_merged_skyframe_analysis_execution in your blazerc to disable skymeld.

Sadly I will have to include that by default for rules_xcodeproj if this ships this way. It breaks Xcode as is.

fmeum · 2023-11-10T14:34:10Z

@joeleba Should we @bazel-io flag this for Bazel 7?

Interestingly Bazel itself depends on source files in .git at

bazel/BUILD

Lines 54 to 61 in 07a571f

    
           filegroup( 
        
               name = "git", 
        
               srcs = glob( 
        
                   [".git/**"], 
        
                   allow_empty = True, 
        
                   exclude = [".git/**/*[*"],  # gitk creates temp files with [] 
        
               ), 
        
           )

. The minimal solution of not symlinking just .git may thus not be sufficiently configurable. Going by .bazelignore should cover this use case though.

joeleba · 2023-11-10T15:56:53Z

@bazel-io flag

Wyverald · 2023-11-10T18:28:58Z

what's the estimated effort to fix this?

keertk · 2023-11-10T22:21:05Z

@bazel-io fork 7.0.0

Surfaced in #19581. PiperOrigin-RevId: 581877718 Change-Id: I8d9e460ac809ab236d04252ba14a5836d2e2d86c

…gnore`. Bazel in skymeld mode eagerly plants the symlinks from the execroot to every entries under the source tree. This leads to some pain points for certain groups of users (#19581). This CL offers a workaround for that issue. The users can now list the paths that they don't want Bazel to symlink to in the `.bazelignore` file in their project. PiperOrigin-RevId: 581919915 Change-Id: I329365b0655fc0bcb2db1eeea91ef74c775c72ae

…gnore`. Bazel in skymeld mode eagerly plants the symlinks from the execroot to every entries under the source tree. This leads to some pain points for certain groups of users (bazelbuild#19581). This CL offers a workaround for that issue. The users can now list the paths that they don't want Bazel to symlink to in the `.bazelignore` file in their project. PiperOrigin-RevId: 581919915 Change-Id: I329365b0655fc0bcb2db1eeea91ef74c775c72ae

…20168) Bazel in skymeld mode eagerly plants the symlinks from the execroot to every entries under the source tree. This leads to some pain points for certain groups of users (#19581). This CL offers a workaround for that issue. The users can now list the paths that they don't want Bazel to symlink to in the `.bazelignore` file in their project. Commit caf1702 PiperOrigin-RevId: 581919915 Change-Id: I329365b0655fc0bcb2db1eeea91ef74c775c72ae Co-authored-by: Googler <leba@google.com>

joeleba · 2023-11-13T14:05:25Z

The commit that tells bazel to skip the paths listed in .bazelignore was cherrypicked into the release. You should be able to use that as a workaround. I'll lower the priority of this bug.

brentleyjones · 2023-11-13T14:21:27Z

@joeleba Thanks! I assume that only applies to when using Skymeld? If we don't use Skymeld, .bazelignore won't fix the originally reported issue?

joeleba · 2023-11-13T16:14:38Z

That's true at least for Skymeld: the symlink planting should repect .bazelignore. I wasn't aware that this was an issue without Skymeld. AFAIK the noskymeld case actually goes through all the packages and only plants symlinks for the necessary ones.

brentleyjones · 2023-11-13T17:11:14Z

AFAIK the noskymeld case actually goes through all the packages and only plants symlinks for the necessary ones

Sure, but the issue is when a target in the root package is built, it will symlink the .git directory, even though nothing has a dependency on that directory.

brentleyjones · 2024-05-28T21:23:46Z

Just confirmed that this is an issue for the noskymeld case. We had to disable Skymeld for a 7.2.0rc1 related bug and we immediately hit this issue.

erikkerber added type: bug untriaged labels Sep 21, 2023

erikkerber assigned iancha1992, Pavank1992 and sgowroji Sep 21, 2023

brentleyjones changed the title ~~Execution root symlink forest includes all file in a package instead of only ones needed~~ Execution root symlink forest includes all files in a package instead of only ones needed Sep 21, 2023

iancha1992 added the team-ExternalDeps External dependency handling, remote repositiories, WORKSPACE file. label Sep 21, 2023

iancha1992 unassigned sgowroji, Pavank1992 and iancha1992 Sep 21, 2023

iancha1992 added team-Local-Exec Issues and PRs for the Execution (Local) team and removed team-ExternalDeps External dependency handling, remote repositiories, WORKSPACE file. labels Sep 21, 2023

joeleba self-assigned this Sep 26, 2023

joeleba added P2 We'll consider working on this in future. (Assignee optional) and removed untriaged labels Sep 26, 2023

joeleba added P1 I'll work on this now. (Assignee required) and removed P2 We'll consider working on this in future. (Assignee optional) labels Nov 10, 2023

bazel-io added the potential release blocker Flagged by community members using "@bazel-io flag". Should be added to a release blocker milestone label Nov 10, 2023

bazel-io mentioned this issue Nov 10, 2023

[7.0.0] Execution root symlink forest includes all files in a package instead of only ones needed #20150

Closed

bazel-io removed the potential release blocker Flagged by community members using "@bazel-io flag". Should be added to a release blocker milestone label Nov 10, 2023

copybara-service bot pushed a commit that referenced this issue Nov 13, 2023

Remove the unused git filegroup.

673cb4a

Surfaced in #19581. PiperOrigin-RevId: 581877718 Change-Id: I8d9e460ac809ab236d04252ba14a5836d2e2d86c

bazel-io mentioned this issue Nov 13, 2023

[7.0.0] [Skymeld] Don't plant the symlinks to the paths specified in #20168

Merged

joeleba added P2 We'll consider working on this in future. (Assignee optional) and removed P1 I'll work on this now. (Assignee required) labels Nov 13, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Execution root symlink forest includes all files in a package instead of only ones needed #19581

Execution root symlink forest includes all files in a package instead of only ones needed #19581

erikkerber commented Sep 21, 2023 •

edited

Loading

erikkerber commented Nov 10, 2023

joeleba commented Nov 10, 2023

brentleyjones commented Nov 10, 2023

brentleyjones commented Nov 10, 2023 •

edited

Loading

fmeum commented Nov 10, 2023 •

edited

Loading

joeleba commented Nov 10, 2023

Wyverald commented Nov 10, 2023

keertk commented Nov 10, 2023

joeleba commented Nov 13, 2023

brentleyjones commented Nov 13, 2023

joeleba commented Nov 13, 2023

brentleyjones commented Nov 13, 2023

brentleyjones commented May 28, 2024

Execution root symlink forest includes all files in a package instead of only ones needed #19581

Execution root symlink forest includes all files in a package instead of only ones needed #19581

Comments

erikkerber commented Sep 21, 2023 • edited Loading

Description of the bug:

Which category does this issue belong to?

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

Which operating system are you running Bazel on?

What is the output of bazel info release?

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

What's the output of git remote get-url origin; git rev-parse master; git rev-parse HEAD ?

Is this a regression? If yes, please try to identify the Bazel commit where the bug was introduced.

Have you found anything relevant by searching the web?

Any other information, logs, or outputs that you want to share?

erikkerber commented Nov 10, 2023

joeleba commented Nov 10, 2023

brentleyjones commented Nov 10, 2023

brentleyjones commented Nov 10, 2023 • edited Loading

fmeum commented Nov 10, 2023 • edited Loading

joeleba commented Nov 10, 2023

Wyverald commented Nov 10, 2023

keertk commented Nov 10, 2023

joeleba commented Nov 13, 2023

brentleyjones commented Nov 13, 2023

joeleba commented Nov 13, 2023

brentleyjones commented Nov 13, 2023

brentleyjones commented May 28, 2024

erikkerber commented Sep 21, 2023 •

edited

Loading

What is the output of `bazel info release`?

If `bazel info release` returns `development version` or `(@non-git)`, tell us how you built Bazel.

What's the output of `git remote get-url origin; git rev-parse master; git rev-parse HEAD` ?

brentleyjones commented Nov 10, 2023 •

edited

Loading

fmeum commented Nov 10, 2023 •

edited

Loading