Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Windows: Bazel cannot share cpp cache between two projects with different workspace name #9172

Closed
meteorcloudy opened this issue Aug 14, 2019 · 5 comments
Assignees
Labels
area-Windows Windows-specific issues and feature requests P1 I'll work on this now. (Assignee required) team-OSS Issues for the Bazel OSS team: installation, release processBazel packaging, website type: bug

Comments

@meteorcloudy
Copy link
Member

If you try to share cache between two projects with different workspace name, when the cache is hit for a C++ compilation, the header check will fail because the header paths printed by MSVC's /showInclude option are absolute paths, which contains the workspace name.

You can reproduce with:
https://github.com/meteorcloudy/my_tests/tree/master/cpp_cache_with_different_workspace_name

$ cd A
$ bazel build --disk_cache=C:/src/tmp/bazel_disk_cache :hello
$ cd ../B
$ bazel build --disk_cache=C:/src/tmp/bazel_disk_cache :hello
Starting local Bazel server and connecting to it...
INFO: Invocation ID: e98dabd3-b255-4c34-86e5-ab5070b346f3
INFO: Analyzed target //:hello (13 packages loaded, 72 targets configured).
INFO: Found 1 target...
ERROR: C:/tools/msys64/home/pcloudy/workspace/my_tests/cpp_cache_with_different_workspace_name/b/BUILD:1:1: undeclared inclusion(s) in rule '//:hello':
this rule is missing dependency declarations for the following files included by 'hello.cc':
  'C:/users/pcloudy/_bazel_pcloudy/7tppxn5m/execroot/a/hello.h'
Target //:hello failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 10.570s, Critical Path: 0.09s
INFO: 0 processes.
FAILED: Build did NOT complete successfully

Note the error message when we build //:hello in B,
'C:/users/pcloudy/_bazel_pcloudy/7tppxn5m/execroot/a/hello.h' is from project A.

This is the root cause of https://buildkite.com/bazel/bazel-federation/builds/19#ee3795da-1088-4c27-b81e-2636e30e3587

FYI @fweikert @laszlocsomor

@meteorcloudy meteorcloudy added type: bug P1 I'll work on this now. (Assignee required) area-Windows Windows-specific issues and feature requests labels Aug 14, 2019
@meteorcloudy meteorcloudy self-assigned this Aug 14, 2019
@meteorcloudy
Copy link
Member Author

We had a similar bug before because output base was different. #6847 (comment)
That issue was fixed at a1aa5c1, but it doesn't take different workspace names into consideration.

@meteorcloudy
Copy link
Member Author

So the question is, should we allow sharing cache between two projects with different workspace name?

@laszlocsomor
Copy link
Contributor

Thanks for the report!

So the question is, should we allow sharing cache between two projects with different workspace name?

Why wouldn't we?

@meteorcloudy
Copy link
Member Author

Why wouldn't we?

Exactly, I don't have any good reason to object it.
Then, I might know how to fix this, will send a change today.

@fweikert
Copy link
Member

Nice, thank you Yun!

@philwo philwo added the team-OSS Issues for the Bazel OSS team: installation, release processBazel packaging, website label Jun 15, 2020
coeuvre added a commit to coeuvre/bazel that referenced this issue Apr 13, 2021
…y default.

When --experimental_sibling_external_layout is set, use the parent directory of execRoot as input root and set working directory to the base name of execRoot. Paths of output files are relative to working directory. When --incompatible_remote_paths_relative_to_input_root is set, paths of output files are relative to input root.

Introduce RemotePathResolver which is used to convert local paths to remote paths and vice versa. We should prefer this class instead of using execRoot directly whenever possible. execRoot usages in RemoteCache are all replaced.

On Windows, shared action results cache for cl.exe across different workspaces causing header dependency checking to fail. This was initially fixed by bazelbuild#9172, but is broken after cl/356735700. The reason test didn't fail is two builds from different workspaces do not share the cache since input files are relative to the parent of exec root which contains workspace name. This change fixes that by adding workspace name as platform property for action running on Windows.

Fixes bazelbuild#13188.
coeuvre added a commit to coeuvre/bazel that referenced this issue Apr 14, 2021
…y default.

When --experimental_sibling_external_layout is set, use the parent directory of execRoot as input root and set working directory to the base name of execRoot. Paths of output files are relative to working directory. When --incompatible_remote_paths_relative_to_input_root is set, paths of output files are relative to input root.

Introduce RemotePathResolver which is used to convert local paths to remote paths and vice versa. We should prefer this class instead of using execRoot directly whenever possible. execRoot usages in RemoteCache are all replaced.

On Windows, shared action results cache for cl.exe across different workspaces causing header dependency checking to fail. This was initially fixed by bazelbuild#9172, but is broken after cl/356735700. The reason test didn't fail is two builds from different workspaces do not share the cache since input files are relative to the parent of exec root which contains workspace name. This change fixes that by adding workspace name as platform property for action running on Windows.

Fixes bazelbuild#13188.
coeuvre added a commit to coeuvre/bazel that referenced this issue Apr 19, 2021
…y default.

When --experimental_sibling_external_layout is set, use the parent directory of execRoot as input root and set working directory to the base name of execRoot. Paths of output files are relative to working directory. When --incompatible_remote_paths_relative_to_input_root is set, paths of output files are relative to input root.

Introduce RemotePathResolver which is used to convert local paths to remote paths and vice versa. We should prefer this class instead of using execRoot directly whenever possible. execRoot usages in RemoteCache are all replaced.

On Windows, shared action results cache for cl.exe across different workspaces causing header dependency checking to fail. This was initially fixed by bazelbuild#9172, but is broken after cl/356735700. The reason test didn't fail is two builds from different workspaces do not share the cache since input files are relative to the parent of exec root which contains workspace name. This change fixes that by adding workspace name as platform property for action running on Windows.

Fixes bazelbuild#13188.
bazel-io pushed a commit that referenced this issue Apr 19, 2021
…y default.

When --experimental_sibling_external_layout is set, use the parent directory of execRoot as input root and set working directory to the base name of execRoot. Paths of output files are relative to working directory.

When --incompatible_remote_paths_relative_to_input_root is set, paths of output files are relative to input root.

Introduce RemotePathResolver which is used to convert local paths to remote paths and vice versa. We should prefer this class instead of using execRoot directly whenever possible in remote module. execRoot usages in RemoteCache are all replaced.

On Windows, shared action results cache for cl.exe across different workspaces causing header dependency checking to fail. This was initially fixed by #9172, but is broken after 24c980b. The reason test didn't fail before this change is that two builds from different workspaces do not share the cache since input files are relative to the parent of exec root which contains workspace name. This change fixes that by adding workspace name as platform property for action running on Windows.

Fixes #13188.

Closes #13339.

PiperOrigin-RevId: 369168230
coeuvre added a commit to coeuvre/bazel that referenced this issue Jul 15, 2021
…y default.

When --experimental_sibling_external_layout is set, use the parent directory of execRoot as input root and set working directory to the base name of execRoot. Paths of output files are relative to working directory.

When --incompatible_remote_paths_relative_to_input_root is set, paths of output files are relative to input root.

Introduce RemotePathResolver which is used to convert local paths to remote paths and vice versa. We should prefer this class instead of using execRoot directly whenever possible in remote module. execRoot usages in RemoteCache are all replaced.

On Windows, shared action results cache for cl.exe across different workspaces causing header dependency checking to fail. This was initially fixed by bazelbuild#9172, but is broken after bazelbuild@24c980b. The reason test didn't fail before this change is that two builds from different workspaces do not share the cache since input files are relative to the parent of exec root which contains workspace name. This change fixes that by adding workspace name as platform property for action running on Windows.

Fixes bazelbuild#13188.

Closes bazelbuild#13339.

PiperOrigin-RevId: 369168230
coeuvre added a commit to coeuvre/bazel that referenced this issue Jul 15, 2021
…y default.

When --experimental_sibling_external_layout is set, use the parent directory of execRoot as input root and set working directory to the base name of execRoot. Paths of output files are relative to working directory.

When --incompatible_remote_paths_relative_to_input_root is set, paths of output files are relative to input root.

Introduce RemotePathResolver which is used to convert local paths to remote paths and vice versa. We should prefer this class instead of using execRoot directly whenever possible in remote module. execRoot usages in RemoteCache are all replaced.

On Windows, shared action results cache for cl.exe across different workspaces causing header dependency checking to fail. This was initially fixed by bazelbuild#9172, but is broken after bazelbuild@24c980b. The reason test didn't fail before this change is that two builds from different workspaces do not share the cache since input files are relative to the parent of exec root which contains workspace name. This change fixes that by adding workspace name as platform property for action running on Windows.

Fixes bazelbuild#13188.

Closes bazelbuild#13339.

PiperOrigin-RevId: 369168230
coeuvre added a commit to coeuvre/bazel that referenced this issue Jul 16, 2021
…y default.

When --experimental_sibling_external_layout is set, use the parent directory of execRoot as input root and set working directory to the base name of execRoot. Paths of output files are relative to working directory.

When --incompatible_remote_paths_relative_to_input_root is set, paths of output files are relative to input root.

Introduce RemotePathResolver which is used to convert local paths to remote paths and vice versa. We should prefer this class instead of using execRoot directly whenever possible in remote module. execRoot usages in RemoteCache are all replaced.

On Windows, shared action results cache for cl.exe across different workspaces causing header dependency checking to fail. This was initially fixed by bazelbuild#9172, but is broken after bazelbuild@24c980b. The reason test didn't fail before this change is that two builds from different workspaces do not share the cache since input files are relative to the parent of exec root which contains workspace name. This change fixes that by adding workspace name as platform property for action running on Windows.

Fixes bazelbuild#13188.

Closes bazelbuild#13339.

PiperOrigin-RevId: 369168230
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area-Windows Windows-specific issues and feature requests P1 I'll work on this now. (Assignee required) team-OSS Issues for the Bazel OSS team: installation, release processBazel packaging, website type: bug
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants