Resolve Windows long path issue during C++ compilation #4149
Comments
meteorcloudy
added
platform: windows
P2
|
I ran into the same limitation while working on #4148. @ulfjack mentioned on 16/11/2017 on an internal mail thread that we are looking into shortening C++ output paths. Here, as well as with #4148, we are duplicating the package path in the output path. AFAICT this is only necessary when I don't know Bazel's C++ compilation machinery enough to tell with certainty whether we could change these output paths, say, to |
|
paging Mr. @mhlopko : what problems do you think we have to solve to get to the state I mentioned in my previous comment? |
|
Can we use a hash string under |
|
We could as well include the object file's name in the hash, so it'd be |
|
Unfortunately I have no idea. I didn't even go that far to find out that we duplicate the path when there are multiple srcs with same basename. How sure are you this is the only case? I was also more interested in In other words, I'm useless to you here :) |
I'm not sure it's the only case, or that it's even the reason we compute paths the way we do. It's just a reason I could think of.
I think you're wrong :P |
|
I'm considering this for Q1-2018 because our users have reported that lack of good documentation and best practices is a real pain. However, I'm leaving it as P2, because there are workaround:
|
|
I've just encountered this issue by myself. I'd like to add that it can be solved in |
|
@excitoon : Thanks for the repro case! We are actively looking for solutions, currently evaluating different approaches. This is one of our highest priority bugs. |
laszlocsomor
added
P1
|
Bumping to P1 because workarounds only get us so far, and it's only a matter of time before more people hit this bug. (Bazel itself started hitting it this week.) |
Have you considered to consequently write response files and then calling |
|
Does that enable the compiler/linker to deal with long paths? |
Invocating Python is pretty slow, and the time will quickly adds up with 1000+ compilation units. Wrapper script is not an ideal solution IMO.
No. In In CMake, it will probably only something like |
I agree, using a wrapper script is not a solution under discussion. We're focusing on how to short the object file path in Bazel. |
That's right, but Python wrapper is already being invoked for each unit on MSVS (it does some flags substitutions and has other logic). |
|
@excitoon : No, there's no Python wrapper for MSVC anymore. Bazel runs the compiler/linker directly. |
No, the wrapper script has been removed since 0.5.3, see 425f249 |
Yes, we made the same observation. @meteorcloudy is currently designing different ways to shorten the object file output paths. Yes, CMake would probably just put the object file next to the source file, which is fine only as long as you don't compile the file with multiple compilers and/or different compiler options within the same build. |
|
I guess, as C++ application, |
|
Yes, @excitoon makes a good point. |
Object file path will no longer be derived from source file path directly. This is a preparation change for[] Related issue bazelbuild#4149 RELNOTES: None PiperOrigin-RevId: 189572213 Change-Id: Iae7ee48e5f1f5500ae03999618598eb9475eaed8
Object file path will no longer be derived from source file path directly. This is a preparation change for[] Related issue #4149 RELNOTES: None PiperOrigin-RevId: 189722421
|
Hey, everyone, this issue has been resolved by bb9ae6a. |
|
@meteorcloudy Just boostraped Bazel with this flag on and it works flawlessly! Will try to build Tensorflow with this flag when I am free (though it looks like Tensorflow's Bazel Windows CI is red again). I notice that the change only affects C++. |
|
\o/ |
|
@rongjiecomputer Glad to hear it worked! I'm working on fixing the TF's Windows build and setting up a presubmit for it. Should be done next week. Yes, this change only affected C++, because it's only |
|
@rongjiecomputer how did you bootstrap Bazel? I tried to build from source with the latest bazel release (0.11) and after 700 seconds of downloading the very first package of the build list i get an error stating there is no such package as androidsdk. I figure this is not the right path and I'm most likely missing something here as I'm new to bazel and just trying to install Tensorflow serving, for which I need the fix that will be implemented in 0.13. |
|
@RickVM I just run |
Closes #4781.
Fix bazelbuild/bazel#4149
RELNOTES:
Users can now pass --experimental_shortened_obj_file_path=true to have a shorter object file path, the object file paths (and all other related paths) will be constructed as following:
If there's no two or more source files with the same base name:
<bazel-bin>/<target_package_path>/_objs/<target_name>/<source_base_name>.<extension>
otherwise:
<bazel-bin>/<target_package_path>/_objs/<target_name>/N/<source_base_name>.<extension>
N = the file?s order among the source files with the same basename, starts from 0.
Examples:
1. Output names for ["lib1/foo.cc", "lib2/bar.cc"] are ["foo", "bar"]
2. Output names for ["foo.cc", "bar.cc", "foo.cpp", "lib/foo.cc"] are
["0/foo", "bar", "1/foo", "2/foo"]
The default value of --experimental_shortened_obj_file_path option is false, but we plan to flip it to true and eventually remove this option.
You shouldn't depend on the format of generated object file path, but if you do and this change breaks you, please use --experimental_shortened_obj_file_path=false to work around it.
PiperOrigin-RevId: 190214375
Object file path will no longer be derived from source file path directly.
This is a preparation change for[]
Related issue bazelbuild/bazel#4149
RELNOTES: None
PiperOrigin-RevId: 189722421
|
A note here, the above link is due to the constructed include path being too long. It would be possible to shorten those paths if, when bazel constructed a virtual includes path, instead of constructing it here:
It constructed as a new path under bazel_output_base:
Then the include path would readlink that instead of using the files directly. |
|
@laramiel This looks like a nice improvement for Bazel, can you file a separate issue as feature request? |
The idea here is to set the existing config "config_msvc" not only when "msvc-cl" is specified but also when "clang-cl" is specified. Supporting clang-cl should be a quick workaround for those who look for a quick solution to cl.exe's path limitation e.g. * bazelbuild/bazel#4149 * bazelbuild/bazel#18683 PiperOrigin-RevId: 630590450
https://stackoverflow.com/questions/47412512/bazel-build-nccl-archive-nccl-could-not-resolve-label-ws2-32-lib
When I tried to help a user build tensorflow_serving on Windows, I encountered an error
Cannot open compiler generated file: '': Invalid argument:It turns out that the object file the compiler trying to write
C:/tmp/_bazel_pcloudy/dpitg86y/execroot/bazel-out/msvc_x64-py3-opt/bin/external/org_tensorflow/tensorflow/core/profiler/internal/advisor/_objs/internal_checker_runner_dummy/external/org_tensorflow/tensorflow/core/profiler/internal/advisor/internal_checker_runner_dummy.ois longer than 270 characters.This happens when I already set the output directory to a short path
C:/tmp.The text was updated successfully, but these errors were encountered: