Windows: protoc.exe built with MSVC cannot handle long path

[meteorcloudy]

Related: #2627
Got following error when enable MSVC as the default toolchain on Windows.

ERROR: C:/tools/msys64/home/pcloudy/workspace/bazel/src/main/java/com/google/devtools/build/lib/buildeventstream/proto/BUILD:10:1: error executing shell command: 'JAR='external/local_jdk/bin/jar.exe' OUTPUT='bazel-out/msvc_x64-fastbuild/bin/src/main/java/com/google/devtools/build/lib/buildeventstream/proto/libbuild_event_stream_java_proto_srcjar.srcjar' PROT...' failed: bash.exe failed: error executing command
  cd C:/tmp/_bazel_pcloudy/7uxoax_v/execroot/bazel
  SET PATH=C:\Program Files\Anaconda3;C:\tools\msys64\home\pcloudy\bin;C:\Program Files\Java\jdk1.8.0_77\bin;C:\tools\msys64\usr\local\bin;C:\tools\msys64\usr\bin;C:\tools\msys64\usr\bin;C:\tools\msys64\opt\bin;C:\Windows\System32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\tools\msys64\usr\bin\site_perl;C:\tools\msys64\usr\bin\vendor_perl;C:\tools\msys64\usr\bin\core_perl
  C:/tools/msys64/usr/bin/bash.exe -c JAR='external/local_jdk/bin/jar.exe' OUTPUT='bazel-out/msvc_x64-fastbuild/bin/src/main/java/com/google/devtools/build/lib/buildeventstream/proto/libbuild_event_stream_java_proto_srcjar.srcjar' PROTO_COMPILER='bazel-out/host/bin/third_party/protobuf/3.0.0/protoc.exe'  SOURCE='src/main/java/com/google/devtools/build/lib/buildeventstream/proto/build_event_stream.proto' bazel-out/host/bin/tools/build_rules/gensrcjar: com.google.devtools.build.lib.shell.BadExitStatusException: Process exited with status 1.
com/google/devtools/build/lib/buildeventstream/BuildEventStreamProtos.java: while trying to create directory bazel-out/msvc_x64-fastbuild/bin/src/main/java/com/google/devtools/build/lib/buildeventstream/proto/libbuild_event_stream_java_proto_srcjar.srcjar.proto_output/com/google/devtools/build/lib/buildeventstream: No such file or directory
gensrcjar: proto_compiler failed
ERROR: C:/tools/msys64/home/pcloudy/workspace/bazel/src/main/protobuf/BUILD:25:2: output 'src/main/protobuf/libcrosstool_config_java_proto_srcjar.srcjar' was not created.
ERROR: C:/tools/msys64/home/pcloudy/workspace/bazel/src/main/protobuf/BUILD:25:2: output 'src/main/protobuf/libintellij_ide_info_java_proto_srcjar.srcjar' was not created.
ERROR: C:/tools/msys64/home/pcloudy/workspace/bazel/src/main/protobuf/BUILD:38:1: output 'src/main/protobuf/libcommand_server_java_proto_srcjar.srcjar' was not created.
ERROR: C:/tools/msys64/home/pcloudy/workspace/bazel/src/main/protobuf/BUILD:25:2: output 'src/main/protobuf/libinvocation_policy_java_proto_srcjar.srcjar' was not created.

I believe the reason is because proto compiler built with MSVC cannot handle long path. The workaround could be using MSYS to build proto.exe, we can still get a msys free bazel in this way, but cannot build bazel without MSYS unless we solve this.

Is it an issue of protobuf instead of Bazel? //cc @dslomov @laszlocsomor

[laszlocsomor]

How do I repro this?

[meteorcloudy]

Add --cpu=x64_windows_msvc --host_cpu=x64_windows_msvc when building Bazel, you'll then get an error like this:
ERROR: missing input file '@local_jdk//:bin/jar'.

It can be fixed by applying the following change. Then build a bazel with this change, use that bazel to run bazel build src:bazel --cpu=x64_windows_msvc --host_cpu=x64_windows_msvc again.

diff --git a/src/main/tools/jdk.BUILD b/src/main/tools/jdk.BUILD
index 87147b5ae..16696913e 100644
--- a/src/main/tools/jdk.BUILD
+++ b/src/main/tools/jdk.BUILD
@@ -24,6 +24,7 @@ filegroup(
     name = "java",
     srcs = select({
        ":windows" : ["bin/java.exe"],
+       ":windows_msvc" : ["bin/java.exe"],
        "//conditions:default" : ["bin/java"],
     }),
 )
@@ -32,6 +33,7 @@ filegroup(
     name = "jar",
     srcs = select({
        ":windows" : ["bin/jar.exe"],
+       ":windows_msvc" : ["bin/jar.exe"],
        "//conditions:default" : ["bin/jar"],
     }),
 )
@@ -40,6 +42,7 @@ filegroup(
     name = "javac",
     srcs = select({
         ":windows" : ["bin/javac.exe"],
+        ":windows_msvc" : ["bin/javac.exe"],
         "//conditions:default" : ["bin/javac"],
     }),
 )
@@ -86,6 +89,7 @@ filegroup(
         # common antivirus software blocks access to npjp2.dll interfering with Bazel,
         # so do not include it in JRE on Windows.
         ":windows" : glob(["jre/bin/**"], exclude = ["jre/bin/plugin2/**"]),
+        ":windows_msvc" : glob(["jre/bin/**"], exclude = ["jre/bin/plugin2/**"]),
         "//conditions:default" : glob(["jre/bin/**"])
     }),
 )
@@ -164,3 +168,9 @@ config_setting(
     values = {"cpu": "x64_windows"},
     visibility = ["//visibility:private"],
 )
+
+config_setting(
+    name = "windows_msvc",
+    values = {"cpu": "x64_windows_msvc"},
+    visibility = ["//visibility:private"],
+)

[laszlocsomor]

I have a fix for this bug.

The problem is that the proto compiler uses POSIX open(2) to open files which on Windows is declared in <io.h> and uses the ANSI Windows API, not the Unicode one. Fortunately the _w variant (_wopen) uses the Unicode API and works with UNC-prefixed Windows paths.

[meteorcloudy]

Nice, that's good to know!