Bazel+RBE failure on Windows with target of length 89 or 90

[veblush]

Description of the problem:

I ran into a weird error when running a test through Bazel RBE on Windows. There is the issue describing it in detail. Its summary is that SymInitialize (Windows Debug API) fails only when the length of the target is 89 or 90. When the target name is shortened or lengthened, this issue is fixed. So something related to the length should be there.

This is a PR to reproduce this simply and I haven't seen this problem when running Bazel directly on Windows. The Bazel RBE team investigated this problem and this might be a result of combination of bazel and docker so I like to get help from the Bazel team.

I guess changing the threshold (from 260 to 258) to use shortened paths on Windows can fix this issue. code

Internal bug with details: b/169902891

Feature requests: what underlying problem are you trying to solve with this feature?

SymInitialize failed only when the length of path is 89 and 90

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

grpc/grpc#24302

What operating system are you running Bazel on?

Windows server 2016

What's the output of bazel info release?

I cannot run bazel info but the version of bazel is 2.2.0.

[jin]

Is this still a P1 bug?

[veblush]

@jin Yep, this is still affecting gRPC debugging.

[meteorcloudy]

@veblush Can you help verify if #12941 fixes this problem? You can cherry-pick this change and build a custom Bazel binary based the Bazel version you were using. Let me know if I can help verify this!

[veblush]

@meteorcloudy Thank for the fix! I don't know how to use a custom build Bazel, though. Have you tried to build my reproduction case described here?

[meteorcloudy]

@veblush

I couldn't reproduce because I don't have access to required RBE resources.

I prepared a branch with the fix based on Bazel 2.2.0:
https://github.com/bazelbuild/bazel/tree/gh-12310-fix-on-2.2.0

Can you fetch this branch and built a custom Bazel binary via bazel build //src:bazel?

[veblush]

@meteorcloudy

Thanks for the fix and I tried to build it on kokoro winserver2016 but failed.

$ bazel --version
bazel 3.7.1

$ git status
On branch gh-12310-fix-on-2.2.0
Your branch is up to date with 'origin/gh-12310-fix-on-2.2.0'.

$ java -version
java version "9.0.1"
Java(TM) SE Runtime Environment (build 9.0.1+11)
Java HotSpot(TM) 64-Bit Server VM (build 9.0.1+11, mixed mode)

$ bazel build //src:bazel-dev

ERROR: C:/cygwin64/home/kbuilder/bazel/src/main/java/com/google/devtools/build/lib/concurrent/BUILD:12:13: Building src/main/java/com/google/devtools/build/lib/concurrent/libconcurrent.jar (22 source files) failed (Exit 1): java.exe failed: error executing command external/remotejdk11_win/bin/java.exe -XX:+UseParallelOldGC -XX:-CompactStrings --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED ... (remaining 15 argument(s) skipped)
src\main\java\com\google\devtools\build\lib\concurrent\MultisetSemaphore.java:89: error: [PreconditionsInvalidPlaceholder] Preconditions only accepts the %s placeholder in error message strings
          "maxNumUniqueValues must be positive (was %d)",
          ^
    (see https://errorprone.info/bugpattern/PreconditionsInvalidPlaceholder)
  Did you mean '"maxNumUniqueValues must be positive (was %s)",'?
Target //src:bazel-dev failed to build

[mai93]

Can you try building it with Bazel 2.1.0? I managed to build this branch using this version.

[meteorcloudy]

@veblush Yeah, like Mai said, can you try to build Bazel 2.2.0 with Bazel 2.1.0, the latest version of Bazel is sometimes unable to build old version of Bazel due to some breaking changes.

[veblush]

Sorry for the late response. I tried it with bazel 2.1.0 but it wasn't able to build it.

$ choco install bazel --version=2.1.0 --force
$ bazel --version
bazel 2.1.0
$ bazel clean --expunge
$ bazel build //src:bazel-dev 
...
ERROR: C:/cygwin64/home/kbuilder/bazel/src/BUILD:321:2: Executing genrule //src:package-zip_jdk_allmodules failed (Exit 2)
bazel-out/host/bin/src/package-bazel-on-host-platform.sh: line 16: $'\r': command not found
: invalid option namec/package-bazel-on-host-platform.sh: line 17: set: pipefail
Target //src:bazel-dev failed to build

[veblush]

You can reproduce this without using bazel_rbe which requires a right credential. Can you try it on windows?

> git clone https://github.com/grpc/grpc.git
> cd grpc
> git fetch origin pull/24302/head:stacktrace-long
> git checkout stacktrace-long
> bazel test //test/core/extra:all

[meteorcloudy]

bazel-out/host/bin/src/package-bazel-on-host-platform.sh: line 16: $'\r': command not found\

Hmm, looks like your files may have the Windows line ending CRLF (\n\r), you can tell git to always use LF by git config --global core.eol lf or just turn off the auto converting git config --global core.autocrlf false

@mai93 Do I understand correctly that you actually verified the fix worked in the RBE build, right?

[mai93]

Yes, all tests passed with the fix but I was working with Bazel 3.7.2. With Bazel 2.2.0 + the fix, I get the following error:

ERROR: C:/users/lenovo/_bazel_lenovo/hhw2zfl5/external/upb/bazel/upb_proto_library.bzl:257:29: aspect() got unexpected keyword argument 'incompatible_use_toolchain_transition'
ERROR: while parsing '//test/core/extra:all': error loading package 'test/core/extra': in D:/bazel/bug-12310/ooo/grpc/bazel/grpc_build_system.bzl: Extension file 'bazel/upb_proto_library.bzl' has errors

[meteorcloudy]

Nice. The error with 2.2.0 should be caused by something else. I think we can merge your fix now.