Fix (and reenable) run_test.test_consistent_command_line_encoding

[dslomov]

Broken on CI like this:
http://ci.bazel.io/job/bazel-tests/BAZEL_VERSION=HEAD,PLATFORM_NAME=linux-x86_64/223/testReport/junit/(root)/(empty)/test_consistent_command_line_encoding/

(other platforms too).
Works locally on some machines.
Probably related to #1766

[aehlig]

Works locally on some machines.

Some remarks:

• At least on our CI, it seems to coincide with the java version (the older
  one, running on ubuntu 14.04, causing the problem).
• The sequence '\xC3\xA4\xC3\xB6\xC3\xBC\xC3\x84\xC3\x96\xC3\x9C\xC3\x9F'
  gets "re-encoded" as '??????????????'---literally 14 question marks (0x3F)!

Klaus Aehlig
Google Germany GmbH, Erika-Mann-Str. 33, 80636 Muenchen
Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg
Geschaeftsfuehrer: Matthew Scott Sucherman, Paul Terence Manicle

[aehlig]

The error happens before RunCommand does anything. printf-debugging with the
following diff

diff --git a/src/main/java/com/google/devtools/build/lib/runtime/commands/RunCommand.java b/src/main/java/com/google/devtools/build/lib/runtime/commands/RunCommand.java
index 26dda27..e3d761f 100644
--- a/src/main/java/com/google/devtools/build/lib/runtime/commands/RunCommand.java
+++ b/src/main/java/com/google/devtools/build/lib/runtime/commands/RunCommand.java
@@ -14,6 +14,10 @@

 package com.google.devtools.build.lib.runtime.commands;

+import static java.nio.charset.StandardCharsets.ISO_8859_1;
+import static java.nio.charset.StandardCharsets.UTF_8;
+
+
 import com.google.common.annotations.VisibleForTesting;
 import com.google.common.collect.ImmutableList;
 import com.google.common.collect.Lists;
@@ -130,6 +134,23 @@ public class RunCommand implements BlazeCommand  {
     // This list should look like: ["//executable:target", "arg1", "arg2"]
     List<String> targetAndArgs = options.getResidue();

+    // DEBUG ONLY
+    for (String arg: targetAndArgs) {
+      System.out.println("XXX");
+      System.out.println(arg);
+      System.out.print("UTF-8 bytes: ");
+      for (byte b : arg.getBytes(UTF_8)) {
+        System.out.print(String.format("%02x ", b));
+      }
+      System.out.println();
+      System.out.print("ISO-8859-1 bytes: ");
+      for (byte b : arg.getBytes(ISO_8859_1)) {
+        System.out.print(String.format("%02x ", b));
+      }
+      System.out.println();
+
+    }
+
     // The user must at the least specify an executable target.
     if (targetAndArgs.isEmpty()) {

Shows that the questionmarks already fall out of the parser (with
the problem probably being earlier); indeed, in the log of the failing
test we find the following.

WARNING: Running Bazel server needs to be killed, because the startup options are different.
XXX
//foo
UTF-8 bytes: 2f 2f 66 6f 6f
ISO-8859-1 bytes: 2f 2f 66 6f 6f
XXX
??????????????
UTF-8 bytes: ef bf bd ef bf bd ef bf bd ef bf bd ef bf bd ef bf bd ef bf bd ef bf bd ef bf bd ef bf bd ef bf bd ef bf bd ef bf bd ef bf bd
ISO-8859-1 bytes: 3f 3f 3f 3f 3f 3f 3f 3f 3f 3f 3f 3f 3f 3f
____Loading package: foo
____Loading...

Klaus Aehlig
Google Germany GmbH, Erika-Mann-Str. 33, 80636 Muenchen
Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg
Geschaeftsfuehrer: Matthew Scott Sucherman, Paul Terence Manicle

[aehlig]

Further printf-debugging shows that the problem is in the way bazel is started.

• In src/main/cpp/blaze_util_posix.cc b/src/main/cpp/blaze_util_posix.cc
  argv[n-1] consists of the octets c3 a4 c3 b6 c3 bc c3 84 c3 96 c3 9c c3 9f 00.
  With that argv, we do execv(exe.c_str(), const_cast<char **>(argv));
• In src/main/java/com/google/devtools/build/lib/bazel/BazelMain.java's
  public static void main(String[] args), the last String of args consists
  of question marks (UTF-8 bytes: ef bf bd ef bf bd ef bf bd ef bf bd ef bf
  bd ef bf bd ef bf bd ef bf bd ef bf bd ef bf bd ef bf bd ef bf bd ef bf bd
  ef bf bd, ISO-8859-1 bytes: 3f 3f 3f 3f 3f 3f 3f 3f 3f 3f 3f 3f 3f 3f).

Klaus Aehlig
Google Germany GmbH, Erika-Mann-Str. 33, 80636 Muenchen
Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg
Geschaeftsfuehrer: Matthew Scott Sucherman, Paul Terence Manicle

[aehlig]

OK, so adding

setenv("LANG", "en_US.UTF-8", 1);
setenv("LC_ALL", "en_US.UTF-8", 1);

in front of the the execv in src/main/cpp/blaze_util_posix.cc (and
including <stdlib.h>) the java invocation gets the arguments without
information loss: the String obtained has as UTF-8 encoding the correct
bytes c3 a4 c3 b6 c3 bc c3 84 c3 96 c3 9c c3 9f. However, when the
the subprocess is called, the string gets rewritten in the default
encoding (which we set to ISO-8859-1) and there it is the octet
sequence e4 f6 fc c4 d6 dc df.

On the other hand, setting the locale to ISO-8859-1 or C, the startup
of java already rewrites the argument to the question marks, so we have
no chance of recovering the original information.

So at the moment, I have no idea on how to fix that encoding mess. On
the other hand, that problem probably has existed since bazel was open
sourced, so probably it is not that urgent.

Klaus Aehlig
Google Germany GmbH, Erika-Mann-Str. 33, 80636 Muenchen
Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg
Geschaeftsfuehrer: Matthew Scott Sucherman, Paul Terence Manicle

[sgowroji]

Hi there! We're doing a clean up of old issues and will be closing this one. Please reopen if you’d like to discuss anything further. We’ll respond as soon as we have the bandwidth/resources to do so.