failed to install bazel on Red Hat 6.7 #760

digitalsword · 2016-01-08T14:33:06Z

I would like to install bazel from source, and use bazel to compile tensorflow on a cluster running redhat 6.7. When I try to install bazel, the glibc version (2.12) is too old. I do not have root access to the cluster. Is it possible to install tensorflow in this case?

My system information:

-bash-4.1$ cat /etc/redhat-release
Red Hat Enterprise Linux Server release 6.7 (Santiago)
-bash-4.1$ which gcc
/usr/bin/gcc
-bash-4.1$ gcc -v
Using built-in specs.
Target: x86_64-redhat-linux
Configured with: ../configure --prefix=/usr --mandir=/usr/share/man --infodir=/usr/share/info --with-bugurl=http://bugzilla.redhat.com/bugzilla --enable-bootstrap --enable-shared --enable-threads=posix --enable-checking=release --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions --enable-gnu-unique-object --enable-languages=c,c++,objc,obj-c++,java,fortran,ada --enable-java-awt=gtk --disable-dssi --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-1.5.0.0/jre --enable-libgcj-multifile --enable-java-maintainer-mode --with-ecj-jar=/usr/share/java/eclipse-ecj.jar --disable-libjava-multilib --with-ppl --with-cloog --with-tune=generic --with-arch_32=i686 --build=x86_64-redhat-linux
Thread model: posix
gcc version 4.4.7 20120313 (Red Hat 4.4.7-16) (GCC) 
-bash-4.1$ ldd --version
ldd (GNU libc) 2.12

The system has newer gcc installed as well. I tried using it, bazel still won't compile.

-bash-4.1$ /usr/local/gcc/4.8.4/bin/gcc -v
Using built-in specs.
COLLECT_GCC=/usr/local/gcc/4.8.4/bin/gcc
COLLECT_LTO_WRAPPER=/usr/local/gcc/4.8.4/libexec/gcc/x86_64-unknown-linux-gnu/4.8.4/lto-wrapper
Target: x86_64-unknown-linux-gnu
Configured with: ../configure --prefix=/usr/local/gcc/4.8.4
Thread model: posix
gcc version 4.8.4 (GCC)

When I was compiling bazel using gcc 4.8.4, I got the following error:

bazel-0.1.1/_bin/build-runfiles: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.14' not found

How can I install the missing dependency locally, and have bazel pick up the right version of glib? What part of tools/cpp/CROSSTOOL should I modify? What environment variables should I set before ./compile.sh

I am trying to install bazel-0.1.1 instead of bazel-0.1.2 because the latter does not compile tensorflow.
tensorflow/tensorflow#469

The text was updated successfully, but these errors were encountered:

damienmg · 2016-01-11T09:54:31Z

Have you updated the path to GCC / libraris in the tools/cpp/CROSSTOOL file?
You should also set correctly CXX, CC, LDFLAGS environment variables and co.

digitalsword · 2016-01-11T14:35:51Z

@damienmg Yes, I have tried. But there are many entries in tools/cpp/CROSSTOOl. I am not sure I have updated everything.

I tried the following but it didn't work.

In CROSSTOOL, I changed cpp path, gcc path and added cxx_builtin_include_directory: "/usr/local/gcc/4.8.4/lib64/", cxx_builtin_include_directory: "/usr/local/gcc/4.8.4/lib/"

toolchain {
  abi_version: "local"
                       abi_libc_version: "local"
                       builtin_sysroot: ""
  compiler: "compiler"
  host_system_name: "local"
  needsPic: true
  supports_gold_linker: false
  supports_incremental_linker: false
  supports_fission: false
  supports_interface_shared_objects: false
  supports_normalizing_ar: false
  supports_start_end_lib: false
  supports_thin_archives: false
  target_libc: "local"
  target_cpu: "local" 
  target_system_name: "local"
  toolchain_identifier: "local_linux"

  tool_path { name: "ar" path: "/usr/bin/ar" }
  tool_path { name: "compat-ld" path: "/usr/bin/ld" }
  tool_path { name: "cpp" path: "/usr/local/gcc/4.8.4/bin/g++" }
  tool_path { name: "dwp" path: "/usr/bin/dwp" }
  tool_path { name: "gcc" path: "/usr/local/gcc/4.8.4/bin/gcc" }
  cxx_flag: "-std=c++0x"
  linker_flag: "-lstdc++"
  linker_flag: "-B/usr/bin/"

  # TODO(bazel-team): In theory, the path here ought to exactly match the path
  # used by gcc. That works because bazel currently doesn't track files at
  # absolute locations and has no remote execution, yet. However, this will need
  # to be fixed, maybe with auto-detection?
  cxx_builtin_include_directory: "/usr/local/gcc/4.8.4/lib64/"
  cxx_builtin_include_directory: "/usr/local/gcc/4.8.4/lib/"
  cxx_builtin_include_directory: "/usr/lib/gcc/"
  cxx_builtin_include_directory: "/usr/local/include"
  cxx_builtin_include_directory: "/usr/include"
  tool_path { name: "gcov" path: "/usr/bin/gcov" }

  # C(++) compiles invoke the compiler (as that is the one knowing where
  # to find libraries), but we provide LD so other rules can invoke the linker.
  tool_path { name: "ld" path: "/usr/bin/ld" }

  tool_path { name: "nm" path: "/usr/bin/nm" }
  tool_path { name: "objcopy" path: "/usr/bin/objcopy" }
  objcopy_embed_flag: "-I"
  objcopy_embed_flag: "binary"
  tool_path { name: "objdump" path: "/usr/bin/objdump" }
  tool_path { name: "strip" path: "/usr/bin/strip" }

  # Anticipated future default.
  unfiltered_cxx_flag: "-no-canonical-prefixes"

  # Make C++ compilation deterministic. Use linkstamping instead of these
  # compiler symbols.
  unfiltered_cxx_flag: "-Wno-builtin-macro-redefined"
  unfiltered_cxx_flag: "-D__DATE__=\"redacted\""
  unfiltered_cxx_flag: "-D__TIMESTAMP__=\"redacted\""
  unfiltered_cxx_flag: "-D__TIME__=\"redacted\""

  # Security hardening on by default.
  # Conservative choice; -D_FORTIFY_SOURCE=2 may be unsafe in some cases.
  # We need to undef it before redefining it as some distributions now have
  # it enabled by default.
  compiler_flag: "-U_FORTIFY_SOURCE"
  compiler_flag: "-D_FORTIFY_SOURCE=1"
  compiler_flag: "-fstack-protector"
  compiler_flag: "-fPIE"
  linker_flag: "-pie"
  linker_flag: "-Wl,-z,relro,-z,now"

  # Enable coloring even if there's no attached terminal. Bazel removes the
  # escape sequences if --nocolor is specified. This isn't supported by gcc
  # on Ubuntu 14.04.
  # compiler_flag: "-fcolor-diagnostics"

  # All warnings are enabled. Maybe enable -Werror as well?
  compiler_flag: "-Wall"
  # Enable a few more warnings that aren't part of -Wall.
  compiler_flag: "-Wunused-but-set-parameter"
  # But disable some that are problematic.
  compiler_flag: "-Wno-free-nonheap-object" # has false positives

  # Keep stack frames for debugging, even in opt mode.
  compiler_flag: "-fno-omit-frame-pointer"

  # Anticipated future default.
  linker_flag: "-no-canonical-prefixes"
  # Have gcc return the exit code from ld.
  linker_flag: "-pass-exit-codes"
  # Stamp the binary with a unique identifier.
  linker_flag: "-Wl,--build-id=md5"
  linker_flag: "-Wl,--hash-style=gnu"
  # Gold linker only? Can we enable this by default?
  # linker_flag: "-Wl,--warn-execstack"
  # linker_flag: "-Wl,--detect-odr-violations"

  compilation_mode_flags {
    mode: DBG
    # Enable debug symbols.
    compiler_flag: "-g"
  }
  compilation_mode_flags {
    mode: OPT

    # No debug symbols.
    # Maybe we should enable https://gcc.gnu.org/wiki/DebugFission for opt or
    # even generally? However, that can't happen here, as it requires special
    # handling in Bazel.
    compiler_flag: "-g0"

    # Conservative choice for -O
    # -O3 can increase binary size and even slow down the resulting binaries.
    # Profile first and / or use FDO if you need better performance than this.
    compiler_flag: "-O2"

    # Disable assertions
    compiler_flag: "-DNDEBUG"

    # Removal of unused code and data at link time (can this increase binary size in some cases?).
    compiler_flag: "-ffunction-sections"
    compiler_flag: "-fdata-sections"
    linker_flag: "-Wl,--gc-sections"
  }
}

I set the environment variable and compile

-bash-4.1$ JAVA_HOME=/usr/local/java/1.8.0_60 CXX=/usr/local/gcc/4.8.4/bin/c++  CC=/usr/local/gcc/4.8.4/bin/gcc LDFLAGS="-L/usr/local/gcc/4.8.4/lib -L/usr/local/gcc/4.8.4/lib64"  ./compile.sh

I got errors.

INFO: You can skip this first step by providing a path to the bazel binary as second argument:
INFO:    ./compile.sh build /path/to/bazel
Building Bazel from scratch............
Building Bazel with Bazel.
bazel/bazel-0.1.1/output/bazel: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.14' not found (required by bazel/bazel-0.1.1/output/bazel)
bazel/bazel-0.1.1/output/bazel: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.15' not found (required by bazel/bazel-0.1.1/output/bazel)

It seems bazel is not picking up the right library. What else should I update in tools/cpp/CROSSTOOL? Am I using the run time environment variables correctly?

damienmg · 2016-01-11T14:43:17Z

There you are failing at the first step you should export CXX, CC, LDFLAGS and CXXFLAGS

I am removing the need for this, but it's still there for now.

digitalsword · 2016-01-11T15:00:46Z

@damienmg

Tried again I still got the same error.

export JAVA_HOME=/usr/local/java/1.8.0_60
export CXX=/usr/local/gcc/4.8.4/bin/c++
export CC=/usr/local/gcc/4.8.4/bin/gcc
export LDFLAGS="-L/usr/local/gcc/4.8.4/lib -L/usr/local/gcc/4.8.4/lib64"
export CXXFLAGS="-L/usr/local/gcc/4.8.4/lib -L/usr/local/gcc/4.8.4/lib64"
./compile.sh

damienmg · 2016-01-11T15:13:29Z

Like exactly the same (at the same place, concerning output/bazel)?

Also can you try a more recent version of bazel (0.1.3 should work nicely)

digitalsword · 2016-01-11T15:37:04Z

@damienmg The error was exactly the same. I also tried 0.1.3. After I export flags and update CROSSTOOL, I got the same error again.

bazel/bazel-0.1.3/output/bazel: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.14' not found (required by bazel/bazel-0.1.3/output/bazel)
bazel/bazel-0.1.3/output/bazel: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.15' not found (required by bazel/bazel-0.1.3/output/bazel)

I suspect I did not update CROSSTOOL correctly. I only changed the following lines:

  tool_path { name: "cpp" path: "/usr/local/gcc/4.8.4/bin/cpp" }
  tool_path { name: "gcc" path: "/usr/local/gcc/4.8.4/bin/gcc" }
  linker_flag: "-lstdc++ -L/usr/local/gcc/4.8.4/lib -L/usr/local/gcc/4.8.4/lib64"

and add following lines

  cxx_builtin_include_directory: "/usr/local/gcc/4.8.4/lib64/"
  cxx_builtin_include_directory: "/usr/local/gcc/4.8.4/lib/"

is there other places I need to update on CROSSTOOLS?

digitalsword · 2016-01-11T20:19:49Z

@damienmg It always search for default libstdc++ in /usr/lib64, not matter how I export environment variable or update CROSSTOOLS. It seems the same problem was observed by other guys (#590).

damienmg · 2016-01-11T20:23:25Z

:( By the end of the week my changes for the bootstrap part should be in so it should go further. I have no idea why it is happening though.

digitalsword · 2016-01-11T22:45:53Z

@damienmg I made some progress. I am getting different errors after I do

export LD_LIBRARY_PATH=/usr/local/gcc/4.8.4/lib:/usr/local/gcc/4.8.4/lib64

But now I am getting:

WARNING: Sandboxed execution is not supported on your system and thus hermeticity of actions cannot be guaranteed. See http://bazel.io/docs/bazel-user-manual.html#sandboxing for more information. You can turn off this warning via --ignore_unsupported_sandboxing.
INFO: Found 1 target...
ERROR: /home/pc/bazel/bazel-0.1.3/src/main/tools/BUILD:26:1: undeclared inclusion(s) in rule '//src/main/tools:build-runfiles':
this rule is missing dependency declarations for the following files included by 'src/main/tools/build-runfiles.cc':
  '/usr/local/gcc/4.8.4/include/c++/4.8.4/map'
  '/usr/local/gcc/4.8.4/include/c++/4.8.4/bits/stl_tree.h'
  '/usr/local/gcc/4.8.4/include/c++/4.8.4/bits/stl_algobase.h'
  '/usr/local/gcc/4.8.4/include/c++/4.8.4/x86_64-unknown-linux-gnu/bits/c++config.h'
  '/usr/local/gcc/4.8.4/include/c++/4.8.4/x86_64-unknown-linux-gnu/bits/os_defines.h'
  '/usr/local/gcc/4.8.4/include/c++/4.8.4/x86_64-unknown-linux-gnu/bits/cpu_defines.h'
  '/usr/local/gcc/4.8.4/include/c++/4.8.4/bits/functexcept.h'
  '/usr/local/gcc/4.8.4/include/c++/4.8.4/bits/exception_defines.h'
  '/usr/local/gcc/4.8.4/include/c++/4.8.4/bits/cpp_type_traits.h'
  '/usr/local/gcc/4.8.4/include/c++/4.8.4/ext/type_traits.h'
  '/usr/local/gcc/4.8.4/include/c++/4.8.4/ext/numeric_traits.h'
  '/usr/local/gcc/4.8.4/include/c++/4.8.4/bits/stl_pair.h'
  '/usr/local/gcc/4.8.4/include/c++/4.8.4/bits/move.h'
  '/usr/local/gcc/4.8.4/include/c++/4.8.4/bits/concept_check.h'
  '/usr/local/gcc/4.8.4/include/c++/4.8.4/type_traits'
  '/usr/local/gcc/4.8.4/include/c++/4.8.4/bits/stl_iterator_base_types.h'
  '/usr/local/gcc/4.8.4/include/c++/4.8.4/bits/stl_iterator_base_funcs.h'
  '/usr/local/gcc/4.8.4/include/c++/4.8.4/debug/debug.h'
  '/usr/local/gcc/4.8.4/include/c++/4.8.4/bits/stl_iterator.h'
  '/usr/local/gcc/4.8.4/include/c++/4.8.4/bits/allocator.h'
  '/usr/local/gcc/4.8.4/include/c++/4.8.4/x86_64-unknown-linux-gnu/bits/c++allocator.h'
  '/usr/local/gcc/4.8.4/include/c++/4.8.4/ext/new_allocator.h'
  '/usr/local/gcc/4.8.4/include/c++/4.8.4/new'
  '/usr/local/gcc/4.8.4/include/c++/4.8.4/exception'
  '/usr/local/gcc/4.8.4/include/c++/4.8.4/bits/atomic_lockfree_defines.h'
  '/usr/local/gcc/4.8.4/include/c++/4.8.4/bits/exception_ptr.h'
  '/usr/local/gcc/4.8.4/include/c++/4.8.4/bits/nested_exception.h'
  '/usr/local/gcc/4.8.4/include/c++/4.8.4/bits/memoryfwd.h'
  '/usr/local/gcc/4.8.4/include/c++/4.8.4/bits/stl_function.h'
  '/usr/local/gcc/4.8.4/include/c++/4.8.4/backward/binders.h'
  '/usr/local/gcc/4.8.4/include/c++/4.8.4/bits/alloc_traits.h'
  '/usr/local/gcc/4.8.4/include/c++/4.8.4/bits/ptr_traits.h'
  '/usr/local/gcc/4.8.4/include/c++/4.8.4/bits/stl_map.h'
  '/usr/local/gcc/4.8.4/include/c++/4.8.4/initializer_list'
  '/usr/local/gcc/4.8.4/include/c++/4.8.4/tuple'
  '/usr/local/gcc/4.8.4/include/c++/4.8.4/utility'
  '/usr/local/gcc/4.8.4/include/c++/4.8.4/bits/stl_relops.h'
  '/usr/local/gcc/4.8.4/include/c++/4.8.4/array'
  '/usr/local/gcc/4.8.4/include/c++/4.8.4/stdexcept'
  '/usr/local/gcc/4.8.4/include/c++/4.8.4/string'
  '/usr/local/gcc/4.8.4/include/c++/4.8.4/bits/stringfwd.h'
  '/usr/local/gcc/4.8.4/include/c++/4.8.4/bits/char_traits.h'
  '/usr/local/gcc/4.8.4/include/c++/4.8.4/bits/postypes.h'
  '/usr/local/gcc/4.8.4/include/c++/4.8.4/cwchar'
  '/usr/local/gcc/4.8.4/include/c++/4.8.4/cstdint'
  '/usr/local/gcc/4.8.4/include/c++/4.8.4/bits/localefwd.h'
  '/usr/local/gcc/4.8.4/include/c++/4.8.4/x86_64-unknown-linux-gnu/bits/c++locale.h'
  '/usr/local/gcc/4.8.4/include/c++/4.8.4/clocale'
  '/usr/local/gcc/4.8.4/include/c++/4.8.4/iosfwd'
  '/usr/local/gcc/4.8.4/include/c++/4.8.4/cctype'
  '/usr/local/gcc/4.8.4/include/c++/4.8.4/bits/ostream_insert.h'
  '/usr/local/gcc/4.8.4/include/c++/4.8.4/bits/cxxabi_forced.h'
  '/usr/local/gcc/4.8.4/include/c++/4.8.4/bits/range_access.h'
  '/usr/local/gcc/4.8.4/include/c++/4.8.4/bits/basic_string.h'
  '/usr/local/gcc/4.8.4/include/c++/4.8.4/ext/atomicity.h'
  '/usr/local/gcc/4.8.4/include/c++/4.8.4/x86_64-unknown-linux-gnu/bits/gthr.h'
  '/usr/local/gcc/4.8.4/include/c++/4.8.4/x86_64-unknown-linux-gnu/bits/gthr-default.h'
  '/usr/local/gcc/4.8.4/include/c++/4.8.4/x86_64-unknown-linux-gnu/bits/atomic_word.h'
  '/usr/local/gcc/4.8.4/include/c++/4.8.4/ext/string_conversions.h'
  '/usr/local/gcc/4.8.4/include/c++/4.8.4/cstdlib'
  '/usr/local/gcc/4.8.4/include/c++/4.8.4/cstdio'
  '/usr/local/gcc/4.8.4/include/c++/4.8.4/cerrno'
  '/usr/local/gcc/4.8.4/include/c++/4.8.4/bits/functional_hash.h'
  '/usr/local/gcc/4.8.4/include/c++/4.8.4/bits/hash_bytes.h'
  '/usr/local/gcc/4.8.4/include/c++/4.8.4/bits/basic_string.tcc'
  '/usr/local/gcc/4.8.4/include/c++/4.8.4/bits/uses_allocator.h'
  '/usr/local/gcc/4.8.4/include/c++/4.8.4/bits/stl_multimap.h'.
Target //src:bazel failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 11.692s, Critical Path: 1.30s

Building output/bazel

Where should I put --ignore_unsupported_sandboxing to disable sandbox? Is sandbox is the culprit?

digitalsword · 2016-01-15T21:16:25Z

@damienmg have you finished the bootstrap part? Maybe I can test it on redhat 6.7?

damienmg · 2016-01-15T21:33:27Z

@digitalsword I am still trying to get them merged, some changes have landed already (0a7a3d2 and 12c68a) but there are more to do. It should land next week. Then I can concentrate on the auto-configuration mechanism for Bazel.

For your specific issue, you need to add /usr/local/gcc/4.8.4/include/c++/4.8.4/ to the tools/cpp/CROSSTOOL file as a cxx_builtin_include_directory (that the kind of thing I want to handle by the auto configuration mechanism)

digitalsword · 2016-01-16T19:16:05Z

@damienmg I am running into the glib version issue again. When you finish auto-configuration part, can you give me instructions on using auto configuration on Redhat 6.7? Thanks!

damienmg · 2016-01-16T19:36:42Z

Sure but this will take a few more months before being ready :(

digitalsword · 2016-01-18T15:25:37Z

@damienmg I am able to compile the head version of bazel, however, when I try to compile tensorflow, I get glib error again.

/tmp/tensorflow_head/_bin/build-runfiles: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.14' not found (required by /tmp/tensorflow_head/_bin/build-runfiles)

I used these commands to compile tensorflow

export JAVA_HOME=/usr/local/java/1.8.0_60
export CXX=/usr/local/gcc/4.8.4/bin/c++
export CC=/usr/local/gcc/4.8.4/bin/gcc
export LDFLAGS="-L/usr/local/gcc/4.8.4/lib -L/usr/local/gcc/4.8.4/lib64"
export CXXFLAGS="-L/usr/local/gcc/4.8.4/lib -L/usr/local/gcc/4.8.4/lib64"
export LD_LIBRARY_PATH=/usr/local/gcc/4.8.4/lib:/usr/local/gcc/4.8.4/lib64

bazel --output_base /tmp/ build -c opt --config=cuda //tensorflow/cc:tutorials_example_trainer

Is there any environment variable to be set when using bazel to compile tensorflow, in other words how can I force bazel to use the newer glib library.

damienmg · 2016-01-18T15:35:14Z

With the latest change, if you pass compile.sh then only the CROSSTOOL files change should be important. How did you modified your CROSSTOOL file?

digitalsword · 2016-01-18T15:38:17Z

@damienmg

change CROSSTOOL file to:

 tool_path { name: "ar" path: "/usr/bin/ar" }
  tool_path { name: "compat-ld" path: "/usr/bin/ld" }
  tool_path { name: "cpp" path: "/usr/local/gcc/4.8.4/bin/cpp" }
  tool_path { name: "dwp" path: "/usr/bin/dwp" }
  tool_path { name: "gcc" path: "/usr/local/gcc/4.8.4/bin/gcc" }
  cxx_flag: "-std=c++0x"
  linker_flag: "-lstdc++ -L/usr/local/gcc/4.8.4/lib -L/usr/local/gcc/4.8.4/lib64"
  linker_flag: "-B/usr/bin/"

  # TODO(bazel-team): In theory, the path here ought to exactly match the path
  # used by gcc. That works because bazel currently doesn't track files at
  # absolute locations and has no remote execution, yet. However, this will need
  # to be fixed, maybe with auto-detection?
  cxx_builtin_include_directory: "/usr/local/gcc/4.8.4/include/c++/4.8.4/"
  cxx_builtin_include_directory: "/usr/local/gcc/4.8.4/lib64/"
  cxx_builtin_include_directory: "/usr/local/gcc/4.8.4/lib/"
  cxx_builtin_include_directory: "/usr/lib/gcc/"
  cxx_builtin_include_directory: "/usr/local/include"
  cxx_builtin_include_directory: "/usr/include"
  tool_path { name: "gcov" path: "/usr/bin/gcov" }

and just compile with:

export JAVA_HOME=/usr/local/java/1.8.0_60
export CXX=/usr/local/gcc/4.8.4/bin/c++
export CC=/usr/local/gcc/4.8.4/bin/gcc
export LDFLAGS="-L/usr/local/gcc/4.8.4/lib -L/usr/local/gcc/4.8.4/lib64"
export CXXFLAGS="-L/usr/local/gcc/4.8.4/lib -L/usr/local/gcc/4.8.4/lib64"
export LD_LIBRARY_PATH=/usr/local/gcc/4.8.4/lib:/usr/local/gcc/4.8.4/lib64
./compile.sh

damienmg · 2016-01-18T15:42:33Z

Invoking @ulfjack who knows more about C++.

ulfjack · 2016-01-21T14:32:13Z

Tricky, since I can't reproduce the issue. I tried downloading a docker image for rhel6.7, but I can't seem to figure out how to install stuff on it.

The env variables are ignored by Bazel, and the compile.sh should no longer compile any C++ code. Therefore, any errors you get have to be from a run of Bazel itself, and the only way to fix them is to change the CROSSTOOL file.

If you have a C++ binary that tries to load the wrong dynamic library, try running ldd on it, and see what that says. The dynamic linker by default uses a search path baked into the binary, and it would be good if we could check that. If that needs to be changed, we'll have to figure out how to do that. I don't know of the top of my head.

damienmg · 2016-01-21T14:55:13Z

(not all the env variable are ignored, JAVA_HOME is used)

weijianwen · 2016-01-22T13:11:01Z

@ulfjack @damienmg thank you for your replies. It is mentioned that

The env variables are ignored by Bazel...

I think that's the reason that bazel ignores GLIBCXX placed in directories among $LD_LIBRARY_PATH.

I worked on Cent OS 6.x. This issue happens when I built bazel 0.1.4 from source with gcc/g++ and libstdc++.so in non-standard directories.

Customization made to `CROSSTOOL`

I specify gcc and necessary header files.

-  tool_path { name: "gcc" path: "/usr/bin/gcc" }
+  tool_path { name: "gcc" path: "/lustre/usr/gcc/4.9/bin/gcc" }

+  cxx_builtin_include_directory: "/lustre/usr/gcc/4.9/lib/gcc/x86_64-unknown-linux-gnu/4.9.3/include"
+  cxx_builtin_include_directory: "/lustre/usr/gcc/4.9/lib/gcc/x86_64-unknown-linux-gnu/4.9.3/include-fixed"
+  cxx_builtin_include_directory: "/lustre/usr/gcc/4.9/include/c++/4.9.3"

Error messages

GCC and JDK I use:

$ which gcc; gcc --version
/lustre/usr/gcc/4.9/bin/gcc
gcc (GCC) 4.9.3
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.


$ which javac; javac -version
/lustre/usr/jdk/1.8/bin/javac
javac 1.8.0_60

Build:

$ pkill java; rm -rf ~/.cache/bazel ;  ./compile.sh
...
.Extracting Bazel installation...
..........
WARNING: Sandboxed execution is not supported on your system and thus hermeticity of actions cannot be guaranteed. See http://bazel.io/docs/bazel-user-manual.html#sandboxing for more information. You can turn off this warning via --ignore_unsupported_sandboxing.
INFO: Found 1 target...
INFO: From Creating runfiles tree bazel-out/local_linux-fastbuild/bin/src/tools/android/java/com/google/devtools/build/android/idlclass/classes.runfiles:
/lustre/home/rpm/.cache/bazel/_bazel_rpm/18dbd9f3d77ee103aaae087f6df97588/bazel/_bin/build-runfiles: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.14' not found (required by /lustre/home/rpm/.cache/bazel/_bazel_rpm/18dbd9f3d77ee103aaae087f6df97588/bazel/_bin/build-runfiles)
ERROR: /tmp/bazel/src/tools/android/java/com/google/devtools/build/android/idlclass/BUILD:10:1: Creating runfiles tree bazel-out/local_linux-fastbuild/bin/src/tools/android/java/com/google/devtools/build/android/idlclass/classes.runfiles failed: build-runfiles failed: error executing command /lustre/home/rpm/.cache/bazel/_bazel_rpm/18dbd9f3d77ee103aaae087f6df97588/bazel/_bin/build-runfiles ... (remaining 2 argument(s) skipped): com.google.devtools.build.lib.shell.BadExitStatusException: Process exited with status 1.
Target //src:bazel failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 11.112s, Critical Path: 4.48s

Building output/bazel

Complaints about GLIBCXX_3.4.14 arises. build-runfiles seems to use the default and outdated C++ lib in /usr/lib64/libstdc++.so.6. However, ldd indicates that the proper lib is available out there.

$ echo $LD_LIBRARY_PATH 
/lustre/usr/gcc/4.9/lib64

$ strings /lustre/usr/gcc/4.9/lib64/libstdc++.so.6 | grep '3.4.14'
GLIBCXX_3.4.14

$ ldd /lustre/home/rpm/.cache/bazel/_bazel_rpm/18dbd9f3d77ee103aaae087f6df97588/bazel/_bin/build-runfiles
        linux-vdso.so.1 =>  (0x00007ffff4dff000)
        libstdc++.so.6 => /lustre/usr/gcc/4.9/lib64/libstdc++.so.6 (0x00002b9213921000)
        libz.so.1 => /lib64/libz.so.1 (0x000000396de00000)
        librt.so.1 => /lib64/librt.so.1 (0x000000396e200000)
        libm.so.6 => /lib64/libm.so.6 (0x000000396d600000)
        libgcc_s.so.1 => /lustre/usr/gcc/4.9/lib64/libgcc_s.so.1 (0x00002b9213c56000)
        libc.so.6 => /lib64/libc.so.6 (0x000000396ce00000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x000000396da00000)
        /lib64/ld-linux-x86-64.so.2 (0x000000396ca00000)

Things that may help

I guess there is something wrong when bazel invokes the build-runfiles command. It'll help if bazel prints every action it takes.

damienmg · 2016-01-22T13:14:08Z

Could you try at HEAD? I removed most C++ compilation from the first steps
to build-runfiles should not be build.

On Fri, Jan 22, 2016 at 2:11 PM 健美猫 notifications@github.com wrote:

@ulfjack https://github.com/ulfjack @damienmg
https://github.com/damienmg thank you for your replies. It is mentioned
that

The env variables are ignored by Bazel...

I think that's the reason that bazel ignores GLIBCXX placed in directories
among $LD_LIBRARY_PATH.

I worked on Cent OS 6.x. This issue happens when I built bazel 0.1.4 from
source with gcc/g++ and libstdc++.so in non-standard directories.
Customization made to CROSSTOOL

I specify gcc and necessary header files.

tool_path { name: "gcc" path: "/usr/bin/gcc" }

tool_path { name: "gcc" path: "/lustre/usr/gcc/4.9/bin/gcc" }

cxx_builtin_include_directory: "/lustre/usr/gcc/4.9/lib/gcc/x86_64-unknown-linux-gnu/4.9.3/include"

cxx_builtin_include_directory: "/lustre/usr/gcc/4.9/lib/gcc/x86_64-unknown-linux-gnu/4.9.3/include-fixed"

cxx_builtin_include_directory: "/lustre/usr/gcc/4.9/include/c++/4.9.3"

Error messages

GCC and JDK I use:

$ which gcc; gcc --version
/lustre/usr/gcc/4.9/bin/gcc
gcc (GCC) 4.9.3
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

$ which javac; javac -version
/lustre/usr/jdk/1.8/bin/javac
javac 1.8.0_60

Build:

$ pkill java; rm -rf ~/.cache/bazel ; ./compile.sh
...
.Extracting Bazel installation...
..........
WARNING: Sandboxed execution is not supported on your system and thus hermeticity of actions cannot be guaranteed. See http://bazel.io/docs/bazel-user-manual.html#sandboxing for more information. You can turn off this warning via --ignore_unsupported_sandboxing.
INFO: Found 1 target...
INFO: From Creating runfiles tree bazel-out/local_linux-fastbuild/bin/src/tools/android/java/com/google/devtools/build/android/idlclass/classes.runfiles:
/lustre/home/rpm/.cache/bazel/_bazel_rpm/18dbd9f3d77ee103aaae087f6df97588/bazel/_bin/build-runfiles: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.14' not found (required by /lustre/home/rpm/.cache/bazel/_bazel_rpm/18dbd9f3d77ee103aaae087f6df97588/bazel/_bin/build-runfiles)
ERROR: /tmp/bazel/src/tools/android/java/com/google/devtools/build/android/idlclass/BUILD:10:1: Creating runfiles tree bazel-out/local_linux-fastbuild/bin/src/tools/android/java/com/google/devtools/build/android/idlclass/classes.runfiles failed: build-runfiles failed: error executing command /lustre/home/rpm/.cache/bazel/_bazel_rpm/18dbd9f3d77ee103aaae087f6df97588/bazel/_bin/build-runfiles ... (remaining 2 argument(s) skipped): com.google.devtools.build.lib.shell.BadExitStatusException: Process exited with status 1.
Target //src:bazel failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 11.112s, Critical Path: 4.48s

Building output/bazel

Complaints about GLIBCXX_3.4.14 arises. build-runfiles seems to use the
default and outdated C++ lib in /usr/lib64/libstdc++.so.6. However, ldd
indicates that the proper lib is available out there.

$ echo $LD_LIBRARY_PATH
/lustre/usr/gcc/4.9/lib64

$ strings /lustre/usr/gcc/4.9/lib64/libstdc++.so.6 | grep '3.4.14'
GLIBCXX_3.4.14

$ ldd /lustre/home/rpm/.cache/bazel/_bazel_rpm/18dbd9f3d77ee103aaae087f6df97588/bazel/_bin/build-runfiles
linux-vdso.so.1 => (0x00007ffff4dff000)
libstdc++.so.6 => /lustre/usr/gcc/4.9/lib64/libstdc++.so.6 (0x00002b9213921000)
libz.so.1 => /lib64/libz.so.1 (0x000000396de00000)
librt.so.1 => /lib64/librt.so.1 (0x000000396e200000)
libm.so.6 => /lib64/libm.so.6 (0x000000396d600000)
libgcc_s.so.1 => /lustre/usr/gcc/4.9/lib64/libgcc_s.so.1 (0x00002b9213c56000)
libc.so.6 => /lib64/libc.so.6 (0x000000396ce00000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x000000396da00000)
/lib64/ld-linux-x86-64.so.2 (0x000000396ca00000)

Things that may help

I guess there is something wrong when bazel invokes the build-runfiles
command. It'll help if bazel prints every action it takes.

—
Reply to this email directly or view it on GitHub
#760 (comment).

damienmg · 2016-01-22T13:14:37Z

For printing all action, use -s. With the compile script, do
EXTRA_BAZEL_ARGS='-s'

On Fri, Jan 22, 2016 at 2:13 PM Damien Martin-guillerez dmarting@google.com
wrote:

Could you try at HEAD? I removed most C++ compilation from the first steps
to build-runfiles should not be build.

On Fri, Jan 22, 2016 at 2:11 PM 健美猫 notifications@github.com wrote:

@ulfjack https://github.com/ulfjack @damienmg
https://github.com/damienmg thank you for your replies. It is
mentioned that

The env variables are ignored by Bazel...

I think that's the reason that bazel ignores GLIBCXX placed in
directories among $LD_LIBRARY_PATH.

I worked on Cent OS 6.x. This issue happens when I built bazel 0.1.4 from
source with gcc/g++ and libstdc++.so in non-standard directories.
Customization made to CROSSTOOL

I specify gcc and necessary header files.

tool_path { name: "gcc" path: "/usr/bin/gcc" }

tool_path { name: "gcc" path: "/lustre/usr/gcc/4.9/bin/gcc" }

cxx_builtin_include_directory: "/lustre/usr/gcc/4.9/lib/gcc/x86_64-unknown-linux-gnu/4.9.3/include"

cxx_builtin_include_directory: "/lustre/usr/gcc/4.9/lib/gcc/x86_64-unknown-linux-gnu/4.9.3/include-fixed"

cxx_builtin_include_directory: "/lustre/usr/gcc/4.9/include/c++/4.9.3"

Error messages

GCC and JDK I use:

$ which gcc; gcc --version
/lustre/usr/gcc/4.9/bin/gcc
gcc (GCC) 4.9.3
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

$ which javac; javac -version
/lustre/usr/jdk/1.8/bin/javac
javac 1.8.0_60

Build:

$ pkill java; rm -rf ~/.cache/bazel ; ./compile.sh
...
.Extracting Bazel installation...
..........
WARNING: Sandboxed execution is not supported on your system and thus hermeticity of actions cannot be guaranteed. See http://bazel.io/docs/bazel-user-manual.html#sandboxing for more information. You can turn off this warning via --ignore_unsupported_sandboxing.
INFO: Found 1 target...
INFO: From Creating runfiles tree bazel-out/local_linux-fastbuild/bin/src/tools/android/java/com/google/devtools/build/android/idlclass/classes.runfiles:
/lustre/home/rpm/.cache/bazel/_bazel_rpm/18dbd9f3d77ee103aaae087f6df97588/bazel/_bin/build-runfiles: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.14' not found (required by /lustre/home/rpm/.cache/bazel/_bazel_rpm/18dbd9f3d77ee103aaae087f6df97588/bazel/_bin/build-runfiles)
ERROR: /tmp/bazel/src/tools/android/java/com/google/devtools/build/android/idlclass/BUILD:10:1: Creating runfiles tree bazel-out/local_linux-fastbuild/bin/src/tools/android/java/com/google/devtools/build/android/idlclass/classes.runfiles failed: build-runfiles failed: error executing command /lustre/home/rpm/.cache/bazel/_bazel_rpm/18dbd9f3d77ee103aaae087f6df97588/bazel/_bin/build-runfiles ... (remaining 2 argument(s) skipped): com.google.devtools.build.lib.shell.BadExitStatusException: Process exited with status 1.
Target //src:bazel failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 11.112s, Critical Path: 4.48s

Building output/bazel

Complaints about GLIBCXX_3.4.14 arises. build-runfiles seems to use the
default and outdated C++ lib in /usr/lib64/libstdc++.so.6. However, ldd
indicates that the proper lib is available out there.

$ echo $LD_LIBRARY_PATH
/lustre/usr/gcc/4.9/lib64

$ strings /lustre/usr/gcc/4.9/lib64/libstdc++.so.6 | grep '3.4.14'
GLIBCXX_3.4.14

$ ldd /lustre/home/rpm/.cache/bazel/_bazel_rpm/18dbd9f3d77ee103aaae087f6df97588/bazel/_bin/build-runfiles
linux-vdso.so.1 => (0x00007ffff4dff000)
libstdc++.so.6 => /lustre/usr/gcc/4.9/lib64/libstdc++.so.6 (0x00002b9213921000)
libz.so.1 => /lib64/libz.so.1 (0x000000396de00000)
librt.so.1 => /lib64/librt.so.1 (0x000000396e200000)
libm.so.6 => /lib64/libm.so.6 (0x000000396d600000)
libgcc_s.so.1 => /lustre/usr/gcc/4.9/lib64/libgcc_s.so.1 (0x00002b9213c56000)
libc.so.6 => /lib64/libc.so.6 (0x000000396ce00000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x000000396da00000)
/lib64/ld-linux-x86-64.so.2 (0x000000396ca00000)

Things that may help

I guess there is something wrong when bazel invokes the build-runfiles
command. It'll help if bazel prints every action it takes.

—
Reply to this email directly or view it on GitHub
#760 (comment).

weijianwen · 2016-01-22T13:30:40Z

I tried the head version(0bf232e). It failed with different messages which are less meaningful compared to version 0.1.4. According to the output given by EXTRA_BAZEL_ARGS='-s', the action taken before the errors arise is //src/main/tools:process-wrapper.

pkill java; rm -rf ~/.cache/bazel ;  ./compile.sh
...
[136 / 461] Writing file src/main/java/com/google/devtools/build/lib/libbazel.jar-2.pa
ERROR: /tmp/bazel/src/main/protobuf/BUILD:22:2: error executing shell command: 'rm -rf bazel-out/local_linux-fastbuild/bin/src/main/protobuf/libinvocation_policy_proto_srcjar.srcjar.proto_output && mkdir bazel-out/local_linux-fastbuild/bin/src/main/protobuf/libinvocation_polic...' failed: bash failed: error executing command 
  (cd /tmp/bazel.d8kSsRIw/out/bazel && \
  exec env - \
    PATH=/lustre/usr/jdk/1.8/bin:/lustre/usr/gcc/4.9/bin:/lustre/usr/emacs/24.5/bin:/lustre/home/rpm/.local/bin:/lustre/home/rpm/bin:/usr/local/bin:/usr/local/sbin:/usr/lib64/qt-3.3/bin:/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/lustre/home/rpm/bin:/usr/local/sbin:/usr/sbin:/sbin:/opt/bin \
  /bin/bash -c 'rm -rf bazel-out/local_linux-fastbuild/bin/src/main/protobuf/libinvocation_policy_proto_srcjar.srcjar.proto_output && mkdir bazel-out/local_linux-fastbuild/
bin/src/main/protobuf/libinvocation_policy_proto_srcjar.srcjar.proto_output && externa
l/bazel_tools/third_party/protobuf/protoc-linux-x86_64.exe --java_out=bazel-out/local_
linux-fastbuild/bin/src/main/protobuf/libinvocation_policy_proto_srcjar.srcjar.proto_o
utput src/main/protobuf/invocation_policy.proto && touch -t 198001010000 $(find bazel-
out/local_linux-fastbuild/bin/src/main/protobuf/libinvocation_policy_proto_srcjar.srcj
ar.proto_output) && external/local-jdk/bin/jar cMf bazel-out/local_linux-fastbuild/bin
/src/main/protobuf/libinvocation_policy_proto_srcjar.srcjar -C bazel-out/local_linux-f
astbuild/bin/src/main/protobuf/libinvocation_policy_proto_srcjar.srcjar.proto_output .
'): com.google.devtools.build.lib.shell.BadExitStatusException: Process exited with st
atus 1: bash failed: error executing command 
  (cd /tmp/bazel.d8kSsRIw/out/bazel && \
  exec env - \
    PATH=/lustre/usr/jdk/1.8/bin:/lustre/usr/gcc/4.9/bin:/lustre/usr/emacs/24.5/bin:/l
ustre/home/rpm/.local/bin:/lustre/home/rpm/bin:/usr/local/bin:/usr/local/sbin:/usr/lib
64/qt-3.3/bin:/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/lustr
e/home/rpm/bin:/usr/local/sbin:/usr/sbin:/sbin:/opt/bin \
  /bin/bash -c 'rm -rf bazel-out/local_linux-fastbuild/bin/src/main/protobuf/libinvoca
tion_policy_proto_srcjar.srcjar.proto_output && mkdir bazel-out/local_linux-fastbuild/
bin/src/main/protobuf/libinvocation_policy_proto_srcjar.srcjar.proto_output && externa
l/bazel_tools/third_party/protobuf/protoc-linux-x86_64.exe --java_out=bazel-out/local_
linux-fastbuild/bin/src/main/protobuf/libinvocation_policy_proto_srcjar.srcjar.proto_o
utput src/main/protobuf/invocation_policy.proto && touch -t 198001010000 $(find bazel-
out/local_linux-fastbuild/bin/src/main/protobuf/libinvocation_policy_proto_srcjar.srcj
ar.proto_output) && external/local-jdk/bin/jar cMf bazel-out/local_linux-fastbuild/bin
/src/main/protobuf/libinvocation_policy_proto_srcjar.srcjar -C bazel-out/local_linux-f
astbuild/bin/src/main/protobuf/libinvocation_policy_proto_srcjar.srcjar.proto_output .
'): com.google.devtools.build.lib.shell.BadExitStatusException: Process exited with st
atus 1.
Target //src:bazel failed to build
INFO: Elapsed time: 4.559s, Critical Path: 2.36s

$ pkill java; rm -rf ~/.cache/bazel ; export EXTRA_BAZEL_ARGS='-s';  ./compile.sh
...
>>>>> # //src/main/tools:process-wrapper [action 'Linking src/main/tools/process-wrapp
er']
(cd /tmp/bazel.XgjN5zHj/out/bazel && \
  exec env - \
  /lustre/usr/gcc/4.9/bin/gcc -o bazel-out/local_linux-fastbuild/bin/src/main/tools/pr
ocess-wrapper -B/usr/bin/ -Wl,-z,relro,-z,now -no-canonical-prefixes -pass-exit-codes 
'-Wl,--build-id=md5' '-Wl,--hash-style=gnu' -Wl,-S -Wl,@bazel-out/local_linux-fastbuil
d/bin/src/main/tools/process-wrapper-2.params)
ERROR: /tmp/bazel/src/main/protobuf/BUILD:22:2: error executing shell command: 'rm -rf
 bazel-out/local_linux-fastbuild/bin/src/main/protobuf/libjava_compilation_proto_srcja
r.srcjar.proto_output && mkdir bazel-out/local_linux-fastbuild/bin/src/main/protobuf/l
ibjava_compilation_...' failed: bash failed: error executing command
...

damienmg · 2016-01-22T13:34:34Z

can you set EXTRA_BAZEL_ARGS='--verbose_failures' and show a bit more of the output log?

weijianwen · 2016-01-22T13:39:35Z

Sure thing. Here it comes. I am planning give it a try, running the long coamnnd exec env... manually.

$ pkill java; rm -rf ~/.cache/bazel ; export EXTRA_BAZEL_ARGS='--verbose_failures';  ./compile.sh
INFO: Found 1 target...
[8 / 76] Writing file src/main/java/com/google/devtools/build/lib/libpython-rules.jar-
[15 / 141] Writing file src/main/java/com/google/devtools/build/lib/libbuild-info.jar-
[61 / 387] Writing file src/main/java/com/google/devtools/build/lib/rules/cpp/libcpp.j
[73 / 392] Writing file src/main/java/com/google/devtools/common/options/liboptions.ja
[77 / 404] Writing file src/main/java/com/google/devtools/build/lib/libpreconditions.j
ERROR: /tmp/bazel/src/main/protobuf/BUILD:22:2: error executing shell command: 'rm -rf bazel-out/local_linux-fastbuild/bin/src/main/protobuf/libextra_actions_base_proto_srcjar.srcjar.proto_output && mkdir bazel-out/local_linux-fastbuild/bin/src/main/protobuf/libextra_actions_b...' failed: bash failed: error executing command 
  (cd /tmp/bazel.NSi9zaov/out/bazel && \
  exec env - \
    PATH=/lustre/usr/jdk/1.8/bin:/lustre/usr/gcc/4.9/bin:/lustre/usr/emacs/24.5/bin:/lustre/home/rpm/.local/bin:/lustre/home/rpm/bin:/usr/local/bin:/usr/local/sbin:/usr/lib64/qt-3.3/bin:/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/lustre/home/rpm/bin:/usr/local/sbin:/usr/sbin:/sbin:/opt/bin \
  /bin/bash -c 'rm -rf bazel-out/local_linux-fastbuild/bin/src/main/protobuf/libextra_actions_base_proto_srcjar.srcjar.proto_output && mkdir bazel-out/local_linux-fastbuild
/bin/src/main/protobuf/libextra_actions_base_proto_srcjar.srcjar.proto_output && exter
nal/bazel_tools/third_party/protobuf/protoc-linux-x86_64.exe --java_out=bazel-out/loca
l_linux-fastbuild/bin/src/main/protobuf/libextra_actions_base_proto_srcjar.srcjar.prot
o_output src/main/protobuf/extra_actions_base.proto && touch -t 198001010000 $(find ba
zel-out/local_linux-fastbuild/bin/src/main/protobuf/libextra_actions_base_proto_srcjar
.srcjar.proto_output) && external/local-jdk/bin/jar cMf bazel-out/local_linux-fastbuil
d/bin/src/main/protobuf/libextra_actions_base_proto_srcjar.srcjar -C bazel-out/local_l
inux-fastbuild/bin/src/main/protobuf/libextra_actions_base_proto_srcjar.srcjar.proto_o
utput .'): com.google.devtools.build.lib.shell.BadExitStatusException: Process exited 
with status 1: bash failed: error executing command 
  (cd /tmp/bazel.NSi9zaov/out/bazel && \
  exec env - \
    PATH=/lustre/usr/jdk/1.8/bin:/lustre/usr/gcc/4.9/bin:/lustre/usr/emacs/24.5/bin:/l
ustre/home/rpm/.local/bin:/lustre/home/rpm/bin:/usr/local/bin:/usr/local/sbin:/usr/lib
64/qt-3.3/bin:/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/lustr
e/home/rpm/bin:/usr/local/sbin:/usr/sbin:/sbin:/opt/bin \
  /bin/bash -c 'rm -rf bazel-out/local_linux-fastbuild/bin/src/main/protobuf/libextra_
actions_base_proto_srcjar.srcjar.proto_output && mkdir bazel-out/local_linux-fastbuild
/bin/src/main/protobuf/libextra_actions_base_proto_srcjar.srcjar.proto_output && exter
nal/bazel_tools/third_party/protobuf/protoc-linux-x86_64.exe --java_out=bazel-out/loca
l_linux-fastbuild/bin/src/main/protobuf/libextra_actions_base_proto_srcjar.srcjar.prot
o_output src/main/protobuf/extra_actions_base.proto && touch -t 198001010000 $(find ba
zel-out/local_linux-fastbuild/bin/src/main/protobuf/libextra_actions_base_proto_srcjar
.srcjar.proto_output) && external/local-jdk/bin/jar cMf bazel-out/local_linux-fastbuil
d/bin/src/main/protobuf/libextra_actions_base_proto_srcjar.srcjar -C bazel-out/local_l
inux-fastbuild/bin/src/main/protobuf/libextra_actions_base_proto_srcjar.srcjar.proto_o
utput .'): com.google.devtools.build.lib.shell.BadExitStatusException: Process exited 
with status 1.
Target //src:bazel failed to build

damienmg · 2016-01-22T13:44:48Z

I still cannot spot the error, can you execute the last command outside of bazel:

(cd /tmp/bazel.NSi9zaov/out/bazel && \
  exec env - \
    PATH=/lustre/usr/jdk/1.8/bin:/lustre/usr/gcc/4.9/bin:/lustre/usr/emacs/24.5/bin:/l
ustre/home/rpm/.local/bin:/lustre/home/rpm/bin:/usr/local/bin:/usr/local/sbin:/usr/lib
64/qt-3.3/bin:/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/lustr
e/home/rpm/bin:/usr/local/sbin:/usr/sbin:/sbin:/opt/bin \
  /bin/bash -c 'rm -rf bazel-out/local_linux-fastbuild/bin/src/main/protobuf/libextra_
actions_base_proto_srcjar.srcjar.proto_output && mkdir bazel-out/local_linux-fastbuild
/bin/src/main/protobuf/libextra_actions_base_proto_srcjar.srcjar.proto_output && exter
nal/bazel_tools/third_party/protobuf/protoc-linux-x86_64.exe --java_out=bazel-out/loca
l_linux-fastbuild/bin/src/main/protobuf/libextra_actions_base_proto_srcjar.srcjar.prot
o_output src/main/protobuf/extra_actions_base.proto && touch -t 198001010000 $(find ba
zel-out/local_linux-fastbuild/bin/src/main/protobuf/libextra_actions_base_proto_srcjar
.srcjar.proto_output) && external/local-jdk/bin/jar cMf bazel-out/local_linux-fastbuil
d/bin/src/main/protobuf/libextra_actions_base_proto_srcjar.srcjar -C bazel-out/local_l
inux-fastbuild/bin/src/main/protobuf/libextra_actions_base_proto_srcjar.srcjar.proto_o
utput .')

If it passes, can you try setting --spawn_strategy=standalone --genrule_strategy=standalone to the extra args?

weijianwen · 2016-01-22T14:05:35Z

Hmm, I am trying to run the long coammand without basel, finding that some intermedia directories are missing.

basel-* (basel-output, basel-bin, ...) are symblic links pointing to temp directories outside basel. These temp directories are generatd during compiling then deleted automatically after exit.
There is no external/ directory.

So, the long command seems to exit all the time as it tries to operate on invalid directories.

damienmg · 2016-01-22T14:10:02Z

oh yes, can you try to comment out that line:
https://github.com/bazelbuild/bazel/blob/master/scripts/bootstrap/buildenv.sh#L49

Then rerun the compile.sh script with verbose_failures and again try to run the failing command outside of bazel?

weijianwen · 2016-01-22T14:27:07Z

It seems that we are making some progress. I got a very clear error message this time. It is ijar. I run the command successfully outside basel. ijar depends on libstdc++.so.6 which is specified in $LD_LIBRARY_PATH. I guess this environment variable should be send to the process who lauches ijar.

pkill java; rm -rf ~/.cache/bazel ; export EXTRA_BAZEL_ARGS='--verbose_failures';  ./compile.sh compile
ERROR: /home/rpmbuild/tmp/bazel/third_party/BUILD:43:1: Extracting interface //third_party:aether failed: ijar failed: error executing command 
  (cd /tmp/bazel.NqqeEfeO/out/bazel && \
  exec env - \
  bazel-out/host/bin/third_party/ijar/ijar third_party/aether/aether-transport-classpath-1.0.0.v20140518.jar bazel-out/local_linux-fastbuild/genfiles/third_party/_ijar/aether/third_party/aether/aether-transport-classpath-1.0.0.v20140518-ijar.jar): com.google.devtools.build.lib.shell.BadExitStatusException: Process exited with status 1: ijar failed: error executing command 
  (cd /tmp/bazel.NqqeEfeO/out/bazel && \
  exec env - \
  bazel-out/host/bin/third_party/ijar/ijar third_party/aether/aether-transport-classpath-1.0.0.v20140518.jar bazel-out/local_linux-fastbuild/genfiles/third_party/_ijar/aether/third_party/aether/aether-transport-classpath-1.0.0.v20140518-ijar.jar): com.google.devtools.build.lib.shell.BadExitStatusException: Process exited with status 1.
Target //src:bazel failed to build

$ echo $LD_LIBRARY_PATH
/lustre/usr/gcc/4.9/lib64

$ strings /lustre/usr/gcc/4.9/lib64/libstdc++.so.6 | grep '3.4.14'
GLIBCXX_3.4.14

$ ldd bazel-out/host/bin/third_party/ijar/ijar
        linux-vdso.so.1 =>  (0x00007fffa1b1c000)
        libz.so.1 => /lib64/libz.so.1 (0x000000396de00000)
        libstdc++.so.6 => /lustre/usr/gcc/4.9/lib64/libstdc++.so.6 (0x00002b1050fda000)
        libgcc_s.so.1 => /lustre/usr/gcc/4.9/lib64/libgcc_s.so.1 (0x00002b10512ec000)
        libc.so.6 => /lib64/libc.so.6 (0x000000396ce00000)
        libm.so.6 => /lib64/libm.so.6 (0x000000396d600000)
        /lib64/ld-linux-x86-64.so.2 (0x000000396ca00000)

$ bazel-out/host/bin/third_party/ijar/ijar third_party/maven_model/maven-aether-provider-3.2.3.jar bazel-out/local_linux-fastbuild/genfiles/third_party/_ijar/maven_model/third_party/maven_model/maven-aether-provider-3.2.3-ijar.jar
(no output messages)

jdoerrie · 2016-04-17T14:48:47Z

I would like to thank everyone involved in this issue for their efforts. I am in a similar situation as @rdipietro, trying to compile bazel on a cluster with a custom gcc location but standard ld location (/usr/bin/ld). Simply adding "-B/usr/bin", after this line https://github.com/bazelbuild/bazel/blob/master/tools/cpp/cc_configure.bzl#L166 fixes my build errors, so your assumption is correct, @damienmg :)

damienmg · 2016-04-19T10:54:01Z

Thanks for the feedback :)

Fixes #760. -- MOS_MIGRATED_REVID=120217217

FokkeDijkstra · 2016-05-31T13:02:53Z

When using an newer GCC on CentOS 6.7 (e.g. GCC 4.9.3) a newer version of binutils is also required. The fix of adding"-B/usr/bin"to the list of gcc options does not work in that case.
I assume the right fix would be to add the correct path of the linker that is actually used.

It could be done by having something like:

"-B" + str(repository_ctx.path(_which(repository_ctx,"ld","/usr/bin")).dirname),

twice in cc_configure.bzl
Although I guess someone familiar with the code would create a better version.

I've managed to compile bazel using this line instead of "-B/usr/bin".
Adding"-B/usr/bin" as well probably causes no harm.

fgeorgatos · 2016-06-09T12:32:28Z

fyi. EasyBuild tool resolves much of the automation needs across binutils+GCC deliveries and does the task consistently; it relies upon environment-modules, in case you care about that part.

i3v · 2016-12-07T12:45:12Z

I was able to build bazel-0.4.1 and tensorflow-0.12.0rc0 from source, without having root privileges. The system only had gcc-4.4.7, so I had to use my own gcc-4.9.4.
To overcome issues like gcc: error trying to exec 'as': execvp: No such file or directory, I've hardcoded paths to ld,nm and as. So, this issue seem to be related to #1713

jimht011 · 2017-01-05T20:30:23Z

It seems most effort here is to use a new gcc and glibc other than the default one in old version OS such as centos 6. But our case is a little different. We use javacpp to load tensorflow trained model . Unfortunately, our J2EE environment is still using RHEL 6.8 which has glibc 2.12. Obviously, the two tensorflow libraries published by google were built in an environment with a glibc equal or higher than 2.14. It seems Bazel needs 2.14 or higher version to run . Is there a way we can run bazel against glibc 2.14 or higher but compile tensorflow against glibc 2.12 ?

Appreciate your comments.

damienmg · 2017-01-09T11:30:10Z

Hi @jimht011: can you open a separate issue for helping for your case?

jcarlostrillo · 2017-02-10T07:58:09Z

I also was able to build Bazel 0.4.4 and Tensorflow v1.0.0-rc2 on CentOS 6.8.
I was using the gcc compiler in 4.8.2 version.

What about the GLIBC library? Well I was using the GLIBC_2.14 version on my CentOS. How? I took the version included by NoMachine (more info here https://www.nomachine.com/download/download&id=5). Once installed, you can import that library in a session terminal (tmux) with the following command: export LD_LIBRARY_PATH="/usr/NX/lib:/opt/glibc-2.14/lib:$LD_LIBRARY_PATH"

blappm · 2017-12-13T09:33:08Z

Always adding -B/usr/bin is wrong, it breaks EasyBuild Tensorflow builds on CentOS6, since the assembler there is too old.

thiell · 2018-01-31T19:19:49Z

I agree with @blappm, hard-coding -B/usr/bin is wrong and break the build on older OS when /usr/bin shouldn't be used at all by gcc. Removing the two hardcoded lines from tools/cpp/unix_cc_configure.bzl does actually fix this particular issue...

jxy · 2018-03-09T04:35:13Z

Why can't bazel simply respect my PATH?

Such was what I kept screaming to google until it led me to this issue. When I set up my compiler tool chain, I expect the first cc and ld found in my PATH to be the working combination. Yet bazel secretly used my cc and instructed my cc to pick a funny ld.

wincle · 2018-05-11T07:02:22Z

@jxy Same problems making me crazy!

caot · 2018-07-14T03:18:20Z

##@jxy while building tensorflow-gpu from source, it worked to change tensorflow-1.7.0/third_party/gpus/crosstool/CROSSTOOL_nvcc.tpl as the follows that might be helpful.

damienmg added the under investigation label Jan 11, 2016

damienmg mentioned this issue Apr 19, 2016

Faile to install bazel in linux #1152

Closed

bazel-io closed this as completed in a20352e Apr 19, 2016

damienmg added a commit that referenced this issue Apr 21, 2016

cc_configure: always add -B/usr/bin to the list of gcc option

42b0c5f

Fixes #760. -- MOS_MIGRATED_REVID=120217217

akors mentioned this issue Apr 21, 2016

Building with ccache fails with auto-configuration #1164

Closed

damienmg added a commit that referenced this issue Apr 22, 2016

cc_configure: always add -B/usr/bin to the list of gcc option

21090b4

Fixes #760. -- MOS_MIGRATED_REVID=120217217

stephenroller mentioned this issue Apr 26, 2016

CentOS 6.7 building with custom GCC #1201

Closed

puneet336 mentioned this issue May 7, 2016

Tensorflow GPU r0.7 compilation => gcc: language cuda not recognized tensorflow/tensorflow#2253

Closed

fgeorgatos mentioned this issue Jun 13, 2016

Question as regards dependency on fresher glibc #1388

Closed

kirilg mentioned this issue Jul 15, 2016

undeclared inclusion(s) in rule '@nanopb_git//:nanopb' tensorflow/serving#125

Closed

frenchwr mentioned this issue Oct 10, 2016

S_IREAD and S_IWRITE errors from gif_archive in CentOS 6.8 Build tensorflow/tensorflow#4882

Closed

sjain-stanford mentioned this issue Nov 17, 2017

Failed to build Bazel 0.7.0 from source on RHEL 6.8: "C++ compilation of rule ... failed" #4107

Closed

thiell unassigned damienmg Jan 31, 2018

georgi-d mentioned this issue Jul 19, 2018

Stop unconditionally adding -B/usr/bin flag to C/C++ toolchains #5634

Closed

YuMan-Tam mentioned this issue Aug 14, 2018

Build Failure: /lib64/libc.so.6: version `GLIBC_2.14' not found (with JAVA_HOME and CC set) #5893

Closed

aiuto added team-Configurability platforms, toolchains, cquery, select(), config transitions and removed z-category: extensibility > configurability labels Feb 4, 2019

failed to install bazel on Red Hat 6.7 #760

failed to install bazel on Red Hat 6.7 #760

Comments

digitalsword commented Jan 8, 2016

damienmg commented Jan 11, 2016

digitalsword commented Jan 11, 2016

damienmg commented Jan 11, 2016

digitalsword commented Jan 11, 2016

damienmg commented Jan 11, 2016

digitalsword commented Jan 11, 2016

digitalsword commented Jan 11, 2016

damienmg commented Jan 11, 2016

digitalsword commented Jan 11, 2016

digitalsword commented Jan 15, 2016

damienmg commented Jan 15, 2016

digitalsword commented Jan 16, 2016

damienmg commented Jan 16, 2016

digitalsword commented Jan 18, 2016

damienmg commented Jan 18, 2016

digitalsword commented Jan 18, 2016

damienmg commented Jan 18, 2016

ulfjack commented Jan 21, 2016

damienmg commented Jan 21, 2016

weijianwen commented Jan 22, 2016

Customization made to CROSSTOOL

Error messages

Things that may help

damienmg commented Jan 22, 2016

damienmg commented Jan 22, 2016

weijianwen commented Jan 22, 2016

damienmg commented Jan 22, 2016

weijianwen commented Jan 22, 2016

damienmg commented Jan 22, 2016

weijianwen commented Jan 22, 2016

damienmg commented Jan 22, 2016

weijianwen commented Jan 22, 2016

jdoerrie commented Apr 17, 2016

damienmg commented Apr 19, 2016

FokkeDijkstra commented May 31, 2016

fgeorgatos commented Jun 9, 2016

i3v commented Dec 7, 2016

jimht011 commented Jan 5, 2017

damienmg commented Jan 9, 2017

jcarlostrillo commented Feb 10, 2017

blappm commented Dec 13, 2017

thiell commented Jan 31, 2018

jxy commented Mar 9, 2018

wincle commented May 11, 2018

caot commented Jul 14, 2018 • edited Loading

Customization made to `CROSSTOOL`

caot commented Jul 14, 2018 •

edited

Loading