Build does not use proper Python path (Following instructions in DEVELOPMENT.md)

[beasteers]

This may be an issue with bazel (2.28.1, the current version), but either way it's a problem for tensorboard build that should at least be documented. I tried the installing 2.23.2, the version found in package.json, but it was too old and another package required a newer version.

I'm trying to build Tensorboard locally using the instructions found in DEVELOPMENT.md.

When I run bazel build tensorboard, I get an error that it's trying to use my system Python2.7 (/usr/bin/python) instead of my virtual environment.

In the error, it says it's using this $PATH: /usr/gnu/bin:/usr/local/bin:/bin:/usr/bin:.

Here's what it should be:

$ echo $PATH
/anaconda3/bin:/anaconda3/condabin:/anaconda3/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Users/bsteers/bin

# and other info
$ which python
/Users/bsteers/Documents/GitHub/tensorboard/tf/bin/python
$ python --version
Python 3.7.3
$ bazel version
Build label: 0.28.1
Build target: bazel-out/darwin-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Fri Jul 19 15:22:50 2019 (1563549770)
Build timestamp: 1563549770
Build timestamp as int: 1563549770

I found out from here that if I run this I get the same path so it seems like it's clearing the environmental variables or something?

$ exec env - /bin/bash -c 'echo $PATH'
/usr/gnu/bin:/usr/local/bin:/bin:/usr/bin:.

I tried adding build --action_env=PATH to my ~/.bazelrc with no luck.

Environment information (required)

Please run diagnose_tensorboard.py (link below) in the same
environment from which you normally run TensorFlow/TensorBoard, and
paste the output here:

Diagnostics

Diagnostics output

--- check: autoidentify
INFO: diagnose_tensorboard.py version 393931f9685bd7e0f3898d7dcdf28819fef54c43

--- check: general
INFO: sys.version_info: sys.version_info(major=3, minor=7, micro=3, releaselevel='final', serial=0)
INFO: os.name: posix
INFO: os.uname(): posix.uname_result(sysname='Darwin', nodename='10-18-151-131.dynapool.wireless.nyu.edu', release='18.0.0', version='Darwin Kernel Version 18.0.0: Wed Aug 22 20:13:40 PDT 2018; root:xnu-4903.201.2~1/RELEASE_X86_64', machine='x86_64')
INFO: sys.getwindowsversion(): N/A

--- check: package_management
INFO: has conda-meta: False
INFO: $VIRTUAL_ENV: '/Users/bsteers/Documents/GitHub/tensorboard/tf'

--- check: installed_packages
INFO: installed: tb-nightly==1.15.0a20190806
INFO: installed: tf-nightly==1.15.0.dev20190809
INFO: installed: tf-estimator-nightly==1.14.0.dev2019080901

--- check: tensorboard_python_version
INFO: tensorboard.version.VERSION: '1.15.0a20190806'

--- check: tensorflow_python_version
WARNING: 

  TensorFlow's `tf-nightly` package will soon be updated to TensorFlow 2.0.

  Please upgrade your code to TensorFlow 2.0:
    * https://www.tensorflow.org/beta/guide/migration_guide

  Or install the latest stable TensorFlow 1.X release:
    * `pip install -U "tensorflow==1.*"`

  Otherwise your code may be broken by the change.

  
INFO: tensorflow.__version__: '1.15.0-dev20190809'
INFO: tensorflow.__git_version__: 'v1.12.1-8422-g0d1a1413c8'

--- check: tensorboard_binary_path
INFO: which tensorboard: b'/Users/bsteers/Documents/GitHub/tensorboard/tf/bin/tensorboard\n'

--- check: readable_fqdn
INFO: socket.getfqdn(): '10-18-151-131.dynapool.wireless.nyu.edu'

--- check: stat_tensorboardinfo
INFO: directory: /var/folders/nc/30stjm_n07gcpz6djh26cf8m0000gn/T/.tensorboard-info
INFO: os.stat(...): os.stat_result(st_mode=16895, st_ino=31416359, st_dev=16777221, st_nlink=2, st_uid=501, st_gid=20, st_size=64, st_atime=1564953516, st_mtime=1565356530, st_ctime=1565356530)
INFO: mode: 0o40777

--- check: source_trees_without_genfiles
INFO: tensorboard_roots (1): ['/Users/bsteers/Documents/GitHub/tensorboard/tf/lib/python3.7/site-packages']; bad_roots (0): []

--- check: full_pip_freeze
INFO: pip freeze --all:
absl-py==0.7.1
astor==0.8.0
gast==0.2.2
google-pasta==0.1.7
grpcio==1.22.0
h5py==2.9.0
Keras-Applications==1.0.8
Keras-Preprocessing==1.1.0
Markdown==3.1.1
numpy==1.17.0
opt-einsum==2.3.2
pip==19.2.1
protobuf==3.9.1
setuptools==41.0.1
six==1.12.0
tb-nightly==1.15.0a20190806
termcolor==1.1.0
tf-estimator-nightly==1.14.0.dev2019080901
tf-nightly==1.15.0.dev20190809
Werkzeug==0.15.5
wheel==0.33.4
wrapt==1.11.2

Next steps

No action items identified. Please copy ALL of the above output,
including the lines containing only backticks, into your GitHub issue
or comment. Be sure to redact any sensitive information.

Steps to reproduce (required)

Please provide explicit commands to reproduce the problem in a fresh
environment (virtualenv, Conda environment, …). Include any necessary
configuration, such as Conda environment.yml files.

First, I installed bazel:

brew tap bazelbuild/tap
brew install bazelbuild/tap/bazel

Then I setup a virtualenv just as explained (I tried conda as well with no luck)

virtualenv tf
source tf/bin/activate
pip install --upgrade pip
pip install tf-nightly

Then to setup and run bazel I did:

cd tensorboard # the repo
bazel build tensorboard

ERROR: /Users/bsteers/Documents/GitHub/tensorboard/tensorboard/plugins/profile/pod_viewer/pod_viewer_common/BUILD:7:1: Compiling 4 TypeScript files //tensorboard/plugins/profile/pod_viewer/pod_viewer_common:pod_viewer_common failed (Exit 1) execrooter failed: error executing command bazel-out/host/bin/tensorboard/scripts/execrooter bazel-out/darwin-fastbuild/bin/tensorboard/plugins/profile/pod_viewer/pod_viewer_common/pod_viewer_common-tsc-execroot.json

Use --sandbox_debug to see verbose messages from the sandbox
Error occurred while attempting to use the default Python toolchain (@bazel_tools//tools/python:autodetecting_toolchain).
According to '/usr/bin/python -V', version is 'Python 2.7.10', but we need version 3. PATH is:

/usr/gnu/bin:/usr/local/bin:/bin:/usr/bin:.

[wchargin]

Indeed. This was a breaking change in Bazel 0.27. We’re “tracking” this
in #2355, but candidly all of us on the TensorBoard team have just
pinned our Bazel versions to 0.26.1, because there doesn’t seem to be an
easy way to resolve this.

I tried the installing 2.23.2, the version found in package.json,
but it was too old and another package required a newer version.

Yes; 0.26.1 is the minimum required version, and also happens to
be the maximum possible version. :-( I’m not sure why package.json
is pointed to 0.23.2; I’ll look into that.

If for any reason you have trouble getting 0.26.1 to work, you should be
able to run TensorBoard even at 0.28.x by invoking

bazel run //tensorboard --incompatible_use_python_toolchains=false -- \
    --logdir ~/tensorboard_data

–note the bare -- to separate Bazel flags from TensorBoard flags. This
likely won’t work forever (eventually Bazel may just remove the option
entirely), but it works for me at 0.28.1.

You’ll have to pass --incompatible_use_python_toolchains=false on
every invocation, though, or add it to a .bazelrc file in your
home directory, so if you can get 0.26.1 installed that will likely be
the easiest.

[wchargin]

And, in case you find them useful, I use the following shell aliases to
manage Bazel versions. Usage: run install_bazel 0.26.1 once, then in
any shell session write PIN_BAZEL=0.26.1 and all your subsequent
bazel invocations for that shell will use 0.26.1.

As written, this assumes that you have gpg set up with a copy of
Bazel’s signing key* trusted, as I use GPG to verify the binaries after
downloading them and before marking them as executable. You can get
Bazel’s key from https://bazel.build/bazel-release.pub.gpg; its
fingerprint is: 71A1D0EFCFEB6281FD0437C93D5919B448457EE0.

This is also written for Linux, but I expect that you could modify it
for macOS without too much trouble by changing the downloaded path
names.

Shell functions for managing Bazel versions

# schmod: securely change mode, only with valid signature
# usage: schmod [SIGNATURE] BINARY
# If not provided, SIGNATURE defaults to BINARY.sig.
schmod() (
    if [ $# -eq 1 ]; then
        sig="$1.sig"
        binary="$1"
    elif [ $# -eq 2 ]; then
        sig="$1"
        binary="$2"
    else
        printf >&2 'usage: schmod [SIGNATURE] binary\n'
        return 1
    fi
    if gpg --verify "${sig}" "${binary}"; then
        printf >&2 'schmod: signature verified; marking executable\n'
        chmod +x "${binary}"
    else
        code=$?
        printf >&2 'schmod: signature NOT verified; leaving mode unchanged\n'
        return "${code}"
    fi
)

# usage: install_bazel VERSION
# where VERSION is like "0.26.1"
install_bazel() (
    cd ~/bin/bazels
    filename="bazel-$1-linux-x86_64"
    base_url="https://github.com/bazelbuild/bazel/releases/download/$1"
    wget -q --show-progress "${base_url}/${filename}" -O "${filename}"
    wget -q --show-progress "${base_url}/${filename}.sig" -O "${filename}.sig"
    schmod "${filename}.sig" "${filename}"
)

bazel() (
    if [ -n "${PIN_BAZEL-}" ]; then
        safe_version="$(printf '%s' "${PIN_BAZEL}" | tr -d -c 'A-Za-z0-9_.-')"
        ~/bin/bazels/bazel-"${safe_version}"-linux-x86_64 "$@"
    else
        command bazel "$@"
    fi
)

[beasteers]

Awesome! thanks for the fast reply. And all of the helpful snippets. I ended up getting it to work by using:

bazel run //tensorboard --extra_toolchains=@bazel_tools//tools/python:autodetecting_toolchain_nonstrict -- ...

It looks like they're both doing about the same thing. I'm glad that it's not just me, but it sucks that it's going to be an ongoing problem.

From what I understand from bazelbuild/bazel#7899, it seems like they expect us to hardcode the interpreter paths using Python toolchains?

This is the example I keep seeing:

# In your BUILD file...
load("@bazel_tools//tools/python/toolchain.bzl", "py_runtime_pair")
py_runtime(
    name = "my_py2_runtime",
    interpreter_path = "/system/python2",
    python_version = "PY2",
)
py_runtime(
    name = "my_py3_runtime",
    interpreter_path = "/system/python3",
    python_version = "PY3",
)
py_runtime_pair(
    name = "my_py_runtime_pair",
    py2_runtime = ":my_py2_runtime",
    py3_runtime = ":my_py3_runtime",
)
toolchain(
    name = "my_toolchain",
    target_compatible_with = [...],  # optional platform constraints
    toolchain = ":my_py_runtime_pair",
    toolchain_type = "@bazel_tools//tools/python:toolchain_type",
)

# In your WORKSPACE...
register_toolchains("//my_pkg:my_toolchain")

But this just seems really extra. I don't get why they can't just use $PATH and $(which python) ...... 🤦‍♀️

[wchargin]

Great! Glad to hear that you’re unblocked.

I don't get why they can't just use $PATH and $(which python)

One of Bazel’s primary goals is to be a fully hermetic system: if it
builds on my machine, it should build on yours, with bit-for-bit
identical outputs. Depending on tools from PATH moves us farther from
that goal, by moving configuration out of the version-controlled project
and into the environment. So I can understand why non-hermetic
toolchains are no longer the default, and I can kind of even understand
why the verbosity required to opt out of hermeticity could be considered
a feature. It’s unfortunate that this doesn’t obviously fit into our use
case, perhaps most notably our need to be compatible with various
versions of TensorFlow (1.x, 2.x, stable, nightlies, GPU builds). Maybe
some day we’ll figure out a way to get the best of all worlds. :-)

I’ll close this for now; please let us know if you run into anything
else!

[beasteers]

Understood. In the style of docker I suppose. I understand clearing $PATH in that respect. Would adding a virtualenv to the build work? Honestly, I'm not sure how it's building using the system python 2 when tf-nightly was only installed in the virtualenv 🤷‍♀️

I think it would be important to note all of this in DEVELOPMENT.md though.
i.e.

---

You have to install Bazel==0.26.1 in order for the build to run successfully.

Due to changes in Bazel, in order to get versions >=0.27 working you will have to add a flag to the build command. This will work temporarily, but at some point Bazel may remove the flag.

bazel run //tensorboard \
    --incompatible_use_python_toolchains=false \
    -- --logdir=/path/to/logs

# or alternatively
bazel run //tensorboard \
    --extra_toolchains=@bazel_tools//tools/python:autodetecting_toolchain_nonstrict \
    -- --logdir=/path/to/logs

This must be run for every build call. For convenience, it can be added to your ~/.bazelrc file (or equivalent, see docs) which will add it as a flag automatically.

---

Anyways, thanks for the help!

[wchargin]

Understood. In the style of docker I suppose.

Yep, exactly.

Would adding a virtualenv to the build work?

Not sure whether you mean “creating a virtualenv as part of the Bazel
build” or “referring to an existing virtualenv from the Bazel build”.
The former is possible, and is kind of what the rules_python
rules do. But for that approach you have to explicitly specify all
the packages and versions (by design), so when I want to test against

• today’s tf-nightly,
• today’s tf-nightly-2.0-preview,
• today’s tf-nightly-gpu-2.0-preview,
• stable tensorflow==1.14,
• tf-nightly from a week ago because something changed in the last
  week and I’m trying to bisect it, and
• no TensorFlow at all

then in the simplest approach I’d need to manually patch the build file
for each different environment. Also, we’d need some kind of bot to bump
the tf-nightly version every day.

It’s possible that we could rig up something more principled by using
Bazel’s config_setting and select functionality, but that
would be more involved, and we just haven’t looked into it yet.

Honestly, I'm not sure how it's building using the system python 2
when tf-nightly was only installed in the virtualenv

With the autodetecting toolchain, the “Python interpreter” is actually a
wrapper provided by Bazel, defined in pywrapper_template.txt
and expanded in tools/python/toolchain.bzl. It just shells out
to which; as long as your virtualenv is active, that will resolve to
the virtualenv’s Python.

I think it would be important to note all of this in DEVELOPMENT.md
though.

Agreed, and thank you for writing this up! Instead of asking developers
to write their own personal .bazelrcs, I’ve gone ahead and added a
workspace-level .bazelrc to TensorBoard itself, so you should be able
to pull in latest master as of #2535 and build without any additional
configuration (i.e., you can remove your user-level .bazelrc if you
like).

Anyways, thanks for the help!

Sure thing.