Using with py_test #2

chrislovecnm · 2020-07-29T21:41:52Z

I have followed your instructions and your example does not seem to work with py_test. How do you cause the installed toolchain to be used instead of the host interpreter?

The text was updated successfully, but these errors were encountered:

kku1993 · 2020-07-29T22:55:42Z

Hi @chrislovecnm, thanks got giving this a try.

I just added a py_test example to the repo and it seems to work for me. Can you share more detail on our setup?

chrislovecnm · 2020-07-29T23:49:01Z

Thanks for the fast response! I am importing your example into my project so that we can compare apples to apples.

chrislovecnm · 2020-07-29T23:58:55Z

have imported your example, into my code, and I am still getting the same error.

Here is my PR: cockroachdb/cockroach-operator#115

Here is my error:

exec ${PAGER:-/usr/bin/less} "$0" || exit 1
Executing tests from //hack:verify_boilerplate_runner
-----------------------------------------------------------------------------
/usr/bin/env: 'python': No such file or directory

I am on a Debian 10 server using bazel 3.4.1. This server does not have a python binary install, it only has python3. Also, I need to do CI on a server with has, of course, python 2 installed on it.

The command that I am running is:

bazel test //hack/...

chrislovecnm · 2020-07-29T23:59:51Z

I broke up the python script into a library in order to see if that fixed the problem. I would rather just have one python file.

chrislovecnm · 2020-07-30T00:04:12Z

Here is my tool chain debug as well:

bazel build --logging=6 --toolchain_resolution_debug --incompatible_py3_is_default --python_version=PY3 //hack/...
DEBUG: Rule 'rules_python' indicated that a canonical reproducible form can be obtained by modifying arguments shallow_since = "1592977862 -0700"
DEBUG: Repository rules_python instantiated at:
  no stack (--record_rule_instantiation_callstack not enabled)
Repository rule git_repository defined at:
  /home/clove/.cache/bazel/_bazel_clove/6daf43ef28bfabb61bbb29fb23d1665c/external/bazel_tools/tools/build_defs/repo/git.bzl:195:33: in <toplevel>
INFO: Build option --toolchain_resolution_debug has changed, discarding analysis cache.
INFO: ToolchainResolution: Selected execution platform @local_config_platform//:host,
INFO: ToolchainResolution: Selected execution platform @local_config_platform//:host,
INFO: ToolchainResolution: Looking for toolchain of type @bazel_tools//tools/cpp:toolchain_type...
INFO: ToolchainResolution: Looking for toolchain of type @bazel_tools//tools/python:toolchain_type...
INFO: ToolchainResolution:   Considering toolchain @local_config_cc//:cc-compiler-armeabi-v7a...
INFO: ToolchainResolution:   Considering toolchain @bazel_tools//tools/python:_autodetecting_py_runtime_pair...
INFO: ToolchainResolution:     Toolchain constraint @platforms//cpu:cpu has value @platforms//cpu:arm, which does not match value @platforms//cpu:x86_64 from the target platform @local_config_platform//:host
INFO: ToolchainResolution:   For toolchain type @bazel_tools//tools/python:toolchain_type, possible execution platforms and toolchains: {@local_config_platform//:host -> @bazel_tools//tools/python:_autodetecting_py_runtime_pair}
INFO: ToolchainResolution:     Toolchain constraint @platforms//os:os has value @platforms//os:android, which does not match value @platforms//os:linux from the target platform @local_config_platform//:host
INFO: ToolchainResolution:   Rejected toolchain @local_config_cc//:cc-compiler-armeabi-v7a, because of target platform mismatch
INFO: ToolchainResolution:   Considering toolchain @local_config_cc//:cc-compiler-k8...
INFO: ToolchainResolution:   For toolchain type @bazel_tools//tools/cpp:toolchain_type, possible execution platforms and toolchains: {@local_config_platform//:host -> @local_config_cc//:cc-compiler-k8}
INFO: ToolchainResolution: Selected execution platform @local_config_platform//:host, type @bazel_tools//tools/cpp:toolchain_type -> toolchain @local_config_cc//:cc-compiler-k8, type @bazel_tools//tools/python:toolchain_type -> toolchain @bazel_tools//tools/python:_autodetecting_py_runtime_pair
INFO: ToolchainResolution: Looking for toolchain of type @bazel_tools//tools/cpp:toolchain_type...
INFO: ToolchainResolution:   Considering toolchain @local_config_cc//:cc-compiler-armeabi-v7a...
INFO: ToolchainResolution:     Toolchain constraint @platforms//cpu:cpu has value @platforms//cpu:arm, which does not match value @platforms//cpu:x86_64 from the target platform @local_config_platform//:host
INFO: ToolchainResolution:     Toolchain constraint @platforms//os:os has value @platforms//os:android, which does not match value @platforms//os:linux from the target platform @local_config_platform//:host
INFO: ToolchainResolution:   Rejected toolchain @local_config_cc//:cc-compiler-armeabi-v7a, because of target platform mismatch
INFO: ToolchainResolution:   Considering toolchain @local_config_cc//:cc-compiler-k8...
INFO: ToolchainResolution:   For toolchain type @bazel_tools//tools/cpp:toolchain_type, possible execution platforms and toolchains: {@local_config_platform//:host -> @local_config_cc//:cc-compiler-k8}
INFO: ToolchainResolution: Selected execution platform @local_config_platform//:host, type @bazel_tools//tools/cpp:toolchain_type -> toolchain @local_config_cc//:cc-compiler-k8
INFO: ToolchainResolution: Looking for toolchain of type @bazel_tools//tools/jdk:runtime_toolchain_type...
INFO: ToolchainResolution:   Considering toolchain @bazel_tools//tools/jdk:dummy_java_runtime...
INFO: ToolchainResolution:   For toolchain type @bazel_tools//tools/jdk:runtime_toolchain_type, possible execution platforms and toolchains: {@local_config_platform//:host -> @bazel_tools//tools/jdk:dummy_java_runtime}
INFO: ToolchainResolution: Selected execution platform @local_config_platform//:host, type @bazel_tools//tools/jdk:runtime_toolchain_type -> toolchain @bazel_tools//tools/jdk:dummy_java_runtime
INFO: ToolchainResolution: Looking for toolchain of type @bazel_tools//tools/jdk:toolchain_type...
INFO: ToolchainResolution:   Considering toolchain @bazel_tools//tools/jdk:dummy_toolchain...
INFO: ToolchainResolution:   For toolchain type @bazel_tools//tools/jdk:toolchain_type, possible execution platforms and toolchains: {@local_config_platform//:host -> @bazel_tools//tools/jdk:dummy_toolchain}
INFO: ToolchainResolution: Selected execution platform @local_config_platform//:host, type @bazel_tools//tools/jdk:toolchain_type -> toolchain @bazel_tools//tools/jdk:dummy_toolchain
INFO: Analyzed 10 targets (0 packages loaded, 15190 targets configured).
INFO: Found 10 targets...
INFO: Elapsed time: 0.926s, Critical Path: 0.03s
INFO: 0 processes.
INFO: Build completed successfully, 1 total action

chrislovecnm · 2020-07-30T00:14:24Z

I'm thinking that this is a bazel version issue. I just downloaded your project and executed:

bazel build //:main
bazel test //:main_test

And I am getting the above error as well. Also, the container rules that you are using do not work with the bazel version as well :(

kku1993 · 2020-07-30T00:17:30Z

I'm on ubuntu 18.04 with bazel 3.1. Let met try it out in a debian container.

One thing I did notice from your PR is that you don't have register_toolchains in your WORKSPACE, see [1]

[1]

bazel-hermetic-python/WORKSPACE

Line 112 in e4f3987

register_toolchains("//:my_py_toolchain")

chrislovecnm · 2020-07-30T00:29:13Z

So the toolchain is now registered. So that is fixed, but still getting the same error.

kku1993 · 2020-07-30T00:54:31Z

I was able to repro the issue on debian 10 with bazel 3.4.1.

I believe this is an issue with bazel - when doing bazel run or bazel test, bazel adds a wrapper script that uses #!/usr/bin/env python. See [1] and its related issue bazelbuild/bazel#8685

One workaround for now is to install symlink python to python3 :/

[1] https://github.com/bazelbuild/bazel/blob/d95bfbded2466bb6ea796254f3f271a73505993b/src/main/java/com/google/devtools/build/lib/bazel/rules/python/python_stub_template.txt#L1

chrislovecnm · 2020-07-30T01:05:34Z

One workaround, for now, is to install symlink python to python3 :/

In the directory that we install python in or on the system?

kku1993 · 2020-07-30T01:08:40Z

On the system.

…

On Wed, Jul 29, 2020, 18:05 Chris Love ***@***.***> wrote: One workaround, for now, is to install symlink python to python3 :/ In the directory that we install python in or on the system? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#2 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AASYHTTXVQTC27DIJHELWB3R6DBOZANCNFSM4PMDOTMA> .

chrislovecnm · 2020-07-30T01:12:51Z

SMH. Yah there goes the hermetic build. LMAO.

kku1993 · 2020-07-30T04:19:30Z

Your code itself is still run using the hermetically built python interpreter. The issue here is that bazel's wrapper script that launches the hermetically built python intepreter depends on the system's python.

chrislovecnm · 2020-07-30T14:19:55Z

I get you, and really appreciate the help!

Hermetic to me means I don’t break outside the sandbox. What is the point of this if I have to mess with the systems python. I may as well use a container. .

Since you understand this a lot better than I do, how should this be fixed? I’m pinging Simone at google to see if the priority of this bug can be raised.

chrislovecnm · 2020-07-30T15:05:09Z

So dumb question. Would this work.

If we add a symbolic link inside of the ~/.cache/bazel/_bazel_clove/6daf43ef28bfabb61bbb29fb23d1665c/external/python_interpreter/bazel_install/bin directory, and then add that folder to the path, then env would find the file.

You can find the base of that path with a bazel info output_base call.

chrislovecnm · 2020-07-30T15:27:17Z

So

bazel test --action_env="PATH=$(bazel info output_base)/external/python_interpreter:$PATH" //hack/...

sorta works, but now I am trying to figure out how to set the path more gracefully.

kku1993 · 2020-07-30T17:00:28Z

Since you understand this a lot better than I do, how should this be fixed? I’m pinging Simone at google to see if the priority of this bug can be raised.

Looks like there's an open PR for this already: bazelbuild/bazel#11434. I think this is part of the fix, and the second part is to pump the python toolchain information to the script.

If we add a symbolic link inside of the

Yeah I think that works! A more graceful way to set it would be to add the --action_env flag to .bazelrc in the root of the repo. See https://docs.bazel.build/versions/master/guide.html#bazelrc-the-bazel-configuration-file

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Using with py_test #2

Using with py_test #2

chrislovecnm commented Jul 29, 2020

kku1993 commented Jul 29, 2020

chrislovecnm commented Jul 29, 2020

chrislovecnm commented Jul 29, 2020 •

edited

Loading

chrislovecnm commented Jul 29, 2020

chrislovecnm commented Jul 30, 2020 •

edited

Loading

chrislovecnm commented Jul 30, 2020

kku1993 commented Jul 30, 2020

chrislovecnm commented Jul 30, 2020

kku1993 commented Jul 30, 2020

chrislovecnm commented Jul 30, 2020

kku1993 commented Jul 30, 2020 via email

chrislovecnm commented Jul 30, 2020

kku1993 commented Jul 30, 2020

chrislovecnm commented Jul 30, 2020 •

edited

Loading

chrislovecnm commented Jul 30, 2020

chrislovecnm commented Jul 30, 2020

kku1993 commented Jul 30, 2020

Using with py_test #2

Using with py_test #2

Comments

chrislovecnm commented Jul 29, 2020

kku1993 commented Jul 29, 2020

chrislovecnm commented Jul 29, 2020

chrislovecnm commented Jul 29, 2020 • edited Loading

chrislovecnm commented Jul 29, 2020

chrislovecnm commented Jul 30, 2020 • edited Loading

chrislovecnm commented Jul 30, 2020

kku1993 commented Jul 30, 2020

chrislovecnm commented Jul 30, 2020

kku1993 commented Jul 30, 2020

chrislovecnm commented Jul 30, 2020

kku1993 commented Jul 30, 2020 via email

chrislovecnm commented Jul 30, 2020

kku1993 commented Jul 30, 2020

chrislovecnm commented Jul 30, 2020 • edited Loading

chrislovecnm commented Jul 30, 2020

chrislovecnm commented Jul 30, 2020

kku1993 commented Jul 30, 2020

chrislovecnm commented Jul 29, 2020 •

edited

Loading

chrislovecnm commented Jul 30, 2020 •

edited

Loading

chrislovecnm commented Jul 30, 2020 •

edited

Loading