pip_install, py_test doesn't work with zeep

[Topher-the-Geek]

🐞 bug report

Affected Rule

pip_install with py_test

Is this a regression?

No

Description

My requirements.txt lists just zeep. My test.py imports that package. I test is using py_test. The import of zeep ultimately fails when zeep in turn imports from lxml.

🔬 Minimal Reproduction

WORKSPACE

load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")

http_archive(
    name = "rules_python",
    url = "https://github.com/bazelbuild/rules_python/releases/download/0.1.0/rules_python-0.1.0.tar.gz",
    sha256 = "b6d46438523a3ec0f3cead544190ee13223a52f6a6765a29eae7b7cc24cc83a0",
)

load("@rules_python//python:pip.bzl", "pip_install")

pip_install(
    name = "pip",
    requirements = "//:requirements.txt"
)

BUILD

load("@rules_python//python:python.bzl", "py_test")
load("@pip//:requirements.bzl", "requirement")

py_test(
    name = "test",
    srcs = ["test.py"],
    deps = [requirement("zeep")]
)

requirements.txt

zeep

test.py

import zeep

if __name__ == "__main__":
    pass

🔥 Exception or Error

bazel test --test_output=errors //:test

Traceback (most recent call last):
  File "/private/var/tmp/_bazel_topher/ca602e7bc266cb702bbae6c3ebbff054/sandbox/darwin-sandbox/35/execroot/__main__/bazel-out/darwin-fastbuild/bin/test.runfiles/__main__/test.py", line 2, in <module>
    import zeep
  File "/private/var/tmp/_bazel_topher/ca602e7bc266cb702bbae6c3ebbff054/sandbox/darwin-sandbox/35/execroot/__main__/bazel-out/darwin-fastbuild/bin/test.runfiles/pip/pypi__zeep/zeep/__init__.py", line 1, in <module>
    from zeep.client import AsyncClient, CachingClient, Client  # noqa
  File "/private/var/tmp/_bazel_topher/ca602e7bc266cb702bbae6c3ebbff054/sandbox/darwin-sandbox/35/execroot/__main__/bazel-out/darwin-fastbuild/bin/test.runfiles/pip/pypi__zeep/zeep/client.py", line 6, in <module>
    from zeep.transports import AsyncTransport, Transport
  File "/private/var/tmp/_bazel_topher/ca602e7bc266cb702bbae6c3ebbff054/sandbox/darwin-sandbox/35/execroot/__main__/bazel-out/darwin-fastbuild/bin/test.runfiles/pip/pypi__zeep/zeep/transports.py", line 11, in <module>
    from zeep.utils import get_media_type, get_version
  File "/private/var/tmp/_bazel_topher/ca602e7bc266cb702bbae6c3ebbff054/sandbox/darwin-sandbox/35/execroot/__main__/bazel-out/darwin-fastbuild/bin/test.runfiles/pip/pypi__zeep/zeep/utils.py", line 5, in <module>
    from lxml import etree
ImportError: cannot import name 'etree' from 'lxml' (/private/var/tmp/_bazel_topher/ca602e7bc266cb702bbae6c3ebbff054/sandbox/darwin-sandbox/35/execroot/__main__/bazel-out/darwin-fastbuild/bin/test.runfiles/pip/pypi__lxml/lxml/__init__.py)

🌍 Your Environment

Operating System:

MacOS BigSur 11.1

Output of bazel version:

bazel 4.0.0-homebrew

Rules_python version:

0.1.0

Anything else relevant?

$ python --version
Python 3.7.7

Switching py_test to py_binary, and running bazel run //:test works.

[thundergolfer]

Thanks for the reproduction. I've set it up and run bazel test successfully on my dev machine, with details shown below.

🌍 My Environment

• Operating System: MacOS Catalina 11.1
• Bazel Version: Bazel 4.0.0
• Rules_python version: 0.1.0 (b/c I copied your repo code)
• Python version 3.9.0

---

ImportError: cannot import name 'etree' from 'lxml' is a good clue to the problem I think. The etree module is a shared object file in lxml. On my machine with Python 3.9 the file is etree.cpython-39-darwin.so, which matches and thus I can import successfully.

The full path under the Bazel workspace working tree is: bazel-<workspace_name>/external/pip/pypi__lxml/lxml/etree.cpython-39-darwin.so

I'd suspect that the Python interpreter version that ran and installed zeep is not the same as the runtime that executes the test. But that doesn't really fit with this statement of yours:

Switching py_test to py_binary, and running bazel run //:test works.

Can you check this platform-specifier part of the etree shared object file? It's this bit: "cpython-39-darwin"

[Topher-the-Geek]

Thank you @thundergolfer. Knowing that you could make it work helped a lot.

I switched to Python 3.9.0, but I still had the issue. Side question: How can I tell which version of Python bazel's using? My Google-fu must be weak. I could not turn up the answer to that question.

I also see that I used bazel 4.0.0-homebrew and you used bazel 4.0.0. So, I bazel clean --expunge, uninstalled my brew bazel and installed it from the binary as explained here. I still had the issue.

I decided to investigate my environment variables, starting with PATH. Again, first I bazel clean --expunge. Then I stripped it down to the minimum of my system's default an ~/bin where bazel lives. In this case, I'm using Python 2.7.16 (Apple missed the memo). I expected bazel test //:test to fail, but to my surprise it succeeded.

Once again, first I bazel clean --expunge. I then added back the paths for pyenv. I did pyenv shell zeep-3.9.0, and the problem returned. So, I seemed to have narrowed down the culprit.

Yes, I see the etree shared object as expected:

$ ls bazel-bazel-zeep/external/pip/pypi__lxml/lxml
ElementInclude.py			doctestcompare.py			lxml.etree.h
__init__.py				etree.cpython-39-darwin.so		lxml.etree_api.h
_elementpath.cpython-39-darwin.so	etree.h					objectify.cpython-39-darwin.so
_elementpath.py				etree_api.h				pyclasslookup.py
builder.cpython-39-darwin.so		html					sax.cpython-39-darwin.so
builder.py				includes				sax.py
cssselect.py				isoschematron				usedoctest.py

I had expected pyenv to be okay, but perhaps its shims are confusing things. Should I setup Python 3.9 differently?

[Topher-the-Geek]

I dug up this hint --action_env=PATH=/bin:/usr/bin:/usr/local/bin in .bazelrc from #62. That worked. I could bazel test //:test despite pyenv having injected it's shims for the Python I run from my prompt.

[Topher-the-Geek]

I have narrowed the problem down. It doesn't involve zeep, lxml, pip, shared objects, the pip toolchain or any of that. I've filed the more focused issue #410. This issue is a red herring, so I'll close it.

[thundergolfer]

Thanks for filing separate issue and closing 👍