Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Build fails/incomplete on requirements.txt schema not followed correctly #2410

Closed
Github-dm-CDE opened this issue May 15, 2024 · 9 comments · Fixed by #2464
Closed

Build fails/incomplete on requirements.txt schema not followed correctly #2410

Github-dm-CDE opened this issue May 15, 2024 · 9 comments · Fixed by #2464
Assignees
Labels

Comments

@Github-dm-CDE
Copy link

Github-dm-CDE commented May 15, 2024

Since python-poetry/poetry-plugin-export#258 was merged, the temporary requirements.txt format's changed. This makes local project dependencies unparseable. The build works, but the pex entrypoints do not exist / changed.

requirements.txt with poetry-plugin-export@1.7.1

--index-url https://gitlab-ci-token:[MASKED]@[MASKED].com/api/v4/projects/13599/packages/pypi/simple

alabaster==0.7.16 ; python_version >= "3.9" and python_version < "3.10"
annotated-types==0.6.0 ; python_version >= "3.9" and python_version < "3.10"
anyascii==0.3.2 ; python_version >= "3.9" and python_version < "3.10"
astroid==3.2.0 ; python_version >= "3.9" and python_version < "3.10"
asttokens==2.4.1 ; python_version >= "3.9" and python_version < "3.10"
autodoc-pydantic==2.2.0 ; python_version >= "3.9" and python_version < "3.10"
babel==2.15.0 ; python_version >= "3.9" and python_version < "3.10"
bda-base-dependencies @ file:///builds/cxa/cxa-ck/spark-sources/bda_base_dependencies ; python_version >= "3.9" and python_version < "3.10"
better-exceptions==0.3.3 ; python_version >= "3.9" and python_version < "3.10"
blinker==1.8.2 ; python_version >= "3.9" and python_version < "3.10"
bokeh==3.4.1 ; python_version >= "3.9" and python_version < "3.10"
certifi==2024.2.2 ; python_version >= "3.9" and python_version < "3.10"
cfgv==3.4.0 ; python_version >= "3.9" and python_version < "3.10"
charset-normalizer==3.3.2 ; python_version >= "3.9" and python_version < "3.10"
ck-spark-sources @ file:///builds/cxa/cxa-ck/spark-sources/sources ; python_version >= "3.9" and python_version < "3.10"
click==8.1.7 ; python_version >= "3.9" and python_version < "3.10"
colorama==0.4.6 ; python_version >= "3.9" and python_version < "3.10" and (sys_platform == "win32" or platform_system == "Windows")
contourpy==1.2.1 ; python_version >= "3.9" and python_version < "3.10"
coverage[toml]==7.5.1 ; python_version >= "3.9" and python_version < "3.10"
decorator==5.1.1 ; python_version >= "3.9" and python_version < "3.10"
distlib==0.3.8 ; python_version >= "3.9" and python_version < "3.10"
dm-knox-token==2.0.18 ; python_version >= "3.9" and python_version < "3.10"
dm-livy-submit==2.3.12 ; python_version >= "3.9" and python_version < "3.10"
dm-webhdfs==9.1.17 ; python_version >= "3.9" and python_version < "3.10"
docutils==0.20.1 ; python_version >= "3.9" and python_version < "3.10"
exceptiongroup==1.2.1 ; python_version >= "3.9" and python_version < "3.10"
executing==2.0.1 ; python_version >= "3.9" and python_version < "3.10"
filelock==3.14.0 ; python_version >= "3.9" and python_version < "3.10"
flask==3.0.3 ; python_version >= "3.9" and python_version < "3.10"
freezegun==1.5.1 ; python_version >= "3.9" and python_version < "3.10"
fsspec==2024.3.1 ; python_version >= "3.9" and python_version < "3.10"
identify==2.5.36 ; python_version >= "3.9" and python_version < "3.10"
idna==3.7 ; python_version >= "3.9" and python_version < "3.10"
imagesize==1.4.1 ; python_version >= "3.9" and python_version < "3.10"
importlib-metadata==7.1.0 ; python_version >= "3.9" and python_version < "3.10"
iniconfig==2.0.0 ; python_version >= "3.9" and python_version < "3.10"
ipdb==0.13.13 ; python_version >= "3.9" and python_version < "3.10"
ipython==8.18.1 ; python_version >= "3.9" and python_version < "3.10"
itsdangerous==2.2.0 ; python_version >= "3.9" and python_version < "3.10"
jedi==0.19.1 ; python_version >= "3.9" and python_version < "3.10"
jinja2==3.1.4 ; python_version >= "3.9" and python_version < "3.10"
json2html==1.3.0 ; python_version >= "3.9" and python_version < "3.10"
loguru==0.7.2 ; python_version >= "3.9" and python_version < "3.10"
markdown-it-py==3.0.0 ; python_version >= "3.9" and python_version < "3.10"
markupsafe==2.1.5 ; python_version >= "3.9" and python_version < "3.10"
matplotlib-inline==0.1.7 ; python_version >= "3.9" and python_version < "3.10"
mdit-py-plugins==0.4.1 ; python_version >= "3.9" and python_version < "3.10"
mdurl==0.1.2 ; python_version >= "3.9" and python_version < "3.10"
myst-parser==3.0.1 ; python_version >= "3.9" and python_version < "3.10"
nodeenv==1.8.0 ; python_version >= "3.9" and python_version < "3.10"
numpy==1.26.4 ; python_version >= "3.9" and python_version < "3.10"
packaging==24.0 ; python_version >= "3.9" and python_version < "3.10"
pandas==2.2.2 ; python_version >= "3.9" and python_version < "3.10"
parso==0.8.4 ; python_version >= "3.9" and python_version < "3.10"
pendulum==3.0.0 ; python_version >= "3.9" and python_version < "3.10"
pexpect==4.9.0 ; python_version >= "3.9" and python_version < "3.10" and sys_platform != "win32"
pillow==10.3.0 ; python_version >= "3.9" and python_version < "3.10"
platformdirs==4.2.1 ; python_version >= "3.9" and python_version < "3.10"
pluggy==1.5.0 ; python_version >= "3.9" and python_version < "3.10"
pre-commit==3.7.1 ; python_version >= "3.9" and python_version < "3.10"
prompt-toolkit==3.0.43 ; python_version >= "3.9" and python_version < "3.10"
ptyprocess==0.7.0 ; python_version >= "3.9" and python_version < "3.10" and sys_platform != "win32"
pure-eval==0.2.2 ; python_version >= "3.9" and python_version < "3.10"
py4j==0.10.9.5 ; python_version >= "3.9" and python_version < "3.10"
pydantic-core==2.18.2 ; python_version >= "3.9" and python_version < "3.10"
pydantic-settings==2.2.1 ; python_version >= "3.9" and python_version < "3.10"
pydantic==2.7.1 ; python_version >= "3.9" and python_version < "3.10"
pygments==2.18.0 ; python_version >= "3.9" and python_version < "3.10"
pyspark==3.3.0 ; python_version >= "3.9" and python_version < "3.10"
pytest-check==2.3.1 ; python_version >= "3.9" and python_version < "3.10"
pytest-cov==5.0.0 ; python_version >= "3.9" and python_version < "3.10"
pytest-flaskserver==1.2.10 ; python_version >= "3.9" and python_version < "3.10"
pytest==8.2.0 ; python_version >= "3.9" and python_version < "3.10"
python-dateutil==2.9.0.post0 ; python_version >= "3.9" and python_version < "3.10"
python-dotenv==1.0.1 ; python_version >= "3.9" and python_version < "3.10"
pytz==2022.7.1 ; python_version >= "3.9" and python_version < "3.10"
pyxattr==0.8.1 ; python_version >= "3.9" and python_version < "3.10"
pyyaml==6.0.1 ; python_version >= "3.9" and python_version < "3.10"
requests==2.31.0 ; python_version >= "3.9" and python_version < "3.10"
rich==13.7.1 ; python_version >= "3.9" and python_version < "3.10"
setuptools==69.5.1 ; python_version >= "3.9" and python_version < "3.10"
six==1.16.0 ; python_version >= "3.9" and python_version < "3.10"
snowballstemmer==2.2.0 ; python_version >= "3.9" and python_version < "3.10"
sphinx-autoapi==3.0.0 ; python_version >= "3.9" and python_version < "3.10"
sphinx-rtd-theme==2.0.0 ; python_version >= "3.9" and python_version < "3.10"
sphinx==7.3.7 ; python_version >= "3.9" and python_version < "3.10"
sphinxcontrib-applehelp==1.0.8 ; python_version >= "3.9" and python_version < "3.10"
sphinxcontrib-devhelp==1.0.6 ; python_version >= "3.9" and python_version < "3.10"
sphinxcontrib-htmlhelp==2.0.5 ; python_version >= "3.9" and python_version < "3.10"
sphinxcontrib-jquery==4.1 ; python_version >= "3.9" and python_version < "3.10"
sphinxcontrib-jsmath==1.0.1 ; python_version >= "3.9" and python_version < "3.10"
sphinxcontrib-qthelp==1.0.7 ; python_version >= "3.9" and python_version < "3.10"
sphinxcontrib-serializinghtml==1.1.10 ; python_version >= "3.9" and python_version < "3.10"
stack-data==0.6.3 ; python_version >= "3.9" and python_version < "3.10"
table-builder @ file:///builds/cxa/cxa-ck/spark-sources/table_builder ; python_version >= "3.9" and python_version < "3.10"
time-machine==2.14.1 ; python_version >= "3.9" and python_version < "3.10" and implementation_name != "pypy"
tomli==2.0.1 ; python_version >= "3.9" and python_version < "3.10"
tornado==6.4 ; python_version >= "3.9" and python_version < "3.10"
traitlets==5.14.3 ; python_version >= "3.9" and python_version < "3.10"
typing-extensions==4.11.0 ; python_version >= "3.9" and python_version < "3.10"
tzdata==2024.1 ; python_version >= "3.9" and python_version < "3.10"
urllib3==2.2.1 ; python_version >= "3.9" and python_version < "3.10"
utils @ file:///builds/cxa/cxa-ck/spark-sources/utils ; python_version >= "3.9" and python_version < "3.10"
virtualenv==20.26.1 ; python_version >= "3.9" and python_version < "3.10"
wcwidth==0.2.13 ; python_version >= "3.9" and python_version < "3.10"
werkzeug==3.0.3 ; python_version >= "3.9" and python_version < "3.10"
win32-setctime==1.1.0 ; python_version >= "3.9" and python_version < "3.10" and sys_platform == "win32"
xyzservices==2024.4.0 ; python_version >= "3.9" and python_version < "3.10"
zipp==3.18.1 ; python_version >= "3.9" and python_version < "3.10"

requirements.txt with poetry-plugin-export@1.8.0

--index-url https://gitlab-ci-token:[MASKED]@[MASKED].com/api/v4/projects/13599/packages/pypi/simple

-e file:///builds/cxa/cxa-ck/spark-sources/bda_base_dependencies ; python_version >= "3.9" and python_version < "3.10"
-e file:///builds/cxa/cxa-ck/spark-sources/sources ; python_version >= "3.9" and python_version < "3.10"
-e file:///builds/cxa/cxa-ck/spark-sources/table_builder ; python_version >= "3.9" and python_version < "3.10"
-e file:///builds/cxa/cxa-ck/spark-sources/utils ; python_version >= "3.9" and python_version < "3.10"
alabaster==0.7.16 ; python_version >= "3.9" and python_version < "3.10"
annotated-types==0.6.0 ; python_version >= "3.9" and python_version < "3.10"
anyascii==0.3.2 ; python_version >= "3.9" and python_version < "3.10"
astroid==3.2.0 ; python_version >= "3.9" and python_version < "3.10"
asttokens==2.4.1 ; python_version >= "3.9" and python_version < "3.10"
autodoc-pydantic==2.2.0 ; python_version >= "3.9" and python_version < "3.10"
babel==2.15.0 ; python_version >= "3.9" and python_version < "3.10"
better-exceptions==0.3.3 ; python_version >= "3.9" and python_version < "3.10"
blinker==1.8.2 ; python_version >= "3.9" and python_version < "3.10"
bokeh==3.4.1 ; python_version >= "3.9" and python_version < "3.10"
certifi==2024.2.2 ; python_version >= "3.9" and python_version < "3.10"
cfgv==3.4.0 ; python_version >= "3.9" and python_version < "3.10"
charset-normalizer==3.3.2 ; python_version >= "3.9" and python_version < "3.10"
click==8.1.7 ; python_version >= "3.9" and python_version < "3.10"
colorama==0.4.6 ; python_version >= "3.9" and python_version < "3.10" and (sys_platform == "win32" or platform_system == "Windows")
contourpy==1.2.1 ; python_version >= "3.9" and python_version < "3.10"
coverage[toml]==7.5.1 ; python_version >= "3.9" and python_version < "3.10"
decorator==5.1.1 ; python_version >= "3.9" and python_version < "3.10"
distlib==0.3.8 ; python_version >= "3.9" and python_version < "3.10"
dm-knox-token==2.0.18 ; python_version >= "3.9" and python_version < "3.10"
dm-livy-submit==2.3.12 ; python_version >= "3.9" and python_version < "3.10"
dm-webhdfs==9.1.16 ; python_version >= "3.9" and python_version < "3.10"
docutils==0.20.1 ; python_version >= "3.9" and python_version < "3.10"
exceptiongroup==1.2.1 ; python_version >= "3.9" and python_version < "3.10"
executing==2.0.1 ; python_version >= "3.9" and python_version < "3.10"
filelock==3.14.0 ; python_version >= "3.9" and python_version < "3.10"
flask==3.0.3 ; python_version >= "3.9" and python_version < "3.10"
freezegun==1.5.1 ; python_version >= "3.9" and python_version < "3.10"
fsspec==2024.3.1 ; python_version >= "3.9" and python_version < "3.10"
identify==2.5.36 ; python_version >= "3.9" and python_version < "3.10"
idna==3.7 ; python_version >= "3.9" and python_version < "3.10"
imagesize==1.4.1 ; python_version >= "3.9" and python_version < "3.10"
importlib-metadata==7.1.0 ; python_version >= "3.9" and python_version < "3.10"
iniconfig==2.0.0 ; python_version >= "3.9" and python_version < "3.10"
ipdb==0.13.13 ; python_version >= "3.9" and python_version < "3.10"
ipython==8.18.1 ; python_version >= "3.9" and python_version < "3.10"
itsdangerous==2.2.0 ; python_version >= "3.9" and python_version < "3.10"
jedi==0.19.1 ; python_version >= "3.9" and python_version < "3.10"
jinja2==3.1.4 ; python_version >= "3.9" and python_version < "3.10"
json2html==1.3.0 ; python_version >= "3.9" and python_version < "3.10"
loguru==0.7.2 ; python_version >= "3.9" and python_version < "3.10"
markdown-it-py==3.0.0 ; python_version >= "3.9" and python_version < "3.10"
markupsafe==2.1.5 ; python_version >= "3.9" and python_version < "3.10"
matplotlib-inline==0.1.7 ; python_version >= "3.9" and python_version < "3.10"
mdit-py-plugins==0.4.0 ; python_version >= "3.9" and python_version < "3.10"
mdurl==0.1.2 ; python_version >= "3.9" and python_version < "3.10"
myst-parser==3.0.1 ; python_version >= "3.9" and python_version < "3.10"
nodeenv==1.8.0 ; python_version >= "3.9" and python_version < "3.10"
numpy==1.26.4 ; python_version >= "3.9" and python_version < "3.10"
packaging==24.0 ; python_version >= "3.9" and python_version < "3.10"
pandas==2.2.2 ; python_version >= "3.9" and python_version < "3.10"
parso==0.8.4 ; python_version >= "3.9" and python_version < "3.10"
pendulum==3.0.0 ; python_version >= "3.9" and python_version < "3.10"
pexpect==4.9.0 ; python_version >= "3.9" and python_version < "3.10" and sys_platform != "win32"
pillow==10.3.0 ; python_version >= "3.9" and python_version < "3.10"
platformdirs==4.2.1 ; python_version >= "3.9" and python_version < "3.10"
pluggy==1.5.0 ; python_version >= "3.9" and python_version < "3.10"
pre-commit==3.7.0 ; python_version >= "3.9" and python_version < "3.10"
prompt-toolkit==3.0.43 ; python_version >= "3.9" and python_version < "3.10"
ptyprocess==0.7.0 ; python_version >= "3.9" and python_version < "3.10" and sys_platform != "win32"
pure-eval==0.2.2 ; python_version >= "3.9" and python_version < "3.10"
py4j==0.10.9.5 ; python_version >= "3.9" and python_version < "3.10"
pydantic-core==2.18.2 ; python_version >= "3.9" and python_version < "3.10"
pydantic-settings==2.2.1 ; python_version >= "3.9" and python_version < "3.10"
pydantic==2.7.1 ; python_version >= "3.9" and python_version < "3.10"
pygments==2.18.0 ; python_version >= "3.9" and python_version < "3.10"
pyspark==3.3.0 ; python_version >= "3.9" and python_version < "3.10"
pytest-check==2.3.1 ; python_version >= "3.9" and python_version < "3.10"
pytest-cov==5.0.0 ; python_version >= "3.9" and python_version < "3.10"
pytest-flaskserver==1.2.10 ; python_version >= "3.9" and python_version < "3.10"
pytest==8.2.0 ; python_version >= "3.9" and python_version < "3.10"
python-dateutil==2.9.0.post0 ; python_version >= "3.9" and python_version < "3.10"
python-dotenv==1.0.1 ; python_version >= "3.9" and python_version < "3.10"
pytz==2022.7.1 ; python_version >= "3.9" and python_version < "3.10"
pyxattr==0.8.1 ; python_version >= "3.9" and python_version < "3.10"
pyyaml==6.0.1 ; python_version >= "3.9" and python_version < "3.10"
requests==2.31.0 ; python_version >= "3.9" and python_version < "3.10"
rich==13.7.1 ; python_version >= "3.9" and python_version < "3.10"
setuptools==69.5.1 ; python_version >= "3.9" and python_version < "3.10"
six==1.16.0 ; python_version >= "3.9" and python_version < "3.10"
snowballstemmer==2.2.0 ; python_version >= "3.9" and python_version < "3.10"
sphinx-autoapi==3.0.0 ; python_version >= "3.9" and python_version < "3.10"
sphinx-rtd-theme==2.0.0 ; python_version >= "3.9" and python_version < "3.10"
sphinx==7.3.7 ; python_version >= "3.9" and python_version < "3.10"
sphinxcontrib-applehelp==1.0.8 ; python_version >= "3.9" and python_version < "3.10"
sphinxcontrib-devhelp==1.0.6 ; python_version >= "3.9" and python_version < "3.10"
sphinxcontrib-htmlhelp==2.0.5 ; python_version >= "3.9" and python_version < "3.10"
sphinxcontrib-jquery==4.1 ; python_version >= "3.9" and python_version < "3.10"
sphinxcontrib-jsmath==1.0.1 ; python_version >= "3.9" and python_version < "3.10"
sphinxcontrib-qthelp==1.0.7 ; python_version >= "3.9" and python_version < "3.10"
sphinxcontrib-serializinghtml==1.1.10 ; python_version >= "3.9" and python_version < "3.10"
stack-data==0.6.3 ; python_version >= "3.9" and python_version < "3.10"
time-machine==2.14.1 ; python_version >= "3.9" and python_version < "3.10" and implementation_name != "pypy"
tomli==2.0.1 ; python_version >= "3.9" and python_version < "3.10"
tornado==6.4 ; python_version >= "3.9" and python_version < "3.10"
traitlets==5.14.3 ; python_version >= "3.9" and python_version < "3.10"
typing-extensions==4.11.0 ; python_version >= "3.9" and python_version < "3.10"
tzdata==2024.1 ; python_version >= "3.9" and python_version < "3.10"
urllib3==2.2.1 ; python_version >= "3.9" and python_version < "3.10"
virtualenv==20.26.1 ; python_version >= "3.9" and python_version < "3.10"
wcwidth==0.2.13 ; python_version >= "3.9" and python_version < "3.10"
werkzeug==3.0.3 ; python_version >= "3.9" and python_version < "3.10"
win32-setctime==1.1.0 ; python_version >= "3.9" and python_version < "3.10" and sys_platform == "win32"
xyzservices==2024.4.0 ; python_version >= "3.9" and python_version < "3.10"
zipp==3.18.1 ; python_version >= "3.9" and python_version < "3.10"

With the pex created based on the requierements.txt of v1.8.0, our pipeline failed:

$ cd tests/test_table_builder/
$ ../../bda_base_dependencies/$BDA_BASE_DEPENDENCIES_DIST_WITH_SPARK_VERSION/bda-base-dependencies.pex -m pytest -m 'not system_test_only' --junitxml=unit_tests_table_builder.xml --cov=table_builder --cov-report term --cov-report xml:unit_tests_table_builder_coverage.xml
ImportError while loading conftest '/builds/cxa/cxa-ck/spark-sources/tests/test_table_builder/conftest.py'.
conftest.py:4: in <module>
    from test_helpers.hive_fixtures import add_database, add_managed_table 
E   ModuleNotFoundError: No module named 'test_helpers'
@Github-dm-CDE Github-dm-CDE changed the title Build fails on requirements.txt schema not followed correctly Build fails/incomplete on requirements.txt schema not followed correctly May 15, 2024
@Github-dm-CDE
Copy link
Author

The current workaround for us is to use the older v1.7.1 version of poetry-plugin-export with poetry self add poetry-plugin-export==1.7.1.

@jsirois
Copy link
Member

jsirois commented May 15, 2024

@Github-dm-CDE can you please provide the exact Pex command line you use to build the PEX? Normally, instead of doing a poetry export then Pex build using the exported requirements, you'd just say pex /path/to/poetry/project ... in one step and skip the export altogether.

@jsirois jsirois self-assigned this May 15, 2024
@jsirois
Copy link
Member

jsirois commented May 15, 2024

@Github-dm-CDE I just looked at Pex's requirements.txt parsing code and and it would seem to handle -e ... correctly:

pex/pex/requirements.py

Lines 264 to 269 in 36dd237

def _strip_requirement_options(line):
# type: (LogicalLine) -> Tuple[bool, Text]
processed_text = re.sub(r"^\s*(-e|--editable)\s+", "", line.processed_text)
editable = processed_text != line.processed_text
return editable, re.sub(r"\s--(global-option|install-option|hash).*$", "", processed_text)
. Now Pants, on the other hand, does not: https://github.com/pantsbuild/pants/blob/bef63e96042d6f79975a662172b4cd85e8ecd0a3/src/python/pants/util/requirements.py#L8-L17

@Github-dm-CDE are you building the PEX using Pants? If so, that's where the bug is. If you can confirm you're using Pants I'll close this as an answered question and you'll have to raise an issue with the Pants project; otherwise, please provide the Pex command line you use and I'll repro and debug.

@Github-dm-CDE
Copy link
Author

Github-dm-CDE commented May 16, 2024

@Github-dm-CDE can you please provide the exact Pex command line you use to build the PEX? Normally, instead of doing a poetry export then Pex build using the exported requirements, you'd just say pex /path/to/poetry/project ... in one step and skip the export altogether.

We are building our pex with help of this small bash script:

#!/bin/sh
set -e

rm -rf test_dist && mkdir test_dist
pip install -U pip 
poetry install --only build
poetry export --with dev --extras sources -f requirements.txt --output test_dist/test_requirements.txt --without-hashes --with-credentials
echo "=========  BUILD PEX ========="
poetry run pex -r test_dist/test_requirements.txt -o test_dist/test_sources_dependencies.pex

This is the content of our pyproject.toml

[tool.poetry]
name = "spark-sources"
version = "0"
description = "spark-sources for CXA-CK Cluster jobs"
readme = "README.md"
authors = ["<[Masked]@[Masked]>"]

[tool.poetry.dependencies]
python = "~3.9"
bda-base-dependencies = { path = "bda_base_dependencies/", develop = true, optional = true }
ck-spark-sources = { path = "sources/", develop = true, optional = true }
utils = { path = "utils/", develop = true, optional = true }
common-utils = { path = "utils/common_utils/", develop = true, optional = true }
datahub-utils = { path = "utils/datahub_utils/", develop = true, optional = true }
kafka-utils = { path = "utils/kafka_utils/", develop = true, optional = true }
logck = { path = "utils/logck/", develop = true, optional = true }
mssql-utils = { path = "utils/mssql_utils/", develop = true, optional = true }
pyspark = { version = "3.3.0", optional = true }
table-builder = { path = "table_builder/", develop = true, optional = true }
test-helpers = { path = "utils/test_helpers/", develop = true, optional = true }
zeppelin-utils = { path = "utils/zeppelin_utils/", develop = true, optional = true }

[tool.poetry.extras]
bda-base = ["bda-base-dependencies"]
pyspark = ["pyspark"]
table-builder = ["bda-base-dependencies", "common-utils", "datahub-utils", "pyspark", "table-builder", "zeppelin-utils"]
sources = [
    "bda-base-dependencies",
    "ck-spark-sources",
    "pyspark",
    "table-builder",
    "utils",
]

[tool.poetry.group.build.dependencies]
pex = "^2.1.117"

[tool.poetry.group.dev.dependencies]
Flask = "^3.0.0"
freezegun = "^1.1.0"
pytest = "^8.0.0"
pytest_check = "^2.0.0"
pytest-cov = "^5.0.0"
pre-commit = "^3.3.3"
rich = "^13.4.0"
ipdb = "^0.13.9"
dm-livy-submit = { version = "^2.3.0", source = "ck-pypi" }
dm-knox-token = { version = "^2.0.1", source = "ck-pypi" }
pytest-flaskserver = "^1.1.3"
sphinx = "^7.1.2"
autodoc-pydantic = "^2.0.1"
sphinx-rtd-theme = "^2.0.0"
sphinx-autoapi = "^3.0.0"
myst-parser = "^3.0.0"


[tool.pytest.ini_options]
addopts = "-ra -v --showlocals -s"
markers = [
    "local_test_only", # marks tests that should not run as system test
    "system_test_only"   # marks tests that should only run as system tests on cluster
]

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

[[tool.poetry.source]]
name = "dm-artifactory-pypi"
url = "https://[Masked].com/artifactory/api/pypi/pypi-remote/simple"
priority = "supplemental"

[[tool.poetry.source]]
name = "ck-pypi"
url = "https://[Masked]/api/v4/projects/13599/packages/pypi/simple/"
priority = "primary"

@jsirois
Copy link
Member

jsirois commented May 17, 2024

@Github-dm-CDE where is the test_helpers module / package found? Is that loose on disk or is it supposed to be in one of the file:/// projects packaged in the Pex?

The more information you can provide the better. For example, PEX's are just zips. You might create both the good PEX and bad PEX, unzip them both in different directories and then run a recursive diff to see what's different. That would be very helpful information.

@xebab
Copy link

xebab commented Jul 9, 2024

Hi @jsirois ,
I have been picking up this issue and I have a minimal example for you. I have checked out an arbitrary Python project (requests in this case) into my local /tmp/ directory and then I have built a PEX with the command pex -r requirements.txt -o foo.pex using the requirements.txt below

pytest
-e file:///tmp/requests

Now pex-tools foo.pex info shows that the requests library and its dependencies are included in the PEX.

{
  "bootstrap_hash": "abae6f731d4b0d818ab6d02e09294593143e0132",
  "build_properties": {
    "pex_version": "2.9.0"
  },
  "code_hash": "4155f382ffd2ab38bcf15a249b859595d933456e",
  "deps_are_wheel_files": false,
  "distributions": {
    "certifi-2024.7.4-py3-none-any.whl": "f51955747a040ad60cc9b281d2b81748fda55a5bb5c4b6b10be65f0354609302",
    "charset_normalizer-3.3.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl": "606334df13ed33c5ab6385635909cf91fb213ef7b11022f40e7a3161c7b2ee98",
    "exceptiongroup-1.2.1-py3-none-any.whl": "f74bb5f1b9d4d177eb069806e20fd043b788e4275f0bd3405badabfc0103dac3",
    "idna-3.7-py3-none-any.whl": "243156850dfb4ab00c4b9604d58d83e722e41eb0f445358feb23af5701231307",
    "iniconfig-2.0.0-py3-none-any.whl": "988e40b3b55723f6802760cae3cd0319699d3e69ae7f34c7793d38fb44d579b7",
    "packaging-24.1-py3-none-any.whl": "25baba288a0f0e2ebe1b0117cad380c5d4c0137aff4a2c7755f92cffd0eb38f8",
    "pluggy-1.5.0-py3-none-any.whl": "6826f70172987927d47205f5bc5b1b18c380201149e5b63bc90e4d1645e7a8d9",
    "pytest-8.2.2-py3-none-any.whl": "ce97a2c0ec29fd42f042f7e76e7f857fb83919d9054169b16988d628f47b43f4",
    "requests-2.32.1-py3-none-any.whl": "7226b06e0a6f282f84ca65816c130920e5d01c2932e0382e9fe7afec1c993928",
    "tomli-2.0.1-py3-none-any.whl": "8af30168c90e438c0488ddb0dcc8bfbdb7f54299570473943266c8f800de58dc",
    "urllib3-2.2.2-py3-none-any.whl": "568f866565573953519a3419f2ab438633b3fb6e932725d0a1aecb46a35d9b12"
  },
  "emit_warnings": true,
  "excluded": [],
  "ignore_errors": false,
  "includes_tools": false,
  "inherit_path": "false",
  "inject_args": [],
  "inject_env": {},
  "inject_python_args": [],
  "interpreter_constraints": [],
  "max_install_jobs": 1,
  "overridden": [],
  "pex_hash": "3f284033281410a36e3ac7a6efebbc97277d6311",
  "pex_path": "",
  "pex_paths": [],
  "requirements": [
    "pytest"
  ],
  "strip_pex_env": true,
  "venv": false,
  "venv_bin_path": "false",
  "venv_copies": false,
  "venv_hermetic_scripts": true,
  "venv_site_packages_copies": false,
  "pex_root": "/home/alex/.pex"
}

However there is an error importing the package, because it is not part of the requirements array.

> ./foo.pex                                                                                                                                                                                                                               

Python 3.9.16 (main, Oct 10 2023, 14:24:18)
[GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> import requests
Traceback (most recent call last):
  File "<console>", line 1, in <module>
ModuleNotFoundError: No module named 'requests'

Hope that helps to debug the issue.

@jsirois
Copy link
Member

jsirois commented Jul 9, 2024

Thanks @xebab, that's a helpful repro case. When I get back to my keyboard on 7/14 I'll take a look and hopefully get a quick fix out.

@jsirois jsirois added the bug label Jul 9, 2024
@jsirois
Copy link
Member

jsirois commented Jul 13, 2024

Ok, this is an embarrassing bug. Although part of the requirement parsing logic handles editable, there is a part that runs before it that short circuits the editable logic. I'll have a fix PR out this afternoon and hopefully a release shortly after.

jsirois added a commit to jsirois/pex that referenced this issue Jul 14, 2024
Previously, editable requirements in requirements files were not parsed
properly by Pex. Although they did not trigger parse errors, PEXes
created from editable requirements would fail to import those
requirements at runtime despite the editable project distribution being
embedded in the PEX file.

Fixes pex-tool#2410
jsirois added a commit that referenced this issue Jul 14, 2024
Previously, editable requirements in requirements files were not parsed
properly by Pex. Although they did not trigger parse errors, PEXes
created from editable requirements would fail to import those
requirements at runtime despite the editable project distribution being
embedded in the PEX file.

Fixes #2410
@jsirois
Copy link
Member

jsirois commented Jul 14, 2024

Alright, a fix for this is now released in 2.10.1:

Thanks @Github-dm-CDE and @xebab for the report and repro case respectively.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants