Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Extras not installed correctly when using isolated_build=true #1510

Closed
janfreyberg opened this issue Jan 31, 2020 · 3 comments
Closed

Extras not installed correctly when using isolated_build=true #1510

janfreyberg opened this issue Jan 31, 2020 · 3 comments
Labels
bug:normal affects many people or has quite an impact

Comments

@janfreyberg
Copy link

I am using flit to manage my package builds. My TOML file looks like:

[build-system]
requires = ["flit"]
build-backend = "flit.buildapi"

[tool.flit.metadata]
module = "pytorch_revgrad"
author = "Jan Freyberg"
author-email = "jan.freyberg@gmail.com"
home-page = "https://github.com/janfreyberg/pytorch-revgrad"
classifiers = ["License :: OSI Approved :: MIT License"]
requires-python = ">=3.5"
description-file="README.md"
requires = [
    "numpy",
    "torch>=1.0.0"
]

[tool.flit.metadata.requires-extra]
test = [
    "tox",
    "pytest",
    "pytest-cov",
    "flake8",
    "pytest-flake8",
    "coveralls"
]

I'd like to use the test requires-extra section to manage my dependencies. Accordingly, my tox.ini file looks like:

[tox]
envlist=py35,py36,py37
isolated_build = true
extras =
    test

[testenv]
commands =
    python -m pytest

However, while I don't fully understand the reason, it seems as if the extra dependencies are not installed:

$ tox -e py37 -rvv
using tox.ini: /Users/jan/pytorch-revgrad/tox.ini (pid 29128)
using tox-3.14.3 from /Users/jan/pytorch-revgrad/.venv/lib/python3.8/site-packages/tox/__init__.py (pid 29128)
.package start: getenv /Users/jan/pytorch-revgrad/.tox/.package
.package cannot reuse: -r flag
.package create: /Users/jan/pytorch-revgrad/.tox/.package
/Users/jan/pytorch-revgrad/.venv/bin/python (/Users/jan/pytorch-revgrad/.venv/bin/python) is {'executable': '/Users/jan/pytorch-revgrad/.venv/bin/python', 'name': 'python', 'version_info': [3, 8, 0, 'final', 0], 'version': '3.8.0 (default, Nov 27 2019, 15:14:08) \n[Clang 11.0.0 (clang-1100.0.20.17)]', 'is_64': True, 'sysplatform': 'darwin'}
.package uses /Users/jan/pytorch-revgrad/.venv/bin/python
setting PATH=/Users/jan/pytorch-revgrad/.tox/.package/bin:/Users/jan/pytorch-revgrad/.venv/bin:/Users/jan/.pyenv/bin:/Users/jan/.pyenv/shims:/Users/jan/scipts:/Users/jan/.local/bin:/usr/local/opt/ruby/bin:/usr/local/lib/ruby/gems/2.6.0/bin:/Users/jan/.cargo/bin:/Users/jan/scripts:/Users/jan/.cabal/bin:/Users/jan/.ghcup/bin:/Users/jan/scripts/google-cloud-sdk/bin:/Users/jan/.poetry/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/share/dotnet:~/.dotnet/tools:/Library/Frameworks/Mono.framework/Versions/Current/Commands
[29131] /Users/jan/pytorch-revgrad/.tox$ /Users/jan/pytorch-revgrad/.venv/bin/python -m virtualenv --no-download --python /Users/jan/pytorch-revgrad/.venv/bin/python .package
Already using interpreter /Users/jan/pytorch-revgrad/.venv/bin/python
Using base prefix '/Users/jan/.pyenv/versions/3.8.0'
New python executable in /Users/jan/pytorch-revgrad/.tox/.package/bin/python
Installing setuptools, pip, wheel...
done.
.package installdeps: flit
setting PATH=/Users/jan/pytorch-revgrad/.tox/.package/bin:/Users/jan/pytorch-revgrad/.venv/bin:/Users/jan/.pyenv/bin:/Users/jan/.pyenv/shims:/Users/jan/scipts:/Users/jan/.local/bin:/usr/local/opt/ruby/bin:/usr/local/lib/ruby/gems/2.6.0/bin:/Users/jan/.cargo/bin:/Users/jan/scripts:/Users/jan/.cabal/bin:/Users/jan/.ghcup/bin:/Users/jan/scripts/google-cloud-sdk/bin:/Users/jan/.poetry/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/share/dotnet:~/.dotnet/tools:/Library/Frameworks/Mono.framework/Versions/Current/Commands
[29136] /Users/jan/pytorch-revgrad$ /Users/jan/pytorch-revgrad/.tox/.package/bin/python -m pip install flit
Looking in indexes: https://pypi.org/simple
Collecting flit
  Using cached https://files.pythonhosted.org/packages/80/f6/b10b8541b07caa4b680bdd84744056a401f142e639c9c316adaa88b9fa78/flit-2.2.0-py3-none-any.whl
Collecting requests
  Using cached https://files.pythonhosted.org/packages/51/bd/23c926cd341ea6b7dd0b2a00aba99ae0f828be89d72b2190f27c11d4b7fb/requests-2.22.0-py2.py3-none-any.whl
Collecting docutils
  Using cached https://files.pythonhosted.org/packages/81/44/8a15e45ffa96e6cf82956dd8d7af9e666357e16b0d93b253903475ee947f/docutils-0.16-py2.py3-none-any.whl
Collecting pytoml
  Using cached https://files.pythonhosted.org/packages/a5/47/c7f8a0f210ad18576840922e0b504f0b7f5f73aea4a52ab14c5b58517edf/pytoml-0.1.21-py2.py3-none-any.whl
Collecting flit_core>=2.2.0
  Using cached https://files.pythonhosted.org/packages/25/4c/0b1ed660937d96ed192c376d3983dd7b052b887c8041ae020c950c0d06f0/flit_core-2.2.0-py2.py3-none-any.whl
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1
  Using cached https://files.pythonhosted.org/packages/e8/74/6e4f91745020f967d09332bb2b8b9b10090957334692eb88ea4afe91b77f/urllib3-1.25.8-py2.py3-none-any.whl
Collecting certifi>=2017.4.17
  Using cached https://files.pythonhosted.org/packages/b9/63/df50cac98ea0d5b006c55a399c3bf1db9da7b5a24de7890bc9cfd5dd9e99/certifi-2019.11.28-py2.py3-none-any.whl
Collecting idna<2.9,>=2.5
  Using cached https://files.pythonhosted.org/packages/14/2c/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.whl
Collecting chardet<3.1.0,>=3.0.2
  Using cached https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl
Installing collected packages: urllib3, certifi, idna, chardet, requests, docutils, pytoml, flit-core, flit
Successfully installed certifi-2019.11.28 chardet-3.0.4 docutils-0.16 flit-2.2.0 flit-core-2.2.0 idna-2.8 pytoml-0.1.21 requests-2.22.0 urllib3-1.25.8
WARNING: You are using pip version 19.3.1; however, version 20.0.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
.package finish: getenv /Users/jan/pytorch-revgrad/.tox/.package after 6.96 seconds
.package start: finishvenv
write config to /Users/jan/pytorch-revgrad/.tox/.package/.tox-config1 as 'ecf9d3c64af8da48f51ade97d1900cd7454f27b5829c29f13cb2cc293f175282 /Users/jan/pytorch-revgrad/.venv/bin/python\n3.14.3 0 0 0\n00000000000000000000000000000000 flit'
.package finish: finishvenv  after 0.01 seconds
.package start: get-build-requires /Users/jan/pytorch-revgrad/.tox/.package
setting PATH=/Users/jan/pytorch-revgrad/.tox/.package/bin:/Users/jan/pytorch-revgrad/.venv/bin:/Users/jan/.pyenv/bin:/Users/jan/.pyenv/shims:/Users/jan/scipts:/Users/jan/.local/bin:/usr/local/opt/ruby/bin:/usr/local/lib/ruby/gems/2.6.0/bin:/Users/jan/.cargo/bin:/Users/jan/scripts:/Users/jan/.cabal/bin:/Users/jan/.ghcup/bin:/Users/jan/scripts/google-cloud-sdk/bin:/Users/jan/.poetry/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/share/dotnet:~/.dotnet/tools:/Library/Frameworks/Mono.framework/Versions/Current/Commands
[29142] /Users/jan/pytorch-revgrad$ /Users/jan/pytorch-revgrad/.tox/.package/bin/python .venv/lib/python3.8/site-packages/tox/helper/build_requires.py flit.buildapi '' >.tox/.package/log/.package-0.log
.package finish: get-build-requires /Users/jan/pytorch-revgrad/.tox/.package after 0.20 seconds
.package start: build_requires /Users/jan/pytorch-revgrad/.tox/.package
setting PATH=/Users/jan/pytorch-revgrad/.tox/.package/bin:/Users/jan/pytorch-revgrad/.venv/bin:/Users/jan/.pyenv/bin:/Users/jan/.pyenv/shims:/Users/jan/scipts:/Users/jan/.local/bin:/usr/local/opt/ruby/bin:/usr/local/lib/ruby/gems/2.6.0/bin:/Users/jan/.cargo/bin:/Users/jan/scripts:/Users/jan/.cabal/bin:/Users/jan/.ghcup/bin:/Users/jan/scripts/google-cloud-sdk/bin:/Users/jan/.poetry/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/share/dotnet:~/.dotnet/tools:/Library/Frameworks/Mono.framework/Versions/Current/Commands
[29143] /Users/jan/pytorch-revgrad$ /Users/jan/pytorch-revgrad/.tox/.package/bin/python -m pip install numpy 'torch>=1.0.0' 'ipython ; extra == "dev"' 'tox ; extra == "test"' 'pytest ; extra == "test"' 'pytest-cov ; extra == "test"' 'flake8 ; extra == "test"' 'pytest-flake8 ; extra == "test"' 'coveralls ; extra == "test"'
Ignoring ipython: markers 'extra == "dev"' don't match your environment
Ignoring tox: markers 'extra == "test"' don't match your environment
Ignoring pytest: markers 'extra == "test"' don't match your environment
Ignoring pytest-cov: markers 'extra == "test"' don't match your environment
Ignoring flake8: markers 'extra == "test"' don't match your environment
Ignoring pytest-flake8: markers 'extra == "test"' don't match your environment
Ignoring coveralls: markers 'extra == "test"' don't match your environment
Looking in indexes: https://pypi.org/simple
Collecting numpy
  Using cached https://files.pythonhosted.org/packages/a7/06/6d616fb5fb423db595b1502cbd873f3f2025f2fd8509046c771a20c4302a/numpy-1.18.1-cp38-cp38-macosx_10_9_x86_64.whl
Collecting torch>=1.0.0
  Using cached https://files.pythonhosted.org/packages/53/e1/f2ba65890bb3ed98eda66ec9195e97b9b1b5824117ff4c8ad3b3f298e464/torch-1.4.0-cp38-none-macosx_10_9_x86_64.whl
Installing collected packages: numpy, torch
Successfully installed numpy-1.18.1 torch-1.4.0
WARNING: You are using pip version 19.3.1; however, version 20.0.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
.package finish: build_requires /Users/jan/pytorch-revgrad/.tox/.package after 9.25 seconds
.package start: finishvenv
.package finish: finishvenv  after 0.01 seconds
.package start: perform-isolated-build /Users/jan/pytorch-revgrad/.tox/.package
setting PATH=/Users/jan/pytorch-revgrad/.tox/.package/bin:/Users/jan/pytorch-revgrad/.venv/bin:/Users/jan/.pyenv/bin:/Users/jan/.pyenv/shims:/Users/jan/scipts:/Users/jan/.local/bin:/usr/local/opt/ruby/bin:/usr/local/lib/ruby/gems/2.6.0/bin:/Users/jan/.cargo/bin:/Users/jan/scripts:/Users/jan/.cabal/bin:/Users/jan/.ghcup/bin:/Users/jan/scripts/google-cloud-sdk/bin:/Users/jan/.poetry/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/share/dotnet:~/.dotnet/tools:/Library/Frameworks/Mono.framework/Versions/Current/Commands
[29145] /Users/jan/pytorch-revgrad$ /Users/jan/pytorch-revgrad/.tox/.package/bin/python .venv/lib/python3.8/site-packages/tox/helper/build_isolated.py .tox/dist flit.buildapi '' >.tox/.package/log/.package-1.log
pytorch_revgrad-0.1.tar.gz

.package finish: perform-isolated-build /Users/jan/pytorch-revgrad/.tox/.package after 1.01 seconds
copying new sdistfile to '/Users/jan/.tox/distshare/pytorch_revgrad-0.1.tar.gz'
package .tmp/package/1/pytorch_revgrad-0.1.tar.gz links to dist/pytorch_revgrad-0.1.tar.gz (/Users/jan/pytorch-revgrad/.tox)
py37 start: getenv /Users/jan/pytorch-revgrad/.tox/py37
py37 cannot reuse: -r flag
py37 create: /Users/jan/pytorch-revgrad/.tox/py37
python3.7 (/Users/jan/.pyenv/shims/python3.7) is {'executable': '/Users/jan/.pyenv/versions/3.7.5rc1/bin/python3.7', 'name': 'python', 'version_info': [3, 7, 5, 'candidate', 1], 'version': '3.7.5rc1 (default, Nov 27 2019, 15:23:39) \n[Clang 11.0.0 (clang-1100.0.20.17)]', 'is_64': True, 'sysplatform': 'darwin'}
py37 uses /Users/jan/.pyenv/versions/3.7.5rc1/bin/python3.7
setting PATH=/Users/jan/pytorch-revgrad/.tox/py37/bin:/Users/jan/pytorch-revgrad/.venv/bin:/Users/jan/.pyenv/bin:/Users/jan/.pyenv/shims:/Users/jan/scipts:/Users/jan/.local/bin:/usr/local/opt/ruby/bin:/usr/local/lib/ruby/gems/2.6.0/bin:/Users/jan/.cargo/bin:/Users/jan/scripts:/Users/jan/.cabal/bin:/Users/jan/.ghcup/bin:/Users/jan/scripts/google-cloud-sdk/bin:/Users/jan/.poetry/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/share/dotnet:~/.dotnet/tools:/Library/Frameworks/Mono.framework/Versions/Current/Commands
[29181] /Users/jan/pytorch-revgrad/.tox$ /Users/jan/pytorch-revgrad/.venv/bin/python -m virtualenv --no-download --python /Users/jan/.pyenv/versions/3.7.5rc1/bin/python3.7 py37
Running virtualenv with interpreter /Users/jan/.pyenv/versions/3.7.5rc1/bin/python3.7
Already using interpreter /Users/jan/.pyenv/versions/3.7.5rc1/bin/python3.7
Using base prefix '/Users/jan/.pyenv/versions/3.7.5rc1'
New python executable in /Users/jan/pytorch-revgrad/.tox/py37/bin/python3.7
Also creating executable in /Users/jan/pytorch-revgrad/.tox/py37/bin/python
Installing setuptools, pip, wheel...
done.
py37 finish: getenv /Users/jan/pytorch-revgrad/.tox/py37 after 3.90 seconds
py37 start: installpkg /Users/jan/pytorch-revgrad/.tox/.tmp/package/1/pytorch_revgrad-0.1.tar.gz
py37 inst: /Users/jan/pytorch-revgrad/.tox/.tmp/package/1/pytorch_revgrad-0.1.tar.gz
write config to /Users/jan/pytorch-revgrad/.tox/py37/.tox-config1 as '13208ff0ed71c6500eb63d0f6f0939d93dc1b46269505dc461860d09ac9bc0c1 /Users/jan/.pyenv/versions/3.7.5rc1/bin/python3.7\n3.14.3 0 0 0'
setting PATH=/Users/jan/pytorch-revgrad/.tox/py37/bin:/Users/jan/pytorch-revgrad/.venv/bin:/Users/jan/.pyenv/bin:/Users/jan/.pyenv/shims:/Users/jan/scipts:/Users/jan/.local/bin:/usr/local/opt/ruby/bin:/usr/local/lib/ruby/gems/2.6.0/bin:/Users/jan/.cargo/bin:/Users/jan/scripts:/Users/jan/.cabal/bin:/Users/jan/.ghcup/bin:/Users/jan/scripts/google-cloud-sdk/bin:/Users/jan/.poetry/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/share/dotnet:~/.dotnet/tools:/Library/Frameworks/Mono.framework/Versions/Current/Commands
[29189] /Users/jan/pytorch-revgrad$ /Users/jan/pytorch-revgrad/.tox/py37/bin/python -m pip install --exists-action w .tox/.tmp/package/1/pytorch_revgrad-0.1.tar.gz
Looking in indexes: https://pypi.org/simple
Processing ./.tox/.tmp/package/1/pytorch_revgrad-0.1.tar.gz
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
    Preparing wheel metadata ... done
Collecting torch>=1.0.0
  Using cached https://files.pythonhosted.org/packages/4a/72/0282449efe6e8a7ab6354ac990b8275bd8c881dcbf95b3ef0a041da3897b/torch-1.4.0-cp37-none-macosx_10_9_x86_64.whl
Collecting numpy
  Using cached https://files.pythonhosted.org/packages/2f/5b/2cc2b9285e8b2ca8d2c1e4a2cbf1b12d70a2488ea78170de1909bca725f2/numpy-1.18.1-cp37-cp37m-macosx_10_9_x86_64.whl
Building wheels for collected packages: pytorch-revgrad
  Building wheel for pytorch-revgrad (PEP 517) ... done
  Created wheel for pytorch-revgrad: filename=pytorch_revgrad-0.1-cp37-none-any.whl size=2764 sha256=6af2264fcfb20aa53ece821009f342d9882248dc13f2b428c81fa803f1df5f08
  Stored in directory: /Users/jan/Library/Caches/pip/wheels/d7/a0/61/edb972be6fa8f71ccd47b95aaa37625a51e11572bf735d2d26
Successfully built pytorch-revgrad
Installing collected packages: torch, numpy, pytorch-revgrad
Successfully installed numpy-1.18.1 pytorch-revgrad-0.1 torch-1.4.0
WARNING: You are using pip version 19.3.1; however, version 20.0.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
py37 finish: installpkg /Users/jan/pytorch-revgrad/.tox/.tmp/package/1/pytorch_revgrad-0.1.tar.gz after 25.06 seconds
py37 start: envreport
setting PATH=/Users/jan/pytorch-revgrad/.tox/py37/bin:/Users/jan/pytorch-revgrad/.venv/bin:/Users/jan/.pyenv/bin:/Users/jan/.pyenv/shims:/Users/jan/scipts:/Users/jan/.local/bin:/usr/local/opt/ruby/bin:/usr/local/lib/ruby/gems/2.6.0/bin:/Users/jan/.cargo/bin:/Users/jan/scripts:/Users/jan/.cabal/bin:/Users/jan/.ghcup/bin:/Users/jan/scripts/google-cloud-sdk/bin:/Users/jan/.poetry/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/share/dotnet:~/.dotnet/tools:/Library/Frameworks/Mono.framework/Versions/Current/Commands
[29209] /Users/jan/pytorch-revgrad$ /Users/jan/pytorch-revgrad/.tox/py37/bin/python -m pip freeze >.tox/py37/log/py37-0.log
py37 finish: envreport  after 0.27 seconds
py37 installed: numpy==1.18.1,pytorch-revgrad==0.1,torch==1.4.0
py37 start: run-test-pre
py37 run-test-pre: PYTHONHASHSEED='2124118408'
py37 finish: run-test-pre  after 0.00 seconds
py37 start: run-test
py37 run-test: commands[0] | python -m pytest
setting PATH=/Users/jan/pytorch-revgrad/.tox/py37/bin:/Users/jan/pytorch-revgrad/.venv/bin:/Users/jan/.pyenv/bin:/Users/jan/.pyenv/shims:/Users/jan/scipts:/Users/jan/.local/bin:/usr/local/opt/ruby/bin:/usr/local/lib/ruby/gems/2.6.0/bin:/Users/jan/.cargo/bin:/Users/jan/scripts:/Users/jan/.cabal/bin:/Users/jan/.ghcup/bin:/Users/jan/scripts/google-cloud-sdk/bin:/Users/jan/.poetry/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/share/dotnet:~/.dotnet/tools:/Library/Frameworks/Mono.framework/Versions/Current/Commands
[29213] /Users/jan/pytorch-revgrad$ /Users/jan/pytorch-revgrad/.tox/py37/bin/python -m pytest
/Users/jan/pytorch-revgrad/.tox/py37/bin/python: No module named pytest
ERROR: InvocationError for command /Users/jan/pytorch-revgrad/.tox/py37/bin/python -m pytest (exited with code 1)
py37 finish: run-test  after 0.02 seconds
py37 start: run-test-post
py37 finish: run-test-post  after 0.00 seconds
__________________________________________________________ summary ___________________________________________________________
ERROR:   py37: commands failed
cleanup /Users/jan/pytorch-revgrad/.tox/.tmp/package/1/pytorch_revgrad-0.1.tar.gz

In particular, I am unclear why these lines occur:

[29143] /Users/jan/pytorch-revgrad$ /Users/jan/pytorch-revgrad/.tox/.package/bin/python -m pip install numpy 'torch>=1.0.0' 'ipython ; extra == "dev"' 'tox ; extra == "test"' 'pytest ; extra == "test"' 'pytest-cov ; extra == "test"' 'flake8 ; extra == "test"' 'pytest-flake8 ; extra == "test"' 'coveralls ; extra == "test"'
Ignoring ipython: markers 'extra == "dev"' don't match your environment
Ignoring tox: markers 'extra == "test"' don't match your environment
Ignoring pytest: markers 'extra == "test"' don't match your environment
Ignoring pytest-cov: markers 'extra == "test"' don't match your environment
Ignoring flake8: markers 'extra == "test"' don't match your environment
Ignoring pytest-flake8: markers 'extra == "test"' don't match your environment
Ignoring coveralls: markers 'extra == "test"' don't match your environment

I tried to look through issues mentioning "extras" or "isolated_build" but couldn't find anything that matches this problem.

Any help would be appreciated!

Jan
@janfreyberg janfreyberg added the bug:normal affects many people or has quite an impact label Jan 31, 2020
@astrojuanlu
Copy link

I confirm I'm observing the same

@gaborbernat
Copy link
Member

Feel free to open a PR against master and fix it. My available efforts at the moment are aimed at fixing this as part of #1394, but that probably will take a while (ETA September).

@gaborbernat
Copy link
Member

I think this has been fixed now in tox version 4. Please check it out! If it still happens can you submit a reproducible to it?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug:normal affects many people or has quite an impact
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@astrojuanlu @gaborbernat @janfreyberg