Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Testing with tox is broken #1012

Closed
zorun opened this issue Apr 2, 2022 · 9 comments
Closed

Testing with tox is broken #1012

zorun opened this issue Apr 2, 2022 · 9 comments

Comments

@zorun
Copy link
Collaborator

zorun commented Apr 2, 2022

See all recent pull requests: CI testing is broken.

I can reproduce locally when testing with tox:

python3 -m venv /tmp/venv-ihm-tox
. /tmp/venv-ihm-tox/bin/activate
pip install -U pip
pip install -U tox
tox -e py

I end up with very weird errors.

Testing "manually" without tox works fine:

python3 -m venv /tmp/venv-ihm-tests
. /tmp/venv-ihm-tests/bin/activate
pip install -e .[dev]
py.test --pyargs ihatemoney.tests

I haven't been able to figure out what's wrong. At first I thought that was caused by a new version of pip (22.X), but even when downgrading pip tox keeps failing.
@almet
Copy link
Member

almet commented Apr 4, 2022 •
edited
Loading

Hey. I'm able to run the tests locally without problems on the latest master and a fresh install. Here's the log I have.

make test
/home/alexis/dev/ihatemoney/.venv/bin/tox
GLOB sdist-make: /home/alexis/dev/ihatemoney/setup.py
py39 recreate: /home/alexis/dev/ihatemoney/.tox/py39
py39 installdeps: -e.[database,dev]
py39 inst: /home/alexis/dev/ihatemoney/.tox/.tmp/package/1/ihatemoney-5.1.2.dev0.zip
py39 installed: alembic==1.7.7,aniso8601==9.0.1,attrs==21.4.0,Babel==2.9.1,black==22.3.0,bleach==4.1.0,blinker==1.4,cachetools==4.2.4,certifi==2021.10.8,cffi==1.15.0,charset-normalizer==2.0.12,click==8.1.2,colorama==0.4.4,commonmark==0.9.1,cryptography==36.0.2,debts==0.5,distlib==0.3.4,dnspython==2.2.1,docutils==0.18.1,email-validator==1.1.3,filelock==3.6.0,flake8==4.0.1,Flask==2.1.1,Flask-Babel==2.0.0,Flask-Cors==3.0.10,Flask-Mail==0.9.1,Flask-Migrate==3.1.0,Flask-RESTful==0.3.9,Flask-SQLAlchemy==2.5.1,flask-talisman==1.0.0,Flask-Testing==0.8.1,Flask-WTF==1.0.1,idna==3.3,ihatemoney @ file:///home/alexis/dev/ihatemoney/.tox/.tmp/package/1/ihatemoney-5.1.2.dev0.zip,importlib-metadata==4.11.3,iniconfig==1.1.1,isort==5.10.1,itsdangerous==2.1.2,jeepney==0.8.0,Jinja2==3.1.1,keyring==23.5.0,Mako==1.2.0,MarkupSafe==2.1.1,mccabe==0.6.1,mypy-extensions==0.4.3,packaging==21.3,pathspec==0.9.0,pkginfo==1.8.2,platformdirs==2.5.1,pluggy==1.0.0,psycopg2-binary==2.9.3,py==1.11.0,pycodestyle==2.8.0,pycparser==2.21,pyflakes==2.4.0,Pygments==2.11.2,PyMySQL==1.0.2,pyparsing==3.0.7,pytest==7.1.1,python-dateutil==2.8.2,pytz==2022.1,readme-renderer==34.0,requests==2.27.1,requests-toolbelt==0.9.1,rfc3986==2.0.0,rich==12.1.0,SecretStorage==3.3.1,six==1.16.0,SQLAlchemy==1.3.24,SQLAlchemy-Continuum==1.3.12,SQLAlchemy-Utils==0.38.2,toml==0.10.2,tomli==2.0.1,tox==3.24.5,twine==4.0.0,typing_extensions==4.1.1,urllib3==1.26.9,vermin==1.3.3,virtualenv==20.14.0,webencodings==0.5.1,Werkzeug==2.1.1,WTForms==3.0.1,zest.releaser==6.22.2,zipp==3.8.0
py39 run-test-pre: PYTHONHASHSEED='3677666617'
py39 run-test: commands[0] | python --version
Python 3.9.12
py39 run-test: commands[1] | py.test --pyargs ihatemoney.tests
==================================================================================== test session starts =====================================================================================
platform linux -- Python 3.9.12, pytest-7.1.1, pluggy-1.0.0
cachedir: /home/alexis/dev/ihatemoney/.tox/py39/.pytest_cache
rootdir: /tmp
collected 102 items                                                                                                                                                                          

api_test.py ................                                                                                                                                                           [ 15%]
budget_test.py .......................................                                                                                                                                 [ 53%]
history_test.py ...........                                                                                                                                                            [ 64%]
import_test.py ................                                                                                                                                                        [ 80%]
main_test.py ....................                                                                                                                                                      [100%]

============================================================================== 102 passed in 103.25s (0:01:43) ===============================================================================
py38 create: /home/alexis/dev/ihatemoney/.tox/py38
py38 installdeps: -e.[database,dev]
py38 inst: /home/alexis/dev/ihatemoney/.tox/.tmp/package/1/ihatemoney-5.1.2.dev0.zip
py38 installed: alembic==1.7.7,aniso8601==9.0.1,attrs==21.4.0,Babel==2.9.1,black==22.3.0,bleach==4.1.0,blinker==1.4,cachetools==4.2.4,certifi==2021.10.8,cffi==1.15.0,charset-normalizer==2.0.12,click==8.1.2,colorama==0.4.4,commonmark==0.9.1,cryptography==36.0.2,debts==0.5,distlib==0.3.4,dnspython==2.2.1,docutils==0.18.1,email-validator==1.1.3,filelock==3.6.0,flake8==4.0.1,Flask==2.1.1,Flask-Babel==2.0.0,Flask-Cors==3.0.10,Flask-Mail==0.9.1,Flask-Migrate==3.1.0,Flask-RESTful==0.3.9,Flask-SQLAlchemy==2.5.1,flask-talisman==1.0.0,Flask-Testing==0.8.1,Flask-WTF==1.0.1,idna==3.3,ihatemoney @ file:///home/alexis/dev/ihatemoney/.tox/.tmp/package/1/ihatemoney-5.1.2.dev0.zip,importlib-metadata==4.11.3,importlib-resources==5.6.0,iniconfig==1.1.1,isort==5.10.1,itsdangerous==2.1.2,jeepney==0.8.0,Jinja2==3.1.1,keyring==23.5.0,Mako==1.2.0,MarkupSafe==2.1.1,mccabe==0.6.1,mypy-extensions==0.4.3,packaging==21.3,pathspec==0.9.0,pkginfo==1.8.2,platformdirs==2.5.1,pluggy==1.0.0,psycopg2-binary==2.9.3,py==1.11.0,pycodestyle==2.8.0,pycparser==2.21,pyflakes==2.4.0,Pygments==2.11.2,PyMySQL==1.0.2,pyparsing==3.0.7,pytest==7.1.1,python-dateutil==2.8.2,pytz==2022.1,readme-renderer==34.0,requests==2.27.1,requests-toolbelt==0.9.1,rfc3986==2.0.0,rich==12.1.0,SecretStorage==3.3.1,six==1.16.0,SQLAlchemy==1.3.24,SQLAlchemy-Continuum==1.3.12,SQLAlchemy-Utils==0.38.2,toml==0.10.2,tomli==2.0.1,tox==3.24.5,twine==4.0.0,typing_extensions==4.1.1,urllib3==1.26.9,vermin==1.3.3,virtualenv==20.14.0,webencodings==0.5.1,Werkzeug==2.1.1,WTForms==3.0.1,zest.releaser==6.22.2,zipp==3.8.0
py38 run-test-pre: PYTHONHASHSEED='3677666617'
py38 run-test: commands[0] | python --version
Python 3.8.13
py38 run-test: commands[1] | py.test --pyargs ihatemoney.tests
==================================================================================== test session starts =====================================================================================
platform linux -- Python 3.8.13, pytest-7.1.1, pluggy-1.0.0
cachedir: /home/alexis/dev/ihatemoney/.tox/py38/.pytest_cache
rootdir: /tmp
collected 102 items                                                                                                                                                                          

api_test.py ................                                                                                                                                                           [ 15%]
budget_test.py .......................................                                                                                                                                 [ 53%]
history_test.py ...........                                                                                                                                                            [ 64%]
import_test.py ................                                                                                                                                                        [ 80%]
main_test.py ....................                                                                                                                                                      [100%]

============================================================================== 102 passed in 103.35s (0:01:43) ===============================================================================
py37 create: /home/alexis/dev/ihatemoney/.tox/py37
py37 installdeps: -e.[database,dev]
py37 inst: /home/alexis/dev/ihatemoney/.tox/.tmp/package/1/ihatemoney-5.1.2.dev0.zip
py37 installed: alembic==1.7.7,aniso8601==9.0.1,attrs==21.4.0,Babel==2.9.1,black==22.3.0,bleach==4.1.0,blinker==1.4,cachetools==4.2.4,certifi==2021.10.8,cffi==1.15.0,charset-normalizer==2.0.12,click==8.1.2,colorama==0.4.4,commonmark==0.9.1,cryptography==36.0.2,debts==0.5,distlib==0.3.4,dnspython==2.2.1,docutils==0.18.1,email-validator==1.1.3,filelock==3.6.0,flake8==4.0.1,Flask==2.1.1,Flask-Babel==2.0.0,Flask-Cors==3.0.10,Flask-Mail==0.9.1,Flask-Migrate==3.1.0,Flask-RESTful==0.3.9,Flask-SQLAlchemy==2.5.1,flask-talisman==1.0.0,Flask-Testing==0.8.1,Flask-WTF==1.0.1,idna==3.3,ihatemoney @ file:///home/alexis/dev/ihatemoney/.tox/.tmp/package/1/ihatemoney-5.1.2.dev0.zip,importlib-metadata==4.2.0,importlib-resources==5.6.0,iniconfig==1.1.1,isort==5.10.1,itsdangerous==2.1.2,jeepney==0.8.0,Jinja2==3.1.1,keyring==23.5.0,Mako==1.2.0,MarkupSafe==2.1.1,mccabe==0.6.1,mypy-extensions==0.4.3,packaging==21.3,pathspec==0.9.0,pkginfo==1.8.2,platformdirs==2.5.1,pluggy==1.0.0,psycopg2-binary==2.9.3,py==1.11.0,pycodestyle==2.8.0,pycparser==2.21,pyflakes==2.4.0,Pygments==2.11.2,PyMySQL==1.0.2,pyparsing==3.0.7,pytest==7.1.1,python-dateutil==2.8.2,pytz==2022.1,readme-renderer==34.0,requests==2.27.1,requests-toolbelt==0.9.1,rfc3986==2.0.0,rich==12.1.0,SecretStorage==3.3.1,six==1.16.0,SQLAlchemy==1.3.24,SQLAlchemy-Continuum==1.3.12,SQLAlchemy-Utils==0.38.2,toml==0.10.2,tomli==2.0.1,tox==3.24.5,twine==4.0.0,typed-ast==1.5.2,typing_extensions==4.1.1,urllib3==1.26.9,vermin==1.3.3,virtualenv==20.14.0,webencodings==0.5.1,Werkzeug==2.1.1,WTForms==3.0.1,zest.releaser==6.22.2,zipp==3.8.0
py37 run-test-pre: PYTHONHASHSEED='3677666617'
py37 run-test: commands[0] | python --version
Python 3.7.13
py37 run-test: commands[1] | py.test --pyargs ihatemoney.tests
==================================================================================== test session starts =====================================================================================
platform linux -- Python 3.7.13, pytest-7.1.1, pluggy-1.0.0
cachedir: /home/alexis/dev/ihatemoney/.tox/py37/.pytest_cache
rootdir: /tmp
collected 102 items                                                                                                                                                                          

api_test.py ................                                                                                                                                                           [ 15%]
budget_test.py .......................................                                                                                                                                 [ 53%]
history_test.py ...........                                                                                                                                                            [ 64%]
import_test.py ................                                                                                                                                                        [ 80%]
main_test.py ....................                                                                                                                                                      [100%]

============================================================================== 102 passed in 100.31s (0:01:40) ===============================================================================
py36 create: /home/alexis/dev/ihatemoney/.tox/py36
py36 installdeps: -e.[database,dev]
py36 inst: /home/alexis/dev/ihatemoney/.tox/.tmp/package/1/ihatemoney-5.1.2.dev0.zip
py36 installed: alembic==1.7.7,aniso8601==9.0.1,attrs==21.4.0,Babel==2.9.1,black==22.3.0,bleach==4.1.0,blinker==1.4,cachetools==4.2.4,certifi==2021.10.8,cffi==1.15.0,charset-normalizer==2.0.12,click==8.0.4,colorama==0.4.4,cryptography==36.0.2,dataclasses==0.8,debts==0.5,distlib==0.3.4,dnspython==2.2.1,docutils==0.18.1,email-validator==1.1.3,filelock==3.4.1,flake8==4.0.1,Flask==2.0.3,Flask-Babel==2.0.0,Flask-Cors==3.0.10,Flask-Mail==0.9.1,Flask-Migrate==3.1.0,Flask-RESTful==0.3.9,Flask-SQLAlchemy==2.5.1,flask-talisman==1.0.0,Flask-Testing==0.8.1,Flask-WTF==1.0.1,idna==3.3,ihatemoney @ file:///home/alexis/dev/ihatemoney/.tox/.tmp/package/1/ihatemoney-5.1.2.dev0.zip,importlib-metadata==4.2.0,importlib-resources==5.4.0,iniconfig==1.1.1,isort==5.10.1,itsdangerous==2.0.1,jeepney==0.7.1,Jinja2==3.0.3,keyring==23.4.1,Mako==1.1.6,MarkupSafe==2.0.1,mccabe==0.6.1,mypy-extensions==0.4.3,packaging==21.3,pathspec==0.9.0,pkginfo==1.8.2,platformdirs==2.4.0,pluggy==1.0.0,psycopg2-binary==2.9.3,py==1.11.0,pycodestyle==2.8.0,pycparser==2.21,pyflakes==2.4.0,Pygments==2.11.2,PyMySQL==1.0.2,pyparsing==3.0.7,pytest==7.0.1,python-dateutil==2.8.2,pytz==2022.1,readme-renderer==34.0,requests==2.27.1,requests-toolbelt==0.9.1,rfc3986==1.5.0,SecretStorage==3.3.1,six==1.16.0,SQLAlchemy==1.3.24,SQLAlchemy-Continuum==1.3.12,SQLAlchemy-Utils==0.38.2,toml==0.10.2,tomli==1.2.3,tox==3.24.5,tqdm==4.64.0,twine==3.8.0,typed-ast==1.5.2,typing_extensions==4.1.1,urllib3==1.26.9,vermin==1.3.3,virtualenv==20.14.0,webencodings==0.5.1,Werkzeug==2.0.3,WTForms==3.0.0,zest.releaser==6.22.2,zipp==3.6.0
py36 run-test-pre: PYTHONHASHSEED='3677666617'
py36 run-test: commands[0] | python --version
Python 3.6.15
py36 run-test: commands[1] | py.test --pyargs ihatemoney.tests
==================================================================================== test session starts =====================================================================================
platform linux -- Python 3.6.15, pytest-7.0.1, pluggy-1.0.0
cachedir: /home/alexis/dev/ihatemoney/.tox/py36/.pytest_cache
rootdir: /tmp
collected 102 items                                                                                                                                                                          

api_test.py ................                                                                                                                                                           [ 15%]
budget_test.py .......................................                                                                                                                                 [ 53%]
history_test.py ...........                                                                                                                                                            [ 64%]
import_test.py ................                                                                                                                                                        [ 80%]
main_test.py ....................                                                                                                                                                      [100%]

============================================================================== 102 passed in 114.33s (0:01:54) ===============================================================================
lint_docs inst-nodeps: /home/alexis/dev/ihatemoney/.tox/.tmp/package/1/ihatemoney-5.1.2.dev0.zip
lint_docs installed: alabaster==0.7.12,alembic==1.7.7,aniso8601==9.0.1,attrs==21.4.0,Babel==2.9.1,black==22.3.0,bleach==4.1.0,blinker==1.4,cachetools==4.2.4,certifi==2021.10.8,cffi==1.15.0,charset-normalizer==2.0.12,click==8.1.2,colorama==0.4.4,commonmark==0.9.1,cryptography==36.0.2,debts==0.5,distlib==0.3.4,dnspython==2.2.1,docutils==0.17.1,email-validator==1.1.3,filelock==3.6.0,flake8==4.0.1,Flask==2.1.1,Flask-Babel==2.0.0,Flask-Cors==3.0.10,Flask-Mail==0.9.1,Flask-Migrate==3.1.0,Flask-RESTful==0.3.9,Flask-SQLAlchemy==2.5.1,flask-talisman==0.8.1,Flask-Testing==0.8.1,Flask-WTF==1.0.1,idna==3.3,ihatemoney @ file:///home/alexis/dev/ihatemoney/.tox/.tmp/package/1/ihatemoney-5.1.2.dev0.zip,imagesize==1.3.0,importlib-metadata==4.11.3,iniconfig==1.1.1,isort==5.10.1,itsdangerous==2.1.2,jeepney==0.8.0,Jinja2==3.1.1,keyring==23.5.0,Mako==1.2.0,markdown-it-py==2.0.1,MarkupSafe==2.1.1,mccabe==0.6.1,mdit-py-plugins==0.3.0,mdurl==0.1.0,mypy-extensions==0.4.3,myst-parser==0.17.0,packaging==21.3,pathspec==0.9.0,pkginfo==1.8.2,platformdirs==2.5.1,pluggy==1.0.0,psycopg2-binary==2.9.3,py==1.11.0,pycodestyle==2.8.0,pycparser==2.21,pyflakes==2.4.0,Pygments==2.11.2,PyMySQL==1.0.2,pyparsing==3.0.7,pytest==7.1.1,python-dateutil==2.8.2,pytz==2022.1,PyYAML==6.0,readme-renderer==34.0,requests==2.27.1,requests-toolbelt==0.9.1,rfc3986==2.0.0,rich==12.1.0,SecretStorage==3.3.1,six==1.16.0,snowballstemmer==2.2.0,Sphinx==4.3.0,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.5,SQLAlchemy==1.3.24,SQLAlchemy-Continuum==1.3.12,SQLAlchemy-Utils==0.38.2,toml==0.10.2,tomli==2.0.1,tox==3.24.5,twine==4.0.0,typing_extensions==4.1.1,urllib3==1.26.9,virtualenv==20.14.0,webencodings==0.5.1,Werkzeug==2.1.1,WTForms==2.3.3,zest.releaser==6.22.2,zipp==3.8.0
lint_docs run-test-pre: PYTHONHASHSEED='3677666617'
lint_docs run-test: commands[0] | black --check --target-version=py36 .
All done! ✨ 🍰 ✨
39 files would be left unchanged.
lint_docs run-test: commands[1] | isort -c .
Skipped 7 files
lint_docs run-test: commands[2] | flake8 ihatemoney
lint_docs run-test: commands[3] | sphinx-build -a -n -b html -d docs/_build/doctrees docs docs/_build/html
Sphinx v4.3.0 en cours d'exécution
Chargement de l'environnement pickled... fait
myst v0.17.0: MdParserConfig(commonmark_only=False, gfm_only=False, enable_extensions=['colon_fence'], linkify_fuzzy_links=True, dmath_allow_labels=True, dmath_allow_space=True, dmath_allow_digits=True, dmath_double_inline=False, update_mathjax=True, mathjax_classes='tex2jax_process|mathjax_process|math|output_area', disable_syntax=[], all_links_external=False, url_schemes=['http', 'https', 'mailto', 'ftp'], ref_domains=None, highlight_code_blocks=True, number_code_blocks=[], title_to_header=False, heading_anchors=None, heading_slug_func=None, html_meta=[], footnote_transition=True, substitutions=[], sub_delimiters=['{', '}'], words_per_minute=200)
Construction en cours [mo] : tous les 0 fichiers po
Construction [html] : tous les fichiers source
Mise à jour de l'environnement : 0 ajouté(s), 0 modifié(s), 0 supprimé(s)
Recherche des fichiers périmés... aucun résultat
Document en préparation... fait
Écriture... [ 14%] api
Écriture... [ 28%] configuration
Écriture... [ 42%] contributing
Écriture... [ 57%] index
Écriture... [ 71%] installation
Écriture... [ 85%] security
Écriture... [100%] upgrade

Génération des index... genindex fait
écriture des pages additionnelles... search fait
copie des fichiers statiques... fait
Copie des fichiers complémentaires... fait
Export de l'index de recherche dans English (code: en)... fait
Export de l'inventaire des objets... fait
La compilation a réussi.

Les pages HTML sont dans docs/_build/html.
__________________________________________________________________________________________ summary ___________________________________________________________________________________________
  py39: commands succeeded
  py38: commands succeeded
  py37: commands succeeded
  py36: commands succeeded
  lint_docs: commands succeeded
  congratulations :)

Can you please post your outputs?

@almet
Copy link
Member

almet commented Apr 4, 2022

In these two scenarii, we're not using tox the same way. When I run tox with make test it works locally, but when I run it with .venv/bin/tox -e py it fails.

Here are the errors I see :

E           alembic.util.exc.CommandError: Path doesn't exist: '/home/alexis/dev/ihatemoney/.tox/py/lib/python3.10/site-packages/ihatemoney/migrations'.  Please use the 'init' command to create a new scripts folder.

Do you have the same kind of errors?

@almet
Copy link
Member

almet commented Apr 4, 2022

So, I tried to run again the make test after running tox manually and… I've got the tests failing. Not sure what's going on, but at least I can reproduce.

python3.{7,8,9} are failing, but python 3.6 is working well.

The installed package versions do not differ between the working run and the failing run. 🤔

@almet
Copy link
Member

almet commented Apr 4, 2022

I don't have much time today to look more into this though…

@zorun
Copy link
Collaborator Author

zorun commented Apr 4, 2022

E alembic.util.exc.CommandError: Path doesn't exist: '/home/alexis/dev/ihatemoney/.tox/py/lib/python3.10/site-packages/ihatemoney/migrations'. Please use the 'init' command to create a new scripts folder.

Yes, I had the same kind of errors. I "fixed" it by adding __init__.py files in migrations/, but then I had other very weird errors on all tests.

This is what led me to believe it was related to wheel building or pip installation, but I haven't got to the bottom of it...

@Glandos
Copy link
Member

Glandos commented Apr 10, 2022

It is related to wheel building. The wheel built with tox doesn't contain any static, templates, etc., directories. The wheel I build when installing into a virtualenv made with my system package (Debian unstable) have all of them.
I strace the creation, and they are all unlinked. I didn't have the time to track where the changed behavior occurred exactly, so this comment is also a reminder for myself.

@Glandos
Copy link
Member

Glandos commented Apr 15, 2022

Bingo: pypa/virtualenv#2327

@Glandos
Copy link
Member

Glandos commented Apr 15, 2022

Closing, since it autosolved with a new version of virtualenv.

@Glandos Glandos closed this as completed Apr 15, 2022
@zorun
Copy link
Collaborator Author

zorun commented Apr 19, 2022 •
edited
Loading

Well done, good detective job, as usual!

It's a bit frustrating that it fixed itself "magically", but at least we (kind of) know where the problem comes from.

@almet almet mentioned this issue Apr 29, 2022
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@almet @Glandos @zorun