Multiple factors aren't respected

[stephenfin]

This may be a misunderstanding of how tox is supposed to work on my end, so apologies if so. Take the following tox.ini file:

[tox]
minversion = 3.1
envlist = py37,functional,pep8
ignore_basepython_conflict = true
skipsdist = true

[testenv]
basepython = python3
whitelist_externals =
  pip
deps =
  requests
commands =
  pip freeze
  python --version

[testenv:functional]
deps =
  {[testenv]deps}
  requests-mock

If I run this with tox -e py37 or tox -e functional, things behave as expected. For example:

$ tox -e functional                                                                                                                                                                                                
functional create: /tmp/tox-test/.tox/functional                                                                                                                                                                   
functional installdeps: requests, requests-mock                                                                                                                                                                    
functional installed: certifi==2019.9.11,chardet==3.0.4,idna==2.8,requests==2.22.0,requests-mock==1.7.0,six==1.12.0,urllib3==1.25.6                                                                                
functional run-test-pre: PYTHONHASHSEED='936107316'                                                                                                                                                                
functional runtests: commands[0] | pip freeze                                                                                                                                                                      
certifi==2019.9.11                                                                                                                                                                                                 
chardet==3.0.4                                                                                                                                                                                                     
idna==2.8                                                                                                                                                                                                          
requests==2.22.0                                                                                                                                                                                                   
requests-mock==1.7.0                                                                                                                                                                                               
six==1.12.0                                                                                                                                                                                                        
urllib3==1.25.6                                                                                                                                                                                                    
functional runtests: commands[1] | python --version                                                                                                                                                                
Python 3.7.4                                                                                                                                                                                                       
_____________________________________________________________________________________________________ summary _____________________________________________________________________________________________________
  functional: commands succeeded                                                                                                                                                                                   
  congratulations :)

My understanding of factors suggests that tox -e functional-py36 should implicitly exist, because py36 is an auto-generated factor that implies some things (namely, the use of basepython=python3.6 instead of python3). However, this is not the case as seen below. What's worse is that the request isn't even outright rejected - it works, but doesn't do what you'd expect:

$ tox -e functional-py36
functional-py36 create: /tmp/tox-test/.tox/functional-py36
functional-py36 installdeps: requests
functional-py36 installed: certifi==2019.9.11,chardet==3.0.4,idna==2.8,requests==2.22.0,urllib3==1.25.6
functional-py36 run-test-pre: PYTHONHASHSEED='1866335307'
functional-py36 runtests: commands[0] | pip freeze
certifi==2019.9.11
chardet==3.0.4
idna==2.8
requests==2.22.0
urllib3==1.25.6
functional-py36 runtests: commands[1] | python --version
Python 3.6.9
_____________________________________________________________________________________________________ summary _____________________________________________________________________________________________________
  functional-py36: commands succeeded
  congratulations :)

Note that deps from testenv:functional isn't used. Rather, the value from testenv is.

I think that one of the following should happen:

• The environment should use a combination of details from the functional testenv and the py36 testenv. As such, it should install requests-mock but use basepython=python3.6 (preferred!)
• The request should be outright rejected since it's not explicitly listed in envlist and isn't one of the auto-generated envs

---

$ tox --version
3.5.3 imported from /usr/lib/python3.7/site-packages/tox/__init__.py
registered plugins:
    tox-docker-1.4.1 at /home/sfinucan/.local/lib/python3.7/site-packages/tox_docker.py

[stephenfin]

Reversing the factors doesn't fix things.

$ tox -e py36-functional
py36-functional create: /tmp/tox-test/.tox/py36-functional
py36-functional installdeps: requests
py36-functional installed: certifi==2019.9.11,chardet==3.0.4,idna==2.8,requests==2.22.0,urllib3==1.25.6
py36-functional run-test-pre: PYTHONHASHSEED='2330775823'
py36-functional runtests: commands[0] | pip freeze
certifi==2019.9.11
chardet==3.0.4
idna==2.8
requests==2.22.0
urllib3==1.25.6
py36-functional runtests: commands[1] | python --version
Python 3.6.9
_____________________________________________________________________________________________________ summary _____________________________________________________________________________________________________
  py36-functional: commands succeeded
  congratulations :)

[stephenfin]

I'm also aware that I can use factors in things like deps, such as:

$ cat tox.ini 
[tox]
minversion = 3.1
envlist = py37,functional,pep8
ignore_basepython_conflict = true
skipsdist = true

[testenv]
basepython = python3
whitelist_externals =
  pip
deps =
  requests
  functional: requests-mock
commands =
  pip freeze
  python --version

However, this doesn't scale for larger tox.ini files with multiple testenvs as the end result is almost impossible to read. It also doesn't change the fact that things are currently failing without any warning.

If either of my solutions are appropriate, I'd be happy to work on fixing this.

[gaborbernat]

This part should not exists:

whitelist_externals =
  pip

There's no reason why you want to pull in an external pip.

envlist = py37,functional,pep8

This does not imply any factorials. You need tox environment names separated with the - so we can talk about factorials. E.g. {py37,py36}-functional, pep would be a such case.

[testenv:functional]

This does not defines options for the functional factorial, but rather the functional environment.

To better understand what's happening I recommend using -vv to see step by step what's happening; and tox --showconfig -e py37-functional to see the generated configuration for a given environment.

deps =
  requests
  functional: requests-mock

Yes, sadly that format is the only way you can define lines that apply to given factorials only. Things might get better with tox 4, in development - see #1394, that will allow one to inherit arbitrary environments, not just from testenv.

Does this helps you?

[stephenfin]

This part should not exists:

whitelist_externals =
  pip

There's no reason why you want to pull in an external pip.

Good point. We can ignore this though. I just whipped this together as an experiment.

envlist = py37,functional,pep8

This does not imply any factorials. You need tox environment names separated with the - so we can talk about factorials. E.g. {py37,py36}-functional, pep would be a such case.

[testenv:functional]

This does not defines options for the functional factorial, but rather the functional environment.

Ahh, this is my mistake. I was confusing factors (factorials?) and environments. Thanks for the clarification.

To better understand what's happening I recommend using -vv to see step by step what's happening; and tox --showconfig -e py37-functional to see the generated configuration for a given environment.

deps =
  requests
  functional: requests-mock

Yes, sadly that format is the only way you can define lines that apply to given factorials only. Things might get better with tox 4, in development - see #1394, that will allow one to inherit arbitrary environments, not just from testenv.

Does this helps you?

It does, yes. I'll just stick with the above approach for now.