0.6.2: self test failure: `static/` missing #403

0-wiz-0 · 2017-11-25T12:52:33Z

When running the self tests for the pypi distfile of 0.6.2 on NetBSD 8.99.7/amd64 with python-3.6.3, I see the following test failure:

httpbin (unittest.loader._FailedTest) ... ERROR

======================================================================
ERROR: httpbin (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: httpbin
Traceback (most recent call last):
  File "/usr/pkg/lib/python3.6/unittest/loader.py", line 462, in _find_test_path
    package = self._get_module_from_name(name)
  File "/usr/pkg/lib/python3.6/unittest/loader.py", line 369, in _get_module_from_name
    __import__(name)
  File "/scratch/www/py-httpbin/work/httpbin-0.6.2/httpbin/__init__.py", line 3, in <module>
    from .core import *
  File "/scratch/www/py-httpbin/work/httpbin-0.6.2/httpbin/core.py", line 61, in <module>
    common = Common(app)
  File "/usr/pkg/lib/python3.6/site-packages/flask_common.py", line 95, in __init__
    self.init_app(app)
  File "/usr/pkg/lib/python3.6/site-packages/flask_common.py", line 112, in init_app
    app.wsgi_app = WhiteNoise(app.wsgi_app, root=url_for('static', filename='')[1:])
  File "/usr/pkg/lib/python3.6/site-packages/whitenoise/base.py", line 61, in __init__
    self.add_files(root, prefix)
  File "/usr/pkg/lib/python3.6/site-packages/whitenoise/base.py", line 96, in add_files
    self.update_files_dictionary(root, prefix)
  File "/usr/pkg/lib/python3.6/site-packages/whitenoise/base.py", line 101, in update_files_dictionary
    stat_cache = dict(scantree(root))
  File "/usr/pkg/lib/python3.6/site-packages/whitenoise/scantree.py", line 21, in scantree
    for entry in scandir(root):
FileNotFoundError: [Errno 2] No such file or directory: 'static/'

The text was updated successfully, but these errors were encountered:

sigmavirus24 · 2017-11-25T14:17:38Z

I don't see this failure. How're you running the tests?

sigmavirus24 · 2017-11-25T14:24:11Z

Judging by the linked issue, you're running python setup.py test which is roughly equivalent to doing python -m unittest.discover .. That's scanning and importing httpbin the application which seems to be unhappy in this particular case. We could commit to supporting this use-case. That said, I don't think that method was ever supported officially and if it worked previously, it was coincidence.

javabrett · 2017-11-28T03:21:19Z

Could try running in a clean virtualenv?

0-wiz-0 · 2017-11-28T12:38:00Z

Sorry, I'm not that used to testing with virtualenv, I usually make separate chroots.
Here's what I tried:

virtualenv vi
cd vi
source ./bin/activate
cd ..
tar xzf httpbin-0.6.2.tar.gz
cd httpbin-0.6.2
python setup.py test

That installs some eggs for a while, but then fails in meinheld:

...
Installed /tmp/a/httpbin-0.6.2/.eggs/Flask-0.12.2-py3.6.egg
Searching for meinheld
Reading https://pypi.python.org/simple/meinheld/
Downloading https://pypi.python.org/packages/6c/b7/0a6f03ba8e12862c71a7acc0c051f902f952f13be2a48878ebd2bc198562/meinheld-0.6.1.tar.gz#md5=84535a88d2e4c7088de1d2737bb330a0
Best match: meinheld 0.6.1
Processing meinheld-0.6.1.tar.gz
Writing /tmp/easy_install-llgg4q42/meinheld-0.6.1/setup.cfg
Running meinheld-0.6.1/setup.py -q bdist_egg --dist-dir /tmp/easy_install-llgg4q42/meinheld-0.6.1/egg-dist-tmp-criuhxyu
Sorry, not support .
error: Setup script exited with 1

because meinheld has not integrated the NetBSD patches yet.

What in particular were you looking for in the test?
I am testing the pypi distfile in an empty sandbox where I only install python-3.6.3 and a lot of python modules, the ones that pkgsrc currently thinks are needed by httpbin or one of its dependencies:

babel-2.5.1
brotlipy-0.7.0
cffi-1.11.2
click-6.7
colorama-0.3.9
cparser-2.18
crayons-0.1.2
dateparser-0.6.0
dateutil-2.6.1
decorator-4.1.2
django-1.11.7
flask-0.12.2
flask-cache-0.13.1
flask-common-0.2.0
Flask-Limiter-0.9.5.1
greenlet-0.4.12nb1
gunicorn-19.7.1nb1
humanize-0.5.1
itsdangerous-0.24
jinja2-2.10
limits-1.2.1
markupsafe-1.0
maya-0.3.3
meinheld-0.6.1nb1
pendulum-1.3.2
pytz-2017.3
raven-4.2.1
regex-2017.11.09
ruamel-yaml-0.15.34
setuptools-36.7.2
six-1.11.0
tzdata-2017.3.1
tzlocal-1.4
werkzeug-0.12.2
whitenoise-4.0b3

sigmavirus24 · 2017-11-28T13:18:00Z

@0-wiz-0 I don't think @javabrett was on the right track.

As I said, it looks as if python setup.py test is doing the equivalent of python -m unittest.discover. That scans every directory and is discovering tests in dependencies. Those tests will not pass because the dependencies don't contain the complete information. If you'd like, you can provide a pull request that will fix this. Otherwise, it may take a while for me to get around to it.

0-wiz-0 · 2017-11-28T13:36:16Z

I'm in no hurry, thanks for looking at this.

sigmavirus24 · 2017-12-09T15:04:19Z

@0-wiz-0 can you confirm that doing python setup.py test --test-suite=test_httpbin works?

Tell unittest.discover specifically what to look for instead of letting them scan the entirety of our directory (which could have other dependencies downloaded with their tests that will fail). Closes #403

0-wiz-0 · 2017-12-09T23:26:45Z

Actually, the suggested command line gives me another error:

/usr/pkg/bin/python3.6 setup.py test --test-suite=test_httpbin
running test
running egg_info
writing httpbin.egg-info/PKG-INFO
writing dependency_links to httpbin.egg-info/dependency_links.txt
writing requirements to httpbin.egg-info/requires.txt
writing top-level names to httpbin.egg-info/top_level.txt
reading manifest file 'httpbin.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching 'README.rst'
warning: no files found matching 'requirements.txt'
writing manifest file 'httpbin.egg-info/SOURCES.txt'
running build_ext
test_httpbin (unittest.loader._FailedTest) ... ERROR

======================================================================
ERROR: test_httpbin (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: test_httpbin
Traceback (most recent call last):
  File "/usr/pkg/lib/python3.6/unittest/loader.py", line 153, in loadTestsFromName
    module = __import__(module_name)
ModuleNotFoundError: No module named 'test_httpbin'


----------------------------------------------------------------------
Ran 1 test in 0.000s

FAILED (errors=1)
Test failed: <unittest.runner.TextTestResult run=1 errors=1 failures=0>
error: Test failed: <unittest.runner.TextTestResult run=1 errors=1 failures=0>

(same when I use it for test-httpbin instead)

sigmavirus24 · 2017-12-10T01:18:23Z

Hm, try out the code in #410? I added test_httpbin.py to the MANIFEST.in so ideally it should be present now.

sigmavirus24 · 2017-12-18T17:34:21Z

@0-wiz-0 I think this should be fixed now in master. Please test that out

0-wiz-0 · 2017-12-23T20:12:36Z

I installed (with my packaging system) the httpbin dependencies into a chroot, did a git clone of httpbin as of a couple minutes ago, and tried python3.6 setup.py test. That gives me:

# python3.6 setup.py test  
running test
running egg_info
writing httpbin.egg-info/PKG-INFO
writing dependency_links to httpbin.egg-info/dependency_links.txt
writing requirements to httpbin.egg-info/requires.txt
writing top-level names to httpbin.egg-info/top_level.txt
reading manifest file 'httpbin.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching 'README.rst'
warning: no files found matching 'requirements.txt'
writing manifest file 'httpbin.egg-info/SOURCES.txt'
running build_ext
error: [Errno 2] No such file or directory: 'static/'
# find . -name static
#

Is this the correct test? Am I missing something?

sigmavirus24 · 2017-12-23T21:47:52Z

@0-wiz-0 I really don't know what could be going wrong. This seems to be an issue with your system. I cannot reproduce it. Can you provide exactly the steps you followed (after creating the chroot)?

Diagnostics for postmanlabs#403.

javabrett · 2017-12-23T22:11:50Z

Error in a Travis build: https://travis-ci.org/javabrett/httpbin/jobs/320788583#L779 .

javabrett · 2017-12-24T00:30:26Z

Looks like this is in whitenoise/scantree.py:

Using /httpbin/.eggs/regex-2017.12.12-py3.6-linux-x86_64.egg
running egg_info
Distribution.get_command_obj(): creating 'egg_info' command object
writing httpbin.egg-info/PKG-INFO
writing dependency_links to httpbin.egg-info/dependency_links.txt
writing requirements to httpbin.egg-info/requires.txt
writing top-level names to httpbin.egg-info/top_level.txt
Distribution.get_command_obj(): creating 'build_py' command object
Distribution.get_command_obj(): creating 'build' command object
reading manifest file 'httpbin.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching 'README.rst'
warning: no files found matching 'requirements.txt'
include README.rst LICENSE AUTHORS requirements.txt test_httpbin.py
recursive-include httpbin/templates *
exclude_pattern: applying regex r'(^|\/)(RCS|CVS|\.svn)\/'
writing manifest file 'httpbin.egg-info/SOURCES.txt'
Distribution.get_command_obj(): creating 'build_ext' command object
running build_ext
error: [Errno 2] No such file or directory: 'static/'
Traceback (most recent call last):
  File "setup.py", line 38, in <module>
    'raven[flask]', 'Flask-Common'
  File "/usr/local/lib/python3.6/site-packages/setuptools/__init__.py", line 129, in setup
    return distutils.core.setup(**attrs)
  File "/usr/local/lib/python3.6/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/usr/local/lib/python3.6/distutils/dist.py", line 955, in run_commands
    self.run_command(cmd)
  File "/usr/local/lib/python3.6/distutils/dist.py", line 974, in run_command
    cmd_obj.run()
  File "/usr/local/lib/python3.6/site-packages/setuptools/command/test.py", line 226, in run
    self.run_tests()
  File "/usr/local/lib/python3.6/site-packages/setuptools/command/test.py", line 248, in run_tests
    exit=False,
  File "/usr/local/lib/python3.6/unittest/main.py", line 94, in __init__
    self.parseArgs(argv)
  File "/usr/local/lib/python3.6/unittest/main.py", line 141, in parseArgs
    self.createTests()
  File "/usr/local/lib/python3.6/unittest/main.py", line 148, in createTests
    self.module)
  File "/usr/local/lib/python3.6/unittest/loader.py", line 219, in loadTestsFromNames
    suites = [self.loadTestsFromName(name, module) for name in names]
  File "/usr/local/lib/python3.6/unittest/loader.py", line 219, in <listcomp>
    suites = [self.loadTestsFromName(name, module) for name in names]
  File "/usr/local/lib/python3.6/unittest/loader.py", line 153, in loadTestsFromName
    module = __import__(module_name)
  File "/httpbin/test_httpbin.py", line 13, in <module>
    import httpbin
  File "/httpbin/httpbin/__init__.py", line 3, in <module>
    from .core import *
  File "/httpbin/httpbin/core.py", line 61, in <module>
    common = Common(app)
  File "/httpbin/.eggs/Flask_Common-0.2.0-py3.6.egg/flask_common.py", line 95, in __init__
    self.init_app(app)
  File "/httpbin/.eggs/Flask_Common-0.2.0-py3.6.egg/flask_common.py", line 112, in init_app
    app.wsgi_app = WhiteNoise(app.wsgi_app, root=url_for('static', filename='')[1:])
  File "/httpbin/.eggs/whitenoise-4.0b4-py3.6.egg/whitenoise/base.py", line 65, in __init__
    self.add_files(root, prefix)
  File "/httpbin/.eggs/whitenoise-4.0b4-py3.6.egg/whitenoise/base.py", line 100, in add_files
    self.update_files_dictionary(root, prefix)
  File "/httpbin/.eggs/whitenoise-4.0b4-py3.6.egg/whitenoise/base.py", line 105, in update_files_dictionary
    stat_cache = dict(scantree(root))
  File "/httpbin/.eggs/whitenoise-4.0b4-py3.6.egg/whitenoise/scantree.py", line 21, in scantree
    for entry in scandir(root):
FileNotFoundError: [Errno 2] No such file or directory: 'static/'

javabrett · 2017-12-24T01:32:23Z

Likely caused by PEP-471 os.scandir() from Python 3.5+ and support for same in Whitenoise. Behaviour is different from os.walk() which tolerates a non-existent root-path passed-in.

…tenoise. This is a workaround for linked-issues: * https://github.com/kennethreitz/httpbin/issues/403 * https://github.com/kennethreitz/flask-common/issues/7 * evansd/whitenoise#166 Fixed postmanlabs#403.

sigmavirus24 · 2017-12-24T14:54:00Z

Cool so it's reproducible. That still doesn't tell me what's going wrong. Can you increase the verbosity of that command? I think python setup.py -vvv test should help

Diagnostics for postmanlabs#403.

javabrett · 2017-12-24T21:37:02Z

I pushed a repro branch for Travis with DISTUTILS_DEBUG=1. You will see the stack-trace starting from https://travis-ci.org/javabrett/httpbin/jobs/321148787#L1672 .

The root cause is between Flask-Common, Whitenoise and changes to support newish os.scandir() which is less-tolerant of a non-existent root. Flask-Common inits Whitenoise with a path static/, which does not always exist. Under earlier versions of Python without scandir, this would be OK, but its presence exposes the missing root requested to scan in Whitenoise.

writing manifest file 'httpbin.egg-info/SOURCES.txt'
Distribution.get_command_obj(): creating 'build_ext' command object
running build_ext
error: [Errno 2] No such file or directory: 'static/'
Traceback (most recent call last):
  File "setup.py", line 38, in <module>
    'raven[flask]', 'Flask-Common'
  File "/home/travis/virtualenv/python3.6.3/lib/python3.6/site-packages/setuptools/__init__.py", line 129, in setup
    return distutils.core.setup(**attrs)
  File "/opt/python/3.6.3/lib/python3.6/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/opt/python/3.6.3/lib/python3.6/distutils/dist.py", line 955, in run_commands
    self.run_command(cmd)
  File "/opt/python/3.6.3/lib/python3.6/distutils/dist.py", line 974, in run_command
    cmd_obj.run()
  File "/home/travis/virtualenv/python3.6.3/lib/python3.6/site-packages/setuptools/command/test.py", line 226, in run
    self.run_tests()
  File "/home/travis/virtualenv/python3.6.3/lib/python3.6/site-packages/setuptools/command/test.py", line 248, in run_tests
    exit=False,
  File "/opt/python/3.6.3/lib/python3.6/unittest/main.py", line 94, in __init__
    self.parseArgs(argv)
  File "/opt/python/3.6.3/lib/python3.6/unittest/main.py", line 141, in parseArgs
    self.createTests()
  File "/opt/python/3.6.3/lib/python3.6/unittest/main.py", line 148, in createTests
    self.module)
  File "/opt/python/3.6.3/lib/python3.6/unittest/loader.py", line 219, in loadTestsFromNames
    suites = [self.loadTestsFromName(name, module) for name in names]
  File "/opt/python/3.6.3/lib/python3.6/unittest/loader.py", line 219, in <listcomp>
    suites = [self.loadTestsFromName(name, module) for name in names]
  File "/opt/python/3.6.3/lib/python3.6/unittest/loader.py", line 153, in loadTestsFromName
    module = __import__(module_name)
  File "/home/travis/build/javabrett/httpbin/test_httpbin.py", line 13, in <module>
    import httpbin
  File "/home/travis/build/javabrett/httpbin/httpbin/__init__.py", line 3, in <module>
    from .core import *
  File "/home/travis/build/javabrett/httpbin/httpbin/core.py", line 61, in <module>
    common = Common(app)
  File "/home/travis/build/javabrett/httpbin/.eggs/Flask_Common-0.2.0-py3.6.egg/flask_common.py", line 95, in __init__
    self.init_app(app)
  File "/home/travis/build/javabrett/httpbin/.eggs/Flask_Common-0.2.0-py3.6.egg/flask_common.py", line 112, in init_app
    app.wsgi_app = WhiteNoise(app.wsgi_app, root=url_for('static', filename='')[1:])
  File "/home/travis/build/javabrett/httpbin/.eggs/whitenoise-4.0b4-py3.6.egg/whitenoise/base.py", line 65, in __init__
    self.add_files(root, prefix)
  File "/home/travis/build/javabrett/httpbin/.eggs/whitenoise-4.0b4-py3.6.egg/whitenoise/base.py", line 100, in add_files
    self.update_files_dictionary(root, prefix)
  File "/home/travis/build/javabrett/httpbin/.eggs/whitenoise-4.0b4-py3.6.egg/whitenoise/base.py", line 105, in update_files_dictionary
    stat_cache = dict(scantree(root))
  File "/home/travis/build/javabrett/httpbin/.eggs/whitenoise-4.0b4-py3.6.egg/whitenoise/scantree.py", line 21, in scantree
    for entry in scandir(root):
FileNotFoundError: [Errno 2] No such file or directory: 'static/'

javabrett · 2017-12-24T21:47:39Z

Does Flask-Common as added in a39de83 provide anything? It initialises Whitenoise obviously.

sigmavirus24 · 2017-12-25T12:16:32Z

I believe it adds middleware that @kennethreitz was looking for but I don't honestly know. It's up to him to answer that definitively.

Diagnostics for postmanlabs#403.

javabrett · 2018-05-06T09:20:08Z

Although this issue is already closed, noting that it will be fixed if Flask-Common can be removed per #442 #447.

… alternative to tox. This will catch installation issues of the type seen in postmanlabs#403.

Tell unittest.discover specifically what to look for instead of letting them scan the entirety of our directory (which could have other dependencies downloaded with their tests that will fail). Closes postmanlabs#403

0-wiz-0 mentioned this issue Nov 25, 2017

0.3.0: test failure: static/ not found kevin1024/pytest-httpbin#46

Closed

sigmavirus24 mentioned this issue Dec 9, 2017

Limit python setup.py test to our tests #410

Merged

kennethreitz closed this as completed in #410 Dec 18, 2017

javabrett added a commit to javabrett/httpbin that referenced this issue Dec 23, 2017

Added Travis job to run 'python3.6 setup.py test' instead of tox.

318ec9e

Diagnostics for postmanlabs#403.

javabrett added a commit to javabrett/httpbin that referenced this issue Dec 23, 2017

Added Travis job to run 'python3.6 setup.py test' instead of tox.

d55c3b5

Diagnostics for postmanlabs#403.

This was referenced Dec 24, 2017

Should fail-fast with clear error if root path does not exist evansd/whitenoise#166

Closed

Non-existent static dir causes failure schedutron/flask-common#7

Closed

javabrett added a commit to javabrett/httpbin that referenced this issue Dec 24, 2017

Added Travis job to run 'python3.6 setup.py test' instead of tox.

f9c7541

Diagnostics for postmanlabs#403.

javabrett added a commit to javabrett/httpbin that referenced this issue May 6, 2018

Added Travis job to run 'python3.6 setup.py test' instead of tox.

fb9e4ca

Diagnostics for postmanlabs#403.

javabrett added a commit to javabrett/httpbin that referenced this issue May 6, 2018

Added Travis job to run 'python3.6 setup.py test' instead of tox.

5574bd8

Diagnostics for postmanlabs#403.

javabrett added a commit to javabrett/httpbin that referenced this issue May 6, 2018

Added Travis job to run 'python3.6 setup.py test' for one build as an…

e30fcd6

… alternative to tox. This will catch installation issues of the type seen in postmanlabs#403.

javabrett mentioned this issue May 6, 2018

Added Travis job to run 'python3.6 setup.py test' instead of tox #448

Closed

javabrett added a commit to javabrett/httpbin that referenced this issue Jul 4, 2018

Added Travis job to run 'python3.6 setup.py test' for one build as an…

33aae87

… alternative to tox. This will catch installation issues of the type seen in postmanlabs#403.

javabrett added a commit to javabrett/httpbin that referenced this issue Jul 24, 2018

Added Travis job to run 'python3.6 setup.py test' for one build as an…

bc834d8

… alternative to tox. This will catch installation issues of the type seen in postmanlabs#403.

javabrett mentioned this issue Dec 10, 2018

Run tests during Docker image build #521

Closed

javabrett added a commit to javabrett/httpbin that referenced this issue Jun 26, 2019

Added Travis job to run 'python3.6 setup.py test' for one build as an…

112b7c4

… alternative to tox. This will catch installation issues of the type seen in postmanlabs#403.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

0.6.2: self test failure: `static/` missing #403

0.6.2: self test failure: `static/` missing #403

0-wiz-0 commented Nov 25, 2017

sigmavirus24 commented Nov 25, 2017

sigmavirus24 commented Nov 25, 2017

javabrett commented Nov 28, 2017

0-wiz-0 commented Nov 28, 2017

sigmavirus24 commented Nov 28, 2017

0-wiz-0 commented Nov 28, 2017

sigmavirus24 commented Dec 9, 2017

0-wiz-0 commented Dec 9, 2017

sigmavirus24 commented Dec 10, 2017

sigmavirus24 commented Dec 18, 2017

0-wiz-0 commented Dec 23, 2017

sigmavirus24 commented Dec 23, 2017

javabrett commented Dec 23, 2017

javabrett commented Dec 24, 2017 •

edited

Loading

javabrett commented Dec 24, 2017

sigmavirus24 commented Dec 24, 2017

javabrett commented Dec 24, 2017 •

edited

Loading

javabrett commented Dec 24, 2017

sigmavirus24 commented Dec 25, 2017

javabrett commented May 6, 2018

0.6.2: self test failure: static/ missing #403

0.6.2: self test failure: static/ missing #403

Comments

0-wiz-0 commented Nov 25, 2017

sigmavirus24 commented Nov 25, 2017

sigmavirus24 commented Nov 25, 2017

javabrett commented Nov 28, 2017

0-wiz-0 commented Nov 28, 2017

sigmavirus24 commented Nov 28, 2017

0-wiz-0 commented Nov 28, 2017

sigmavirus24 commented Dec 9, 2017

0-wiz-0 commented Dec 9, 2017

sigmavirus24 commented Dec 10, 2017

sigmavirus24 commented Dec 18, 2017

0-wiz-0 commented Dec 23, 2017

sigmavirus24 commented Dec 23, 2017

javabrett commented Dec 23, 2017

javabrett commented Dec 24, 2017 • edited Loading

javabrett commented Dec 24, 2017

sigmavirus24 commented Dec 24, 2017

javabrett commented Dec 24, 2017 • edited Loading

javabrett commented Dec 24, 2017

sigmavirus24 commented Dec 25, 2017

javabrett commented May 6, 2018

0.6.2: self test failure: `static/` missing #403

0.6.2: self test failure: `static/` missing #403

javabrett commented Dec 24, 2017 •

edited

Loading

javabrett commented Dec 24, 2017 •

edited

Loading