read_html failing in many tests: AssertionError: No tables found #4214

yarikoptic · 2013-07-11T16:03:58Z

typical error with

======================================================================
FAIL: test_banklist (pandas.io.tests.test_html.TestReadHtmlBase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/yoh/deb/gits/pkg-exppsy/build-area/pandas-0.11.0+git43-g7b2eaa4/debian/tmp/usr/lib/python2.7/dist-packages/pandas/io/tests/test_html.py", line 114, in test_banklist
    attrs={'id': 'table'})
  File "/home/yoh/deb/gits/pkg-exppsy/build-area/pandas-0.11.0+git43-g7b2eaa4/debian/tmp/usr/lib/python2.7/dist-packages/pandas/io/tests/test_html.py", line 67, in run_read_html
    return read_html(*args, **kwargs)
  File "/home/yoh/deb/gits/pkg-exppsy/build-area/pandas-0.11.0+git43-g7b2eaa4/debian/tmp/usr/lib/python2.7/dist-packages/pandas/io/html.py", line 900, in read_html
    attrs)
  File "/home/yoh/deb/gits/pkg-exppsy/build-area/pandas-0.11.0+git43-g7b2eaa4/debian/tmp/usr/lib/python2.7/dist-packages/pandas/io/html.py", line 769, in _parse
    raise retained
AssertionError: No tables found

The text was updated successfully, but these errors were encountered:

yarikoptic · 2013-07-11T16:04:36Z

I guess another related failure

======================================================================
FAIL: pandas.io.tests.test_html.test_bs4_finds_tables
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/home/yoh/deb/gits/pkg-exppsy/build-area/pandas-0.11.0+git43-g7b2eaa4/debian/tmp/usr/lib/python2.7/dist-packages/pandas/io/tests/test_html.py", line 453, in test_bs4_finds_tables
    assert get_elements_from_url(filepath, 'table')
AssertionError

jreback · 2013-07-11T16:07:26Z

can you show ci/print_versions.py?

yarikoptic · 2013-07-11T16:16:56Z

Here is what I have ATM (without running the build with adjusted
matplotlib backend and PYTHONPATH... which shouldn't matter -- just want
to make clear if I am not missing anything):

$> ci/print_versions.py

INSTALLED VERSIONS

Python: 2.7.5.final.0
OS: Linux 3.2.0-4-amd64 #1 SMP Debian 3.2.41-2 x86_64
LC_ALL: None
LANG: en_US

Cython: 0.19
Numpy: 1.7.1
Scipy: 0.10.1
statsmodels: 0.4.2
patsy: Not installed
scikits.timeseries: Not installed
dateutil: 1.5
pytz: 2012c
bottleneck: Not installed
PyTables: 2.3.1
numexpr: 2.0.1
matplotlib: 1.1.1rc2
openpyxl: 1.6.1
xlrd: 0.6.1
xlwt: 0.7.4
sqlalchemy: 0.7.9
lxml: 3.2.0
bs4: 4.2.0
html5lib: 0.95-dev

On Thu, 11 Jul 2013, jreback wrote:

can you show ci/print_versions.py?

—
Reply to this email directly or [1]view it on GitHub.

References

Visible links

read_html failing in many tests: AssertionError: No tables found #4214 (comment)

Yaroslav O. Halchenko, Ph.D.
http://neuro.debian.net http://www.pymvpa.org http://www.fail2ban.org
Senior Research Associate, Psychological and Brain Sciences Dept.
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419
WWW: http://www.linkedin.com/in/yarik

jreback · 2013-07-11T16:25:02Z

@cpcloud ?

cpcloud · 2013-07-11T16:31:30Z

@yarikoptic this is master correct?

cpcloud · 2013-07-11T16:38:32Z

7097368 should have fixed these issues

it looks like you're using an older version of html5lib. i believe 1.0b2 is out.

also, check out the reading html gotchas

yarikoptic · 2013-07-11T16:41:46Z

yes -- master 0.11.0+git43-g7b2eaa4

thanks for the fix -- I will check it out

html5lib -- well, I have the freshiest non-beta release ;) thanks for the note though!

cpcloud · 2013-07-11T16:45:00Z

i'll try this on a vagrant box

the commit i referenced above is already in master

cpcloud · 2013-07-11T16:46:04Z

@yarikoptic is it of any significance that your version says 0.11.0?

$ git describe
v0.12.0rc1-43-g7b2eaa4

cpcloud · 2013-07-11T16:52:16Z

@yarikoptic there are no guarantees that your bs4 version will work...

see the optional dependencies docs

also your lxml version might need to be upgraded.

i would say: leave html5lib alone for now that should be fine, but change your bs4 from 4.2.0 to either 4.2.1, 4.1.3 or 4.0.2. then run tests. if lxml still fails upgrade to 3.2.1

if none of this works then i'll mark it as a bug

cpcloud · 2013-07-11T16:54:13Z

there's a release of bs4 that really shouldn't have existed, i believe it is 4.2.0.

cpcloud · 2013-07-11T16:55:27Z

fyi travis installs bs4 4.0.2 and that works...i'm not sure how different ubuntu 12.04 is from debian sid, but i imagine they share a lot of similarity, i'm sure you know much more than i about this

jreback · 2013-07-11T16:57:14Z

It think @cpcloud should setup a version dependency hotline, kind of like AA :)

cpcloud · 2013-07-11T16:59:28Z

Hello, this is read_html-one-one, what's your emergency?

yarikoptic · 2013-07-11T16:59:59Z

ah -- thanks -- should have been indeed 0.12.0~rc1+git... as the Debian
perspective version -- fixed in my debian branch.

On Thu, 11 Jul 2013, Phillip Cloud wrote:

[1]@yarikoptic is it of any significance that your version says 0.11.0?

$ git describe
v0.12.0rc1-43-g7b2eaa4

Yaroslav O. Halchenko, Ph.D.
http://neuro.debian.net http://www.pymvpa.org http://www.fail2ban.org
Senior Research Associate, Psychological and Brain Sciences Dept.
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419
WWW: http://www.linkedin.com/in/yarik

cpcloud · 2013-07-11T17:00:15Z

excellent!

cpcloud · 2013-07-11T17:00:35Z

does that mean bs4==4.2.0 works for you?

yarikoptic · 2013-07-11T17:01:41Z

yikes -- good to know... well at least in current Debian stable we have
4.1.0 -- is that good enough? ;-)

On Thu, 11 Jul 2013, Phillip Cloud wrote:

there's a release of bs4 that really shouldn't have existed, i believe it
is 4.2.0.

Yaroslav O. Halchenko, Ph.D.
http://neuro.debian.net http://www.pymvpa.org http://www.fail2ban.org
Senior Research Associate, Psychological and Brain Sciences Dept.
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419
WWW: http://www.linkedin.com/in/yarik

cpcloud · 2013-07-11T17:03:35Z

that should be ok, as long as it passes for you. does this also fix #4212? if so please close both, thanks for the report 😄

cpcloud · 2013-07-11T17:06:23Z

also the other issues you've raised today might be fixed now that your version is correct

yarikoptic · 2013-07-11T17:13:25Z

sorry -- I guess my mental pipelining broke -- what should have fixed my issue? downgrade of bs4 to < 4.2.0?
that commit mentioned 7097368 has been in master for a while so was already included in the version I have built -- or am I wrong?

cpcloud · 2013-07-11T17:38:05Z

The bs4 downgrade should be the fix.

yarikoptic · 2013-07-11T17:45:54Z

ok -- will give it a try. but as for the reports correction it would be better if pandas internally switched between backends/skipped the tests if broken bs4 is present; since otherwise it becomes impractical for me to build them across all debian/ubuntu releases having the broken bs4

yarikoptic · 2013-07-11T17:47:13Z

btw -- what about bs4 4.2.1 -- should that be "good enough"?

cpcloud · 2013-07-11T17:52:03Z

Re bs4 skip good idea. I'll add a warning to the code as well since iirc 4.0.2 worked fine on my arch Linux box.

4.2.1 should be fine.

cpcloud · 2013-07-11T17:52:32Z

I mean 4.2.0 worked fine.

yarikoptic · 2013-07-11T17:58:49Z

... so bs4 4.2.0 is not at fault? (anyways -- rebuilding/testing now with 4.2.1 installed)

cpcloud · 2013-07-11T18:06:00Z

no bs4 4.2.0 is at fault

yarikoptic · 2013-07-11T18:21:49Z

then what "I mean 4.2.0 worked fine. " is about? ;) or it is at fault only in some deployments?

cpcloud · 2013-07-11T18:22:49Z

works fine on my arch linux machine, but fails on debian distros, not sure exactly why

cpcloud · 2013-07-11T18:22:58Z

let me check that though...

yarikoptic · 2013-07-11T18:40:03Z

wow -- with 4.2.1 only #4215 is left failing:

FAIL: test_invalid_colormap (pandas.tests.test_graphics.TestDataFrameGroupByPlots)

Traceback (most recent call last):
File "/home/yoh/deb/gits/pkg-exppsy/build-area/pandas-0.12.0~rc1+git43-g7b2eaa4/debian/tmp/usr/lib/python2.7/dist-packages/pandas/tests/test_graphics.py", line 995, in test_invalid_colormap
self.assertRaises(ValueError, df.plot, colormap='invalid_colormap')
AssertionError: ValueError not raised

Ran 3622 tests in 743.502s

FAILED (SKIP=83, failures=1)

On Thu, 11 Jul 2013, Phillip Cloud wrote:

let me check that though...

Yaroslav O. Halchenko, Ph.D.
http://neuro.debian.net http://www.pymvpa.org http://www.fail2ban.org
Senior Research Associate, Psychological and Brain Sciences Dept.
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419
WWW: http://www.linkedin.com/in/yarik

cpcloud · 2013-07-11T18:42:37Z

that good news! not sure where that is coming from...i'll see what i can do about it

jreback · 2013-07-12T13:02:14Z

@cpcloud ok....you are going to add a skip test / warning message ? if 4.2.0 is installed (maybe just raise in read_html...hey user your 4.2.0 is broken, this won't work.....

cpcloud · 2013-07-13T16:09:38Z

i'm going to actually raise if no tables found and bs4==4.2.0, since there's nothing else to fallback on after that. i've cooked up a fairly informative error message

cpcloud · 2013-07-13T16:52:22Z

hard to test for this...we don't have an extensive list of OSes that work with bs4

yarikoptic · 2013-07-16T02:59:38Z

with 0.12.0~rc1+git79-g50eff60 and bs4 4.2.0 installed (on sparc)

======================================================================
FAIL: pandas.io.tests.test_html.test_bs4_finds_tables
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/home/yoh/deb/gits/pkg-exppsy/pandas/debian/tmp/usr/lib/python2.7/dist-packages/pandas/io/tests/test_html.py", line 472, in test_bs4_finds_tables
    assert get_elements_from_url(filepath, 'table')
AssertionError

yarikoptic · 2013-07-16T03:00:00Z

ah -- just now mentioned that it is closed -- let me know if I need to open a new one

cpcloud · 2013-07-16T03:20:06Z

4.2.0 won't work.

yarikoptic · 2013-07-16T03:23:53Z

then test should be skipped?

On Mon, 15 Jul 2013, Phillip Cloud wrote:

4.2.0 won't work.

—
Reply to this email directly or [1]view it on GitHub.

References

Visible links

read_html failing in many tests: AssertionError: No tables found #4214 (comment)

Yaroslav O. Halchenko, Ph.D.
http://neuro.debian.net http://www.pymvpa.org http://www.fail2ban.org
Senior Research Associate, Psychological and Brain Sciences Dept.
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419
WWW: http://www.linkedin.com/in/yarik

cpcloud · 2013-07-16T03:35:27Z

R u sure u have master?

yarikoptic · 2013-07-16T04:07:30Z

as I have mentioned I had 50eff60 which was the master few hours back, now there is some new changes:

*$> git log --pretty=oneline 0.12.0~rc1+git79-g50eff60..origin/master
1e69dade1309b494e43f26f9831326c2ce63f7de Merge pull request #4248 from cpcloud/mpl-1.1.1-build
5be1e5a3f27e43c8b8cae57fed684e62587580fb Merge pull request #4243 from kjordahl/master
5b908a5a1d801f67b78d179b934f860bffe37533 BLD: use mpl 1.1.1 in python 2.7 production travis build
53b7a74fd756cfd2e6fca58326a577558ab97909 Merge pull request #4247 from jreback/series_dups
bbcfd929205ef79c00a403050f107c5e05e0b300 ENH: implement non-unique indexing in series (GH4246)
56e6b173f1c94162bfe5b7dfacb58062011dc96b DOC: Fix typos in CONTRIBUTING.md

cpcloud · 2013-07-16T13:21:32Z

i'll open a straw man issue for your most recent find -- will be knocked down by the other html test fixes pr...

@yarikoptic thanks for all the reports!

yarikoptic mentioned this issue Jul 11, 2013

BLD: _check_plot_works(df.plot) in test results in RuntimeError: Error building image #4213

Closed

cpcloud mentioned this issue Jul 13, 2013

CLN/TST: clean up and raise on bs4 version and no tables #4232

Merged

ghost assigned cpcloud Jul 13, 2013

cpcloud closed this as completed in #4232 Jul 13, 2013

cpcloud reopened this Jul 16, 2013

cpcloud mentioned this issue Jul 16, 2013

TST: no need for flavor testing in skip #4257

Merged

cpcloud closed this as completed in #4257 Jul 16, 2013

wesm unassigned cpcloud Oct 12, 2016

read_html failing in many tests: AssertionError: No tables found #4214

read_html failing in many tests: AssertionError: No tables found #4214

Comments

yarikoptic commented Jul 11, 2013

yarikoptic commented Jul 11, 2013

jreback commented Jul 11, 2013

yarikoptic commented Jul 11, 2013

INSTALLED VERSIONS

jreback commented Jul 11, 2013

cpcloud commented Jul 11, 2013

cpcloud commented Jul 11, 2013

yarikoptic commented Jul 11, 2013

cpcloud commented Jul 11, 2013

cpcloud commented Jul 11, 2013

cpcloud commented Jul 11, 2013

cpcloud commented Jul 11, 2013

cpcloud commented Jul 11, 2013

jreback commented Jul 11, 2013

cpcloud commented Jul 11, 2013

yarikoptic commented Jul 11, 2013

cpcloud commented Jul 11, 2013

cpcloud commented Jul 11, 2013

yarikoptic commented Jul 11, 2013

cpcloud commented Jul 11, 2013

cpcloud commented Jul 11, 2013

yarikoptic commented Jul 11, 2013

cpcloud commented Jul 11, 2013

yarikoptic commented Jul 11, 2013

yarikoptic commented Jul 11, 2013

cpcloud commented Jul 11, 2013

cpcloud commented Jul 11, 2013

yarikoptic commented Jul 11, 2013

cpcloud commented Jul 11, 2013

yarikoptic commented Jul 11, 2013

cpcloud commented Jul 11, 2013

cpcloud commented Jul 11, 2013

yarikoptic commented Jul 11, 2013

FAIL: test_invalid_colormap (pandas.tests.test_graphics.TestDataFrameGroupByPlots)

cpcloud commented Jul 11, 2013

jreback commented Jul 12, 2013

cpcloud commented Jul 13, 2013

cpcloud commented Jul 13, 2013

yarikoptic commented Jul 16, 2013

yarikoptic commented Jul 16, 2013

cpcloud commented Jul 16, 2013

yarikoptic commented Jul 16, 2013

cpcloud commented Jul 16, 2013

yarikoptic commented Jul 16, 2013

cpcloud commented Jul 16, 2013