-
Notifications
You must be signed in to change notification settings - Fork 10.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Issue with running scrapy spider from script. #2473
Comments
I would try to downgrade your twisted version from 2017-01-02 14:25:00 [scrapy] ERROR: Error downloading <GET http://www.citysearch.com/profile/645344264/jackson_ms/wright_patrick_b_md_patrick_b_wright_md.html>
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1297, in _inlineCallbacks
result = result.throwExceptionIntoGenerator(g)
File "/usr/local/lib/python2.7/dist-packages/twisted/python/failure.py", line 393, in throwExceptionIntoGenerator
return g.throw(self.type, self.value, self.tb)
File "/usr/local/lib/python2.7/dist-packages/scrapy/core/downloader/middleware.py", line 43, in process_request
defer.returnValue((yield download_func(request=request,spider=spider)))
File "/usr/local/lib/python2.7/dist-packages/scrapy/utils/defer.py", line 45, in mustbe_deferred
result = f(*args, **kw)
File "/usr/local/lib/python2.7/dist-packages/scrapy/core/downloader/handlers/__init__.py", line 65, in download_request
return handler.download_request(request, spider)
File "/usr/local/lib/python2.7/dist-packages/scrapy/core/downloader/handlers/http11.py", line 60, in download_request
return agent.download_request(request)
File "/usr/local/lib/python2.7/dist-packages/scrapy/core/downloader/handlers/http11.py", line 285, in download_request
method, to_bytes(url, encoding='ascii'), headers, bodyproducer)
File "/usr/local/lib/python2.7/dist-packages/twisted/web/client.py", line 1631, in request
parsedURI.originForm)
File "/usr/local/lib/python2.7/dist-packages/twisted/web/client.py", line 1408, in _requestWithEndpoint
d = self._pool.getConnection(key, endpoint)
File "/usr/local/lib/python2.7/dist-packages/twisted/web/client.py", line 1294, in getConnection
return self._newConnection(key, endpoint)
File "/usr/local/lib/python2.7/dist-packages/twisted/web/client.py", line 1306, in _newConnection
return endpoint.connect(factory)
File "/usr/local/lib/python2.7/dist-packages/twisted/internet/endpoints.py", line 779, in connect
EndpointReceiver, self._hostText, portNumber=self._port
File "/usr/local/lib/python2.7/dist-packages/twisted/internet/_resolver.py", line 174, in resolveHostName
onAddress = self._simpleResolver.getHostByName(hostName)
File "/usr/local/lib/python2.7/dist-packages/scrapy/resolver.py", line 21, in getHostByName
d = super(CachingThreadedResolver, self).getHostByName(name, timeout)
File "/usr/local/lib/python2.7/dist-packages/twisted/internet/base.py", line 276, in getHostByName
timeoutDelay = sum(timeout)
TypeError: 'float' object is not iterable After, downgrading to the version I had( command: |
#2479 This is related to this one as well. |
@tituskex , did you manage to make it work? |
Downgrading Twisted worked for me too. |
@pembeci what is your Scrapy version? |
@kmike The latest from pip install: 1.3.2. I am running on an old machine which is not upgraded for a while: Ubuntu 12.04 LTS - 32 bit |
@pembeci what was the exception? |
@pembeci I would recommend to use (mini)conda to have the latest releases without having to upgrade system libraries in old systems. |
+1 . Same problem with scrapy (1.3.2) and twisted (17.1.0) .
|
@wzpan what is your pyOpenSSL version? Twisted dropped support for pyOpenSSL < 16.0.0 in Twisted 16.4.0 release (see http://twistedmatrix.com/trac/ticket/8441); in fact it worked for some time, but they recently removed some of the supporting code as well. Is upgrading it an option? You can check pyOpenSSL version by running |
@kmike awesome! 👍 |
I run into this problem, too.
Version:
Any help would be appreciated. |
@noprom Try doing these: pip install --upgrade scrapy
pip install --upgrade twisted
pip install --upgrade pyopenssl |
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
also solved mine, but tbh backwards incompatibility is a shame |
I couldn't even run scrapy by itself with out the SSL error until I downgraded Twisted from 17 to 16.4.1 per @rapliandras |
For the record, we've released "packaging fix" versions that prevent Twisted>=17 getting installed, because branches 1.0.x, 1.1.x and 1.2.x only support Twisted<=16.6
Master branch (and the recent v1.3.3) are compatible with Twisted 17+ |
So it seems that latest Twisted does require pyOpenSSL>=0.16, but provided you add the |
@redapple I haven't realized it is just a warning, not an error. If adding |
It seems that pip < 6.1.0 raises an error if extra requirement is unknown intead of showing a warning - see pypa/pip#2142. I'm not sure what happens if Twisted < 15.0 is already installed, user has pip < 6.1.0 (e.g. pip 1.5 is still popular), and runs |
Good point @kmike . It does not work if one asks for Twisted<15:
If we consider upgrades to latest Twisted, it works though, because latest Twisted has the extra:
Is it fair to say that installing and upgrading via |
I was asking about a different case:
It seems it can fail (I've execute this in a clean virtualenv):
|
Alright, so the best we can do until scrapy requires twisted[tls]>=15 is
to document this in the faq perhaps. And maybe suggest either `pip install
--upgrade scrapy` or downgrade Twisted. Thoughts ?
Le 28 mars 2017 13:26, "Mikhail Korobov" <notifications@github.com> a
écrit :
… I was asking about a different case:
1. User already has Twisted < 15 installed (e.g. from system
packages), but doesn't have Scrapy installed.
2. Then user runs pip install scrapy, without --upgrade or specifying
a version.
It seems this can fail in a clean virtualenv:
> pip install 'pip < 6.1.0'
..snip..
> pip install 'twisted<15'
..snip..
> pip install twisted[tls]
You are using pip version 6.0.8, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Requirement already satisfied (use --upgrade to upgrade): twisted[tls] in /Users/kmike/envs/tst-scrapy/lib/python2.7/site-packages
Exception:
Traceback (most recent call last):
File "/Users/kmike/envs/tst-scrapy/lib/python2.7/site-packages/pip/basecommand.py", line 232, in main
status = self.run(options, args)
File "/Users/kmike/envs/tst-scrapy/lib/python2.7/site-packages/pip/commands/install.py", line 339, in run
requirement_set.prepare_files(finder)
File "/Users/kmike/envs/tst-scrapy/lib/python2.7/site-packages/pip/req/req_set.py", line 436, in prepare_files
req_to_install.extras):
File "/Users/kmike/envs/tst-scrapy/lib/python2.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2504, in requires
"%s has no such extra feature %r" % (self, ext)
UnknownExtra: Twisted 14.0.2 has no such extra feature 'tls'
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#2473 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AA2GGAR3b9URqFQT9lCAjtyAUzp6dB1Pks5rqO5ZgaJpZM4LY7Vt>
.
|
For the record, both debian jessie and ubuntu 14.04 use pip 1.5 and twisted < 15.0, so these baslines are affected. Suggesting |
What do you think about providing scrapy[tls] extra? After bumping requirements to |
I am not very fond of introducing a "tls" extra at Scrapy level as well, as I think it could be hard to explain that it does not mean TLS support ON or OFF, when to use it etc. It's just a shame we cannot says something like |
Fair enough. I'm fine with documenting this in FAQ, or maybe in a new Troubleshooting section in Install docs ("got AttributeError: 'module' object has no attribute 'OP_NO_TLSv1_1' exception? This happens because Twisted dropped support for older pyOpenSSL versions. Either downgrade Twisted to ... or upgrade PyOpenSSL to 0.16+). |
Have tried to deal with a problem Just follow up step working fine |
+1 for a new Troubleshooting section in Install docs. Could be hard to keep updated, but I believe we have some common cases in StackOverflow and here |
Rhel 7/centos 7 works for me pip install Twisted==16.4.1 |
This comment has been minimized.
This comment has been minimized.
I'm using when I run EDIT: just thought I'd mention that I've also tried putting |
@wzpan Cool! you solved my problem, Thanks |
In scrapy=1.5.0 still exist this problem, need install Twisted==16.4.1 |
Its initial content covers the workaround for scrapy#2473.
Its initial content covers the workaround for scrapy#2473.
Hi, |
Its initial content covers the workaround for scrapy#2473.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
I wonder if it's still needed with modern Scrapy? |
We actually covered this in the documentation as part of #3517 But now I wonder if we should remove that from the documentation now, if this were not needed nowadays. |
Hi, I'm trying to run scrapy from a script like this:
However, when I run this script I get the following error:
Does anyone know how to fix this? Thanks in advance.
The text was updated successfully, but these errors were encountered: