getting 401

[zinyosrim]

I installed aquarium with the default options. I looks fine, I get lots of messages like:

splash0_1  | 2018-01-22 12:11:06.702841 [-] "172.21.0.6" - - [22/Jan/2018:12:11:05 +0000] "GET / HTTP/1.0" 200 7677 "-" "-"
splash1_1  | 2018-01-22 12:11:07.039685 [-] "172.21.0.6" - - [22/Jan/2018:12:11:06 +0000] "GET / HTTP/1.0" 200 7677 "-" "-"
splash2_1  | 2018-01-22 12:11:07.442394 [-] "172.21.0.6" - - [22/Jan/2018:12:11:06 +0000] "GET / HTTP/1.0" 200 7677 "-" "-"

I'm running locally on mac, therefore my settings.py has the line SPLASH_URL = 'http://localhost:8050'

When launching the crawler I'm getting

2018-01-22 12:51:45 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <401 http://www.example.com>: HTTP status code is not handled or not allowed

and I can't see the request in the splash terminal. How can I fix this?

Thanks
Zin

[zinyosrim]

I fixed this by adding

http_user = 'user'
http_pass = 'userpass'

as class attributes of my spider

[chipzzz]

@zinyosrim , where did you add this?

[zinyosrim]

@chipzzz here:

[chipzzz]

@zinyosrim Thanks!

In another issue I found this to be working for me

yield SplashRequest(url, callback=self.parse, endpoint='render.html',
             meta={'start_url': url}, args={'wait': 3}, splash_headers={'Authorization': basic_auth_header('user', 'userpass')})

Still no luck on getting these requests to go through my 3rd party proxy though.

[tayyab-elahi]

I fixed this by adding

http_user = 'user'
http_pass = 'userpass'

as class attributes of my spider

Not working for me though.