Skip to content
This repository has been archived by the owner on Feb 20, 2022. It is now read-only.

Commit

Permalink
Restoring RST readme version
Browse files Browse the repository at this point in the history
  • Loading branch information
alecxe committed Sep 23, 2016
1 parent 0f1eca8 commit c786e38
Show file tree
Hide file tree
Showing 4 changed files with 56 additions and 39 deletions.
36 changes: 0 additions & 36 deletions README.md

This file was deleted.

55 changes: 55 additions & 0 deletions README.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
.. image:: https://badge.fury.io/py/scrapy-fake-useragent.svg
:target: http://badge.fury.io/py/scrapy-fake-useragent
:alt: PyPI version

.. image:: https://requires.io/github/alecxe/scrapy-fake-useragent/requirements.svg?branch=master
:target: https://requires.io/github/alecxe/scrapy-fake-useragent/requirements/?branch=master
:alt: Requirements Status

scrapy-fake-useragent
=====================

Random User-Agent middleware based on
`fake-useragent <https://pypi.python.org/pypi/fake-useragent>`__. It
picks up ``User-Agent`` strings based on `usage
statistics <http://www.w3schools.com/browsers/browsers_stats.asp>`__
from a `real world database <http://useragentstring.com/>`__.

Configuration
-------------

Turn off the built-in ``UserAgentMiddleware`` and add
``RandomUserAgentMiddleware``.

In Scrapy >=1.0:

::

DOWNLOADER_MIDDLEWARES = {
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware': None,
'scrapy_fake_useragent.middleware.RandomUserAgentMiddleware': 400,
}

In Scrapy <1.0:

::

DOWNLOADER_MIDDLEWARES = {
'scrapy.contrib.downloadermiddleware.useragent.UserAgentMiddleware': None,
'scrapy_fake_useragent.middleware.RandomUserAgentMiddleware': 400,
}

Usage with `scrapy-proxies`
-------------

To use with middlewares of random proxy such as `scrapy-proxies <https://github.com/aivarsk/scrapy-proxies>`_, you need:

1. set ``RANDOM_UA_PER_PROXY`` to True to allow switch per proxy

2. set priority of ``RandomUserAgentMiddleware`` to be greater than ``scrapy-proxies``, so that proxy is set before handle UA


.. |GitHub version| image:: https://badge.fury.io/gh/alecxe%2Fscrapy-fake-useragent.svg
:target: http://badge.fury.io/gh/alecxe%2Fscrapy-fake-useragent
.. |Requirements Status| image:: https://requires.io/github/alecxe/scrapy-fake-useragent/requirements.svg?branch=master
:target: https://requires.io/github/alecxe/scrapy-fake-useragent/requirements/?branch=master
1 change: 1 addition & 0 deletions setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
name='scrapy-fake-useragent',
version='1.0.1',
description='Use a random User-Agent provided by fake-useragent every request',
long_description=open('README.rst').read(),
keywords='scrapy proxy user-agent web-scraping',
license='New BSD License',
author="Alexander Afanasyev",
Expand Down
3 changes: 0 additions & 3 deletions upload.sh

This file was deleted.

0 comments on commit c786e38

Please sign in to comment.