This is a scrapy project that is about Scrapy + selenium + ProxyAgent + UserAgent, and using a suprising package called 'CloudScraper' to help me to promote my project. But unfortunately, I still couldn't make a perfect job, Im guessing that I should use Redis? or not? I hope that my project could be a reference for someone.

这个scrapy项目使用了 selenium+代理池(User-agent、IPpool) 来进行的爬取，以及一个应对简单的CloudFlare的库，我希望这对你能有帮助。