Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

可以增加一些友好的配置项 #11

Open
zhengjianyang opened this issue Jul 25, 2021 · 4 comments
Open

可以增加一些友好的配置项 #11

zhengjianyang opened this issue Jul 25, 2021 · 4 comments
Assignees
@OxOOo

Comments

@zhengjianyang
Copy link

可以增加一些友好的配置项
比如：
为每个spider设置等待时间、因为是爬取站点免费公开信息、更新时间难免不同、统一的等待难免对部分更新快的站点不友好
为spider快捷设置代理、爬取过程中频繁或者根本就是被墙站点就会无法访问、如果能带着库里的代理ip就会友好很多
@OxOOo
Copy link
Owner

OxOOo commented Jul 28, 2021

好主意

@OxOOo OxOOo self-assigned this Jul 28, 2021
@chalan630
Copy link

好主意

“可用代理”那个页面会显示全部可用代理，数据量过大导致响应时间过长，后期能否进行分页显示呢？

@OxOOo
Copy link
Owner

OxOOo commented Sep 5, 2021

“可用代理”那个页面会显示全部可用代理，数据量过大导致响应时间过长，后期能否进行分页显示呢？

已经增加了分页功能。

@gobyto
Copy link

gobyto commented Nov 17, 2022

建议增加ip所属国家分类提取不同地区的ip

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@OxOOo OxOOo

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@OxOOo @chalan630 @gobyto @zhengjianyang