Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

更方便組裝的爬蟲 #31

Closed
4 of 5 tasks
ddio opened this issue Jun 9, 2019 · 0 comments · Fixed by #93
Closed
4 of 5 tasks

更方便組裝的爬蟲 #31

ddio opened this issue Jun 9, 2019 · 0 comments · Fixed by #93
Assignees
@ddio

Comments

@ddio
Copy link
Contributor

ddio commented Jun 9, 2019 •
edited
Loading

為了減少新專案重造輪子的複雜度，要將通常會共用的部份拆出來，方便寫 scrapy 的人降低開發成本。

待作：

  1. 一個可以針對所有租屋網站的爬蟲，並以目前已有的 591 為例子
  2. 方便調整，插入其他功能的架構
  3. 盡量避免依賴第三方套件的，例如目前專案裡使用的 Django + GeoDjango
  4. 說明文件
  5. 整合到原本的爬蟲程式裡
@ddio ddio self-assigned this Jun 9, 2019
ddio added a commit that referenced this issue Jun 11, 2019
@ddio
Migrate 591 list spider from original code, #31
8dd4605
ddio added a commit that referenced this issue Jun 11, 2019
@ddio
Migrate 591 detail spider from original code and add simple e2e testi…
e02cf85 
…ng, #31
ddio added a commit that referenced this issue Jun 11, 2019
@ddio
Finish 1st version of scrapy package. Still need to finish last examp…
68f0069 
…le and integrate into original crawler, #31
ddio added a commit that referenced this issue Jun 11, 2019
@ddio
Add package url, #31
7cec0ab
ddio added a commit that referenced this issue Jun 11, 2019
@ddio
Update example readme to mark unfinished item, #31
a2a58e4
ddio added a commit that referenced this issue Jun 11, 2019
@ddio
Remove redundent file, #31
bed88fc
@ddio ddio mentioned this issue Jun 19, 2019
Open
This was referenced Oct 25, 2021
Merged
Closed
ddio added a commit that referenced this issue Oct 27, 2021
@ddio
feat: use scrapy_twrh in list591 spider, #89, #31
139ecc3
ddio added a commit that referenced this issue Oct 27, 2021
@ddio
feat: use scrapy_twrh in detail591 spider, #89, #31
3651c6f
@ddio ddio mentioned this issue Oct 27, 2021
Merged
@ddio ddio closed this as completed in #93 Oct 27, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ddio ddio

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Apply scrapy package g0v/tw-rental-house-data
1 participant
@ddio