Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add example to use write_posts_to_csv #992

Merged

Conversation

PierreMesure
Copy link
Contributor

Hi, I tried to scrape from a rather large group and I got a Timeout error several times.

I was using the get_posts() method so I had to restart from page 1 every time and that was neither reliable nor resource effective.

So I dug a little more in the code and found out about the write_posts_to_csv method. I got it to work with a loop to do just what I want.

I'm not sure if you want to keep that method internal but it's already partly documented in the code and I see you're recommending its use so I added an example in the README. Hopefully, that will help people trying to do the same and reduce the amount of support you get.

Some more things that I was thinking of improving:

  • The function's arguments are not all documented in the code. format, resume_file...
  • There is potential to make the function even smarter at dealing with an existing file. It could for instance overwrite it or just continue filling it from where it left off if resume_file is set? Would that be interesting to implement in your opinion?

@neon-ninja neon-ninja merged commit 5f101fa into kevinzg:master Oct 30, 2023
@PierreMesure PierreMesure deleted the Add-example-for-write_posts_to_csv branch October 30, 2023 07:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants