How to have some sort of setup phase?

[afuetterer]

Hi,

I am a new user of the library and I really enjoy the experience so far.

I have a question: Can I do some sort of "setup" for the browser / context? I would like to have a PlaywrightCrawler, that is "logged in" and selected a few common parameters in the UI of a website. The crawling/scraping of the following urls should use this state.

Is this possible or out of scope for the library, because it can spawn a lot of browser instances?

Hoping for a hint.

[janbuchar]

Hello @afuetterer and thanks for your interest in Crawlee! Maybe @vdusek knows a better way to do this.

When I needed to do a similar thing for a demo scraper of the Nike website, I made a context manager and wrapped all my request handlers with that - https://crawlee.dev/blog/infinite-scroll-using-python#accept-cookies-context-manager.

Also, once pre_navigation_hooks (#427) are implemented and released, you may want to use those.

[afuetterer]

Thanks, I will have a look at your demo scraper.

The context manager could log in and select the parameters could be a solution, yes. But it will do these steps for every request, right?

This needs to be done once per browser, that per request, as the log in and auth steps are slow.

If I understand correctly, pre_navigation_hooks might be the way for achieving this, but not available right now?

[janbuchar]

Yes, you understand both right. The pre-navigation hooks are also per-request. You can always check if the "setup" is necessary for each request separately. Perhaps you could utilize page.context to determine the browser that owns each page and keep a record of these somewhere?

[afuetterer]

Thanks, I will give this a try.