AsyncPlaywrightCrawlerStrategy page-evaluate context destroyed by navigation

[dvschuyl]

Crawling webpage URLs (https://www.metanous.be/ in the example) sometimes raise the following error when using the default crawling strategy:

[INIT].... → Crawl4AI 0.3.744
[ERROR]... × https://www.metanous.be/... | Error: 
┌───────────────────────────────────────────────────────────────────────────────┐
│ × async_crawler_strategy.py:_crawleb(): Page.evaluate: Execution context was  │
│ destroyed, most likely because of a navigation                                │
└───────────────────────────────────────────────────────────────────────────────┘

I have tracked down the error and found that it is raised when executing the following line of code in the async_crawler_strategy._crawl_web-function:

crawl4ai/crawl4ai/async_crawler_strategy.py

Line 923 in b0419ed

await page.evaluate(update_image_dimensions_js)

By inserting a await page.wait_for_load_state() just before, I no longer get any errors.

[INIT].... → Crawl4AI 0.3.744
[FETCH]... ↓ https://www.metanous.be/... | Status: True | Time: 1.39s
[SCRAPE].. ◆ Processed https://www.metanous.be/... | Time: 33ms
[COMPLETE] ● https://www.metanous.be/... | Status: True | Total: 1.43s

---

FYI: The (simplified) code I used to crawl a website:

async def main():
    async with AsyncWebCrawler(verbose=True) as crawler:
        # Crawling the given URL
        result = await crawler.arun(
            url=...,
            extraction_strategy=None,
            cache_mode=CacheMode.BYPASS,
            magic=True,
        )

        # Printing the result
        print(len(result.markdown))
 
asyncio.run(main())

[unclecode]

Your pull request has been merged, thh