chrome feature not working

[mrcnski]

Potentially related: https://github.com/orgs/spider-rs/discussions/142#discussioncomment-7317218

Problem

I noticed that when I enable the chrome feature my crawler no longer returns pages. I tried running spider's chrome example, but the output is inconsistent. Sometimes it manages to crawl a page, but usually it does not:

$ cargo run --example chrome --features chrome
[...]
Time elapsed in website.crawl(https://example.com) is: 313.451042ms for total pages: 0
Time elapsed in website.crawl(https://choosealicense.com) is: 323.886042ms for total pages: 0
"https://jeffmendez.com"
Time elapsed in website.crawl(https://jeffmendez.com) is: 674.120625ms for total pages: 1

$ cargo run --example chrome --features chrome
[...]
"https://choosealicense.com"
Time elapsed in website.crawl(https://jeffmendez.com) is: 1.1972115s for total pages: 0
Time elapsed in website.crawl(https://example.com) is: 1.19657375s for total pages: 0
"https://choosealicense.com/licenses/mit/"
"https://choosealicense.com/no-permission/"
[...]
"https://choosealicense.com/licenses/ms-pl/"
Time elapsed in website.crawl(https://choosealicense.com) is: 16.551638916s for total pages: 76

Solutions Tried

I installed Chrome and confirmed that the GUI works at least.

chromedriver:

$ chromedriver -v
ChromeDriver 132.0.6834.159 (2d77d3fc4452661469b78f115e0aed4d71269739-refs/branch-heads/6834@{#4408})

I started headless chrome as per https://github.com/orgs/spider-rs/discussions/142#discussioncomment-7317218 and set .with_chrome_connection(Some("http://127.0.0.1:9222/json/version".into())). This made things significantly better for one run, but after that I was getting issues again.

I tried both crawl and crawl_smart (the latter of which never finished, seeming to hang indefinitely). I set the features chrome, chrome_intercept, and smart.

[j-mendez]

Hi, I would use a proxy when running multiple times using chrome. jeffmendez.com you will get blocked shortly etc. Also I would use chrome-headless-shell to start the chrome instance.