Dom class change? (no srp class)

[migliori]

Hello,

Thanks for this wonderful work - really -

I'm using your Search Engine at https://www.hack-hunt.com/find-illegal-shared-links.html, just finished the setup and it works fine now, but I had to edit vendor/serps/search-engine-google/src/Page/GoogleSerp.php.

In javascriptIsEvaluated() function:
I get the body content, but the body class is empty so the function throws an InvalidDOMException.
I forced it to return true, I know this is not the best way but I couldn't find a better solution for now.

Here's the body content I get with var_dump($body);

Thanks for your answer

google-body.txt

[gsouf]

Hi @migliori

I don't have any issue at this moment. Can you give me more details on how you are querying google? Are you using CURL?

[migliori]

Thanks for the quick answer. Yes I'm using Curl, and proxies from https://www.myprivateproxy.net
I tried with different standard user agents. I wrote a custom class named 'scraper', and basically call different parameters on setup:

$browser_language = "en-US,en;q=0.8";
$browser = new Browser(new CurlClient(), $userAgent, $browser_language);
$proxy = Proxy::createFromString('https://[...]');
$browser->setProxy($proxy);
$google_client = new GoogleClient($browser);
$google_url = new GoogleUrl();
$google_url->setParam('q', $scraper->q, true);
$google_url->setParam('pws', 0); // disabled personalized results
$google_url->setParam('filter', 0); // disabled ommited results
$google_url->setPage($scraper->page_number);
$google_url->setResultsPerPage($scraper->results_per_page);
$response = $google_client->query($google_url);
$results = $response->getNaturalResults(); // here comes the troubles

I understand it can be difficult for you to help if you can't reproduce the issue. If you want I can give you an ftp access.

I'm available on chat at https://www.phpformbuilder.pro when I'm online.

Thanks again,

Gilles

[gsouf]

Did you try without the proxy?

[migliori]

Yes, the issue comes from the proxy. Without proxy it works like a charm.
I'm still waiting for proxy support answert, that's why I didn't answer you before.
I'll tell you if they help & give a useful answer.
Thanks again for your great work & support.

[migliori]

I've got the answer of the proxies provider, and they finally couldn't help.
If you want to make inverstigations you can contact me and send me your IP, I'll authorize it and you'll be able to test with my proxies.