different results with MacOS and Amazon Linux

[AndriiOmelianenko]

What versions are you running?

MacOS

$ go list -m github.com/chromedp/chromedp
github.com/chromedp/chromedp v0.6.5
$ chrome --version
Google Chrome 88.0.4324.192
$ go version
go version go1.15.3 darwin/amd64

Amazon Linux

$ go list -m github.com/chromedp/chromedp
github.com/chromedp/chromedp v0.6.5
$ google-chrome --version
Google Chrome 88.0.4324.182
$ go version
go version go1.15.8 linux/amd64

What did you do? Include clear steps.

run this code

package main

import (
	"context"
	"log"
	"time"

	"github.com/chromedp/chromedp"
)

func main() {
	log.Println("app is running")
	log.SetFlags(log.LstdFlags | log.Llongfile)
	ctx, cancel := chromedp.NewExecAllocator(context.Background(), chromedp.NoSandbox, chromedp.Headless)
	defer cancel()
	ctx, cancel = chromedp.NewContext(ctx, chromedp.WithDebugf(log.Printf))
	defer cancel()

	// create a timeout
	ctx, cancel = context.WithTimeout(ctx, 5*time.Second)
	defer cancel()

	u := `https://www.whatismybrowser.com/detect/what-is-my-user-agent`
	selector := `#detected_value`
	log.Println("requesting", u)
	log.Println("selector", selector)
	var result string
	err := chromedp.Run(ctx,
		chromedp.Navigate(u),
		chromedp.WaitReady(selector),
		chromedp.OuterHTML(selector, &result),
	)
	if err != nil {
		log.Fatal(err)
	}
	log.Printf("result:\n%s", result)
}

What did you expect to see?

same results on darwin and linux

What did you see instead?

MacOS

works well
https://gist.github.com/AndriiOmelianenko/d4dcacf0902eb1e222c95535f2c5c8b1#file-macos-log

Linux

fails
https://gist.github.com/AndriiOmelianenko/d4dcacf0902eb1e222c95535f2c5c8b1#file-linux-log

[ZekeLu]

The element #detected_value is generated dynamically.

It works as expected on MacOS because the element has been generated (line 186):

2021/02/23 10:23:09 /Users/user/go/pkg/mod/github.com/chromedp/chromedp@v0.6.5/conn.go:103: <- {"method":"DOM.setChildNodes","params":{"parentId":26,"nodes":[{"nodeId":44,"parentId":26,"backendNodeId":17,"nodeType":1,"nodeName":"DIV","localName":"div","nodeValue":"","childNodeCount":1,"attributes":["class","value","id","detected_value"]}]},"sessionId":"906FEFC58D8A67ED7288DC19861FBEA1"}

And on Linux, it doesn't (line 50, the response status code is 403, which means that this IP is blocked).

2021/02/23 08:28:17 /home/ec2-user/go/pkg/mod/github.com/chromedp/chromedp@v0.6.6/conn.go:103: <- {"method":"Network.responseReceived","params":{"requestId":"F39F52358CE95FC1E2F076D02A15287E","loaderId":"F39F52358CE95FC1E2F076D02A15287E","timestamp":1317.749537,"type":"Document","response":{"url":"https://www.whatismybrowser.com/detect/what-is-my-user-agent","status":403,"statusText":"","headers":{"server":"nginx","date":"Tue, 23 Feb 2021 08:28:17 GMT","content-type":"text/html; charset=utf-8","vary":"Accept-Language","content-language":"en","x-content-type-options":"nosniff\nnosniff","referrer-policy":"same-origin\nunsafe-url","x-frame-options":"DENY","accept-ch":"ua,ua-full-version,ua-platform,ua-platform-version,arch,architecture,model,mobile,viewport-width,dpr,device-memory,rtt,downlink,ect","accept-ch-lifetime":"60","strict-transport-security":"max-age=31536000; includeSubDomains;","feature-policy":"microphone 'none'; geolocation 'none'","x-xss-protection":"1; mode=block","content-encoding":"gzip"},"mimeType":"text/html","requestHeaders":{":method":"GET",":authority":"www.whatismybrowser.com",":scheme":"https",":path":"/detect/what-is-my-user-agent","upgrade-insecure-requests":"1","user-agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/88.0.4324.182 Safari/537.36","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9","sec-fetch-site":"none","sec-fetch-mode":"navigate","sec-fetch-user":"?1","sec-fetch-dest":"document","accept-encoding":"gzip, deflate, br","accept-language":"en-US"},"connectionReused":false,"connectionId":13,"remoteIPAddress":"3.234.51.167","remotePort":443,"fromDiskCache":false,"fromServiceWorker":false,"fromPrefetchCache":false,"encodedDataLength":464,"timing":{"requestTime":1317.31249,"proxyStart":-1,"proxyEnd":-1,"dnsStart":12.587,"dnsEnd":13.752,"connectStart":13.752,"connectEnd":302.252,"sslStart":80.601,"sslEnd":302.217,"workerStart":-1,"workerReady":-1,"workerFetchStart":-1,"workerRespondWithSettled":-1,"sendStart":302.485,"sendEnd":302.7,"pushStart":0,"pushEnd":0,"receiveHeadersEnd":434.631},"responseTime":1.614068897817586e+12,"protocol":"h2","securityState":"secure","securityDetails":{"protocol":"TLS 1.2","keyExchange":"ECDHE_RSA","keyExchangeGroup":"X25519","cipher":"AES_256_GCM","certificateId":0,"subjectName":"*.whatismybrowser.com","sanList":["*.whatismybrowser.com","whatismybrowser.com"],"issuer":"Sectigo RSA Domain Validation Secure Server CA","validFrom":1573430400,"validTo":1644364799,"signedCertificateTimestampList":[],"certificateTransparencyCompliance":"unknown"}},"frameId":"1565D797426647DFDE00B1BA2131EE43"},"sessionId":"1A5616844638C1143B54A4A33E818455"}

You can request the URL on your Linux server directly, and you will see the 403 response:

curl -i 'https://www.whatismybrowser.com/detect/what-is-my-user-agent'
HTTP/2 403
server: nginx
...

[AndriiOmelianenko]

@ZekeLu thanks, haven't noticed the 403 error