Unable to send any headers to site using https connection.

[DChistyakov]

chistyak...@gmail.com commented:

Hello,
first of all let me please thank you for such a greate job! It`s really very useful and cool!

Have just downloaded and compilled current master e960279.

I`ve tried the same script with https and http hosts usind proxy. Here is the example of headers using https (dump from proxy server):

array(4) {
["_requst"]=>
string(35) "CONNECT www.google.com:443 HTTP/1.1"
["proxy-connection"]=>
string(10) "keep-alive"
["user-agent"]=>
string(11) "Mozilla/5.0"
["host"]=>
string(14) "www.google.com"
}

And here the same script headers but using http (dump from proxy server):

array(10) {
["requst"]=>
string(60) "GET http://www.google.com/accounts/ServiceLoginAuth HTTP/1.1"
["user-agent"]=>
string(117) "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_7; en-us) AppleWebKit/533.4 (KHTML, like Gecko) Version/4.1 Safari/533.4"
["proxy-authorization"]=>
string(86) "Basic ............................"
["............"]=>
string(12) "............"
["expect"]=>
string(0) ""
["accept"]=>
string(90) "application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,/_;q=0.5"
["connection"]=>
string(10) "Keep-Alive"
["accept-encoding"]=>
string(4) "gzip"
["accept-language"]=>
string(4) "en,*"
["host"]=>
string(14) "www.google.com"
}

Please make it works the same with http and https. Or at least add on opportunity to set a useragent or headers at all.
Thanks a lot for such great code!

FreeBSD serv233.sp 8.2-RELEASE FreeBSD 8.2-RELEASE #0: Thu Feb 17 02:41:51 UTC 2011 root@mason.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC amd64

QMake version 2.01a
Using Qt version 4.7.3 in /usr/local/lib/qt4

Disclaimer:
This issue was migrated on 2013-03-15 from the project's former issue tracker on Google Code, Issue #233.
🌟   3 people had starred this issue at the time of migration.

[ariya]

roejame...@gmail.com commented:

Or at least add on opportunity to set a useragent
You can set the useragent like so:

page.settings.userAgent = 'myagent'

Please see the API reference for more:
http://code.google.com/p/phantomjs/wiki/Interface#settings_%28object%29

[ariya]

ariya.hi...@gmail.com commented:

You may want to see the network traffic from PhantomJS itself. Refer to the examples/netlog.js.

[DChistyakov]

chistyak...@gmail.com commented:

No, im wrong!
Sent according netlog using http through proxy:

requested: {
"headers": [
{
"name": "User-Agent",
"value": "Mozilla/5.0 (Windows; U; Windows NT 5.0; ru-ru) AppleWebKit/533.16 (KHTML, like Gecko) Version/4.1 Safari/533.16\r\nProxy-Authorization: Basic Y2hpc3R5YWtvdi5kbWl0cnlAZ21haWwuY29tOmUxMGFkYzM5NDliYTU5YWJiZTU2ZTA1N2YyMGY4ODNl\r\nProxy-Interface: black\r\nExpect: "
},
{
"name": "Accept",
"value": "application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,/;q=0.5"
}
],
"id": 1,
"method": "GET",
"time": "2011-09-15T07:13:09.926Z",
"url": "http://www.google.com/accounts/ServiceLoginAuth"
}

recived on proxy:

string(532) "GET http://www.google.com/accounts/ServiceLoginAuth HTTP/1.1
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; ru-ru) AppleWebKit/533.16 (KHTML, like Gecko) Version/4.1 Safari/533.16
Proxy-Authorization: Basic Y2hpc3R5YWtvdi5kbWl0cnlAZ21haWwuY29tOmUxMGFkYzM5NDliYTU5YWJiZTU2ZTA1N2YyMGY4ODNl
Proxy-Interface: black
Expect:
Accept: application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,/;q=0.5
Connection: Keep-Alive
Accept-Encoding: gzip
Accept-Language: en,*
Host: www.google.com"

Sent according netlog using the same https through the same proxy:
requested: {
"headers": [
{
"name": "User-Agent",
"value": "Mozilla/13.0 (Macintosh; U; Intel Mac OS X 10_6_6; en-gb) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.4 Safari/533.20.27\r\nProxy-Authorization: Basic Y2hpc3R5YWtvdi5kbWl0cnlAZ21haWwuY29tOmUxMGFkYzM5NDliYTU5YWJiZTU2ZTA1N2YyMGY4ODNl\r\nProxy-Interface: black\r\nExpect: "
},
{
"name": "Accept",
"value": "application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,/;q=0.5"
}
],
"id": 1,
"method": "GET",
"time": "2011-09-15T07:12:22.284Z",
"url": "https://www.google.com/accounts/ServiceLoginAuth"
}
received: {
"contentType": null,
"headers": [],
"id": 0,
"redirectURL": null,
"stage": "end",
"status": null,
"statusText": null,
"text": "",
"time": "2011-09-15T07:12:22.288Z",
"url": "https://www.google.com/accounts/ServiceLoginAuth"
}

recived on proxy:

string(116) "CONNECT www.google.com:443 HTTP/1.1
Proxy-Connection: keep-alive
User-Agent: Mozilla/5.0
Host: www.google.com"

So phantomjs doesnt send any headers using https proxy but prints linke it sent. Bug(

[DChistyakov]

chistyak...@gmail.com commented:

I deleted my wrong comment #3.

[ariya]

ariya.hi...@gmail.com commented:

In the case of proxy log, using https:

User-Agent: Mozilla/5.0

Why is the proxy showing a wrong user agent?

[DChistyakov]

chistyak...@gmail.com commented:

I can send any user agent to proxy, it will show Mozilla/5.0. Have just sent Mozilla/13.0. Result is: Mozilla/5.0

[DChistyakov]

chistyak...@gmail.com commented:

useragent GodZilla)

load started
requested: {
"headers": [
{
"name": "User-Agent",
"value": "GodZilla_aaamamama!\r\nProxy-Authorization: Basic Y2hpc3R5YWtvdi5kbWl0cnlAZ21haWwuY29tOmUxMGFkYzM5NDliYTU5YWJiZTU2ZTA1N2YyMGY4ODNl\r\nProxy-Interface: 93.170.121.7\r\nExpect: "
},
{
"name": "Accept",
"value": "application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,/;q=0.5"
}
],
"id": 1,
"method": "GET",
"time": "2011-09-15T07:34:30.544Z",
"url": "https://www.google.com/accounts/ServiceLoginAuth"
}
received: {
"contentType": null,
"headers": [],
"id": 0,
"redirectURL": null,
"stage": "end",
"status": null,
"statusText": null,
"text": "",
"time": "2011-09-15T07:34:30.552Z",
"url": "https://www.google.com/accounts/ServiceLoginAuth"
}
load finished

server:

string(116) "CONNECT www.google.com:443 HTTP/1.1
Proxy-Connection: keep-alive
User-Agent: Mozilla/5.0
Host: www.google.com"

[DChistyakov]

chistyak...@gmail.com commented:

Gentlemen, problem is still actual! It`s unable to use ssl + proxy + additional headers!

[ariya]

ariya.hi...@gmail.com commented:

I am still confused. Does the proxy server see the actual user agent from PhantomJS or not?

[DChistyakov]

chistyak...@gmail.com commented:

The problem is:
I can set any UserAgent in script. And start it without or with http proxy and it will works fine. But if try to set any UserAgent and start script with https proxy - proxy servre will recive such:

string(116) "CONNECT www.google.com:443 HTTP/1.1
Proxy-Connection: keep-alive
User-Agent: Mozilla/5.0
Host: www.google.com"

And useragent is always Mozilla/5.0 or indefined.

[zackw]

On behalf of the PhantomJS team, I apologize for letting this bug linger unaddressed for years.

I think what's going on here is, PhantomJS will send custom headers to an origin server over proxied HTTPS just fine -- at least, I assume it will -- but the CONNECT command sent to the proxy doesn't get any of those headers. I do not know what is supposed to happen here; I'm unfamiliar with explicit HTTPS proxies. (It would be entirely plausible to me if the proxy was NOT supposed to receive any but the most generic of request headers under these conditions.)

To do anything about this, we need to figure out:

• What the relevant Web standards say about this.
• Whether the situation is still the same in PhantomJS 2.0.
• Whether it makes sense to be able to specify custom headers for the proxy independent of custom headers for the origin server, and if so, what the API for that should look like.

I think this is all stuff that we (the PhantomJS devs) can figure out for ourselves, but we may not get to it anytime soon; if you have time to investigate it even a little, that would be helpful.

[ariya]

Due to our very limited maintenance capacity (see #14541 for more details), we need to prioritize our development focus on other tasks. Therefore, this issue will be closed. In the future, if we see the need to attend to this issue again, then it will be reopened.
Thank you for your contribution!