" Not Secure " while browsing with selenium-wire #120

hobocow · 2020-06-07T21:14:59Z

On some websites, requests cannot be made through the browser because the connection is not considered as a secure connection.

I tried installing the certificate on my computer (mac) and tried to start selenium wire with a local chrome profile too, still not working.

Same on firefox.

Any workaround ?

wkeeling · 2020-06-08T07:53:31Z

Thanks for raising this. What does the browser say on websites that it can't connect to - do you get a specific error message?

hobocow · 2020-06-08T13:29:40Z

Hi,

It's not so much about the browser itself, i can browse any websites as I would on a regular browser / selenium. But for example if I try to login on some websites, the request wouldn't get through.

An example would be this :
https://www.nike.com/login

Working fine on regular browser/seleniuim but in selenium wire it's just an unknown error systematically.

I think its because in selenium wire there is always the " not secure " message at the top left corner of the browser ? In regular selenium i dont have that 👍

wkeeling · 2020-06-08T15:05:52Z

Ok thanks. The "not secure" indicator in the browser address bar indicates the browser doesn't trust Selenium Wire's root certificate, but this wouldn't normally stop you from accessing or logging into a site.

I'll see if I can reproduce with nike.com.

hobocow · 2020-06-08T15:15:39Z

Thank you, that is what i thought too.

In the mean time is there any way to make the browser trust selenium wire's certificate ?

wkeeling · 2020-06-09T10:57:35Z

Have you manually installed the ca.crt into Chrome? If you do that, Chrome should then trust the certificate and the "not secure" indicator will be replaced with a padlock icon.

To get the ca.crt you can run:

python -m seleniumwire extractcert

Or alternatively download it from here: https://raw.githubusercontent.com/wkeeling/selenium-wire/master/seleniumwire/proxy/ca.crt

hobocow · 2020-06-09T19:05:29Z

Yes I have tried installing it on both mac and windows, however, im not sure what you mean by install it into chrome ?

I only installed the certificate in both system : in the key chain access app on mac os and just double clicking on windows, that should do the job ? I couldnt find anyway to install a certificate directly in chrome

edit : i managed to install it directly in my chrome in windows and managed to login ! thank you so much ! however i cant see the post requests it makes with

for request in driver.requests: if request.response: print( request.path, request.response.status_code, request.response.headers['Content-Type'] )

edit 2 : it couldnt get all the requests because ome requests dont have content-type and it made the script stop, problem solved and everything is working perfectly now, thank you so much for your time and your work !!!!! 😄

wkeeling · 2020-06-10T07:27:10Z

Great, glad to hear it's working!

P-Leo10 · 2020-07-21T07:23:20Z

@hobocow @wkeeling Hi bros, I have the same problem to access https://www.nike.com.br. Could you tell me how to install ca.crt for windows? I do not even know where to begin with. I´m using Chrome and the new version of Selenium Wire (v2.00)

edit1: I got it thanks to this answer #97 (comment)

wkeeling · 2020-07-21T07:59:26Z

@P-Leo10 The root certificate is here: https://raw.githubusercontent.com/wkeeling/selenium-wire/master/seleniumwire/proxy/ca.crt

Copy all the text (including the BEGIN and END lines) and save it in a new file called ca.crt. Then go to Chrome's settings, then to "Certificates" (you can search the settings for "Certificates"). From there locate the "Certificate Authorities" section and use the "Import" function to import the ca.crt file you just created.

P-Leo10 · 2020-07-21T08:06:10Z

@wkeeling thank you very much for the help, it worked. The only problems that have not yet come out are:
-this error, client_socket_impl.cc(959)] handshake failed; returned -1, SSL error code 1, net_error -101
-this error in the browser, runned with selenium-wire ERR_PROXY_CONNECTION_FAILED after the script execution

wkeeling · 2020-07-21T08:17:42Z

@P-Leo10 could you post the code that you're using to create the webdriver and the full error traceback you're seeing?

P-Leo10 · 2020-07-21T08:31:34Z

@wkeeling
I'm working with a larger and more complex code but I did a test with this simple code and even in this script it informs me this error in the terminal

code:

import requests
from seleniumwire import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.chrome.options import Options

driver = webdriver.Chrome('./chromedriver')
driver.get('https://www.nike.com.br/')

error:

[86924:97464:0721/051946.790:ERROR:ssl_client_socket_impl.cc(959)] handshake failed; returned -1, SSL error code 1, net_error -101
[86924:97464:0721/051946.851:ERROR:ssl_client_socket_impl.cc(959)] handshake failed; returned -1, SSL error code 1, net_error -101
[86924:97464:0721/051946.855:ERROR:ssl_client_socket_impl.cc(959)] handshake failed; returned -1, SSL error code 1, net_error -101
[86924:97464:0721/051946.856:ERROR:ssl_client_socket_impl.cc(959)] handshake failed; returned -1, SSL error code 1, net_error -101
[86924:97464:0721/051946.899:ERROR:ssl_client_socket_impl.cc(959)] handshake failed; returned -1, SSL error code 1, net_error -101
[86924:97464:0721/051946.901:ERROR:ssl_client_socket_impl.cc(959)] handshake failed; returned -1, SSL error code 1, net_error -101
[86924:97464:0721/051946.917:ERROR:ssl_client_socket_impl.cc(959)] handshake failed; returned -1, SSL error code 1, net_error -101
[86924:97464:0721/051946.961:ERROR:ssl_client_socket_impl.cc(959)] handshake failed; returned -1, SSL error code 1, net_error -101
[86924:97464:0721/051946.963:ERROR:ssl_client_socket_impl.cc(959)] handshake failed; returned -1, SSL error code 1, net_error -101
[86924:97464:0721/051946.977:ERROR:ssl_client_socket_impl.cc(959)] handshake failed; returned -1, SSL error code 1, net_error -101
[86924:97464:0721/051947.023:ERROR:ssl_client_socket_impl.cc(959)] handshake failed; returned -1, SSL error code 1, net_error -101

moreover, thanks to your Selenium-wire, I am trying to create a way to overcome recaptcha v3, if you can give me a means to communicate with you in a non-public way, it would be great.

wkeeling · 2020-07-21T09:16:13Z

It might be worth switching the backend to see if that makes a difference. Try that first, but happy to continue the conversation non-publicly if no joy.

P-Leo10 · 2020-07-21T19:05:09Z

@wkeeling i try with

options  = {
     'backend' : 'mitmproxy'
}

but when i used this, the problem continues and comes back the problem with 'Not secure'.
I expressed myself wrong, I wanted to speak to you in private so as not to publish my idea publicly, I'm using the google translator so I don't know if I'm writing 100% right hahaha

wkeeling · 2020-07-22T07:54:00Z

With switching the backend to mitmproxy, you will need to ensure that you install the mitmproxy root certificate (it is different to the ca.crt that you already installed). Once installed, that should make the "Not secure" message go away. You can find instructions on installing the mitmproxy certificate here: https://docs.mitmproxy.org/stable/concepts-certificates/#installing-the-mitmproxy-ca-certificate-manually

shanebenlolo · 2021-07-11T18:44:10Z

@P-Leo10 The root certificate is here: https://raw.githubusercontent.com/wkeeling/selenium-wire/master/seleniumwire/proxy/ca.crt

Copy all the text (including the BEGIN and END lines) and save it in a new file called ca.crt. Then go to Chrome's settings, then to "Certificates" (you can search the settings for "Certificates"). From there locate the "Certificate Authorities" section and use the "Import" function to import the ca.crt file you just created.

Hey wkeeling, I tried to follow this link, as I am having the same problem as the original poster and am trying to work the issue out by adding the root certificate to chrome, but the link now appears to be dead. Do you have a working link to this file anymore? Thanks so much

wkeeling · 2021-07-11T18:51:16Z

@shanebenlolo Thanks for the message. The certificate has moved since that comment containing the link was posted. You should be able to download it here: https://github.com/wkeeling/selenium-wire/raw/master/seleniumwire/ca.crt (also accessible from the Certificates section in the doc).

You can also get the certificate by running the following command:

python -m seleniumwire extractcert

KarateKugler · 2022-02-15T11:28:07Z

It's important to import the certificate in the "Trusted Certificates" Tab.

Esaanjum · 2022-06-16T15:49:20Z

On some websites, requests cannot be made through the browser because the connection is not considered as a secure connection.

I tried installing the certificate on my computer (mac) and tried to start selenium wire with a local chrome profile too, still not working.

Same on firefox.

Any workaround ?

This is the only solution I know of:
use selenium webdriver
instead of seleniumwire webdriver

wkeeling · 2022-06-16T15:58:39Z

If you're using a Linux based system the certificate can also be installed from the terminal using the following command:

certutil -d sql:$HOME/.pki/nssdb -A -t TC -n "Selenium Wire" -i /path/to/ca.crt

Change /path/to/ca.crt appropriately.

If you get a "command not found" then certutil can be installed with:

apt install libnss3-tools

reeemsalah · 2022-09-07T10:20:39Z

It's important to import the certificate in the "Trusted Certificates" Tab.

@KarateKugler I can't find that tab inside google chrome settings, I can only find "Trusted People" and "Trusted Publishers". I have tried importing the certificate to both tabs but still, when I try to open any website using selenium-wire, it fails to find a certificate for it.

wkeeling · 2022-09-07T11:12:09Z

@reeemsalah there should be a tab that looks like the following:

If you don't have that, it's possible it may have been disabled by your organisation - e.g. by a company-wide security policy.

reeemsalah · 2022-09-07T11:56:11Z

@wkeeling Thank you so much, it worked 💯

yeyuchen198 · 2022-09-19T16:01:06Z

seleniumwire和undetected_chromedriver，证书不安全警告问题！

解决办法，新开一个CMD窗口，输入命令：
python -m seleniumwire extractcert
就会在同级文件夹生成一个ca.crt证书文件，然后在Chrome浏览器 > 设置里面搜索证书，管理证书，在【受信任的根证书颁发机构】那一栏，导入ca.crt这个证书！这样就没有证书不安全的警告了！！

cheezypotatoes · 2023-05-16T03:05:00Z

@wkeeling

should I continue?

cheezypotatoes · 2023-05-16T05:47:18Z

nvm, I realize you own this package

omid-adibfar · 2023-06-20T09:58:55Z

@P-Leo10 The root certificate is here: https://raw.githubusercontent.com/wkeeling/selenium-wire/master/seleniumwire/proxy/ca.crt

Copy all the text (including the BEGIN and END lines) and save it in a new file called ca.crt. Then go to Chrome's settings, then to "Certificates" (you can search the settings for "Certificates"). From there locate the "Certificate Authorities" section and use the "Import" function to import the ca.crt file you just created.

As an update, the link is deprecated, you can download ca.crt here: https://raw.githubusercontent.com/wkeeling/selenium-wire/master/seleniumwire/ca.crt

somT-oss · 2023-07-06T15:34:26Z

@wkeeling I tried your solution for Linux-based systems, but it did not work.

Is there any workaround?

A4Abhiraj · 2023-07-11T18:14:00Z

Try this:
This should extract the certificate in the dir you run this command from.
python -m seleniumwire extractcert

Open your chrome, go to settings > privacy & security > security > manage device certificates > authorities > click import
select the ca.crt that the previous command had generated.

Act0r1 · 2023-11-13T01:37:09Z

@P-Leo10 The root certificate is here: https://raw.githubusercontent.com/wkeeling/selenium-wire/master/seleniumwire/proxy/ca.crt

Copy all the text (including the BEGIN and END lines) and save it in a new file called ca.crt. Then go to Chrome's settings, then to "Certificates" (you can search the settings for "Certificates"). From there locate the "Certificate Authorities" section and use the "Import" function to import the ca.crt file you just created.

when i try to add it, i get error: The Private Key for this Client Certificate is missing or invalid

ravigadila · 2023-11-15T13:26:21Z

@Act0r1 same issue
Got "The Private Key for this Client Certificate is missing or invalid"
when trying to import ca.crt

Arrowar · 2023-11-28T14:37:10Z

Doesnt work -> Client certificate is missing or invalid

unchaineddev · 2023-11-28T15:24:07Z

Are you adding it in the “Trusted Root Certification Authority" section? @Ghost6446 Try this

Act0r1 · 2023-11-28T15:32:15Z

@Act0r1 same issue

Got "The Private Key for this Client Certificate is missing or invalid"

when trying to import ca.crt

First, generate the certificate - python -m seleniumwire extractcert
then install it - certutil -d sql:$HOME/.pki/nssdb -A -t TC -n "Selenium Wire" -i /path/to/ca.crt

Maybe it will help someone
And I highly recommend just use playwrig, there no error like this

milahu · 2023-12-04T10:33:05Z

summary:

the ca.crt file must be imported as an *authority* certificate:
chrome > settings > security > manage device certificates > authorities > import
the certificate is stored in $HOME/.pki/nssdb/cert9.db and $HOME/.pki/nssdb/key4.db.
it is not stored in chromium's user-data-dir
adding the certificate can be automated with certutil from nss-tools:
mkdir -p $HOME/.pki/nssdb; certutil -d sql:$HOME/.pki/nssdb -A -t TC -n "Selenium Wire" -i /path/to/ca.crt

i made nssdb-py to replace certutil -A
to add a ca.crt certificate to $HOME/.pki/nssdb/cert9.db

problem is: cloudflare will block the https proxy with self-signed certificate
see also #656

premrakh-vision · 2023-12-20T09:43:12Z

the ca.crt file must be imported as an authority certificate:
chrome > settings > security > manage device certificates > authorities > import

when I follow above step to handle Not Secure protocol its work but these setting do manually
I want to fix this issue via coding if possible so please answer it

milahu · 2023-12-20T12:15:42Z

adding the certificate can be automated with certutil

wkeeling closed this as completed Jun 10, 2020

ElSnoMan mentioned this issue Nov 2, 2022

NO SSL, slow driver execution ElSnoMan/pyleniumio#294

Closed

A4Abhiraj mentioned this issue Jul 11, 2023

Not secure (unvalid certificate) #696

Closed

" Not Secure " while browsing with selenium-wire #120

" Not Secure " while browsing with selenium-wire #120

Comments

hobocow commented Jun 7, 2020

wkeeling commented Jun 8, 2020

hobocow commented Jun 8, 2020

wkeeling commented Jun 8, 2020

hobocow commented Jun 8, 2020

wkeeling commented Jun 9, 2020 • edited Loading

hobocow commented Jun 9, 2020 • edited Loading

wkeeling commented Jun 10, 2020

P-Leo10 commented Jul 21, 2020 • edited Loading

wkeeling commented Jul 21, 2020

P-Leo10 commented Jul 21, 2020

wkeeling commented Jul 21, 2020

P-Leo10 commented Jul 21, 2020 • edited Loading

wkeeling commented Jul 21, 2020

P-Leo10 commented Jul 21, 2020 • edited Loading

wkeeling commented Jul 22, 2020 • edited Loading

shanebenlolo commented Jul 11, 2021

wkeeling commented Jul 11, 2021

KarateKugler commented Feb 15, 2022

Esaanjum commented Jun 16, 2022

wkeeling commented Jun 16, 2022

reeemsalah commented Sep 7, 2022

wkeeling commented Sep 7, 2022 • edited Loading

reeemsalah commented Sep 7, 2022

yeyuchen198 commented Sep 19, 2022

cheezypotatoes commented May 16, 2023

cheezypotatoes commented May 16, 2023

omid-adibfar commented Jun 20, 2023

somT-oss commented Jul 6, 2023

A4Abhiraj commented Jul 11, 2023

Act0r1 commented Nov 13, 2023

ravigadila commented Nov 15, 2023

Arrowar commented Nov 28, 2023

unchaineddev commented Nov 28, 2023

Act0r1 commented Nov 28, 2023

milahu commented Dec 4, 2023 • edited Loading

premrakh-vision commented Dec 20, 2023

the ca.crt file must be imported as an authority certificate: chrome > settings > security > manage device certificates > authorities > import

milahu commented Dec 20, 2023

wkeeling commented Jun 9, 2020 •

edited

Loading

hobocow commented Jun 9, 2020 •

edited

Loading

P-Leo10 commented Jul 21, 2020 •

edited

Loading

P-Leo10 commented Jul 21, 2020 •

edited

Loading

P-Leo10 commented Jul 21, 2020 •

edited

Loading

wkeeling commented Jul 22, 2020 •

edited

Loading

wkeeling commented Sep 7, 2022 •

edited

Loading

milahu commented Dec 4, 2023 •

edited

Loading

the ca.crt file must be imported as an authority certificate:
chrome > settings > security > manage device certificates > authorities > import