Thank you very much for the author's sharing. I would like to ask if it supports remote connection drivers.

[mytokendemo]

Thank you very much for the author's sharing. I would like to ask if it supports remote connection drivers.

[auror44]

For now you can patch the chromedriver that is used by remote with patch_exe function in https://github.com/ultrafunkamsterdam/undetected-chromedriver/blob/master/undetected_chromedriver/patcher.py.

[mytokendemo]

@auror44 thank you very much

[mytokendemo]

@auror44 Hello, is my error related to the version python: 3.8 undetected_chromedriver: 3.0.3
import undetected_chromedriver as uc
uc.patcher(
executable_path='http://127.0.0.1:4444/wd/hub',
force = False,
)
uc.TARGET_VERSION = 100
driver = uc.Chrome()
test = driver.get('https://www.google.com.hk/')
print(driver.title)
driver.close()

[Wamy-Dev]

For now you can patch the chromedriver that is used by remote with patch_exe function in https://github.com/ultrafunkamsterdam/undetected-chromedriver/blob/master/undetected_chromedriver/patcher.py.

how do you go about doing this?

[Anticope12]

Hi @Wamy-Dev , I'm wondering if it's possible to guide us through this process.
This is relevant to the following threads.
#394
#462

[Wamy-Dev]

@Anticope12 Thanks for bringing this topic back to my mind. A good fix would to add the "Privacy Pass" chrome extension, Here. Just add this extension to your chrome options:

from selenium import webdriver
uc = webdriver
chrome_options = webdriver.ChromeOptions()
chrome_options.add_extension('privacypass.crx')

You will have to download the extension as a crx (just google it, its easy) and then add it to your project. You will then have to manually go to each of the cloudflare protected pages and completely the human verification manually. Its a drag, but it works. This is the only solution I have found so far.

[Anticope12]

Thank you @Wamy-Dev , I've similar options already in place. just want to reduce the RAM usage, that's why I'm avoiding the extensions solution, It's kinda pitty to not be able to run UC (via remote Command_executor)

[Wamy-Dev]

Ah, I understand, the ram usage isn't too much increased, usually less than 100MB. But I understand not wanting to use extensions, I on the other hand use 3 total extensions and the increase for all 3 is less than 200MB total.

[Anticope12]

Thanks for the input here! Just tested it out on Gmail login. and unfortunately, it's not solving the sign-in problem.

[mytokendemo]

@Wamy-Dev @Anticope12 Hello, have you found a solution, can you share it with me, I haven't solved the problem above, I hope you can help me

[Wamy-Dev]

Thanks for the input here! Just tested it out on Gmail login. and unfortunately, it's not solving the sign-in problem.

Thats a little different problem, check here for a solution for that.

[Wamy-Dev]

@Wamy-Dev @Anticope12 Hello, have you found a solution, can you share it with me, I haven't solved the problem above, I hope you can help me

What exactly are you trying to do? Access cloudflare protected pages or log in to services or something else? Please check my above comment for how to bypass cloudflare pages.

[mytokendemo]

@Wamy-Dev Thanks for your reply, #594 (comment) Have you found a solution to this problem? Remote support reported an error

[Wamy-Dev]

@Wamy-Dev Thanks for your reply, #594 (comment) Have you found a solution to this problem? Remote support reported an error

What was the error

[mytokendemo]

[Wamy-Dev]

May I see your code please, just anything that deals with the driver.

[mytokendemo]

@Wamy-Dev 我的错误与python版本有关吗：3.8 undetected_chromedriver: 3.0.3
import undetected_chromedriver as uc
uc.patcher(
executable_path=' http://127.0.0.1:4444/wd/hub ',
force = False,
)
uc。 TARGET_VERSION = 100
driver = uc.Chrome()
test = driver.get(' https://www.google.com.hk/ ')
print(driver.title)
driver.close()

[Wamy-Dev]

@Wamy-Dev 我的错误与python版本有关吗：3.8 undetected_chromedriver: 3.0.3

import undetected_chromedriver as uc

uc.patcher(

executable_path=' http://127.0.0.1:4444/wd/hub ',

force = False,

)

uc。 TARGET_VERSION = 100

driver = uc.Chrome()

test = driver.get(' https://www.google.com.hk/ ')

print(driver.title)

driver.close()

just use the normal webdriver from selenium. Right now it is impossible to use undetected chromedriver remotely. @auror44 said to patch it but I have no idea what that means.

[mytokendemo]

@Wamy-Dev thank you

[Anticope12]

Hi @mytokendemo , @auror44 ,
I'm doing some search and I've found this https://github.com/wkeeling/selenium-wire a Selenium-wire that is supported with a Remote driver. Could you please test it out and let me know if it works?

Chrome and Firefox

For Chrome and Firefox you don't need to do anything special. Just instantiate the webdriver as you would normally with webdriver.Chrome() or webdriver.Firefox() passing in any desired capabilities and browser specific options for Chrome or Firefox , such as the executable path, headless mode etc. Selenium Wire also has it's own options that can be passed in the seleniumwire_options attribute.

Remote

Selenium Wire has limited support for using the remote webdriver client. When you create an instance of the remote webdriver, you need to specify the hostname or IP address of the machine (or container) running Selenium Wire. This allows the remote instance to communicate back to Selenium Wire with its requests and responses.

options = {
'addr': 'hostname_or_ip' # Address of the machine running Selenium Wire. Explicitly use 127.0.0.1 rather than localhost if remote session is running locally.
}
driver = webdriver.Remote(
command_executor='http://www.example.com/',
seleniumwire_options=options
)
If the machine running the browser needs to use a different address to talk to the machine running Selenium Wire you need to configure the browser manually. wkeeling/selenium-wire#220 goes into more detail.

[Wamy-Dev]

Hi @mytokendemo , @auror44 ,

I'm doing some search and I've found this https://github.com/wkeeling/selenium-wire a Selenium-wire that is supported with a Remote driver. Could you please test it out and let me know if it works?

Chrome and Firefox

For Chrome and Firefox you don't need to do anything special. Just instantiate the webdriver as you would normally with webdriver.Chrome() or webdriver.Firefox() passing in any desired capabilities and browser specific options for Chrome or Firefox , such as the executable path, headless mode etc. Selenium Wire also has it's own options that can be passed in the seleniumwire_options attribute.

Remote

Selenium Wire has limited support for using the remote webdriver client. When you create an instance of the remote webdriver, you need to specify the hostname or IP address of the machine (or container) running Selenium Wire. This allows the remote instance to communicate back to Selenium Wire with its requests and responses.

options = {

'addr': 'hostname_or_ip' # Address of the machine running Selenium Wire. Explicitly use 127.0.0.1 rather than localhost if remote session is running locally.

}

driver = webdriver.Remote(

command_executor='http://www.example.com/',

seleniumwire_options=options

)

If the machine running the browser needs to use a different address to talk to the machine running Selenium Wire you need to configure the browser manually. wkeeling/selenium-wire#220 goes into more detail.

Wow this does look really promising. I will take a look later tonight, thank you for finding this

[Anticope12]

Hi @mytokendemo , @auror44 ,
I'm doing some search and I've found this https://github.com/wkeeling/selenium-wire a Selenium-wire that is supported with a Remote driver. Could you please test it out and let me know if it works?
Chrome and Firefox
For Chrome and Firefox you don't need to do anything special. Just instantiate the webdriver as you would normally with webdriver.Chrome() or webdriver.Firefox() passing in any desired capabilities and browser specific options for Chrome or Firefox , such as the executable path, headless mode etc. Selenium Wire also has it's own options that can be passed in the seleniumwire_options attribute.
Remote
Selenium Wire has limited support for using the remote webdriver client. When you create an instance of the remote webdriver, you need to specify the hostname or IP address of the machine (or container) running Selenium Wire. This allows the remote instance to communicate back to Selenium Wire with its requests and responses.
options = {
'addr': 'hostname_or_ip' # Address of the machine running Selenium Wire. Explicitly use 127.0.0.1 rather than localhost if remote session is running locally.
}
driver = webdriver.Remote(
command_executor='http://www.example.com/',
seleniumwire_options=options
)
If the machine running the browser needs to use a different address to talk to the machine running Selenium Wire you need to configure the browser manually. wkeeling/selenium-wire#220 goes into more detail.

Wow this does look really promising. I will take a look later tonight, thank you for finding this

No worries! please let me know if it's any good! I'm traveling and I was bored and I've found it.

Holding my fingers crossed :)

[Anticope12]

Hi @mytokendemo , Any good news? :)

[gonnsuarez]

For now you can patch the chromedriver that is used by remote with patch_exe function in https://github.com/ultrafunkamsterdam/undetected-chromedriver/blob/master/undetected_chromedriver/patcher.py.

Hi @auror44, thanks for this idea,

I was trying it, downloading a fresh version of the chrome driver and in a separate python file running only the following lines of code from the patcher.py file:

def gen_random_cdc():
    cdc = random.choices(string.ascii_lowercase, k=26)
    cdc[-6:-4] = map(str.upper, cdc[-6:-4])
    cdc[2] = cdc[0]
    cdc[3] = "_"
    return "".join(cdc).encode()

linect = 0
replacement = gen_random_cdc()
with io.open("/pathto/chromedriver.exe", "r+b") as fh:
    for line in iter(lambda: fh.readline(), b""):
        if b"cdc_" in line:
            fh.seek(-len(line), 1)
            newline = re.sub(b"cdc_.{22}", replacement, line)
            fh.write(newline)
            linect += 1

It applies the patch without errors, those line changes excecutes perfectly, but then, when I try to run a normal selenium instance using this modified chromedriver instead of the original, it's not seems to take any effect, it opens the browser saying "Chrome is being controlled by automated test sofware" and is not able to pass the "nowsecure.nl" check

The idea is that if I can confirm that it works, apply this patch into the chromedriver file of the docker container, and then run a normal remote selenium session, the only difference will be the patched chromedriver inside the container. But maybe I'm not getting the whole picture here and missing some big things.

I wil appreciate some guidence on this!

[Anticope12]

Hi @gonnsuarez , Have you checked the New Docker ChromeDriver that was posted by UltraFunkamsterdam?

#634

I think it can help with the issue, but I don't have the great tech skills to look into it.
Let me know what do you think?
https://hub.docker.com/r/ultrafunk/undetected-chromedriver

[Wamy-Dev]

Hi @gonnsuarez , Have you checked the New Docker ChromeDriver that was posted by UltraFunkamsterdam?

#634

I think it can help with the issue, but I don't have the great tech skills to look into it.

Let me know what do you think?

https://hub.docker.com/r/ultrafunk/undetected-chromedriver

This is actually a pretty good solution, although its not exactly remote.

[Anticope12]

Hi @gonnsuarez , Have you checked the New Docker ChromeDriver that was posted by UltraFunkamsterdam?
#634
I think it can help with the issue, but I don't have the great tech skills to look into it.
Let me know what do you think?
https://hub.docker.com/r/ultrafunk/undetected-chromedriver

This is actually a pretty good solution, although its not exactly remote.

@Wamy-Dev , do you think we could patch this to make it work ?

I'm trying even to find a freelancer to have this sorted it out, because having virtual machine to run my code, I'll get 10x memory Consumption.

[Wamy-Dev]

Hi @gonnsuarez , Have you checked the New Docker ChromeDriver that was posted by UltraFunkamsterdam?
#634
I think it can help with the issue, but I don't have the great tech skills to look into it.
Let me know what do you think?
https://hub.docker.com/r/ultrafunk/undetected-chromedriver

This is actually a pretty good solution, although its not exactly remote.

@Wamy-Dev , do you think we could patch this to make it work ?

I'm trying even to find a freelancer to have this sorted it out, because having virtual machine to run my code, I'll get 10x memory Consumption.

I honestly have no idea, I have yet to try the previous idea of using selenium wire, which supposedly can support undetected chrome driver remotely, and this docker image could be a pretty good idea too. For the first idea it is just a matter of switching out a couple lines of code. For the second idea, it would mean just adding a couple lines of code to the script, and then making a new docker image based off of this experimental undetected chrome driver container. I am out of town until tomorrow so I have not had time to try either. If you can get a freelancer to figure it out that would work too, although it may be a bit pricey.

[Wamy-Dev]

For the new docker image a docker file would look like,

FROM ultrafunk/undetected-chromedriver
COPY ./requirements.txt /tmp/
RUN pip install --requirement /tmp/requirements.txt
COPY  .  ${PWD}/data

and the command to run would be

docker run -it -v ${PWD}/data:/data yourusername/yourimagename ipython yourscriptname.py

note: you would need to change the imports in your script as well

import undetected_chromedriver as uc
options = uc.ChromeOptions()
options.arguments.extend(["--no-sandbox", "--disable-setuid-sandbox"])
driver = uc.Chrome(options)

[Anticope12]

Hi @Wamy-Dev ,
Thank you for sharing the above, I'll test it out.

I'm using Selnoid as the main approach due to the Great UI and easy structure, I'm not sure if I can work,

Happy to hear your thoughts about it :)
https://aerokube.com/selenoid/

[gonnsuarez]

I made this simple sketchy test, maybe it can help in deciding what could be the best approach to the issue:

after having a running selenium hub with four chrome nodes, I just entered into one of the docker nodes with:

docker exec -it selenium-chrome-2 /bin/bash

installed undetected chrome driver inside there and run the example:

sudo apt-get update
sudo apt-get install pip
pip install undetected_chromedriver
python3
import undetected_chromedriver as uc
driver = uc.Chrome()
driver.get('https://nowsecure.nl')

and it works!

the problem with this is that the selenium hub not recognize an active session inside that node, and of course you cannot control that chrome session remotely with webdriver.Remote().
In order to take that screenshot, I had to run in a separate python file that webdriver.Remote() to get an active session inside that node, and then open the vnc console when the selenium hub recognized that was an active session.

But maybe around here there is a correct way of doing it.

[mytokendemo]

你好@mytokendemo， 有没有好消息？:)

@Anticope12 The test results are invalid for me, what are your results?

[Wamy-Dev]

@gonnsuarez @mytokendemo, I wrote a small script using nano, this is the result. It navigates successfully to the page, but I never actually get to see the result as the sessions tab on the remote driver is not there. It does open the page though... I can conclude it as a success. I guess the next step is to actually build a docker container around it all

[Anticope12]

Just catching up on the thread here!
Glad to see it was double to some extent, I'm wondering if anyone managed to have a successful docker container built around it. @Wamy-Dev , @auror44 :)

[Wamy-Dev]

@Anticope12, thanks for the reminder, ill actually try this right now, it may take me a few hours. Expect to hear back some results later tonight.

[Wamy-Dev]

Alright, I have some results. I am going to mention everybody for this. This will include code, and results that I have found after running quite a few tests on my system. @gonnsuarez, @auror44, @Anticope12, @mytokendemo. This is regarding @ultrafunkamsterdam's docker container. I have yet to test "selenium wire" for this purpose.

For the code side of things

Integrating was pretty easy. I just used code from the docker repo and copy and pasted it. For now I just commented out all of my other chrome options. This is all regarding my production application Rezi.one so I cannot share code snippets but the code works and switching over is pretty easy. So in step form for the code. 1. Switch from whatever selenium driver you are using in your imports and move to "undected_chromedriver" instead at the top of your code. 2. Switch the chrome options by adding that line from the docker repo: options.arguments.extend(["--no-sandbox", "--disable-setuid-sandbox"]) for me I just had my chrome options set as chrome_options = uc.ChromeOptions() so I just renamed the previous line to chrome_options.arguments.extend(["--no-sandbox", "--disable-setuid-sandbox"]). Then just use your code as normal as it was a local chromedriver.

For the docker side of things

Just a note, if you wanted to see the vnc of what the container is doing like the remote... you can't (unless there is a way that I don't know about). It seems like this container by @ultrafunkamsterdam is just a local undetected chromedriver smooshed into a container, and irrc the local selenium doesnt include vnc of any kind. Maybe @ultrafunkamsterdam it could be an update in the future? Idk how you would do it but it would be nice. Any ways, regarding creating the docker container for your own projects. I have created a dockerfile (it is named "dockerfile", no extensions) that works. It does rely on local files and building it on your local machine so if you work out of github (like I do sometimes), then the dockerfile will have to be modified accordingly. Here it is, its pretty simple:

FROM ultrafunk/undetected-chromedriver
COPY . .
RUN pip install --requirement requirements.txt

If you are not familiar with docker, this is all this file does. 1. The "FROM" line gets the original image that you want to build off of, in our case its the undected-chromedriver image. 2. The "COPY" line copies all of the files that are in the same directory as the dockerfile (plan for this) into the container 3. Then the "RUN" line just installs all of our requirements that we copied over. This file will default to the latest release of the container so you can just keep this dockerfilelike this and when @ultrafunkamsterdam updates the docker container you will also get the latest update.

My findings

This is just gonna be the TLDR of the post, but I do recommend reading all of my post. The first thing to note before I say anything is that I am out of town as I am writing this on a dual core macbook air, building and running this docker container, so some of these things may be due to that, but will have to be tested at a later time. The first thing is that this driver crashes a lot. For the first 3 times I kept on getting WebDriverException: Message: unknown error: session deleted because of page crash. Again there are a lot of things that could go into this, including this crappy laptop, my unoptimized code, etc... It might not just be because of the container. I will test it on my 16core desktop in the future (lol). I could just build it, and push it to my homeserver registry to test it, but Im on vacation and I really don't want to do that haha. So to anyone reading this: do tests before production. I think that this is a great project, and that this is a good solution for this problem, and the one I will most likely use for my own project. There is the option of selenium-wire which is highly documented and already seems to have a lot of support, even though I am not sure how well it works remotely (which is why this issue exists). Thank you everyone, I can consider this closed for myself, but please also comment your findings in this test as well. I have tried to make it as easy as possible to integrate with your own projects as some of you may not know docker or even python very well.

TLDR: The container works, implementing is pretty easy, it just needs more development from @ultrafunkamsterdam and a lot of testing by the users who want to use it.

[Anticope12]

Thank you for sharing @Wamy-Dev , I'll look into it :)

[Originn]

@Wamy-Dev Thanks a lot for this thread, it was super helpful. I have set up the container and running a flask app that do some scraping with the following options

chrome_options = uc.ChromeOptions()
chrome_options.arguments.extend(["--no-sandbox", "--disable-setuid-sandbox"]) driver = uc.Chrome(options=chrome_options)

I setup that the flask app will run on port 80 and runned the container on -p 80:80 and I can access the flask app on 127.0.0.1 but I have an error. log:

I have multiple messages:

Fatal server error:
(EE) Server is already active for display 1
If this server is no longer running, remove /tmp/.X1-lock
and start again.

Any idea what's going on?

[Wamy-Dev]

There are too many problems with the docker image. I just reverted back to what I had originally before this entire thread. Once @ultrafunkamsterdam updates the image. I will switch back to it.

[Originn]

@Wamy-Dev I am struggling to find a solution to my problem - running headless chrome in a flask app which is being detected by website protection. If I can overcome this detection then all is going to run well. Do you have a suggestion? What do you use currently to avoid being detected?

P.S I figured out that the issue was in the a function within entrypoint.sh as it dosen't detect the pid of Xvfb although if running pgrep Xvfb it does find it. I now have issue that the chrome browser is not opening and I get "cannot connect to chrome at 127.0.0.1:47465". I have upgraded chrome to v104, but I am assuming that it can't run on 127.0.0.1 as its already taken by the flask app, can anyone point me how to fix this? can I run chrome on different address?

[Schtil]

Greetings!

Can you tell me if there are any updates on running undected_chromedriver inside selenium node chrome (in the context of selenium grid)?

Tried to do it by trivial overwriting chromedriver in the "node chrome" docker container - didn't start

From @gonnsuarez's reply, I understood that I managed to start the session directly via python script inside the node chrome container, but it is not starting via hub.

[Hudrolax]

It looks like it's just not possible to run this inside docker

[ttraxxrepo]

Has anyone made any progress on this?

is it possible to run a patched UC chromedriver as a standalone that selenium hub connects to?

perhaps in here https://github.com/SeleniumHQ/docker-selenium/blob/trunk/NodeChrome/Dockerfile we build a node that runs the patched undetected_chromdriver (override the one it downloads) and then just webdriver.Remote() into it with the normal selenium library?