Selenium 4 beta 3 high memory usage

[marsh0lion]

🐛 Bug Report

When compared to running Selenium v.3.141.59 in hub and node setup, Selenium 4 beta consumes considerably more memory.

Usually when running Selenium 3.141.59 I would have 1 hub and 2 nodes - one with usually 5 threads running through xvfb and another 1 with 1 thread for confirming scenario's. The hub would rarely exceed 500mb memory usage, and the node running 5 threads rarely exceeded 1.5 gb after running for a few days.

After running overnight (~16 hours), Selenium 4 beta hub is consuming 4gb and the node is using 3.4gb (capability of 8 threads, but only 1 running) (note I also had a Firefox tab pointing to the server page (<server_url>:/ui/index.html#/)

To Reproduce

Detailed steps to reproduce the behavior:
Run a large suite overnight.
Moodle (moodle.org) is a large open source project that has a large suite that could be used (although this is not what I tested with, and may need some modifications to get to work with Selenium 4)

Expected behavior

RAM Usage for hub and node should be under 2 gb

Test script or set of commands reproducing this issue

Run a large suite

Environment

OS: Ubuntu 20.04 LTS
Browser: chrome
Browser version: 91.0.4472.77 (Official Build) (64-bit)
Browser Driver version: ChromeDriver 90
Language Bindings version: PHP
Selenium Grid version (if applicable): 4 beta 3

Java Version:
openjdk 11.0.11 2021-04-20
OpenJDK Runtime Environment (build 11.0.11+9-Ubuntu-0ubuntu2.20.04)
OpenJDK 64-Bit Server VM (build 11.0.11+9-Ubuntu-0ubuntu2.20.04, mixed mode, sharing)

[diemol]

This depends on the number of processors available when the jars are started. We use internally AsyncHttpClient, which implcitely uses a thread pool to handle connections. The default size of this pool is # of available processors * 2. As the pool grows up to its limit, more memory is being used up to the point where the pool is full, where memory usage gets stable, which is simply how the client works. Therefore, memory usage is proportional to the pool size.

We have also done some stress test in the new Grid, running up to 60 tests in parallel for about one hour. The Hub had 16GB of RAM and 4CPUs, and the memory did not go up over 2.5GB since there were 4 CPUs.

A more complete and detailed analysis can be seen in #9152, so please have a look and we'll leave this issue open for a while in case you have questions.

[marsh0lion]

Thanks @diemol, that makes sense for how the node uses a large amount of RAM (although in my mind, it's still a bit excessive - possibly an issue with AsyncHttpClient), but in the case I mentioned above the hub takes even more RAM than the node. While I'm not familiar with the architecture of the hub (I'm also not a java developer), it doesn't seem to make sense to tie the thread pool to the number of processors (as I would not expect the hub to be CPU bound).

When I look at the stress tests that have been done, they don't appear to have been running for a long period of time. One of our use cases involves running behat with over 160k steps single threaded on a PHP application talking to a MSSQL database (we do the same runs on MySQL and Postgres as well) which takes over 48 hours (yes there are other improvements we could be doing).

When I've got a bit more time, I'll have a look to see if the Java memory flags (eg. -Xmx) help. The new drain functionality looks promising to help us here as well. Given what you've mentioned, I'll also figure out how I can constrain the thread pool to see how much difference that makes (I've just started checking out Selenium 4, and I like what I'm seeing).

I'm testing on an i7 (4 physical cores running as 8 virtual) with 32Gb of RAM, but it is my development machine so RAM is well utilised.

[marsh0lion]

I did another full run of our suite over the weekend (again a single thread for 160,000 behat steps). This time starting the hub and node with the following commands:

• java -jar selenium-server-4.0.0-beta-3.jar hub --allow-cors true -p 4000 --max-threads 3 which is now at ~4Gb RAM
• java -jar selenium-server-4.0.0-beta-3.jar node --max-sessions 2 --max-threads 2 which is now at ~5.5Gb RAM

As a comparison, I also had a selenium 3 hub and node setup (max of 5 threads, and was maxing out most of the time the v4 beta was running) which is currently using a combined 2.5GB RAM. There were also a few more failures (which are probably our fault) this time as well. This tells me that I haven't triggered #9152 (As I would expect RAM usage to have dropped by the --max-threads and --max-sessions arguments)

[elie-delorme]

I'm also experiencing abnormal memory usage on the dockerized version of selenium node (docker image: selenium/node-firefox:4.0.0-rc-1-prerelease-20210618).
I'm running simple tests, they are all executed on a single firefox node machine limited to one session (concurrency=1) and the amount of threads used by selenium keeps increasing (and so does the memory usage).
Thread dumps seem to suggest a lot of Threads named "Driver Service Executor" are being created and they seem to be sticking around for a while.

Here is a thread dump taken after 9 days uptime, this is one of 7 pods we are running, they only run a couple tests every 5 minute:
td.txt

There are 6197 threads in the "Driver Service Executor" thread group, and I suspect this is only going to increase.
Here is a report for that thread dump: https://fastthread.io/my-thread-report.jsp?p=c2hhcmVkLzIwMjEvMDYvMjMvLS10ZF82ay50eHQtLTE5LTM2LTM=&

We've rolled-back to docker tag version 4.0.0-beta-3-20210426 for the time being

[pujagani]

I'm also experiencing abnormal memory usage on the dockerized version of selenium node (docker image: selenium/node-firefox:4.0.0-rc-1-prerelease-20210618).
I'm running simple tests, they are all executed on a single firefox node machine limited to one session (concurrency=1) and the amount of threads used by selenium keeps increasing (and so does the memory usage).
Thread dumps seem to suggest a lot of Threads named "Driver Service Executor" are being created and they seem to be sticking around for a while.

Here is a thread dump taken after 9 days uptime, this is one of 7 pods we are running, they only run a couple tests every 5 minute:
td.txt

There are 6197 threads in the "Driver Service Executor" thread group, and I suspect this is only going to increase.
Here is a report for that thread dump: https://fastthread.io/my-thread-report.jsp?p=c2hhcmVkLzIwMjEvMDYvMjMvLS10ZF82ay50eHQtLTE5LTM2LTM=&

We've rolled-back to docker tag version 4.0.0-beta-3-20210426 for the time being

Thank you for providing the details. The growing thread count for "Driver Service Executor" was a leak. It has been fixed as part of #9677.

[diemol]

Thanks for adding these details, @pujagani.

We can close this for now as it will be part of the next release.

[marsh0lion]

HI @diemol,
Unfortunately this doesn't appear fixed (in beta-4). My node is currently showing 6.6 GB RAM usage whereas my hub is using 4.9 GB (after running a suite with 160,000 behat steps over the weekend).

I wouldn't be surprised if there are multiple bugs leading to this situation.

Kind regards,
Brian Barnes

[pujagani]

HI @diemol,
Unfortunately this doesn't appear fixed (in beta-4). My node is currently showing 6.6 GB RAM usage whereas my hub is using 4.9 GB (after running a suite with 160,000 behat steps over the weekend).

I wouldn't be surprised if there are multiple bugs leading to this situation.

Kind regards,
Brian Barnes

Thank you for following up. The changes are made after Beta-4 hence will the fix will be available as part of the next release.
Meanwhile, Diego has helped in providing prerelease https://github.com/SeleniumHQ/docker-selenium/releases/download/4.0.0-rc-1-prerelease-20210804/selenium-server-4.0.0-prerelease-rc-1-c498dad8c5.jar. I would recommend please try it out since it has the changes that might fix the leak and provide the feedback in case the memory usage is still high.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.