Occasional EADDRINUSE by chromedriver causes crash

[JosephGarrone]

Meta -

OS: Windows 10
Selenium Version: 3.0.0-beta-2 (NodeJS)
Browser: Chrome
Browser Version: 53.0.2785.116 (Official Build) m (32-bit)

Expected Behavior -

Attempt to resolve connection if possible, or at least allow users to provide their own error handler.

Actual Behavior -

Unsure if this is a chromedriver issue, selenium webdriver issue or an operator issue. Occasionally during a portion of my tests where I conduct a fair bit of waiting in the form:

// Wait up to 10 seconds for the control to appear
driver.wait(Until.elementLocated(By.xpath('/html/body/div[4]/div/div[2]/div/div[2]/div[1]')), 10000).then(function(element) {
    expect(element).to.not.be.null;

    // Wait up to 10 seconds for the control to disappear
    driver.wait(Until.elementIsNotVisible(element), 10000).then(function(element) {
        done();
    });
});

I receive an error from https://github.com/SeleniumHQ/selenium/blob/master/javascript/node/selenium-webdriver/http/index.js#L238.

Without having delved into the actual implementation, I assume that this code, under the hood, will open many repeat connections to the chromedriver in order to poll for element state. After a while, this eventually causes an EADDRINUSE error to be thrown by the above linked error handler, causing the webdriver to stop. This issue does not happen when using firefox. Error in question:

Uncaught Error: EADDRINUSE connect EADDRINUSE 127.0.0.1:57941                                                                                                                            
      at ClientRequest.<anonymous> (d:\SVN\<redacted>\Testing\trunk\GUITesting\Selenium\node_modules\selenium-webdriver\http\index.js:238:15)                                           
      at Socket.socketErrorListener (_http_client.js:308:9)                                                                                                                                   
      at emitErrorNT (net.js:1272:8)                                                                                                                                                          
      at _combinedTickCallback (internal/process/next_tick.js:74:11)                                                                                                                          
      at process._tickCallback (internal/process/next_tick.js:98:9)                                                                                                                           
  From: Task: WebElement.getTagName()                                                                                                                                                         
      at Driver.schedule (d:\SVN\<redacted>\Testing\trunk\GUITesting\Selenium\node_modules\selenium-webdriver\lib\webdriver.js:414:17)                                                  
      at WebElement.schedule_ (d:\SVN\<redacted>\Testing\trunk\GUITesting\Selenium\node_modules\selenium-webdriver\lib\webdriver.js:1820:25)                                            
      at WebElement.getTagName (d:\SVN\<redacted>\Testing\trunk\GUITesting\Selenium\node_modules\selenium-webdriver\lib\webdriver.js:2012:17)                                           
      at d:\SVN\<redacted>\Testing\trunk\GUITesting\Selenium\node_modules\selenium-webdriver\lib\until.js:282:20                                                                        
      at d:\SVN\<redacted>\Testing\trunk\GUITesting\Selenium\node_modules\selenium-webdriver\lib\webdriver.js:753:14                                                                    
      at TaskQueue.execute_ (d:\SVN\<redacted>\Testing\trunk\GUITesting\Selenium\node_modules\selenium-webdriver\lib\promise.js:2736:14)                                                
      at TaskQueue.executeNext_ (d:\SVN\<redacted>\Testing\trunk\GUITesting\Selenium\node_modules\selenium-webdriver\lib\promise.js:2719:21)                                            
      at asyncRun (d:\SVN\<redacted>\Testing\trunk\GUITesting\Selenium\node_modules\selenium-webdriver\lib\promise.js:2642:25)                                                          
      at d:\SVN\<redacted>\Testing\trunk\GUITesting\Selenium\node_modules\selenium-webdriver\lib\promise.js:639:7                                                                       
  From: Task: <anonymous>                                                                                                                                                                     
      at Timeout.pollCondition [as _onTimeout] (d:\SVN\<redacted>\Testing\trunk\GUITesting\Selenium\node_modules\selenium-webdriver\lib\promise.js:2178:16)                             
  From: Task: Waiting element to become stale                                                                                                                                                 
      at ControlFlow.wait (d:\SVN\<redacted>\Testing\trunk\GUITesting\Selenium\node_modules\selenium-webdriver\lib\promise.js:2171:17)                                                  
      at Driver.wait (d:\SVN\<redacted>\Testing\trunk\GUITesting\Selenium\node_modules\selenium-webdriver\lib\webdriver.js:749:29)                                                      
      at d:\SVN\<redacted>\Testing\trunk\GUITesting\Selenium\spec\setup\products\productDetails.spec.js:28:28                                                                           
      at ManagedPromise.invokeCallback_ (d:\SVN\<redacted>\Testing\trunk\GUITesting\Selenium\node_modules\selenium-webdriver\lib\promise.js:1315:14)                                    
      at TaskQueue.execute_ (d:\SVN\<redacted>\Testing\trunk\GUITesting\Selenium\node_modules\selenium-webdriver\lib\promise.js:2736:14)                                                
      at TaskQueue.executeNext_ (d:\SVN\<redacted>\Testing\trunk\GUITesting\Selenium\node_modules\selenium-webdriver\lib\promise.js:2719:21)                                            
      at asyncRun (d:\SVN\<redacted>\Testing\trunk\GUITesting\Selenium\node_modules\selenium-webdriver\lib\promise.js:2595:27)                                                          
      at d:\SVN\<redacted>\Testing\trunk\GUITesting\Selenium\node_modules\selenium-webdriver\lib\promise.js:639:7                                           

I fixed this issue locally by replacing above error handler with:

request.on('error', function(e) {
  if (e.code === 'ECONNRESET' || e.code === 'EADDRINUSE') {
    setTimeout(function() {
      sendRequest(options, onOk, onError, opt_data, opt_proxy);
    }, e.code === 'EADDRINUSE' ? 50 : 15);
  } else {
    var message = e.message;
    if (e.code) {
      message = e.code + ' ' + message;
    }
    onError(new Error(message));
  }
});

This works fine. If you add a console log above for each retry it retries maybe 5-10 times before it starts working again. This makes me think that the issue is with clogging up the communications (netstat -a shows the ports hitting 65535 with requests to the what I can only assume is the chromedriver).

I have tried running chromedriver in verbose mode with logging turned on, there are no events being displayed from within chromedriver to indicate that it is failing.

If this is an issue with chromedriver instead of webdriver please close this and I will see what they have to say on the issue.

Steps to reproduce -

Can't easily provide as the point at which this occurs is variable, however it always happens inside tests with the above wait-based code.

[jleyba]

Underlying problem is the chromedriver does not use keep-alive, so the wait loop is exhausting your ephemeral ports with short-lived connections while it polls the server. Unfortunately, I haven't been able to trigger this on my machine, even with an indefinite wait.

You could try tweaking the number of maximum connections on http.globalAgent (I need to expose the ability to use a custom HTTP agent with the chromedriver)

[mijay]

I've got the same problem while trying to run multiple chromedrivers in parallels (and hence using lot's of sockets), tweaking neither maxSockets nor maxFreeSockets of both global or custom agent doesn't solve it. While the aforementioned hack does.

[cgoldberg]

It's an OS limit you are hitting. Windows doesn't come out of the box tuned for high throughput. You need to adjust the TCP settings of your OS to increase pool size for ephemeral ports.

[alohaninja]

@cgoldberg - thanks for the ephemeral ports suggestion! Posted a powershell script on serverfault that will adjust these windows settings without having to manually dig thru the registry or deal with netsh commands.

We were also seeing the EADDRINUSE error when calling Webdriver.quit()

[szikszail]

Hello, Guys!
So can this issue be fixed in selenium, in chromedriver? Or we must tweak OS settings on every machinve we're running tests to avoid this issue?
If it's really in issue in how selenium-webdriver handles requests (as @jleyba mentioned, w/o keep-alive), then I would be glad if it would be fixed there.

[szikszail]

Any info, Guys? We are facing this issue more&more regularly with Chrome.

[p0deje]

If it's really in issue in how selenium-webdriver handles requests (as @jleyba mentioned, w/o keep-alive), then I would be glad if it would be fixed there.

selenium-webdriver can issue keep-alive connections, but ChromeDriver doesn't support them, so there is nothing we can fix here.

[szikszail]

@p0deje thank you for making it clear!

[seanpoulter]

This is a huge, intermittent pain in the ass for my tests on a Windows 10 PC.

@p0deje , would it make sense to:

• add an option to sendRequest() to enable Joseph's workaround to retry on EADDRINUSE
• enable the option when using ChromeDriver (or other drivers) that don't support keep-alive (if that's possible)

[p0deje]

I think it makes sense to retry on EADDRINUSE all the time. At least I see that Ruby bindings do it.

However, I'm not very familiar with NodeJS bindings, so either Jason needs to implement it or pull request is welcome!

[radiantmediaplayer]

Ran into this issue on my local Win 10 dev machine - all tests would start erroring with EADDRINUSE after about 60 sec. After a bit a of digging I found the official MSDN doc with clear steps for tuning Win-based machine to workaround this issue. I am posting it here for reference.

[patrikmolsson]

Had this issue as well. We came up with a solution, however please keep in mind it's a jest-specific solution, and not a general solution for this issue.

Our issue arose when we ran our jest testing and multiple workers spun up different instances of the chromedriver. By decreasing the maximum allowed number of workers (parallel instances), using the --maxWorkers option , we did not encounter the issue anymore.

jest --maxWorkers=4