Auth error: ETIMEDOUT #283

stephenplusplus · 2018-02-13T21:56:36Z

Copied from original issue: googleapis/nodejs-pubsub#30

@jamesholcomb
January 5, 2018 3:50 AM

Environment details

Note: App is running external from GCP

OS: Ubuntu 16.04.3 LTS
Node.js version: 8.9.1
npm version: 5.5.1
@google-cloud/pubsub version: 0.14.5

Steps to reproduce

UNKNOWN

Auth error:Error: connect ETIMEDOUT 172.217.4.109:443
Auth error:Error: connect ETIMEDOUT 172.217.4.109:443
Auth error:Error: connect ETIMEDOUT 172.217.4.109:443
Auth error:Error: connect ETIMEDOUT 172.217.4.109:443
Dec 31, 2017 7:05 AM ERROR  { message: 'Getting metadata from plugin failed with error: con
nect ETIMEDOUT 172.217.4.109:443',
  stack: 'Error: Getting metadata from plugin failed with error: connect ETIMEDOUT 172.217.4.109:44
3\n    at ClientDuplexStream.onConnectionStatus (/home/jholcomb/ridealert.server/node_modules/@goog
le-cloud/pubsub/src/connection-pool.js:270:21)\n    at emitOne (events.js:116:13)\n    at ClientDup
lexStream.emit (events.js:211:7)\n    at ClientDuplexStream._emitStatusIfDone (/home/jholcomb/ridea
lert.server/node_modules/grpc/src/node/src/client.js:260:10)\n    at ClientDuplexStream._receiveSta
tus (/home/jholcomb/ridealert.server/node_modules/grpc/src/node/src/client.js:233:8)\n    at /home/
jholcomb/ridealert.server/node_modules/grpc/src/node/src/client.js:757:12',
  code: 16 }

This error occured after the server app had been connected and processing messages for several days. Is it expected that the pubsub lib automatically reconnects (eventually) or does a reconnect in this situation have to be handled by the client?

The text was updated successfully, but these errors were encountered:

stephenplusplus · 2018-02-13T21:56:36Z

January 9, 2018 5:53 PM

Sorry for the delay @jamesholcomb. @callmehiphop do you know?

stephenplusplus · 2018-02-13T21:56:37Z

@jamesholcomb
January 18, 2018 2:49 AM

On a possibly related note, I started getting this error after updating to 0.16.2.

Sending messages seems to continue working. But I have to restart the app in order to receive messages on subscriptions.

Jan 17, 2018 7:31 PM ERROR { message: 'Retry total timeout exceeded before anyresponse was received', stack: 'Error: Retry total timeout exceeded before anyresponse was received at repeat (/Users/jamesholcomb/code/ridealert.server/node_modules/google-gax/lib/api_callable.js:224:18) at Timeout._onTimeout (/Users/jamesholcomb/code/ridealert.server/node_modules/google-gax/lib/api_callable.js:256:13) at ontimeout (timers.js:475:11) at tryOnTimeout (timers.js:310:5) at Timer.listOnTimeout (timers.js:270:5)' }

stephenplusplus · 2018-02-13T21:56:37Z

@jonparrott
February 9, 2018 7:05 PM

Related: googleapis/google-auth-library-python#211

This is due to Compute Engine's metadata server being occasionally unavailable. We should move this to the nodejs auth library and set a reasonable timeout and retry when talking to the GCE metadata server.

In the meantime, you can use a service account private key file as describe in cloud.google.com/docs/authentication/getting-started

stephenplusplus · 2018-02-13T21:56:38Z

@jonparrott
February 9, 2018 7:06 PM

@stephenplusplus can you handle moving this to the appropriate place?

stephenplusplus · 2018-02-13T21:56:38Z

February 9, 2018 7:22 PM

Hmm, I'm not sure which library of code is making the request, whether it's our handwritten layer, the gapic, or the auth library itself. @callmehiphop do you know?

stephenplusplus · 2018-02-13T21:56:39Z

@callmehiphop
February 9, 2018 9:03 PM

@stephenplusplus hard to say, when creating the gapic clients we attempt to get the project id from the auth library, after which all requests are made by the gapic. If this error occurs multiple times, I would think it was from the gapic since we cache the project id.

stephenplusplus · 2018-02-13T21:56:39Z

February 9, 2018 9:18 PM

@jamesholcomb do you know the minimum required code to track down the method that is making the request that returns that error? Not so much concerned with reproducing, but just knowing which method is making the call.

stephenplusplus · 2018-02-13T21:56:40Z

@jamesholcomb
February 10, 2018 3:30 PM

@stephenplusplus I've only encountered this error once. The server where it was running did not have full debug/trace enabled at the time, so I don't have much to go on.

stephenplusplus · 2018-02-13T21:56:40Z

@ctavan
February 13, 2018 10:05 AM

@stephenplusplus @callmehiphop I have encountered a similar error (see below) while running my branch from #65 (which is based on current master). I'll base my branch on the v0.16.2 tag and test again.

Here's the error, I'm getting ECONNRESET instead of ETIMEDOUT:

{ Error: Getting metadata from plugin failed with error: connect ECONNRESET 169.254.169.254:80
    at /app/node_modules/grpc/src/client.js:554:15 code: 16, metadata: Metadata { _internal_repr: {} } } },
  id: '06f12d14-5870-4aea-ad1c-f76ccf5d17fe' }

stephenplusplus · 2018-02-13T21:58:37Z

@ofrobots @JustinBeckwith -- I moved this here, as it seems like this error comes from a call to the GCP metadata server. Note that we are still using 0.12 in @google-cloud/pubsub, where this issue was opened originally.

Any ideas how to approach this one?

theacodes · 2018-02-13T21:59:07Z

I'm guessing this is probably just a case of unreliable network if it's not reliably reproducible.

theacodes · 2018-02-13T21:59:25Z

If we want to address this at all, see #283 (comment)

ctavan · 2018-02-13T22:04:57Z

Concerning my comment: I've encountered this actually during a faulty deploy of a service that reads from Pubsub and writes to BigQuery. The service was badly configured to consume much faster than it was able to write which simply overloaded the node process.

In my experience node processes that max out one CPU thread cannot be trusted anyways (as events pile up on the event loop and memory consumption grows and grows…). I believe in my case this was rather an artifact of a badly configured setup and not really a pointer towards a serious issue.

I'm sorry for the confusion I may have caused.

JustinBeckwith · 2018-02-13T22:34:31Z

@jonparrott @stephenplusplus @ofrobots The answer here is "It's complicated". In the current release, we generally do NOT retry when trying to refresh the token:
https://github.com/google/google-auth-library-nodejs/blob/v1.2.1/src/auth/computeclient.ts#L61

In this commit, we moved almost all of our interaction with the GCE metadata server into the gcp-metadata npm module. I say almost because the refreshToken method linked above provides the ability to override the base tokenUrl, which we decided not to support in gcp-metadata. Getting rid of this field (which I think we should do) is a breaking change, and tracked in #280.

Even if that call uses gcp-metadata, that module is currently configured to (by default) retry the request 3 times in the case of a non-200 response, but fail immediately in the case of an ETIMEDOUT or ENOTFOUND error:
https://github.com/stephenplusplus/gcp-metadata/blob/master/src/index.ts#L47

We do this to ensure that calls to the metadata service fail fast on non-GCP environments. Summing this up - there are a few separate tasks at hand:

Getting all of our gcp-metadata handling code in one place. This work is mostly done, there's just that last nagging spot :) We can either a.) break the API and do a semver major release, or b.) add the ability to override the tokenUrl to the gcp-metadata module.
Deciding across languages how to handle retries. We took the approach of failing fast on failed requests (with no response). How are we handling this in other languages? We have a fairly robust retry module now, so it's just a matter of working through the desired details.

Thoughts?

theacodes · 2018-02-13T22:39:21Z

We do this to ensure that calls to the metadata service fail fast on non-GCP environments.

You should do what Python does - for ADC use a quick check with no retries and timeouts to determine if the metadata server is present and then when using metadata server credentials allow retries.

Deciding across languages how to handle retries

At the API level, this is pretty clear - retries are configured with the gapic configuration.

jjhesk · 2019-05-06T11:15:37Z

got the same in here: Error while trying to retrieve access token { FetchError: request to https://oauth2.googleapis.com/token failed, reason: connect ETIMEDOUT 216.58.200.10:443

more details:


Error while trying to retrieve access token { FetchError: request to https://oauth2.googleapis.com/token failed, reason: connect ETIMEDOUT 216.58.200.10:443
    at ClientRequest.<anonymous> (/Users/hesk/Documents/localize-spreadsheet-bot/node_modules/node-fetch/lib/index.js:1453:11)
    at ClientRequest.emit (events.js:180:13)
    at TLSSocket.socketErrorListener (_http_client.js:395:9)
    at TLSSocket.emit (events.js:180:13)
    at emitErrorNT (internal/streams/destroy.js:64:8)
    at process._tickCallback (internal/process/next_tick.js:178:19)
  message: 'request to https://oauth2.googleapis.com/token failed, reason: connect ETIMEDOUT 216.58.200.10:443',
  type: 'system',
  errno: 'ETIMEDOUT',
  code: 'ETIMEDOUT',
  config:
   { method: 'POST',
     url: 'https://oauth2.googleapis.com/token',
     data: 'code=4%2FQgFCT-LEUxcnDljD1DMn9olKwYQVJ9bVxiaZJMmUgT7fPAyu5Gc14Ro&client_id=875537178561-5j56883h195m8e8lrggah3fes3gh253t.apps.googleusercontent.com&client_secret=6IkI8HtPvcmXU7XORCgKg7TR&redirect_uri=urn%3Aietf%3Awg%3Aoauth%3A2.0%3Aoob&grant_type=authorization_code&code_verifier=',
     headers:
      { 'Content-Type': 'application/x-www-form-urlencoded',
        'User-Agent': 'google-api-nodejs-client/3.1.2',
        Accept: 'application/json' },
     params: {},
     paramsSerializer: [Function: paramsSerializer],
     body: 'code=4%2FQgFCT-LEUxcnDljD1DMn9olKwYQVJ9bVxiaZJMmUgT7fPAyu5Gc14Ro&client_id=875537178561-5j56883h195m8e8lrggah3fes3gh253t.apps.googleusercontent.com&client_secret=6IkI8HtPvcmXU7XORCgKg7TR&redirect_uri=urn%3Aietf%3Awg%3Aoauth%3A2.0%3Aoob&grant_type=authorization_code&code_verifier=',
     validateStatus: [Function: validateStatus],
     responseType: 'json' } }

1c7 · 2019-12-09T05:49:28Z

May I ask, any update? same problem

FetchError: request to https://oauth2.googleapis.com/token failed, reason: connect ETIMEDOUT 216.58.194.170:443
    at ClientRequest.<anonymous> (/Users/remote_edit/Documents/1c7-vue-electron/node_modules/node-fetch/lib/index.js:1455:11)
    at ClientRequest.emit (events.js:200:13)
    at TLSSocket.socketErrorListener (_http_client.js:402:9)
    at TLSSocket.emit (events.js:200:13)
    at emitErrorNT (internal/streams/destroy.js:91:8)
    at emitErrorAndCloseNT (internal/streams/destroy.js:59:3)
    at processTicksAndRejections (internal/process/task_queues.js:84:9) {
  message: 'request to https://oauth2.googleapis.com/token ' +
    'failed, reason: connect ETIMEDOUT ' +
    '216.58.194.170:443',
  type: 'system',
  errno: 'ETIMEDOUT',
  code: 'ETIMEDOUT',
  config: {
    method: 'POST',
    url: 'https://oauth2.googleapis.com/token',
    data: 'code=4%2FuAFRFmJVrsftXjIyl-OI7yg53vdIRK-0MN1ZQ9lrKiJh_X1HTHovnbi9-geBoKk7OTlApzp-6uG8znaSYYJun8I&client_id=1058382469405-em0mtqch877g3etc1ddo8j4sk9uafia6.apps.googleusercontent.com&client_secret=&redirect_uri=http%3A%2F%2Flocalhost%3A3000&grant_type=authorization_code&code_verifier=',
    headers: {
      'Content-Type': 'application/x-www-form-urlencoded',
      'User-Agent': 'google-api-nodejs-client/5.6.1',
      'x-goog-api-client': 'gl-node/12.4.0 auth/5.6.1',
      Accept: 'application/json'
    },
    params: [Object: null prototype] {},
    paramsSerializer: [Function: paramsSerializer],
    body: 'code=4%2FuAFRFmJVrsftXjIyl-OI7yg53vdIRK-0MN1ZQ9lrKiJh_X1HTHovnbi9-geBoKk7OTlApzp-6uG8znaSYYJun8I&client_id=1058382469405-em0mtqch877g3etc1ddo8j4sk9uafia6.apps.googleusercontent.com&client_secret=&redirect_uri=http%3A%2F%2Flocalhost%3A3000&grant_type=authorization_code&code_verifier=',
    validateStatus: [Function: validateStatus],
    responseType: 'json'
  }
}

JustinBeckwith · 2019-12-09T14:23:06Z

Greetings! Could we trouble you to open a new issue?

1c7 · 2019-12-09T15:14:59Z

@JustinBeckwith Thank for reply!
I have solve the problem by:

process.env.HTTP_PROXY = "http://127.0.0.1:1087"
process.env.HTTPS_PROXY = "http://127.0.0.1:1087"

Should I still open a new issue anyway?

JustinBeckwith · 2019-12-09T16:06:26Z

If it works for you, no need!

pnowakaero · 2020-09-03T08:49:32Z

Hello. I have the same error like user 1c7 but I don't quite understand on which port should I set the proxy. Should be same port that my nodejs app is running?

KAEOnIt · 2021-03-17T08:53:35Z

Hi All, Same problem on my side and It works when disabling the Firewall. I've authorised ports 5228 to 5230 and tried again with no success. Does any one solved the issue? To me Oauth is HTTPS so should works tru 443.
Thx

oneWalker · 2022-09-09T06:05:15Z

@JustinBeckwith Thank for reply! I have solve the problem by:
process.env.HTTP_PROXY = "http://127.0.0.1:1087"
process.env.HTTPS_PROXY = "http://127.0.0.1:1087"
Should I still open a new issue anyway?

add one note:
I adjust the port with my personal prxoy port in .env and it runs successfully.
the project will not read it from pc's global setting but .env

stephenplusplus mentioned this issue Feb 13, 2018

Auth error: ETIMEDOUT googleapis/nodejs-pubsub#30

Closed

JustinBeckwith mentioned this issue Feb 14, 2018

fix: retry connections to gce metadata server #284

Merged

JustinBeckwith self-assigned this Feb 14, 2018

JustinBeckwith added the type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns. label Feb 14, 2018

JustinBeckwith closed this as completed in #284 Feb 16, 2018

stephenplusplus mentioned this issue Feb 20, 2018

Cloud Spanner throw Auth error:Error: read ECONNRESET intermittently googleapis/nodejs-spanner#118

Closed

tanishiking mentioned this issue Mar 7, 2020

Add more information on "Retry total timeout exceeded before any response was received" error. googleapis/gax-nodejs#741

Closed

yoshi-automation added 🚨 This issue needs some love. triage me I really want to be triaged. labels Apr 7, 2020

JustinBeckwith added the electron label Jun 25, 2020

anilvpatel21 mentioned this issue Aug 18, 2020

timeout field isn't being considered while fetching the Bearer token #948

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Auth error: ETIMEDOUT #283

Auth error: ETIMEDOUT #283

stephenplusplus commented Feb 13, 2018

stephenplusplus commented Feb 13, 2018

stephenplusplus commented Feb 13, 2018

stephenplusplus commented Feb 13, 2018

stephenplusplus commented Feb 13, 2018

stephenplusplus commented Feb 13, 2018

stephenplusplus commented Feb 13, 2018

stephenplusplus commented Feb 13, 2018

stephenplusplus commented Feb 13, 2018

stephenplusplus commented Feb 13, 2018

stephenplusplus commented Feb 13, 2018

theacodes commented Feb 13, 2018

theacodes commented Feb 13, 2018

ctavan commented Feb 13, 2018 •

edited

Loading

JustinBeckwith commented Feb 13, 2018 •

edited

Loading

theacodes commented Feb 13, 2018

jjhesk commented May 6, 2019 •

edited

Loading

1c7 commented Dec 9, 2019

JustinBeckwith commented Dec 9, 2019

1c7 commented Dec 9, 2019 •

edited

Loading

JustinBeckwith commented Dec 9, 2019

pnowakaero commented Sep 3, 2020

KAEOnIt commented Mar 17, 2021

oneWalker commented Sep 9, 2022

Auth error: ETIMEDOUT #283

Auth error: ETIMEDOUT #283

Comments

stephenplusplus commented Feb 13, 2018

Copied from original issue: googleapis/nodejs-pubsub#30

Environment details

Steps to reproduce

stephenplusplus commented Feb 13, 2018

January 9, 2018 5:53 PM

stephenplusplus commented Feb 13, 2018

stephenplusplus commented Feb 13, 2018

stephenplusplus commented Feb 13, 2018

stephenplusplus commented Feb 13, 2018

February 9, 2018 7:22 PM

stephenplusplus commented Feb 13, 2018

stephenplusplus commented Feb 13, 2018

February 9, 2018 9:18 PM

stephenplusplus commented Feb 13, 2018

stephenplusplus commented Feb 13, 2018

stephenplusplus commented Feb 13, 2018

theacodes commented Feb 13, 2018

theacodes commented Feb 13, 2018

ctavan commented Feb 13, 2018 • edited Loading

JustinBeckwith commented Feb 13, 2018 • edited Loading

theacodes commented Feb 13, 2018

jjhesk commented May 6, 2019 • edited Loading

1c7 commented Dec 9, 2019

May I ask, any update? same problem

JustinBeckwith commented Dec 9, 2019

1c7 commented Dec 9, 2019 • edited Loading

JustinBeckwith commented Dec 9, 2019

pnowakaero commented Sep 3, 2020

KAEOnIt commented Mar 17, 2021

oneWalker commented Sep 9, 2022

ctavan commented Feb 13, 2018 •

edited

Loading

JustinBeckwith commented Feb 13, 2018 •

edited

Loading

jjhesk commented May 6, 2019 •

edited

Loading

1c7 commented Dec 9, 2019 •

edited

Loading