Cloud Spanner throw Auth error:Error: read ECONNRESET intermittently

[manasi99]

I have a cloud function deployed on GCP which is triggered by Cloud Pub/Sub topic. It connects to cloud spanner to insert the pub/sub event data. It is a high volume pub/sub topic and make simultaneous insert requests in different cloud spanner table. However, it keep giving below errors intermittently. Application Default Credentials are used to do authentication of Cloud Spanner.

ERROR: { Error: Getting metadata from plugin failed with error: read ECONNRESET at /user_code/node_modules/@google-cloud/spanner/node_modules/grpc/src/client.js:554:15 code: 16, metadata: Metadata { _internal_repr: {} }, note: 'Exception occurred in retry method that was not classified as transient' }

Below are the dependencies in package.json

 "dependencies": {
      "safe-buffer": "5.1.1",
      "@google-cloud/spanner": "1.0.3",
      "google-gax": "^0.14.5"
}

Environment details

• Node.js version: v8.5.0
• npm version: 5.3.0
• @google-cloud/spanner version: 1.0.3

[stephenplusplus]

It sounds like you might need to write some retry logic around the code that's causing problems. But to be sure there's not anything we can do, can you show the code that calls the method which is returning the error?

[vkedia]

But this does not look like an error that users should be seeing. We do not want users to be retrying operations. If the error is retryable, then the library should retry it.

[callmehiphop]

@manasi99 could you put together a minimal amount of code that reproduces this issue for us? That should help us resolve this issue for you quickly.

[manasi99]

@stephenplusplus @callmehiphop : This is the simple version of actual code where the error is caught in catch block:

const Spanner = require('@google-cloud/spanner');
const config = require('./config.js');

const spannerClient = new Spanner({
	projectId: config.PROJECT_ID
});

const spannerInstance = spannerClient.instance(config.INSTANCE_ID);

const spannerDatabase = spannerInstance.database(config.DATABASE_ID);

exports.upsertRows = function(tableName, rows) {
return new Promise((resolve, reject) => {
	try {
		const spannerTable = spannerDatabase.table(tableName);
		spannerTable.upsert(rows)
			.then(() => {
				console.log('Upserted data.');
				resolve(true);
			})
			.catch(err => {
				console.error(`Upsert `, err);
				reject(err);
			});
		});
	} catch (err) {
		console.error(`upsertRows `, err);
		reject(err);
	}
};

[stephenplusplus]

This has been addressed by a dependency, google-auth-library: googleapis/google-auth-library-nodejs#283. It won't be pulled in automatically, however, we are still determining our game plan to get it out (stephenplusplus/google-auto-auth#39).

I'll mark this as blocked, but hopefully not for long.

[pembeci]

We are also seeing these "Auth error:Error: read ECONNRESET" errors in our Cloud Function environment which utilizes Spanner. Since our setup is too complex we were running different narrowed down versions of our code and trying this patch to understand if it is caused by something on our part or a more general Cloud issue. Now that we noticed this issue, at least let me show some of our logs. All of the below logs belong to the same function execution_id but we have a similar pattern in many other calls:

10:20:48.771 Function execution started
10:20:48.875 {... "runId":"SygFY--vIM", "respCode":200, "ALL":98....}
10:20:48.875 Function execution took 104 ms, finished with status code: 200
11:10:28.435 Auth error:Error: read ECONNRESET
11:10:28.734 Auth error:Error: read ECONNRESET
11:10:28.834 Auth error:Error: read ECONNRESET
11:10:28.834 Auth error:Error: read ECONNRESET
11:10:28.934 Auth error:Error: read ECONNRESET
11:10:35.635 Auth error:Error: read ECONNRESET
11:10:35.636 Auth error:Error: read ECONNRESET
11:10:39.034 Unhandled rejection
11:10:39.134 Error: Getting metadata from plugin failed with error: read ECONNRESET
    at /user_code/node_modules/grpc/src/client.js:554:15
11:10:42.035 [runId: S1SwZWv8M] Unhandled promise rejection: 'Error: Getting metadata from plugin failed with error: read ECONNRESET'

What is interesting is that we are getting these Auth errors 50 minutes after the original request is already completed successfully. Although console.log reports them with the same function execution_id, our own corresponding runId (on 2nd and last logs) shows that the error was logged while another request was handled. @stephenplusplus, do you think we have the same issue discussed here or shall we continue digging in our code? Thanks.

[stephenplusplus]

This should be resolved now, as we have upgraded to the latest google-auth-library via google-auto-auth. To pull in the changes, just un- and re-install @google-cloud/spanner. Please let me know if the problem persists, and we will re-investigate.

[pembeci]

Thanks. We'll test it immediately.