GRPC-JS Error: Metadata key contains illegal characters

[daaain]

Environment details

• OS: Debian (using node:10.15.0 Docker image)
• Node.js version: v10.15.0
• npm version: 6.4.1 (using yarn 1.12.3 though)
• @google-cloud/logging-winston version: 1.1.0

Steps to reproduce

1. Send logs via this logger
2. Wait
3. Get error caught by a reporting library, like Sentry in our case

I've been seeing this error coming up sporadically, with different characters in the key each time:

Error: Metadata key "���é" contains illegal characters
  File "app:///../node_modules/@grpc/grpc-js/build/src/client.js", line 101, col 45, in Http2CallStream.call.on
    const error = Object.assign(new Error(status.details), status);
  File "events.js", line 187, col 15, in Http2CallStream.emit
  File "domain.js", line 459, col 23, in Http2CallStream.EventEmitter.emit
  File "app:///../node_modules/@grpc/grpc-js/build/src/call-stream.js", line 71, col 22, in process.nextTick
    this.emit('status', status);
  File "internal/process/next_tick.js", line 61, col 11, in process._tickCallback

It only appeared once we had a deployment with logging using this library, so I'm pretty sure it's the logging transport, but can't be 100% as the stack trace starts in the grpc-js dependency.

Since I don't know where it is coming from, I don't really know how debug it. I'm guessing because the work is offloaded to the next tick I wouldn't even be able to catch the library error in my client code either 🤷‍♀

I found this related upstream issue: grpc/grpc-node#519

[DominicKramer]

Thank you for opening this issue. I'm transferring the issue to nodejs-logging since interactions with the logging API are not done by this library but are instead done using the nodejs-logging library. As such, I don't think this issue is specific to the nodejs-logging-winston library.

[callmehiphop]

Hi @daaain can I ask what version of google-gax is installed in your project? I think a fix went out for this in v1.1.12.

[daaain]

@callmehiphop just had a look, apparently it was an older version according to yarn:

google-gax@^1.0.0:
  version "1.1.2"
  resolved "https://registry.yarnpkg.com/google-gax/-/google-gax-1.1.2.tgz#e526c375275759696ac4a0424bd5ce476a66729b"
  integrity sha512-kl/7U5x0Db9TwY4qYKomyBjKAgy3ztb1iqpx/wC84CtUVMNa1JNyPVlVdjgJS2JgOUF0yxMp9z6amfvss6tSIA==

I'll bump the version and report back!

[daaain]

Wait, it looks like that fix was in 1.1.2 after all: https://github.com/googleapis/gax-nodejs/releases/tag/v1.1.2

The thing I noticed though is that it fixes encoding for values, but not keys, so maybe that's the issue?

I honestly have no idea where these non-ASCII characters come from and not sure if URL encoding them would help, but I guess it's still better to have normalised strings for keys than the library exploding 🤷‍♀

Edit: oh no, it's only the test which tests for values only, but the actual method does encode the keys too...

[daaain]

Oh, and also noticed that googleapis/nodejs-datastore#415 and googleapis/nodejs-pubsub#667 and googleapis/nodejs-pubsub#680 are similar, but wondering if the issue is solved for those guys after all?

[callmehiphop]

@daaain oh whoops, that was a typo on my part, 1.1.2 is correct. @alexander-fenster any insight here?

[alexander-fenster]

Hi @daaain,

We have released google-gax v1.1.5 that pulls @grpc/grpc-js v0.5.2 in which the invalid headers (that are coming from Node.js http2 module) are printed in a warning, but do not throw exception. Please upgrade your dependencies, it should fix the problem for you.