ChannelOptions doesn't work well in client tls connection

[feuyeux]

Problem description

In TLS connection scenaio, the client implemented by nodejs(which depends on grpc-js) cannot connect to server.
I got the error messages from c++ server ssl_transport_security.cc:1847] No match found for server name: localhost., and no error message from nodejs server.

The certs is signed with a special dns name, and the client uses localhost to connect. So, I use ChannelOptions in node/ChannelArguments in c++ to define the grpc.default_authority. In this way, the c++ client(actually, including java/kotlin/csharp/python/rust/go) works well.

I'm not sure if it's an issue or my mistake. Please help to check:

my nodejs client node

let address = connectTo + ":" + port
    let secure = process.env.GRPC_HELLO_SECURE
    if (typeof secure !== 'undefined' && secure !== null) {
        logger.info("Connect With TLS(%s)", port)
        let rootCertContent = fs.readFileSync(certChain);
        let privateKeyContent = fs.readFileSync(certKey);
        let certChainContent = fs.readFileSync(certChain);
        const credentials = grpc.credentials.createSsl(rootCertContent, privateKeyContent, certChainContent);
        //https://grpc.github.io/grpc/core/group__grpc__arg__keys.html
        const channelOptions = {
            'grpc.default_authority': serverName
        }
        return new services.LandingServiceClient(address, credentials, channelOptions)
    } else {
        logger.info("Connect With InSecure(%s)", port)
        return new services.LandingServiceClient(address, grpc.credentials.createInsecure())
    }

my c++ client code

const string &port = Utils::getBackendPort();
        const basic_string<char, char_traits<char>, allocator<char>> &target = Utils::getBackend() + ":" + port;
        const string &secure = Utils::getSecure();
        if (!secure.empty() && secure == "Y") {
            grpc::SslCredentialsOptions ssl_opts;
            ssl_opts.pem_root_certs = Connection::getFileContent(certChain);
            ssl_opts.pem_private_key = Connection::getFileContent(certKey);
            ssl_opts.pem_cert_chain = Connection::getFileContent(certChain);
            grpc::ChannelArguments channel_args;
            channel_args.SetString("grpc.default_authority", serverName);
            LOG(INFO) << "Connect with TLS(" << port << ")";
            return grpc::CreateCustomChannel(target, grpc::SslCredentials(ssl_opts), channel_args);
        } else {
            LOG(INFO) << "Connect with InSecure(" << port << ")";
            return grpc::CreateChannel(target, grpc::InsecureChannelCredentials());
        }

Reproduction steps

Clone and go to this folder:
https://github.com/feuyeux/hello-grpc/tree/main/grpc/hello-grpc-nodejs

Run the below scripts on two terminals:

export GRPC_HELLO_SECURE="Y" 
sh server_start.sh

export GRPC_HELLO_SECURE="Y" 
sh client_start.sh

Environment

• OS name, version and architecture: macOS 11.6
• Node version: v16.13.0 and v17.0.1(need to remove sleep lib)
• Node installation method: brew
• Package name and version: package.json

Additional context

https://github.com/feuyeux/hello-grpc

[murgatroid99]

In Node, in order to validate a server certificate against a different name, you also need to set the option grpc.ssl_target_name_override to the same value that you are setting for grpc.default_authority

[feuyeux]

I tried with both options, it didn't work, yet.

const options = {
            "grpc.default_authority": serverName,
            "grpc.ssl_target_name_override": serverName
        };
return new LandingServiceClient(address, credentials, options)

[murgatroid99]

Are you getting the exact same error No match found for server name: localhost. even with both of those options set? And what is the value of serverName? It should be just a domain name. For example, in some of our own test code, we set those options with the value foo.test.google.fr.

[feuyeux]

Yes, I got localhost error even with both sets, and the serverName is "hello.grpc.io" in this line: https://github.com/feuyeux/hello-grpc/blob/e255fd966627d2fd88ac46c78ac17b928c5a35b0/grpc/hello-grpc-nodejs/common/connection.js#L9

[murgatroid99]

And your certificate is for a server named hello.grpc.io?

[feuyeux]

Yes, Michael. I have verified server/client tls in java/go/c++/.. with same certificates(https://github.com/feuyeux/hello-grpc/tree/main/grpc), and only node client doesn't work.

[murgatroid99]

I managed to run your Node example. When I have the Node client communicate with the Node server, the server does not report any error like the one you mentioned, but the client actually gets an error that says "unsupported certificate purpose". I can't tell why that is happening in this example, but from my research that seems to be some kind of problem with the "X509v3 Extended Key Usage" field in the certificate.

I am still trying to get your C++ example to build, so I have not tested the Node client with the C++ server yet, but one suggestion I have is to not use client authentication (don't provide the certificate or key to credentials.createSsl) just to simplify the TLS interaction that we are trying to debug. Does that still result in the same error? In addition, can you run the client with the environment variables GRPC_TRACE=subchannel and GRPC_VERBOSITY=DEBUG and look for a line that starts like this:

D 2021-12-16T15:19:14.817Z | subchannel | (3) 127.0.0.1:9996 connection closed with error...

We know what error the server sees, but that line should provide more information about what error the client is seeing when the connection fails.

[murgatroid99]

I also want to mention that I will be off for the rest of the year. I will continue to look into this in January, but I do not expect to respond again until then.

[feuyeux]

COMMAND:

export GRPC_HELLO_SECURE=Y
export GRPC_TRACE=subchannel
export GRPC_VERBOSITY=DEBUG

node proto_client.js

OUTPUT:

D 2021-12-17T02:52:56.846Z | subchannel | (2) ::1:9996 Subchannel constructed with options {
  "grpc.default_authority": "hello.grpc.io",
  "grpc.ssl_target_name_override": "hello.grpc.io"
}
D 2021-12-17T02:52:56.847Z | subchannel | (3) 127.0.0.1:9996 Subchannel constructed with options {
  "grpc.default_authority": "hello.grpc.io",
  "grpc.ssl_target_name_override": "hello.grpc.io"
}
D 2021-12-17T02:52:56.848Z | subchannel | (2) ::1:9996 IDLE -> CONNECTING
D 2021-12-17T02:52:56.849Z | subchannel | (3) 127.0.0.1:9996 IDLE -> CONNECTING
D 2021-12-17T02:52:56.849Z | subchannel | (2) ::1:9996 creating HTTP/2 session
D 2021-12-17T02:52:56.852Z | subchannel | (3) 127.0.0.1:9996 creating HTTP/2 session
D 2021-12-17T02:52:56.855Z | subchannel | (2) ::1:9996 connection closed with error connect ECONNREFUSED ::1:9996
D 2021-12-17T02:52:56.856Z | subchannel | (2) ::1:9996 connection closed
D 2021-12-17T02:52:56.856Z | subchannel | (2) ::1:9996 CONNECTING -> TRANSIENT_FAILURE
D 2021-12-17T02:52:56.861Z | subchannel | (3) 127.0.0.1:9996 connection closed with error unsupported certificate purpose
D 2021-12-17T02:52:56.861Z | subchannel | (3) 127.0.0.1:9996 connection closed
D 2021-12-17T02:52:56.861Z | subchannel | (3) 127.0.0.1:9996 CONNECTING -> TRANSIENT_FAILURE
2021-12-17T02:52:56.862Z [error] 14 UNAVAILABLE: No connection established
2021-12-17T02:52:56.862Z [error] 14 UNAVAILABLE: No connection established
2021-12-17T02:52:56.862Z [error] 14 UNAVAILABLE: No connection established
2021-12-17T02:52:56.862Z [error] 14 UNAVAILABLE: No connection established
D 2021-12-17T02:52:57.848Z | subchannel | (2) ::1:9996 TRANSIENT_FAILURE -> CONNECTING
D 2021-12-17T02:52:57.849Z | subchannel | (2) ::1:9996 creating HTTP/2 session
D 2021-12-17T02:52:57.850Z | subchannel | (2) ::1:9996 connection closed with error connect ECONNREFUSED ::1:9996
D 2021-12-17T02:52:57.850Z | subchannel | (2) ::1:9996 connection closed
D 2021-12-17T02:52:57.850Z | subchannel | (2) ::1:9996 CONNECTING -> TRANSIENT_FAILURE
D 2021-12-17T02:52:57.850Z | subchannel | (2) ::1:9996 TRANSIENT_FAILURE -> IDLE
D 2021-12-17T02:52:57.851Z | subchannel | (3) 127.0.0.1:9996 TRANSIENT_FAILURE -> IDLE

[urko-b]

@murgatroid99 Any update from here? I've got build a go grpc server and I expect same behavior. When I connect a go client to my server (which now is deployed on a VPS exposed through domain name) it is working fine but when I connect through a nodejs client it gives me same error.

[murgatroid99]

The log shows the same error I saw: "unsupported certificate purpose". I think the problem is that there is something about the certificate that Node's TLS library can't handle. gRPC is just passing the certificate along to the TLS library, so unfortunately I don't really have any special insight about what the problem might be.

[urko-b]

@murgatroid99 is this library https://nodejs.org/api/tls.html ?

[murgatroid99]

Yes, that is what I was referring to.

[urko-b]

ok I think I will link this issue to their repository. if they have I've not checked yet 😄

[urko-b]

@murgatroid99 I've created this link on nodejs repo. So we will see how it is working.