Issue connecting with Google gRPC APIs

[Hirevo]

Bug Report

Version

tonic = { version = "0.1.0-alpha.4", features = ["rustls", "prost"] }
tonic-build = { version = "0.1.0-alpha.4" }

Platform

OS: macOS Catalina (10.15)
rustc version: 1.39.0-nightly (2019-09-10)

# output from: `uname -a`
Darwin MBP 19.0.0 Darwin Kernel Version 19.0.0: Wed Sep 25 20:18:50 PDT 2019; root:xnu-6153.11.26~2/RELEASE_X86_64 x86_64

Description

Hello,

I was trying to write an abstraction library for using Google Cloud Platform APIs.
More specifically, I was writing the one for the Pub/Sub service of GCP.

So, I cloned the protobuf definitions of their services (https://github.com/googleapis/googleapis), and attempted to call one of their endpoints.

Here is the build.rs file I used:

fn main() {
    tonic_build::configure()
        .build_client(true)
        .build_server(false)
        .format(true)
        .out_dir("src/api")
        .compile(&["protos/google/pubsub/v1/pubsub.proto"], &["protos"])
        .unwrap();
    println!("cargo:rerun-if-changed=protos/google/pubsub/v1/pubsub.proto");
}

protos/ is the directory containing the clone of Google's protobufs.

Then, I attempt to connect and make a call to the service, like this:

use http::HeaderValue;
use tonic::transport::{Certificate, Channel, ClientTlsConfig};
use tonic::Request;

mod api {
    // Generated protobuf bindings
    include!("api/google.pubsub.v1.rs");
}

const ENDPOINT: &str = "https://pubsub.googleapis.com";
const SCOPES: [&str; 2] = [
    "https://www.googleapis.com/auth/cloud-platform",
    "https://www.googleapis.com/auth/pubsub",
];

#[tokio::main]
async fn main() {
    let certs = tokio::fs::read("roots.pem").await.unwrap();

    let mut tls_config = ClientTlsConfig::with_rustls();
    tls_config.ca_certificate(Certificate::from_pem(certs.as_slice()));
    tls_config.domain_name("pubsub.googleapis.com");

    let channel = Channel::from_static(ENDPOINT)
        .intercept_headers(|headers| {
            let token = "some-valid-google-oauth-token";
            let value = format!("Bearer {0}", token);
            let value = HeaderValue::from_str(value.as_str()).unwrap();
            headers.insert("authorization", value);
        })
        .tls_config(&tls_config)
        .channel();

    let mut service = api::client::PublisherClient::new(channel);

    let response = service.list_topics(Request::new(api::ListTopicsRequest {
        project: format!("projects/{0}", "some-gcp-project-name"),
        page_size: 10,
        page_token: String::default(),
    }));

    dbg!(response.await);
}

The roots.pem file I use here is the sample PEM file from Google Trust Services (https://pki.goog/roots.pem).

The dbg!(response.await) statement always yields:

[src/sample.rs:42] response.await = Err(
    Status {
        code: Internal,
        message: "Unexpected compression flag: 60",
    },
)

This happens regardless of the validity of the OAuth token.

To see if the connection flow was valid, I reproduced the experiment using Golang's google.golang.org/grpc and github.com/golang/protobuf packages:

package main

import (
	"context"
	"fmt"
	"os"

	"google.golang.org/grpc"
	"google.golang.org/grpc/credentials"
	"google.golang.org/grpc/metadata"

	pubsub "google.golang.org/genproto/googleapis/pubsub/v1" // generated protobuf bindings
)

func clientInterceptor(
	ctx context.Context,
	method string,
	req interface{},
	reply interface{},
	cc *grpc.ClientConn,
	invoker grpc.UnaryInvoker,
	opts ...grpc.CallOption,
) error {
	md := metadata.New(map[string]string{
		"authorization": "Bearer some-valid-google-oauth-token",
	})
	ctx = metadata.NewOutgoingContext(ctx, md)

	return invoker(ctx, method, req, reply, cc, opts...)
}

func main() {
	creds, err := credentials.NewClientTLSFromFile("roots.pem", "pubsub.googleapis.com")
	if err != nil {
		fmt.Fprintf(os.Stderr, "error: %s\n", err.Error())
	}

	conn, err := grpc.Dial("pubsub.googleapis.com:443", grpc.WithTransportCredentials(creds), grpc.WithUnaryInterceptor(clientInterceptor))
	if err != nil {
		fmt.Fprintf(os.Stderr, "error: %s\n", err.Error())
	}
	defer conn.Close()

	pub := pubsub.NewPublisherClient(conn)

	resp, err := pub.ListTopics(context.Background(), &pubsub.ListTopicsRequest{Project: "projects/some-gcp-project-name", PageSize: 10})
	if err != nil {
		fmt.Fprintf(os.Stderr, "error: %s\n", err.Error())
	}
	fmt.Printf("topics: %#v\n", resp)
}

This Go code, provided with valid credentials, works as expected whereas the Rust code, with the same credentials, encounters the invalid compression flag error.

[LucioFranco]

@Hirevo just for more debugging purposes by chance could you see what the headers are that gcp returns to you?

So I know we don't quite yet support compression but I am curious if we can disable it.

[LucioFranco]

And also since you have an interceptor does changing it to this work?

        .intercept_headers(|headers| {
            let token = "some-valid-google-oauth-token";
            let value = format!("Bearer {0}", token);
            let value = HeaderValue::from_str(value.as_str()).unwrap();
            headers.insert("authorization", value);
			headers.insert("grpc-accept-encoding", HeaderValue::from_str("identity").unwrap());
        })

The docs are not clear specifically if this will work but pretty much for now we need to tell their grpc server that we don't accept any compression.

[Hirevo]

@LucioFranco Yes, here are the response headers that the Go code received for the gRPC call:

metadata.MD{
    "alt-svc":[]string{
        "quic=\":443\"; ma=2592000; v=\"46,43\",h3-Q048=\":443\"; ma=2592000,h3-Q046=\":443\"; ma=2592000,h3-Q043=\":443\"; ma=2592000"
    },
    "content-disposition":[]string{"attachment"},
    "content-type":[]string{"application/grpc"},
    "date":[]string{"Mon, 28 Oct 2019 21:12:24 GMT"}
}

I have also just tried the additional header you proposed, but the error has not changed, sadly.

Also, I published both the Rust and Go versions of these tests, with the same setup and crate versions that I have:
Rust: https://github.com/Hirevo/rust-gcp-test
Go: https://github.com/Hirevo/go-gcp-test

[LucioFranco]

Ok I dug into this and got it working if I change the content type to application/grpc from application/grpc+proto which doesn't make much sense. I will dig in a bit more into this...

[peter05010402]

I got the same Err!

[LucioFranco]

@peter05010402 what version of tonic are you using?

[frostRed]

I also had the same Error when using grpc steram rpc. It doesn't have the error at program startup but occurs at many minutes after. Then I re-connect to the grpc server and the error is gone but many minutes it will come again.

The server side is golang. And I tried tonic 0.2 and 0.3.1, not change anything.

[LucioFranco]

@frostRed you are getting the "unexpected compression flag error" after startup? Usually this only happens when you have an incorrectly configured channel. value 60 for this flag isn't even valid iirc.

[frostRed]

No, after startup everything is ok, it only occurs on after running half hour or longer.

[LucioFranco]

Not startup for the process but for the connection? If that is still the issue something that can reproduce this or that we can inspect the frames would be very helpful.

[xiaochuanyu]

I arrived here googling "invalid compression flag: 60" and found out that I get a similar error message if I use the wrong endpoint URL e.g. using pubsub URL for endpoint with firestore client.
Its a stupid mistake while copy pasting from gcp example but error message is a little cryptic.

    const ENDPOINT: &str = "https://pubsub.googleapis.com";  // wrong endpoint
    let token = "some valid token";
    let token_metadata: MetadataValue<_> = format!("Bearer {}", token).parse().unwrap();

    let channel = Channel::from_static(ENDPOINT)
        .connect()
        .await.unwrap();

    let mut client = FirestoreClient::with_interceptor(channel, move |mut req: Request<()>| {
        req.metadata_mut()
            .insert("authorization", token_metadata.clone());
        Ok(req)
    });
    // ...

Response struct I get if I call client.create_document(...):

Status {
code:
  Internal,
  message: "protocol error: received message with invalid compression flag: 60 (valid flags are 0 and 1) while receiving response with status: 404 Not Found",
  metadata: MetadataMap{
    headers: {
      "date": "Sun, 24 Jul 2022 14:57:20 GMT",
      "content-type": "text/html; charset=UTF-8",
      "server": "ESF",
      "content-length": "1605",
      "x-xss-protection": "0",
      "x-frame-options": "SAMEORIGIN",
      "x-content-type-options": "nosniff",
      "alt-svc": "h3=\":443\"; ma=2592000,h3-29=\":443\"; ma=2592000,h3-Q050=\":443\"; ma=2592000,h3-Q046=\":443\"; ma=2592000,h3-Q043=\":443\"; ma=2592000,quic=\":443\"; ma=2592000; v=\"46,43\""
    }
  },
  source: None
}