fix: Handle cancel in ReleasingClientCall and rethrow the exception in start

[mutianf]

We got a customer issue and the stacktrace didn't show us where the call was cancelled (b/263968439)

When trying to reproduce this error, we noticed some inconsistent behaviors when we cancel the stream in onStart()

client.readRowsCallable().call(Query.create("t"), new ResponseObserver<Row>() {
    @Override
    public void onStart(StreamController streamController) {
        streamController.cancel();
    }
    @Override
    public void onResponse(Row row) {
    }
    @Override
    public void onError(Throwable throwable) {
    }
    @Override
    public void onComplete() {
    }
});

When the above code is called, we got exception:

Caused by: java.util.concurrent.CancellationException: User cancelled stream
	at com.google.api.gax.rpc.ServerStreamingAttemptCallable.onCancel(ServerStreamingAttemptCallable.java:309)

However, if we make another call before it:

Iterator<Row> stream = client.readRowsCallable().call(Query.create("t")).iterator();
client.readRowsCallable().call(Query.create("t"), new ResponseObserver<Row>() {
    @Override
    public void onStart(StreamController streamController) {
        streamController.cancel();
    }
   ...
}

We got exception:

java.lang.IllegalStateException: Not started
	at com.google.common.base.Preconditions.checkState(Preconditions.java:502)
	at io.grpc.internal.ClientCallImpl.sendMessageInternal(ClientCallImpl.java:511)
	at io.grpc.internal.ClientCallImpl.sendMessage(ClientCallImpl.java:504)

Which is very difficult to debug.

I think there are 2 problems in the ReleasingClientCall:

1. It's not tracking cancellation status: After the call is cancelled, it continued to go into start() logic and eventually will call ClientCallImpl#startInternal https://github.com/grpc/grpc-java/blob/v1.51.1/core/src/main/java/io/grpc/internal/ClientCallImpl.java#L199, where the cancelCalled will be true and the state check will fail. In this case we won't see the actual stacktrace that cancelled the stream.
2. It's not rethrowing the error in onStart(): The IllegalStateException thrown from ClientCallImpl#startInternal will be caught, GrpcDirectStreamController will move on to ClientCallImpl#sendMessage and eventually fail when checking if the stream is started https://github.com/grpc/grpc-java/blob/v1.51.1/core/src/main/java/io/grpc/internal/ClientCallImpl.java#L514.

[igorbernstein2]

lgtm after nits

[mutianf]

@burkedavison Do you think it's ok to merge this PR? Do you have any other concerns? Thanks!

[burkedavison]

@burkedavison Do you think it's ok to merge this PR? Do you have any other concerns? Thanks!

Hey @mutianf ,
LGTM, but I'd like to request approval from someone more familiar with gax-java development. @blakeli0 / @meltsufin ?

[sonarqubecloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
1 Code Smell

87.5% Coverage
0.0% Duplication