RATIS-920. Fix error use of onCompleted in onError

[runzhiwang]

Apache jira:
https://issues.apache.org/jira/browse/RATIS-920

What's the problem ?
When run test, a lot of StatusRuntimeException: CANCELLED: call already cancelled was thrown, as the image shows.

What's the reason ?

1. When run into onError, the call has already been cancelled, there is no need to call onCompleted. You can find from the following doc of onError, it's the last method call, and
   no further calls to any method are allowed.

void onError(Throwable t)
Receives a terminating error from the stream.
May only be called once and if called it must be the last method called. In particular if an exception is thrown by an implementation of onError no further calls to any method are allowed.

t should be a StatusException or StatusRuntimeException, but other Throwable types are possible. Callers should generally convert from a Status via Status.asException() or Status.asRuntimeException(). Implementations should generally convert to a Status via Status.fromThrowable(Throwable).

2. You can find why the exception was thrown in the following code, copied from gRPC.

2.1. Before call requestObserver.onError, it will first set responseObserver.cancelled = true. Besides the onCancelHandler is null, because it was not set in ratis.

      @Override
      public void onCancel() {
        responseObserver.cancelled = true;
        if (responseObserver.onCancelHandler != null) {
          responseObserver.onCancelHandler.run();
        }
        if (!halfClosed) {
          requestObserver.onError(
              Status.CANCELLED
                  .withDescription("cancelled before receiving half close")
                  .asRuntimeException());
        }
      }

2.2. When call responseObserver.onCompleted, because cancelled == true and onCancelHandler == null, then exception was threw.

    @Override
    public void onCompleted() {
      if (cancelled) {
        if (onCancelHandler == null) {
          throw Status.CANCELLED.withDescription("call already cancelled").asRuntimeException();
        }
      } else {
        call.close(Status.OK, new Metadata());
        completed = true;
      }
    }

[runzhiwang]

@lokeshj1703 Could you help review this patch ? Thank you very much.

[lokeshj1703]

@runzhiwang Thanks for working on this! I think this scenario will occur when client cancels the call. How about we extract the status in onError(using Status.fromThrowable(Throwable)) and check if it is cancelled. If it is cancelled we do not make the onCompleted call. Let's see if that solves the problem.

[runzhiwang]

@lokeshj1703 Thanks for review. The suggestion does work. I have updated.

[lokeshj1703]

@runzhiwang Thanks for updating the PR! The changes look good to me. +1.

[lokeshj1703]

@runzhiwang Thanks for the contribution! I have committed the PR to master branch.