Implement new requirement of completing protocol handshaking before 'READY'

[dfawley]

Detailed description of upcoming changes:

gRPC-Go clients currently (in the 1.17 release and earlier) do not wait for the HTTP/2 handshake to complete before considering a connection ready for sending RPCs. However, the client does not consider the connection to be "successful" unless the server's handshake is received before a connection error occurs. An "unsuccessful" connection results in a backoff being performed before redialing.

This behavior will be changed during development for the 1.18 release. The new default behavior will be to wait for the server handshake before considering the connection ready for sending RPCs. This behavior can be tested in 1.17 by setting the environment variable GRPC_GO_REQUIRE_HANDSHAKE=on. (This is the current behavior of gRPC-Java.)

Setting the environment variable to "off" will cause gRPC-Go clients to not wait for the handshake before considering the connection ready. However, if a connection error occurs, backoff will not be performed and the client will attempt to connect to the first address returned by the name resolver. This can lead to DOS problems and is not recommended. (This is the current behavior of gRPC-C and was the behavior of gRPC-Go ~1 yr ago before #1648.)

During development for the 1.19 release, support for changing this behavior via the environment variable will be removed entirely. Also, the grpc.WithWaitForHandshake() DialOption (was "experimental"; now "deprecated") will be removed.

Users impacted: as far as we are aware, the only usage that may be impacted by the new behavior is cmux. cmux has a workaround for Java using MatchWithWriters to allow it to continue working in the face of this behavior.

For more information, see: grpc/grpc#17006

When security is disabled, not waiting for the HTTP/2 handshake can lead to DoS-style behavior. For details, see: #954. This requirement will incur an extra half-RTT latency before the first RPC can be sent under plaintext, but this is negligible and unencrypted connections are rarer than secure ones.

Under TLS, the server will effectively send its part of the HTTP/2 handshake along with its final TLS "server finished" message, which the client must wait for before transmitting any data securely. This means virtually no extra latency is incurred by this requirement.

Go had attempted to separate "connection ready" with "connection successful" (Issue: #1444 PR: #1648). However, this is confusing to users and introduces an arbitrary distinction between these two events. It has led to several bugs in our reconnection logic (e.g.s #2380, #2391, #2392), due to the complexity, and it makes custom transports (grpc/proposal#103) more difficult for users to implement.

We are aware of some use cases (in particular, soheilhy/cmux) expecting the behavior of transmitting an RPC before the HTTP/2 handshake is completed. Before making behavior changes to implement this, we will reach out to our users to the best of our abilities.

[dfawley]

Minor update:

During development for the 1.19 release, support for changing this behavior via the environment variable will be removed entirely. Also, the grpc.WithWaitForHandshake() DialOption (was "experimental"; now "deprecated") will be removed.

This will be removed shortly, but not in the 1.19 release.