Add timeout to connection.Connect()

If connection.Connect() gets stuck it will block infinitely, never
reporting an error to the library's consumer.

We have received reports of this happening even when a _valid_ address
is passed in node-driver-registrar because of a race condition involving
pod restarts. Specifically, there is an unlucky sequence of events when
restarting both the CSI Driver and node-driver-registrar, wheere the
node-driver-registar will attempt to Connect() to the CSI Driver, but
will get stuck doing so on an old file descriptor from the previously
running CSI Driver (and thus, get stuck infinitely).

There is no mechanism to pass a connection timeout to Connect, so this
commit adds a reasonbly long default timeout so these cases will
eventually return an error rather than getting stuck infinitely.

Signed-off-by: Connor Catlett <conncatl@amazon.com>

connection/connection.go

@@ -70,7 +70,7 @@ func SetMaxGRPCLogLength(characterCount int) {
 // For other connections, the default behavior from gRPC is used and
 // loss of connection is not detected reliably.
 func Connect(address string, metricsManager metrics.CSIMetricsManager, options ...Option) (*grpc.ClientConn, error) {
-	return connect(address, metricsManager, []grpc.DialOption{}, options)
+	return connect(address, metricsManager, []grpc.DialOption{grpc.WithTimeout(time.Second * 30)}, options)
 }
 
 // Option is the type of all optional parameters for Connect.