Add TLS verify callback for Python

[zizhong]

Is your feature request related to a problem? Please describe.

Currently no way to set SSL verify callback when using a python grpc client.

Describe the solution you'd like

A clear and concise description of what you want to happen.

#12656 could work but didn't get merged.

[octaviansima]

If @zizhong or I were to contribute this, would it be reviewed and hopefully accepted into the project?

[zizhong]

Friendly ping. @gnossen @XuanWang-Amos Please let us know if it makes sense to you guys! Thanks!

[gnossen]

I would definitely welcome this contribution, but I'll also give @ZhenLian a chance to comment.

The place to start on this would be with a gRFC.

[gtcooke94]

Some thoughts - There exists this functionality in C as an experimental API -

grpc/include/grpc/grpc_security.h

Lines 961 to 1042 in 2cd1501

	/**
	* EXPERIMENTAL API - Subject to change
	*
	* A struct containing all the necessary functions a custom external verifier
	* needs to implement to be able to be converted to an internal verifier.
	*/
	typedef struct grpc_tls_certificate_verifier_external {
	void* user_data;
	/**
	* A function pointer containing the verification logic that will be
	* performed after the TLS handshake is done. It could be processed
	* synchronously or asynchronously.
	* - If expected to be processed synchronously, the implementer should
	* populate the verification result through |sync_status| and
	* |sync_error_details|, and then return true.
	* - If expected to be processed asynchronously, the implementer should return
	* false immediately, and then in the asynchronous thread invoke |callback|
	* with the verification result. The implementer MUST NOT invoke the async
	* |callback| in the same thread before |verify| returns, otherwise it can
	* lead to deadlocks.
	*
	* user_data: any argument that is passed in the user_data of
	* grpc_tls_certificate_verifier_external during construction time
	* can be retrieved later here.
	* request: request information exposed to the function implementer.
	* callback: the callback that the function implementer needs to invoke, if
	* return a non-zero value. It is usually invoked when the
	* asynchronous verification is done, and serves to bring the
	* control back to gRPC.
	* callback_arg: A pointer to the internal ExternalVerifier instance. This is
	* mainly used as an argument in |callback|, if want to invoke
	* |callback| in async mode.
	* sync_status: indicates if a connection should be allowed. This should only
	* be used if the verification check is done synchronously.
	* sync_error_details: the error generated while verifying a connection. This
	* should only be used if the verification check is done
	* synchronously. the implementation must allocate the
	* error string via gpr_malloc() or gpr_strdup().
	* return: return 0 if |verify| is expected to be executed asynchronously,
	* otherwise return a non-zero value.
	*/
	int (*verify)(void* user_data,
	grpc_tls_custom_verification_check_request* request,
	grpc_tls_on_custom_verification_check_done_cb callback,
	void* callback_arg, grpc_status_code* sync_status,
	char** sync_error_details);
	/**
	* A function pointer that cleans up the caller-specified resources when the
	* verifier is still running but the whole connection got cancelled. This
	* could happen when the verifier is doing some async operations, and the
	* whole handshaker object got destroyed because of connection time limit is
	* reached, or any other reasons. In such cases, function implementers might
	* want to be notified, and properly clean up some resources.
	*
	* user_data: any argument that is passed in the user_data of
	* grpc_tls_certificate_verifier_external during construction time
	* can be retrieved later here.
	* request: request information exposed to the function implementer. It will
	* be the same request object that was passed to verify(), and it
	* tells the cancel() which request to cancel.
	*/
	void (*cancel)(void* user_data,
	grpc_tls_custom_verification_check_request* request);
	/**
	* A function pointer that does some additional destruction work when the
	* verifier is destroyed. This is used when the caller wants to associate some
	* objects to the lifetime of external_verifier, and destroy them when
	* external_verifier got destructed. For example, in C++, the class containing
	* user-specified callback functions should not be destroyed before
	* external_verifier, since external_verifier will invoke them while being
	* used.
	* Note that the caller MUST delete the grpc_tls_certificate_verifier_external
	* object itself in this function, otherwise it will cause memory leaks. That
	* also means the user_data has to carries at least a self pointer, for the
	* callers to later delete it in destruct().
	*
	* user_data: any argument that is passed in the user_data of
	* grpc_tls_certificate_verifier_external during construction time
	* can be retrieved later here.
	*/
	void (*destruct)(void* user_data);
	} grpc_tls_certificate_verifier_external;

So it shouldn't be too difficult for a wrapped language to do this, just with the understanding that it is still experimental at this point

[rockspore]

@ZhenLian does not work on gRPC security any more. cc @erm-g as the current lead.

Just to supplement some context to what @gtcooke94 said, it seems #12656 and #16395 gave way to grpc/proposal#98 and subsequently grpc/proposal#205, which includes the grpc_tls_certificate_verifier_external above. Apart from it being experimental, using this also means you would have to switch your client to using this new set of APIs entirely to set up channel credentials.

[zizhong]

Thanks! Those are very helpful information. Will look into it! cc @octaviansima

[XuanWang-Amos]

Again, we welcome any contributions, for this particular case, I think the following needs to be done:

1. Verify the C experimental API grpc_tls_certificate_verifier_external is working as intended now.
2. Design a way to use this API in Python and create a gRFC (One thing we're interested to know is how do you plan to call back to Python).
3. Review the gRFC and implement it.
4. Don't forget tests :).

[shevisj]

Would love to see this implemented! Specifically, the ability to disable certificate verification on the client side when using a secure channel would be extremely helpful in cases where the server generates certificates on the fly and is hidden behind a load balancer/proxy. This is something that pretty much every other language's client implementation supports today.