GRPC Auth problem with GCP IAP

[elia-secchi]

Hi all,

Working with Google Cloud, Kubeflow and Seldon, I would like to establish a GRPC connection with a pod in my cluster.
My current setup includes a istio-ingressgateway configured with Cloud Identity-Aware Proxy as authorization layer.
I already have in place a virtual service to redirect my traffic from the ingress-gateway to the GRPC pod port.

When I local port-forward to my ingress-gateway http2 port, I am unable to send requests getting the following error:

grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
        status = StatusCode.UNAUTHENTICATED
        details = "Origin authentication failed"
        debug_error_string = "{"created":"@1587401119.875413000","description":"Error received from peer ipv6:[::1]:8080","file":"src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"Origin authentication failed","grpc_status":16}"

From the error code I can notice that is a problem in authenticating my GRPC Channel.

The problem looks even stranger when I look to my ingress-gateway logs, where I can see the following for my failed request:

"POST /seldon.protos.Seldon/Predict HTTP/2" 200 - "-" 0 0 0 - "10.50.7.24" "grpc-python/1.27.2 grpc-c/9.0.0 (osx; chttp2; guantao)"

which in the logs looks successful.
My current security policy is:

apiVersion: authentication.istio.io/v1alpha1
kind: Policy
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"authentication.istio.io/v1alpha1","kind":"Policy","metadata":{"annotations":{},"labels":{"app.kubernetes.io/component":"iap-ingress","app.kubernetes.io/instance":"iap-ingress-v1.0.0","app.kubernetes.io/managed-by":"kfctl","app.kubernetes.io/name":"iap-ingress","app.kubernetes.io/part-of":"kubeflow","app.kubernetes.io/version":"v1.0.0","kustomize.component":"iap-ingress"},"name":"ingress-jwt","namespace":"istio-system"},"spec":{"origins":[{"jwt":{"audiences":["TO_BE_PATCHED"],"issuer":"https://cloud.google.com/iap","jwksUri":"https://www.gstatic.com/iap/verify/public_key-jwk","jwtHeaders":["x-goog-iap-jwt-assertion"],"trigger_rules":[{"excluded_paths":[{"exact":"/healthz/ready"},{"prefix":"/.well-known/acme-challenge"}]}]}}],"principalBinding":"USE_ORIGIN","targets":[{"name":"istio-ingressgateway","ports":[{"number":80}]}]}}
  creationTimestamp: "2020-04-18T09:43:46Z"
  generation: 2
  labels:
    app.kubernetes.io/component: iap-ingress
    app.kubernetes.io/instance: iap-ingress-v1.0.0
    app.kubernetes.io/managed-by: kfctl
    app.kubernetes.io/name: iap-ingress
    app.kubernetes.io/part-of: kubeflow
    app.kubernetes.io/version: v1.0.0
    kustomize.component: iap-ingress
  name: ingress-jwt
  namespace: istio-system
  resourceVersion: "8170"
  selfLink: /apis/authentication.istio.io/v1alpha1/namespaces/istio-system/policies/ingress-jwt
  uid: *********
spec:
  origins:
  - jwt:
      audiences:
      - /projects/2********/global/backendServices/
      issuer: https://cloud.google.com/iap
      jwksUri: https://www.gstatic.com/iap/verify/public_key-jwk
      jwtHeaders:
      - x-goog-iap-jwt-assertion
      trigger_rules:
      - excluded_paths:
        - exact: /healthz/ready
        - prefix: /.well-known/acme-challenge
  principalBinding: USE_ORIGIN
  targets:
  - name: istio-ingressgateway
    ports:
    - number: 80

Using REST I was able to authenticate successfully my requests, adding a token to my request following what described here
I've tried various ways to authenticate my requests in GRPC with no success:

• https://grpc.io/docs/guides/auth/#authenticate-with-google-using-a-jwt
• https://grpc.io/docs/guides/auth/#authenticate-with-google-using-an-oauth2-token
• Adding as metadata of the request ('authentication', 'Bearer {}'.format(token)

Is there any example in creating a secure channel with Cloud IAP?

Many Thanks

Environment:

Kubeflow version: 1.0.1
kfctl version: kfctl v1.0.1-0-gf3edb9b
Kubernetes platform: GKE
Kubernetes version: 1.14.10

[ukclivecox]

OK. We don't have a good example at present.

There are some notes in https://github.com/SeldonIO/seldon-core/blob/master/notebooks/seldon_client.ipynb

The seldon_client could also be tried:

seldon-core/python/seldon_core/seldon_client.py

Lines 1905 to 1947 in 34ab0c3

	else:
	# If one of root cert & cert chain are provided, both must be provided
	# otherwise there is a null pointer exception in the Go underlying impl
	if (
	channel_credentials.private_key_file
	and channel_credentials.root_certificates_file
	and channel_credentials.certificate_chain_file
	):
	grpc_channel_credentials = grpc.ssl_channel_credentials(
	root_certificates=open(
	channel_credentials.root_certificates_file, "rb"
	).read(),
	private_key=open(channel_credentials.private_key_file, "rb").read(),
	certificate_chain=open(
	channel_credentials.certificate_chain_file, "rb"
	).read(),
	)
	# For most usecases only providing the root cert file is enough
	elif channel_credentials.root_certificates_file:
	grpc_channel_credentials = grpc.ssl_channel_credentials(
	root_certificates=open(
	channel_credentials.root_certificates_file, "rb"
	).read()
	)
	# This piece also allows for blank SSL Channel credentials in case this is required
	else:
	grpc_channel_credentials = grpc.ssl_channel_credentials()
	if channel_credentials.verify == False:
	# If Verify is set to false then we add the SSL Target Name Override option
	options += [
	("grpc.ssl_target_name_override", gateway_endpoint.split(":")[0])
	]
	if not call_credentials is None:
	grpc_call_credentials = grpc.metadata_call_credentials(
	lambda context, callback: callback(
	(("x-auth-token", call_credentials.token),), None
	)
	)
	credentials = grpc.composite_channel_credentials(
	grpc_channel_credentials, grpc_call_credentials
	)
	else:

Hope we can help you can an example so we can add to docs for gRPC.