How can I bypass Dex or use a proper secure way to access inferenceservices?

[yurkoff-mv]

In version 1.5 i was able to bypass Dex using extension provider.
I added it to the istio configmap:

    extensionProviders:
    - name: "dex-auth-provider"
      envoyExtAuthzHttp:
        service: "authservice.istio-system.svc.cluster.local"
        port: "8080"
        includeHeadersInCheck: ["authorization", "cookie", "x-auth-token"]
        headersToUpstreamOnAllow: ["kubeflow-userid"]

Next I created two AuthorizationPolicy:
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: dex-auth
  namespace: istio-system
spec:
  selector:
    matchLabels:
      istio: ingressgateway
  action: CUSTOM
  provider:
    # The provider name must match the extension provider defined in the mesh config.
    name: dex-auth-provider
  rules:
  # The rules specify when to trigger the external authorizer.
  - to:
    - operation:
        notPaths: ["/v1*"]

and

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: allow-inference-services
  namespace: istio-system
spec:
  selector:
    matchLabels:
  action: ALLOW
  rules:
  - from:
    - source:
        principals: ["cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account"]
  - to:
    - operation:
        methods: ["POST"]
        paths: ["/v1*"]

Then I removed the filter and restarted istiod:
microk8s kubectl delete -n istio-system envoyfilters.networking.istio.io authn-filter
microk8s kubectl rollout restart deployment/istiod -n istio-system

After these manipulations, I could access Inference Services at the address http://host_ip:host_port/v1/models/model_name:predict without the need for authorization, since all requests from /v1* went bypassing Dex. Now I'm getting a 404 error.

It seems that requests are redirected to the central dashboard...
microk8s kubectl logs -n istio-system istio-ingressgateway-8f46b776-8sbt5

"POST /v1/models/my-model:predict HTTP/1.1" 404 - via_upstream - "-" 221 172 7 6 "192.168.21.45" "python-requests/2.31.0" "bf880dc0-918d-422b-a418-2d608de132bf" "my-model.my-namespace.example.com" "10.1.251.224:8082" outbound|80||centraldashboard.kubeflow.svc.cluster.local 10.1.251.225:43688 10.1.251.225:8080 192.168.21.45:6542 - -

Request:

curl -v POST 'http://127.0.01:32333/v1/models/my-model:predict' -H 'Host:my-model.my-namespace.example.com' -d '{"instances": [{"data": {"req": ["My request"]}}]}'

How do I bypass Dex?

P.S. I can't get session.

[yurkoff-mv]

This is what the log looks like in KubeFlow 1.5 with the Dex bypass described above:

[2023-11-22T06:00:46.148Z] "POST /v1/models/my-model:predict HTTP/1.1" 200 - via_upstream - "-" 221 777 8433 8433 "192.168.21.44" "python-requests/2.31.0" "835184ee-8830-92d5-a9ba-f8be36fe192a" "my-model-predictor-default.my-namespace.svc.cluster.local" "10.1.1.95:8081" outbound|80||knative-local-gateway.istio-system.svc.cluster.local 10.1.1.94:53100 10.1.1.94:8080 192.168.21.44:54926 - -

Requests go to Knative and not to Central Dashboard.
How can I achieve the same behavior in version 1.8?

[yurkoff-mv]

There is a difference in the output for the command kubectl get inferenceservices -n my-namespace
For version 1.5:

NAME               URL                                                      READY   PREV   LATEST   PREVROLLEDOUTREVISION   LATESTREADYREVISION                        AGE
my-model           http://my-model.my-namespace.example.com                 True           100                              my-model -predictor-default-00001          3m56s

For version 1.8:

NAME             URL                                                          READY   PREV   LATEST   PREVROLLEDOUTREVISION   LATESTREADYREVISION              AGE
my-model          http://my-model.my-namespace.svc.cluster.local              True           100                              my-model -predictor-00001        15s

In the first case, the service address ends with example.com, in the second with svc.cluster.local.
How can I perform inference for a deployed model?

[yurkoff-mv]

There is a difference in the output for the command kubectl get virtualservices -n my-namespace
For version 1.5:

NAMESPACE            NAME                                             GATEWAYS                                                                HOSTS                                                                                                                                                                                                                                                         AGE
auth                 dex                                              ["kubeflow/kubeflow-gateway"]                                           ["*"]                                                                                                                                                                                                                                                         23h
kubeflow             centraldashboard                                 ["kubeflow-gateway"]                                                    ["*"]                                                                                                                                                                                                                                                         23h
kubeflow             jupyter-web-app-jupyter-web-app                  ["kubeflow-gateway"]                                                    ["*"]                                                                                                                                                                                                                                                         23h
kubeflow             katib-ui                                         ["kubeflow-gateway"]                                                    ["*"]                                                                                                                                                                                                                                                         23h
kubeflow             metadata-grpc                                    ["kubeflow-gateway"]                                                    ["*"]                                                                                                                                                                                                                                                         23h
kubeflow             ml-pipeline-ui                                   ["kubeflow-gateway"]                                                    ["*"]                                                                                                                                                                                                                                                         23h
kubeflow             profiles-kfam                                    ["kubeflow-gateway"]                                                    ["*"]                                                                                                                                                                                                                                                         23h
kubeflow             tensorboards-web-app-tensorboards-web-app        ["kubeflow-gateway"]                                                    ["*"]                                                                                                                                                                                                                                                         23h
kubeflow             volumes-web-app-volumes-web-app                  ["kubeflow-gateway"]                                                    ["*"]                                                                                                                                                                                                                                                         23h
kubeflow             kserve-models-web-app                            ["kubeflow/kubeflow-gateway"]                                           ["*"]                                                                                                                                                                                                                                                         23h
my-namespace         notebook-my-namespace-megaputer-notebook         ["kubeflow/kubeflow-gateway"]                                           ["*"]                                                                                                                                                                                                                                                         23h
my-namespace         my-model-predictor-default-mesh                  ["mesh"]                                                                ["my-model-predictor-default.my-namespace","my-model-predictor-default.my-namespace.svc","my-model-predictor-default.my-namespace.svc.cluster.local"]                                                                                                         22h
my-namespace         my-model-predictor-default-ingress               ["knative-serving/knative-local-gateway","kubeflow/kubeflow-gateway"]   ["my-model-predictor-default.my-namespace","my-model-predictor-default.my-namespace.example.com","my-model-predictor-default.my-namespace.svc","my-model-predictor-default.my-namespace.svc.cluster.local"]                                                   22h
my-namespace         my-model                                         ["knative-serving/knative-local-gateway","kubeflow/kubeflow-gateway"]   ["my-model.my-namespace.svc.cluster.local","my-model.my-namespace.example.com"]                                                                                                                                                                               22h

For version 1.8:

NAMESPACE            NAME                                             GATEWAYS                                    HOSTS                                                                                                                                                               AGE
auth                 dex                                              ["kubeflow/kubeflow-gateway"]               ["*"]                                                                                                                                                               22h
kubeflow             centraldashboard                                 ["kubeflow-gateway"]                        ["*"]                                                                                                                                                               22h
kubeflow             jupyter-web-app-jupyter-web-app                  ["kubeflow-gateway"]                        ["*"]                                                                                                                                                               22h
kubeflow             katib-ui                                         ["kubeflow-gateway"]                        ["*"]                                                                                                                                                               22h
kubeflow             metadata-grpc                                    ["kubeflow-gateway"]                        ["*"]                                                                                                                                                               22h
kubeflow             ml-pipeline-ui                                   ["kubeflow-gateway"]                        ["*"]                                                                                                                                                               22h
kubeflow             profiles-kfam                                    ["kubeflow-gateway"]                        ["*"]                                                                                                                                                               22h
kubeflow             tensorboards-web-app-tensorboards-web-app        ["kubeflow-gateway"]                        ["*"]                                                                                                                                                               22h
kubeflow             volumes-web-app-volumes-web-app                  ["kubeflow-gateway"]                        ["*"]                                                                                                                                                               22h
kubeflow             kserve-models-web-app                            ["kubeflow/kubeflow-gateway"]               ["*"]                                                                                                                                                               22h
my-namespace         notebook-my-namespace-megaputer-notebook         ["kubeflow/kubeflow-gateway"]               ["*"]                                                                                                                                                               21h
my-namespace         my-model-predictor-mesh                          ["mesh"]                                    ["my-model-predictor.my-namespace","my-model-predictor.my-namespace.svc","my-model-predictor.my-namespace.svc.cluster.local"]                                       19h
my-namespace         my-model-predictor-ingress                       ["knative-serving/knative-local-gateway"]   ["my-model-predictor.my-namespace","my-model-predictor.my-namespace.svc","my-model-predictor.my-namespace.svc.cluster.local"]                                       19h
my-namespace         my-model                                         ["knative-serving/knative-local-gateway"]   ["my-model.my-namespace.svc.cluster.local"]                                                                                                                         19h

No kubeflow/kubeflow-gateway specified for version 1.8.
How can this be fixed?

[yurkoff-mv]

@juliusvonkohout, it seems impossible to do inference from the outside. Virtual Services of Inference Services do not contain a gateway looking out (kubeflow-gateway).
I'm not sure, but maybe it's a missing line local-gateway.mesh: "mesh" in the file net-istio.yaml.

[juliusvonkohout]

Can you join the next manifest WG meeting for discussion? It has been possible before. Can you use path based routing? Why do you not want the oidc-authservice token based authorization?

[yurkoff-mv]

Yes, I can join the discussion.
Previously (in KubeFlow 1.5) this was possible and I used it.
The fact is that a large number of requests end up in the Dex queue, which ultimately breaks it and the cluster.
Most likely the problem is not with Dex bypass, but with the inability to perform Inference from outside the cluster.
I think this issue is related to this.

[ksgnextuple]

Any conclusion to this issue? Looking for the exact same solution where I want to bypass the oidc authuservice when exposing the Inference service to outside the cluster while making use of istio. Facing the same issue where the requests end up going to central dashboard.

[ksgnextuple]

Patching the knative configmap config-domain with a custom domain atleast updates the virtual service so that it makes use of the kubeflow-gateway for external traffic.

[ksgnextuple]

Got it to work with the authentication where I could pass the Cookie authservice_session. But would be better if we could have an option to bypass this.

[yurkoff-mv]

Unfortunately, I was never able to beat Dex in version 1.8. In addition, I was never able to access the InferenceService from the outside.

@ksgnextuple, are you able to at least access InferenceService with a session?

[ksgnextuple]

Yes @yurkoff-mv

[yurkoff-mv]

@ksgnextuple, сan you share the output of the following command?
kubectl get virtualservices -n my-namespace

[ksgnextuple]

@yurkoff-mv I am not using dex, but the oidc-uservice.

Here is the output:

sklearn-v2-iris-predictor-ingress   ["knative-serving/knative-local-gateway","kubeflow/kubeflow-gateway"]   ["sklearn-v2-iris-predictor-dev.example.com","sklearn-v2-iris-predictor.dev","sklearn-v2-iris-predictor.dev.svc","sklearn-v2-iris-predictor.dev.svc.cluster.local"]   5d20h
sklearn-v2-iris-predictor-mesh      ["mesh"]                                                                ["sklearn-v2-iris-predictor.dev","sklearn-v2-iris-predictor.dev.svc","sklearn-v2-iris-predictor.dev.svc.cluster.local"]                                                5d20h

[ksgnextuple]

I also had to update knative config domain, that was the key

kind: ConfigMap
metadata:
  name: config-domain
  namespace: knative-serving
  labels:
    app.kubernetes.io/name: knative-serving
    app.kubernetes.io/component: controller
    app.kubernetes.io/version: "1.10.2"
  annotations:
    knative.dev/example-checksum: "26c09de5"
data:
  example.com: ""
  _example: | ```

[ksgnextuple]

@yurkoff-mv Which ConfigMap are you adding the extension provider? I haven't tried the steps you had mentioned for bypassing auth on my setup yet

[ksgnextuple]

@yurkoff-mv Was able to achieve it. Disabled session token as well for Infer endpoints. Though not recommended for a production setup. We wanted it for our nonprod env.

[yurkoff-mv]

Thank you! It's interesting that kubeflow/kubeflow-gateway doesn't appear in my VirtualService.

@yurkoff-mv Which ConfigMap are you adding the extension provider? I haven't tried the steps you had mentioned for bypassing auth on my setup yet

I didn't change the ConfigMap

@yurkoff-mv Was able to achieve it. Disabled session token as well for Infer endpoints. Though not recommended for a production setup. We wanted it for our nonprod env.

How did you disable the session token?

[ksgnextuple]

@yurkoff-mv The same steps you had mentioned above for disabling the session token. For the kubeflow-gateway to come up the knative configmap needs to be updated in the way I had mentioned. Reference on Knative website -> https://knative.dev/docs/serving/using-a-custom-domain/

[ksgnextuple]

https://kserve.github.io/website/0.10/admin/serverless/serverless/#1-install-knative-serving - Look at the warning here too. The changes was mostly from knative side

[yurkoff-mv]

@ksgnextuple, thank you very much! It worked for me!
It's strange that this behavior is not enabled by default.
For future readers, I will write that in the source code of the inference it is worth replacing authservice_session with oauth2_proxy_kubeflow.
It's strange that these steps are not described in this repository. My problem has been hanging around for several months now and only you could help me. Thank you again!

Now I will try to bypass authorization for the inference service.

[yurkoff-mv]

@ksgnextuple, Could you please provide your ConfigMap for clarity?
kubectl edit configmap istio -n istio-system

[ksgnextuple]

data:
  mesh: |-
    extensionProviders:
    - name: "oidc-auth-provider"
      envoyExtAuthzHttp:
        service: "authservice.istio-system.svc.cluster.local"
        port: "8080"
        includeHeadersInCheck: ["authorization", "cookie", "x-auth-token"]
        headersToUpstreamOnAllow: ["kubeflow-userid"]
    accessLogFile: /dev/stdout
    defaultConfig:
      discoveryAddress: istiod.istio-system.svc:15012
      proxyMetadata: {}
      tracing: {}
    enablePrometheusMerge: false
    rootNamespace: istio-system
    tcpKeepalive:
      interval: 5s
      probes: 3
      time: 10s
    trustDomain: cluster.local
  meshNetworks: 'networks: {}'
kind: ConfigMap

[yurkoff-mv]

It’s similar for me. Only when I try to perform inference I get RBAC: access denied.

curl -v -H "Host: sklearn-iris.kubeflow-user-example-com.example.com" -H "Content-Type: application/json"

"http://localhost:32333/v1/models/sklearn-iris:predict" -d @./iris-input.json
*   Trying 127.0.0.1:32333...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 32333 (#0)
> POST /v1/models/sklearn-iris:predict HTTP/1.1
> Host: sklearn-iris.kubeflow-user-example-com.example.com
> User-Agent: curl/7.68.0
> Accept: */*
> Content-Type: application/json
> Content-Length: 76
> 
* upload completely sent off: 76 out of 76 bytes
* Mark bundle as not supporting multiuse
< HTTP/1.1 403 Forbidden
< content-length: 19
< content-type: text/plain
< date: Thu, 04 Apr 2024 06:33:56 GMT
< server: istio-envoy
< connection: close
< 
* Closing connection 0
RBAC: access denied

It’s also strange that there was no EnvoyFilter authn-filter by default.

[ksgnextuple]

I had to remove the authn-filter Envoy filter. This RBAC access denied i think seems to be triggered by the global deny auth policy. Not too sure though.

[yurkoff-mv]

@ksgnextuple, thank you for the help.
It didn’t work for me because by default OAuth2-proxy is used as authorization instead of OIDC AuthService. The instructions given at the beginning of the topic work for OIDC AuthService. Therefore, you need to either immediately deploy KubeFlow with OIDC AuthService, or think about how to bypass OAuth2-proxy. There is no EnvoyFilter authn-filter in OAuth2-proxy. In theory, it would be enough to deploy the two above AuthorizationPolicies. Only in the first one replace dex-auth-provider with oauth2-proxy. But for some reason this doesn't work.

This is what the Extension Provider looks like for OAuth2-proxy:

    extensionProviders:
    - envoyExtAuthzHttp:
        headersToDownstreamOnDeny:
        - content-type
        - set-cookie
        headersToUpstreamOnAllow:
        - authorization
        - path
        - x-auth-request-email
        - x-auth-request-groups
        - x-auth-request-user
        includeRequestHeadersInCheck:
        - authorization
        - cookie
        service: oauth2-proxy.oauth2-proxy.svc.cluster.local
        port: 80
      name: oauth2-proxy

Its definition can be found in the following yaml file.

[yurkoff-mv]

Yes, I did it!
As a result, with OAuth2-proxy it was even easier to bypass Dex...
You just need to edit AuthorizationPolicy istio-ingressgateway-oauth2-proxy, adding rules to it:
kubectl edit AuthorizationPolicy -n istio-system istio-ingressgateway-oauth2-proxy
It should look like this:

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: istio-ingressgateway-oauth2-proxy
  namespace: istio-system
spec:
  action: CUSTOM
  provider:
    name: oauth2-proxy
  selector:
    matchLabels:
      app: istio-ingressgateway
  rules:
  - to:
    - operation:
        notPaths: ["/v1*"]

For testing in Python:

KUBEFLOW_ENDPOINT = "http://127.0.0.1:32333"     # Cluster IP and port
KUBEFLOW_USERNAME = "user@example.com"
KUBEFLOW_PASSWORD = "12341234"
MODEL_NAME = "sklearn-iris"
SERVICE_HOSTNAME = "sklearn-iris.kubeflow-user-example-com.example.com"
PREDICT_ENDPOINT = f"{KUBEFLOW_ENDPOINT}/v1/models/{MODEL_NAME}:predict"
iris_input = {"instances": [[6.8, 2.8, 4.8, 1.4], [6.0, 3.4, 4.5, 1.6]]}
res = requests.post(
    url=PREDICT_ENDPOINT,
    headers={"Host": SERVICE_HOSTNAME, "Content-Type": "application/json"},
    # cookies=jar,
    json=iris_input,
    timeout=200,
)
print("Status Code: ", res.status_code)
print("Response: ", res.json())

Output:

Status Code:  200
Response:  {'predictions': [1, 1]}

Alternative for curl in console...
Create file iris-input.json:

cat <<EOF > "./iris-input.json"                                                                                                                                                           
{
  "instances": [
    [6.8,  2.8,  4.8,  1.4],
    [6.0,  3.4,  4.5,  1.6]
  ]
}                           
EOF

Request:

curl -v -H "Host: sklearn-iris.kubeflow-user-example-com.example.com" -H "Content-Type: application/json" 

Response:

"http://127.0.0.1:32333/v1/models/sklearn-iris:predict" -d @./iris-input.json
*   Trying 127.0.0.1:32333...
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 32333 (#0)
> POST /v1/models/sklearn-iris:predict HTTP/1.1
> Host: sklearn-iris.kubeflow-user-example-com.example.com
> User-Agent: curl/7.68.0
> Accept: */*
> Content-Type: application/json
> Content-Length: 76
> 
* upload completely sent off: 76 out of 76 bytes
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< content-length: 21
< content-type: application/json
< date: Fri, 05 Apr 2024 05:22:58 GMT
< server: istio-envoy
< x-envoy-upstream-service-time: 4
< 
* Connection #0 to host 127.0.0.1 left intact
{"predictions":[1,1]}

[juliusvonkohout]

@yurkoff-mv @ksgnextuple this looks more like introducing a security flaw than using proper tokens from serviceaccounts with oauth2-proxy. I am very open to merge something with proper authentication.

CC @kromanow94

[ksgnextuple]

@juliusvonkohout It was actually just more of a 'is it possible scenario' to bypass auth, ideally not recommended off course. And the changes we had to do were at auth, Isio and Knative level.

[kromanow94]

I agree with Julius that this is making the setup less secure but if you really want to do this, the AuthorizationPolicy provided in this comment #2575 (comment) would be the way to go. In general, using AuthorizationPolicies is the preferred way of configuring authorization with the recent changes for Authz introduced in #2544.

But, please consider using M2M tokens, functionality natively integrated with oauth2-proxy/istio, also configured as part of latest changes around Authz.

Considering the above example in python, it should be enough to do:

export TOKEN="$(kubectl -n kubeflow-user-example-com create token default-editor)"
export AUTHORIZATION = "Bearer: ${TOKEN}"

AUTHORIZATION = os.getenv("AUTHORIZATION")
KUBEFLOW_ENDPOINT = "http://127.0.0.1:32333"     # Cluster IP and port
MODEL_NAME = "sklearn-iris"
SERVICE_HOSTNAME = "sklearn-iris.kubeflow-user-example-com.example.com"
PREDICT_ENDPOINT = f"{KUBEFLOW_ENDPOINT}/v1/models/{MODEL_NAME}:predict"
iris_input = {"instances": [[6.8, 2.8, 4.8, 1.4], [6.0, 3.4, 4.5, 1.6]]}
res = requests.post(
    url=PREDICT_ENDPOINT,
    headers={"Host": SERVICE_HOSTNAME, "Content-Type": "application/json", "Authorization": AUTHORIZATION},
    json=iris_input,
    timeout=200,
)
print("Status Code: ", res.status_code)
print("Response: ", res.json())

With the above, you should be able to just use the latest master version of kubeflow/manifests, most probably without additional changes around istio and knative. I welcome you to try and provide feedback in this setup.

[juliusvonkohout]

I agree with Julius that this is making the setup less secure but if you really want to do this, the AuthorizationPolicy provided in this comment #2575 (comment) would be the way to go. In general, using AuthorizationPolicies is the preferred way of configuring authorization with the recent changes for Authz introduced in #2544.

But, please consider using M2M tokens, functionality natively integrated with oauth2-proxy/istio, also configured as part of latest changes around Authz.

Considering the above example in python, it should be enough to do:

export TOKEN="$(kubectl -n kubeflow-user-example-com create token default-editor)"
export AUTHORIZATION = "Bearer: ${TOKEN}"

AUTHORIZATION = os.getenv("AUTHORIZATION")
KUBEFLOW_ENDPOINT = "http://127.0.0.1:32333"     # Cluster IP and port
MODEL_NAME = "sklearn-iris"
SERVICE_HOSTNAME = "sklearn-iris.kubeflow-user-example-com.example.com"
PREDICT_ENDPOINT = f"{KUBEFLOW_ENDPOINT}/v1/models/{MODEL_NAME}:predict"
iris_input = {"instances": [[6.8, 2.8, 4.8, 1.4], [6.0, 3.4, 4.5, 1.6]]}
res = requests.post(
    url=PREDICT_ENDPOINT,
    headers={"Host": SERVICE_HOSTNAME, "Content-Type": "application/json", "Authorization": AUTHORIZATION},
    json=iris_input,
    timeout=200,
)
print("Status Code: ", res.status_code)
print("Response: ", res.json())

With the above, you should be able to just use the latest master version of kubeflow/manifests, most probably without additional changes around istio and knative. I welcome you to try and provide feedback in this setup.

@diegolovison is this something for the /contrib/kserve documentation?

[diegolovison]

Looking at https://www.kubeflow.org/docs/external-add-ons/kserve/first_isvc_kserve/ I don't see any reference about token. Any idea why ?

[kromanow94]

@diegolovison probably because this is something considered to be relatively new. The setup with tokens and oauth2-proxy was introduced in #2544

[diegolovison]

maybe if we document the secured way and have a note about how to disable it?

[yurkoff-mv]

"Authorization"

The method didn't work for me. Redirects to http://127.0.0.1:32333/dex/auth?approval_prompt=force&client_id=kubeflow-oidc-authservice&code_challenge=C0155PrMpdTfBQl_e-_8MNDGph-JACiFFforSNGulGg&code_challenge_method=S256&redirect_uri=%2 Foauth2%2Fcallback&response_type=code&scope=profile+email+groups+openid&state=GEEipbxgOIyZ -TD71Dqi9E74OQ6oI6SdtvEkZIynS_Y%3A%2Fv1%2Fmodels%2Fsklearn-iris%3Apredict

[kromanow94]

I did some investigation and I found out that this is because the VirtualServices created by kserve are configured by default to use cluster-local-gateway. istio-ingressgateway is configured with AuthorizationPolicy istio-ingressgateway-oauth2-proxy which enforces the traffic to go through oauth2-proxy. There is no such AuthorizationPolicy for cluster-local-gateway.

So, I see two options:

1. Configure Istio auth for current setup with cluster-local-gateway
   1. Create cluster-local-gateway-oauth2-proxy AuthorizationPolicy to enforce authentication with oauth2-proxy:

      apiVersion: security.istio.io/v1
      kind: AuthorizationPolicy
      metadata:
        name: cluster-local-gateway-oauth2-proxy
        namespace: istio-system
      spec:
        action: CUSTOM
        provider:
          name: oauth2-proxy
        rules:
        - {}
        selector:
          matchLabels:
            app: cluster-local-gateway

   2. Depending on your setup, if the model is deployed in Kubeflow managed namespace (KF Profile, for example kubeflow-user-example-com), you also have to configure access to the sklearn-iris deployment:

      apiVersion: security.istio.io/v1beta1
      kind: AuthorizationPolicy
      metadata:
        name: sklearn-iris-predictor-allow
        namespace: kubeflow-user-example-com
      spec:
        selector:
          matchLabels:
            serving.knative.dev/service: sklearn-iris-predictor
        action: ALLOW
        rules:
        - {}

   3. Testing with curl:

      $ curl -XPOST -v "http://sklearn-iris.kubeflow-user-example-com.svc.cluster.local/v1/models/sklearn-iris:predict" -H "Authorization: Bearer $(cat /run/secrets/kubernetes.io/serviceaccount/token)" -d '{"instances": [[6.8,  2.8,  4.8,  1.4], [6.0,  3.4,  4.5,  1.6]]}' -H "Content-Type: application/json"
      Note: Unnecessary use of -X or --request, POST is already inferred.
      * Host sklearn-iris.kubeflow-user-example-com.svc.cluster.local:80 was resolved.
      * IPv6: (none)
      * IPv4: 172.20.1.23
      *   Trying 172.20.1.23:80...
      * Connected to sklearn-iris.kubeflow-user-example-com.svc.cluster.local (172.20.1.23) port 80
      > POST /v1/models/sklearn-iris:predict HTTP/1.1
      > Host: sklearn-iris.kubeflow-user-example-com.svc.cluster.local
      > User-Agent: curl/8.7.1
      > Accept: */*
      > Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6Ikh3ZUQ2enNYYnRZNUFZQk8xX1ZKc3ZCZGwwRmR3dTdwRURiQXpDN3c5MncifQ.eyJhdWQiOlsiaHR0cHM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwiLCJodHRwczovL2t1YmVybmV0ZXMuZGVmYXVsdC5zdmMiLCJodHRwczovL2t1YmVybmV0ZXMuZGVmYXVsdCJdLCJleHAiOjIwMjg1NDA4NzAsImlhdCI6MTcxMzE4MDg3MCwiaXNzIjoiaHR0cHM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwiLCJrdWJlcm5ldGVzLmlvIjp7Im5hbWVzcGFjZSI6Imt1YmVmbG93LXVzZXItZXhhbXBsZS1jb20iLCJwb2QiOnsibmFtZSI6ImN1cmwiLCJ1aWQiOiI3ZGI2ZjliNC0zZTliLTQ3ZDUtOWI4ZC0yZjhiMWVkNTVhZjkifSwic2VydmljZWFjY291bnQiOnsibmFtZSI6ImRlZmF1bHQtZWRpdG9yIiwidWlkIjoiODZhZmM3OGYtMTIzYS00MDMwLWI5YjQtZDllYWQ5YmE2NTc4In19LCJuYmYiOjE3MTMxODA4NzAsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDprdWJlZmxvdy11c2VyLWV4YW1wbGUtY29tOmRlZmF1bHQtZWRpdG9yIn0.iY9WY7vqFQvxv3mzFYlnKQ3arG631movAfkIM1eWH_UdsQuWUupIz7wak81pOM23gBPpYxMT5HR1ZgVHYWG07Neh4e1ySUzhmPNNfydSIs-jUP1P8BjEPq3BdSQ9j_1pGggMDXFM4msFnEdAjlmpl23yDKOoJCj0RDV3fZIiA-mf7wLyiv_E38ah1ygZXYjrTCdzstCH02aZ7VCLc1dPETttE7nlF3YoaurwHJzZF6WHXmQlVdU2yMg0RT8uRDBUDI6WTq_guxjuEBEJrj166pXbp1MBvslMBUYXPV3StQ-AXnvQUyCBDoa5NOlJKOht3UOhGeS_-1A50ctjsl8xKw
      > Content-Type: application/json
      > Content-Length: 65
      > 
      * upload completely sent off: 65 bytes
      < HTTP/1.1 200 OK
      < content-length: 21
      < content-type: application/json
      < date: Mon, 15 Apr 2024 12:47:28 GMT
      < server: envoy
      < x-envoy-upstream-service-time: 9
      < 
      * Connection #0 to host sklearn-iris.kubeflow-user-example-com.svc.cluster.local left intact
      {"predictions":[1,1]}

2. Change the kserve config to use istio-ingressgateway instead of cluster-local-gateway. This touches kserve which I don't have a lot of experience with. I tried changing the inferenceservice-config ConfigMap to define the "localGatewayService": "istio-ingressgateway.istio-system.svc.cluster.local" and "localGateway": "kubeflow/kubeflow-gateway" but that didn't work for some reason, probably something is missing...

@juliusvonkohout do you think we should add this AuthorizationPolicy for cluster-local-gateway to the manifests?

[juliusvonkohout]

@juliusvonkohout do you think we should add this AuthorizationPolicy for cluster-local-gateway to the manifests?

Yes we should do so, because at some point this should be possible out of the Box. I have to deep dive into the gateways and kserve again, but maybe that is something for 1.10, because i will be very busy wit other stuff over the next weeks, especially GSOC.

[yurkoff-mv]

@kromanow94, This is don't working for external access. In general, it is necessary to add an initial Knative configuration to access scripts from outside the cluster. I don't see the point in using a cluster locally.
Knative ConfigMap:

kind: ConfigMap
metadata:
  name: config-domain
  namespace: knative-serving
  labels:
    app.kubernetes.io/name: knative-serving
    app.kubernetes.io/component: controller
    app.kubernetes.io/version: "1.10.2"
data:
  example.com: ""

Please try:

$ curl -XPOST -v "http:127.0.0.1:xxxxx/v1/models/sklearn-iris:predict" -H "Authorization: Bearer $(cat /run/secrets/kubernetes.io/serviceaccount/token)" -d '{"instances": [[6.8,  2.8,  4.8,  1.4], [6.0,  3.4,  4.5,  1.6]]}' -H "Content-Type: application/json"

[yurkoff-mv]

My example:

cpu-1@cpu-1:~$ curl -v -H "Host: sklearn-iris.kubeflow-user-example-com" -H "Content-Type: application/json"  -H "Authorization: Bearer ${TOKEN}" "http://127.0.0.1:32333/v1/models/sklearn-iris:predict" -d @./iris-input.json
*   Trying 127.0.0.1:32333...
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 32333 (#0)
> POST /v1/models/sklearn-iris:predict HTTP/1.1
> Host: sklearn-iris.kubeflow-user-example-com
> User-Agent: curl/7.68.0
> Accept: */*
> Content-Type: application/json
> Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IjVZaUxvLU53LVVxaDZDZnpaY1Jja2tudjNmWEZtY01CNTRkTUNPMUxuaVUifQ.eyJhdWQiOlsiaHR0cHM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjIl0sImV4cCI6MTcxMzE5NTUzMSwiaWF0IjoxNzEzMTkxOTMxLCJpc3MiOiJodHRwczovL2t1YmVybmV0ZXMuZGVmYXVsdC5zdmMiLCJrdWJlcm5ldGVzLmlvIjp7Im5hbWVzcGFjZSI6Imt1YmVmbG93LW1lZ2FwdXRlciIsInNlcnZpY2VhY2NvdW50Ijp7Im5hbWUiOiJkZWZhdWx0LWVkaXRvciIsInVpZCI6IjAxYTRjMjk1LWY0NmEtNGQzMC1hNTI0LTg0MTc0NTAyNTI3ZiJ9fSwibmJmIjoxNzEzMTkxOTMxLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6a3ViZWZsb3ctbWVnYXB1dGVyOmRlZmF1bHQtZWRpdG9yIn0.ToJ5e4nuMZnVf5Ru3asirs0nj_yhXm9yY-oHxyGWj6Ned8epzmh1TbIApzaurTSkQ3fBVbZ8QCHsFcot3fuNMVYOsWduEO1oAk80sa8t2Nu5aJxe1wmUKpNPZwLQq_Pe9bP1JBKhb3BKsRzpD6lFCugi6QPnz0lA1_jGCdLxEqwa-PNdbl7ta1Xi2SfH2-xmqpzbs0N-3larEZkOQpLL3hLcXjsO6Z6I81m_KHKbJvLBIx3mOTyivAT1GmhBApulFr60cIZaljGUDNqgn9r-1ogVqlrhZyHCJKITmGZsFtDdbyOuJ-cWOlkuEuDTG7w5pBvwYd2ksUByk7gvyfIkmQ
> Content-Length: 76
> 
* upload completely sent off: 76 out of 76 bytes
* Mark bundle as not supporting multiuse
< HTTP/1.1 302 Found
< location: /dex/auth?approval_prompt=force&client_id=kubeflow-oidc-authservice&code_challenge=MuFB9QUDGAzgqlbk1Ym2LSsIvqrEuE-I2Mho1fKANt8&code_challenge_method=S256&redirect_uri=%2Foauth2%2Fcallback&response_type=code&scope=profile+email+groups+openid&state=fgOjHnFlKUFLXwr5RGOVAe_X6EndbfbUxIQlt2-HfkM%3A%2Fv1%2Fmodels%2Fsklearn-iris%3Apredict
< set-cookie: oauth2_proxy_kubeflow_csrf=bR-mOan5LNHd5EUL18WmNdc35zMHs3fhYbqQrk_wz90pxPs9yuXtuBG-cWKn3Lp_IE2fY4IrmcjgxSzH8L-GO_zaEHpJVssxvpwCSMaMn0Cuga4Q6L__Y07_RcbTYG6gHSt1vQ6Ki5fYaYa2SRLvqhefDARii20u2vfXc3Jz4AZJltZmB-345OlzxQ-VhOHxTBz2ofBzGghZLRQ07GiFRGzn4HNCMPJ7-O8lpX4WR0GDhRN9or-m3HSoJZ3dYA==|1713192017|pUiorzMPjysl3iLQX8bIt_qz4updR4OdYVpSefconHQ=; Path=/; Expires=Mon, 15 Apr 2024 14:55:17 GMT; HttpOnly
< date: Mon, 15 Apr 2024 14:40:17 GMT
< server: istio-envoy
< content-length: 0
< 
* Connection #0 to host 127.0.0.1 left intact

[juliusvonkohout]

you also have to expose kserve with different routing and probably an additional virtual service. Because the hostname based kserve routing does not work trough the ingressgateway. This is how i got it working years ago. But like i said i might come back to this in proper detail over the next months.

[muratyarali]

Thanks Guys for the info. This worked for me.

• First of all, We should patch the knative configmap config-domain with a custom domain. Configure config-domain as described below

kubectl edit cm -n knative-serving config-domain

...
data:
  www.example.com: ""
...

• Add Dex extension provider to the istio configmap:

kubectl edit cm -n istio-system istio

data:
  mesh: |-
    extensionProviders:
    - name: "dex-auth-provider"
      envoyExtAuthzHttp:
        service: "authservice.istio-system.svc.cluster.local"
        port: "8080"
        includeHeadersInCheck: ["authorization", "cookie", "x-auth-token"]
        headersToUpstreamOnAllow: ["kubeflow-userid"]
    accessLogFile: /dev/stdout

• Create two AuthorizationPolicy:

kubectl apply -f authorizationpolicy.yaml

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: dex-auth
  namespace: istio-system
spec:
  selector:
    matchLabels:
      istio: ingressgateway
  action: CUSTOM
  provider:
    # The provider name must match the extension provider defined in the mesh config.
    name: dex-auth-provider
  rules:
  # The rules specify when to trigger the external authorizer.
  - to:
    - operation:
        notPaths: ["/v1*"]

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: allow-inference-services
  namespace: istio-system
spec:
  selector:
    matchLabels:
  action: ALLOW
  rules:
  - from:
    - source:
        principals: ["cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account"]
  - to:
    - operation:
        methods: ["POST"]
        paths: ["/v1*"]

• Remove the filter

kubectl delete -n istio-system envoyfilters.networking.istio.io authn-filter

• Rollout deployment/istiod

kubectl rollout restart deployment/istiod -n istio-system

[jxdn]

Thanks Guys for the info. This worked for me.

• First of all, We should patch the knative configmap config-domain with a custom domain. Configure config-domain as described below

kubectl edit cm -n knative-serving config-domain

...
data:
  www.example.com: ""
...

• Add Dex extension provider to the istio configmap:

kubectl edit cm -n istio-system istio

data:
  mesh: |-
    extensionProviders:
    - name: "dex-auth-provider"
      envoyExtAuthzHttp:
        service: "authservice.istio-system.svc.cluster.local"
        port: "8080"
        includeHeadersInCheck: ["authorization", "cookie", "x-auth-token"]
        headersToUpstreamOnAllow: ["kubeflow-userid"]
    accessLogFile: /dev/stdout

• Create two AuthorizationPolicy:

kubectl apply -f authorizationpolicy.yaml

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: dex-auth
  namespace: istio-system
spec:
  selector:
    matchLabels:
      istio: ingressgateway
  action: CUSTOM
  provider:
    # The provider name must match the extension provider defined in the mesh config.
    name: dex-auth-provider
  rules:
  # The rules specify when to trigger the external authorizer.
  - to:
    - operation:
        notPaths: ["/v1*"]

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: allow-inference-services
  namespace: istio-system
spec:
  selector:
    matchLabels:
  action: ALLOW
  rules:
  - from:
    - source:
        principals: ["cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account"]
  - to:
    - operation:
        methods: ["POST"]
        paths: ["/v1*"]

• Remove the filter

kubectl delete -n istio-system envoyfilters.networking.istio.io authn-filter

• Rollout deployment/istiod

kubectl rollout restart deployment/istiod -n istio-system

this will open all the the whole kubeflow dashboard.

is thre any solution just only open the inference servive only

[juliusvonkohout]

Lets continue with secure stuff in #2811
I will close issues where people recommend to disable security.

[jaffe-fly]

Whether the request was successfully received outside the cluster inferenceservices

[juliusvonkohout]

#2811 i will lock this issue here.