Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add tensorboards-web-app.yaml networkpolicy #2304

Merged
merged 3 commits into from
Nov 16, 2022

Conversation

juliusvonkohout
Copy link
Member

@kimwnasptd users complained that this one is missing so i am adding it.

Checklist:

  • Unit tests pass:
    Make sure you have installed kustomize == 3.2.1
    1. make generate-changed-only
    2. make test

@juliusvonkohout juliusvonkohout changed the title Add tensorboards-web-app.yaml networkpolicy WIP Add tensorboards-web-app.yaml networkpolicy Oct 21, 2022
@kimwnasptd
Copy link
Member

@juliusvonkohout is there any reason this is still WIP? It LGTM

@juliusvonkohout
Copy link
Member Author

@kimwnasptd. i discussed it a lot with @TobiasGoerke and he said that there are even more problems. He told me that the istio sidecar is explicitly disabled for tensorboads-web-application and there is no istio authorizationpolicy. He told me that it just checks the easily fakeable userid header. So yes, we can merge that, but it does not improve security much until there is a istio sidecar and only traffic from centraldashboard is allowed via an authorizationpolicy, as it is the case for the other web applications.

@juliusvonkohout juliusvonkohout changed the title WIP Add tensorboards-web-app.yaml networkpolicy Add tensorboards-web-app.yaml networkpolicy Nov 2, 2022
@juliusvonkohout
Copy link
Member Author

kustomize build "https://github.com/kubeflow/manifests/example" | grep "name: tensorboards-web-app-deployment" -B 6 -A 15

shows a disabled istio sidecar

[~]$ kustomize build "https://github.com/kubeflow/manifests/example" | grep "name: tensorboards-web-app-deployment" -B 6 -A 15
2022/11/02 08:35:51 nil value at `valueFrom.configMapKeyRef.name` ignored in mutation attempt
2022/11/02 08:35:51 nil value at `valueFrom.secretKeyRef.name` ignored in mutation attempt
2022/11/02 08:35:51 well-defined vars that were never replaced: kfp-app-name,kfp-app-version
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: tensorboards-web-app
    kustomize.component: tensorboards-web-app
  name: tensorboards-web-app-deployment
  namespace: kubeflow
spec:
  replicas: 1
  selector:
    matchLabels:
      app: tensorboards-web-app
      kustomize.component: tensorboards-web-app
  template:
    metadata:
      annotations:
        sidecar.istio.io/inject: "false"
      labels:
        app: tensorboards-web-app
        kustomize.component: tensorboards-web-app
    spec:

@TobiasGoerke
Copy link

@kimwnasptd. i discussed it a lot with @TobiasGoerke and he said that there are even more problems. He told me that the istio sidecar is explicitly disabled for tensorboads-web-application and there is no istio authorizationpolicy. He told me that it just checks the easily fakeable userid header. So yes, we can merge that, but it does not improve security much until there is a istio sidecar and only traffic from centraldashboard is allowed via an authorizationpolicy, as it is the case for the other web applications.

FYI: the tensoboards-web-app is copied off of the volumes-web-app, which consequentially has the same vulnerability.
These web-apps use ensure_authorized, which calls get_username. This in return simply checks the userid header to create the SubjectAccessReview which, as @juliusvonkohout said, can be overwritten without an istio sidecar to spoof requests.

Slightly related to this / security: naming tensorboards the same as notebooks leads to resources like virtualservices being overwritten due to naming conflicts, causing either the notebook or tensorboard to crash. I'll open a separate issue later.

@juliusvonkohout
Copy link
Member Author

juliusvonkohout commented Nov 2, 2022

@TobiasGoerke Yes i can an easily fake my userid, this is why the pipeline authorizationpolicies block such nonsense and only allow the userid header when it comes directly from the dashboard or the ingressgateway that sets it in a secure manner.

@DomFleischmann this is becoming quite important for 1.7

@kimwnasptd
Copy link
Member

@TobiasGoerke @juliusvonkohout we'll need to ensure that:

  1. The web apps have a sidecar
  2. Only the IGW can talk to the apps

This way only Istio's IGW will be able to talk with the apps that are kubeflow-userid header-sensitive. If someone would like to have other in-cluster pods to talk to the web apps then they would also need to ensure that there's an AuthorizationPolicy in place for allowing only the IGW to set this header

I'll create a tracking issue for this to ensure the apps under kubeflow/kubeflow/components all have sidecars, so that we can unblock this effort as well

@juliusvonkohout
Copy link
Member Author

juliusvonkohout commented Nov 8, 2022

@kimwnasptd now that #6702 is merged this here can be merged as well

@kimwnasptd
Copy link
Member

/lgtm
/approve

@google-oss-prow
Copy link

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: juliusvonkohout, kimwnasptd

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@google-oss-prow google-oss-prow bot merged commit b11f8f1 into kubeflow:master Nov 16, 2022
kevin85421 pushed a commit to juliusvonkohout/manifests that referenced this pull request Feb 28, 2023
* Create tensorboards-web-app

* Update kustomization.yaml

* Rename tensorboards-web-app to tensorboards-web-app.yaml
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants