diff --git a/.gitignore b/.gitignore index 9384f959..394cf7dd 100644 --- a/.gitignore +++ b/.gitignore @@ -1,2 +1,6 @@ coverage_unit.txt jupyter-apis +.vscode +.env +.task +__debug_bin \ No newline at end of file diff --git a/README.md b/README.md index 05b15734..755c0185 100644 --- a/README.md +++ b/README.md @@ -9,6 +9,13 @@ A Golang replacement for the **[Kubeflow][kubeflow]** Jupyter Web APIs. See **[CONTRIBUTING.md](CONTRIBUTING.md)** ## Development Environment +***Note that the frontend will report errors when calling `/api/namespaces` when run locally. This*** +***issue does not arise in production, as the `/api/namespaces` endpoint is unused.*** + +To initialize the `.env` file for the development environment, use `task env`. +You will need to fill out your kubeflow cloud account and kubeflow namespace information manually. +The `thunder-tests` folder contains configuration for testing requests against the backend. Use the `vscode` +`THUNDER CLIENT` extension to load the tests. ### Run API Server @@ -19,15 +26,42 @@ context. See _Connecting a Kubeflow cluster_ below for options. 2. Change directory to project root: `cd jupyter-apis` 3. Run `go run . -spawner-config samples/spawner_ui_config.yaml` +Alternatively, + +1. `task go:dev -w -- -spawner-config samples/spawner_ui_config.yaml` will live-reload the Go server upon changes. + +_Recommended_ + +You can use the vscode debugger to run the backend, just copy the below contents to a file at path `.vscode/launch.json`. +```json +{ + "version": "0.2.0", + "configurations": [ + { + "name": "Debug jupyter-api backend", + "type": "go", + "request": "launch", + "mode": "debug", + "program": ".", + "args": [ + "-spawner-config", + "samples/spawner_ui_config.yaml", + ], + "envFile": "${workspaceFolder}/.env" + } + ] +} +``` + ### Run Front-End The front-end is configured to proxy requests to the local API server. It requires an environment variable (`KF_USER_ID`) to specify the current user – this is passed to the API server as an HTTP header. - The following can be pasted in a script and executed. This uses the latest node lts version(v16.16.0). **NOTE**: `user` is when using vagrant. Use the email adress if it is the dev cluser (please never connect to prod directly) + ``` cd frontend/common/kubeflow-common-lib npm i @@ -44,6 +78,7 @@ KF_USER_ID=user npm start For the kubecost data to be retrievable, the following will need to be executed `kubectl port-forward -n kubecost-system deployment/kubecost-cost-analyzer 9090` ### Older instructions + 1. ~Change directory to front-end folder: `cd frontend`~ 2. ~Install dependencies: `npm install`~ 3. ~Run the front-end `KF_USER_ID= npm start`~ @@ -103,3 +138,31 @@ and run `vagrant up`. [go]: https://golang.org/dl/ [kubeflow]: https://github.com/kubeflow/kubeflow + +## Whats Different? + +Routes are defined in this repository [here](./main.go). + +[Upstream](https://github.com/kubeflow/kubeflow/tree/v1.6.0/components/crud-web-apps/jupyter/backend/apps/common/routes), the endpoints are structures via request type (e.g. `GET`, `PUT`, `DELETE`). + +_Note_ + +- _that not all endpoints are included in the golang implementation_ +- _to find the upstream endpoint, load the [Upstream](https://github.com/kubeflow/kubeflow/tree/v1.6.0/components/crud-web-apps/jupyter/backend/apps/common/routes) + and use search with the endpoint text!_ + +| Request Type | Golang Endpoint | Upstream Python Endpoint | Purpose | +| ------------ | ------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------- | +| GET | /api/config | [/api/config](https://github.com/kubeflow/kubeflow/blob/v1.6.0/components/crud-web-apps/jupyter/backend/apps/common/routes/get.py#L9) | | +| GET | /api/gpus | [/api/gpus](https://github.com/kubeflow/kubeflow/blob/v1.6.0/components/crud-web-apps/jupyter/backend/apps/common/routes/get.py#L52) | | +| GET | /api/storageclasses/default | [/api/storageclasses/default](https://github.com/kubeflow/kubeflow/blob/v1.6.0/components/crud-web-apps/common/backend/kubeflow/kubeflow/crud_backend/routes/get.py#L26) | | +| GET | /api/namespaces/{namespace}/cost/aggregated | Not found | Get the aggregated kubecost | +| GET | /api/namespaces | [/api/namespaces](https://github.com/kubeflow/kubeflow/blob/v1.6.0/components/crud-web-apps/common/backend/kubeflow/kubeflow/crud_backend/routes/get.py#L10) | Get the list of namespaces | +| GET | /api/namespaces/{namespace} | Not found | Get namespace metadata | +| GET | /api/namespaces/{namespace}/notebooks | [/api/namespaces/\/notebooks](https://github.com/kubeflow/kubeflow/blob/v1.6.0/components/crud-web-apps/jupyter/backend/apps/common/routes/get.py#L44) | Get the list of notebooks | +| POST | /api/namespaces/{namespace}/notebooks | [/api/namespaces/\/notebooks](https://github.com/kubeflow/kubeflow/blob/v1.6.0/components/crud-web-apps/jupyter/backend/apps/default/routes/post.py#L11) | Create a notebook | +| DELETE | /api/namespaces/{namespace}/notebooks/{notebook} | [/api/namespaces/\/notebooks/](https://github.com/kubeflow/kubeflow/blob/v1.6.0/components/crud-web-apps/jupyter/backend/apps/common/routes/delete.py#L9) | Update a notebook | +| PATCH | /api/namespaces/{namespace}/notebooks/{notebook} | [/api/namespaces/\/notebooks//pvc](https://github.com/kubeflow/kubeflow/blob/v1.6.0/components/crud-web-apps/jupyter/backend/apps/common/routes/get.py#L15) | List `PVC`s | +| DELETE | /api/namespaces/{namespace}/pvcs/{pvc} | [/api/namespaces/\/pvcs/](https://github.com/kubeflow/kubeflow/blob/v1.6.0/components/crud-web-apps/volumes/backend/apps/default/routes/delete.py#L11) | Delete a `PVC` | +| GET | /api/namespaces/{namespace}/poddefaults | [/api/namespaces/\/poddefaults](https://github.com/kubeflow/kubeflow/blob/v1.6.0/components/crud-web-apps/jupyter/backend/apps/common/routes/get.py#L25) | Get `PodDefault`s for a given namespace | diff --git a/Taskfile.yml b/Taskfile.yml index cc8c3adc..32d3e961 100644 --- a/Taskfile.yml +++ b/Taskfile.yml @@ -75,7 +75,13 @@ tasks: desc: Push the built container image cmds: - docker push {{.CONTAINER_IMAGE}} - + go:dev: + desc: run go code locally for development + cmds: + - go run . {{.CLI_ARGS}} + - sleep 1 + sources: + - "*.go" # Source: https://github.com/arduino/tooling-project-assets/blob/main/workflow-templates/assets/go-task/Taskfile.yml go:build: desc: Build the Go code @@ -163,3 +169,14 @@ tasks: desc: stop knative environment cmds: - "k3d cluster stop {{.CLUSTER_NAME}}" + + env: + prefix: ⚙ > env + desc: setup dev environment + cmds: + - echo "KF_NAMESPACE=">>.env + - echo "KF_USER_ID=">>.env + - echo "KF_NOTEBOOK_NAME=swag">>.env + - echo "KF_PVC_NAME=swag">>.env + sources: + - .env \ No newline at end of file diff --git a/access.go b/access.go index c28f5c29..20124007 100644 --- a/access.go +++ b/access.go @@ -74,7 +74,7 @@ func (s *server) checkAccess(subjectAccessReviewTemplate authorizationv1.Subject // If the user is not permitted, log and return the error and do not process the request if !resp.Status.Allowed { - msg := fmt.Sprintf("User %s is not permitted to %s %s.%s.%s for namespace: %s", sar.Spec.User, sar.Spec.ResourceAttributes.Verb, sar.Spec.ResourceAttributes.Group, sar.Spec.ResourceAttributes.Version, sar.Spec.ResourceAttributes.Resource, sar.Spec.ResourceAttributes.Namespace) + msg := fmt.Sprintf("User %s is not permitted to %s %s.%s.%s for namespace: '%s'", sar.Spec.User, sar.Spec.ResourceAttributes.Verb, sar.Spec.ResourceAttributes.Group, sar.Spec.ResourceAttributes.Version, sar.Spec.ResourceAttributes.Resource, sar.Spec.ResourceAttributes.Namespace) log.Println(msg) diff --git a/main.go b/main.go index 8fa40f49..de2dd0c0 100644 --- a/main.go +++ b/main.go @@ -11,7 +11,6 @@ import ( "os/signal" "path" "path/filepath" - "sync" "time" kubeflowv1 "github.com/StatCan/kubeflow-apis/apis/kubeflow/v1" @@ -54,8 +53,6 @@ type clientsets struct { } type server struct { - mux sync.Mutex - Config Configuration clientsets clientsets @@ -126,7 +123,7 @@ func main() { log.Fatal(err) } - err = s.setupListers(gctx) + _ = s.setupListers(gctx) // Generate the Gorilla Mux router router := mux.NewRouter() @@ -163,8 +160,8 @@ func main() { Spec: authorizationv1.SubjectAccessReviewSpec{ ResourceAttributes: &authorizationv1.ResourceAttributes{ Group: corev1.SchemeGroupVersion.Group, - Verb: "list", - Resource: "pods", + Verb: "get", + Resource: "namespaces", Version: corev1.SchemeGroupVersion.Version, }, }, diff --git a/thunder-tests/thunderActivity.json b/thunder-tests/thunderActivity.json new file mode 100644 index 00000000..0637a088 --- /dev/null +++ b/thunder-tests/thunderActivity.json @@ -0,0 +1 @@ +[] \ No newline at end of file diff --git a/thunder-tests/thunderCollection.json b/thunder-tests/thunderCollection.json new file mode 100644 index 00000000..d17f7bc3 --- /dev/null +++ b/thunder-tests/thunderCollection.json @@ -0,0 +1,22 @@ +[ + { + "_id": "bf518063-1214-41a0-a602-702e12d98039", + "colName": "Golang Kubeflow", + "created": "2022-09-23T13:47:44.945Z", + "sortNum": 10000, + "folders": [], + "settings": { + "headers": [ + { + "name": "kubeflow-userid", + "value": "{{KF_USER_ID}}" + } + ], + "tests": [], + "options": { + "baseUrl": "http://localhost:5000" + }, + "envId": "0d8a7513-38dd-4e46-93b3-2c0ccb2ee056" + } + } +] \ No newline at end of file diff --git a/thunder-tests/thunderEnvironment.json b/thunder-tests/thunderEnvironment.json new file mode 100644 index 00000000..6493a36e --- /dev/null +++ b/thunder-tests/thunderEnvironment.json @@ -0,0 +1,12 @@ +[ + { + "_id": "0d8a7513-38dd-4e46-93b3-2c0ccb2ee056", + "name": "golang-backend", + "default": true, + "sortNum": 10000, + "created": "2022-09-23T13:47:57.807Z", + "modified": "2022-09-23T13:48:16.638Z", + "data": [], + "envFile": "../.env" + } +] \ No newline at end of file diff --git a/thunder-tests/thunderclient.json b/thunder-tests/thunderclient.json new file mode 100644 index 00000000..dc53943a --- /dev/null +++ b/thunder-tests/thunderclient.json @@ -0,0 +1,258 @@ +[ + { + "_id": "5f88a11f-dc74-4074-8ad4-d82893376a09", + "colId": "bf518063-1214-41a0-a602-702e12d98039", + "containerId": "", + "name": "/api/config", + "url": "/api/config", + "method": "GET", + "sortNum": 10000, + "created": "2022-09-23T13:47:44.950Z", + "modified": "2022-09-23T13:47:44.950Z", + "headers": [], + "params": [], + "tests": [] + }, + { + "_id": "9f5dd3ac-6dfa-4463-aac3-b641d7eb583d", + "colId": "bf518063-1214-41a0-a602-702e12d98039", + "containerId": "", + "name": "/api/gpus", + "url": "/api/gpus", + "method": "GET", + "sortNum": 15000, + "created": "2022-09-23T13:47:44.951Z", + "modified": "2022-09-23T13:47:44.951Z", + "headers": [], + "params": [], + "tests": [] + }, + { + "_id": "2b047a64-db79-4a13-afb6-4c6e82453c98", + "colId": "bf518063-1214-41a0-a602-702e12d98039", + "containerId": "", + "name": "/api/storageclasses/default", + "url": "http://localhost:5000/api/storageclasses/default", + "method": "GET", + "sortNum": 16250, + "created": "2022-09-23T13:47:44.952Z", + "modified": "2022-09-23T13:47:44.952Z", + "headers": [], + "params": [], + "tests": [] + }, + { + "_id": "74116746-f35f-4308-8f2a-e588b21d0c0e", + "colId": "bf518063-1214-41a0-a602-702e12d98039", + "containerId": "", + "name": "/api/namespaces/{namespace}/cost/aggregated", + "url": "http://localhost:5000/api/namespaces/{namespace}/cost/aggregated", + "method": "GET", + "sortNum": 17500, + "created": "2022-09-23T13:47:44.953Z", + "modified": "2022-09-23T13:47:44.953Z", + "headers": [ + { + "name": "KF_USER_ID", + "value": "christian.boin@cloud.statcan.ca" + } + ], + "params": [ + { + "name": "namespace", + "value": "{{KF_NAMESPACE}}", + "isPath": true + } + ], + "tests": [] + }, + { + "_id": "b330877f-f4e1-4ebe-8e18-ff1e63c0bd90", + "colId": "bf518063-1214-41a0-a602-702e12d98039", + "containerId": "", + "name": "/api/namespaces", + "url": "/api/namespaces", + "method": "GET", + "sortNum": 18750, + "created": "2022-09-23T13:47:44.954Z", + "modified": "2022-09-23T13:47:44.954Z", + "headers": [], + "params": [], + "tests": [] + }, + { + "_id": "82282dcc-de4e-41ce-aeb6-9e293478a60c", + "colId": "bf518063-1214-41a0-a602-702e12d98039", + "containerId": "", + "name": "/api/namespaces/{namespace}", + "url": "/api/namespaces/{namespace}", + "method": "GET", + "sortNum": 60000, + "created": "2022-09-23T13:47:44.955Z", + "modified": "2022-09-23T13:47:44.955Z", + "headers": [], + "params": [ + { + "name": "namespace", + "value": "{{KF_NAMESPACE}}", + "isPath": true + } + ], + "tests": [] + }, + { + "_id": "237a4715-fe2f-4092-a206-fad69c9c6d23", + "colId": "bf518063-1214-41a0-a602-702e12d98039", + "containerId": "", + "name": "/api/namespaces/{namespace}/notebooks", + "url": "/api/namespaces/{namespace}/notebooks", + "method": "GET", + "sortNum": 65000, + "created": "2022-09-23T13:47:44.956Z", + "modified": "2022-09-23T13:47:44.956Z", + "headers": [], + "params": [ + { + "name": "namespace", + "value": "{{KF_NAMESPACE}}", + "isPath": true + } + ], + "tests": [] + }, + { + "_id": "a3f525e5-fb55-4543-a038-ea845d980a2a", + "colId": "bf518063-1214-41a0-a602-702e12d98039", + "containerId": "", + "name": "/api/namespaces/{namespace}/notebooks", + "url": "http://localhost:5000/api/namespaces/{namespace}/notebooks", + "method": "POST", + "sortNum": 70000, + "created": "2022-09-23T13:47:44.957Z", + "modified": "2022-09-23T13:47:44.957Z", + "headers": [], + "params": [ + { + "name": "namespace", + "value": "{{KF_NAMESPACE}}", + "isPath": true + } + ], + "tests": [] + }, + { + "_id": "712b4552-8b7c-482b-8c93-58975015eced", + "colId": "bf518063-1214-41a0-a602-702e12d98039", + "containerId": "", + "name": "/api/namespaces/{namespace}/notebooks/{notebook}", + "url": "http://localhost:5000/api/namespaces/{namespace}/notebooks/{notebook}", + "method": "DELETE", + "sortNum": 80000, + "created": "2022-09-23T13:47:44.958Z", + "modified": "2022-09-23T13:50:19.620Z", + "headers": [], + "params": [ + { + "name": "namespace", + "value": "{{KF_NAMESPACE}}", + "isPath": true + }, + { + "name": "notebook", + "value": "{{KF_NOTEBOOK_NAME}}", + "isPath": true + } + ], + "tests": [] + }, + { + "_id": "c99b0b2b-a7c7-432e-b111-1d6c94b4db0c", + "colId": "bf518063-1214-41a0-a602-702e12d98039", + "containerId": "", + "name": "/api/namespaces/{namespace}/notebooks/{notebook}", + "url": "http://localhost:5000/api/namespaces/{namespace}/notebooks/{notebook}", + "method": "PATCH", + "sortNum": 120000, + "created": "2022-09-23T13:47:44.959Z", + "modified": "2022-09-23T13:50:14.465Z", + "headers": [], + "params": [ + { + "name": "namespace", + "value": "{{KF_NAMESPACE}}", + "isPath": true + }, + { + "name": "notebook", + "value": "{{KF_NOTEBOOK_NAME}}", + "isPath": true + } + ], + "tests": [] + }, + { + "_id": "cfa60ce5-8b79-4e60-80ce-dc47bdc629e1", + "colId": "bf518063-1214-41a0-a602-702e12d98039", + "containerId": "", + "name": "/api/namespaces/{namespace}/pvcs", + "url": "http://localhost:5000/api/namespaces/{namespace}/pvcs", + "method": "GET", + "sortNum": 130000, + "created": "2022-09-23T13:47:44.960Z", + "modified": "2022-09-23T13:47:44.960Z", + "headers": [], + "params": [ + { + "name": "namespace", + "value": "{{KF_NAMESPACE}}", + "isPath": true + } + ], + "tests": [] + }, + { + "_id": "9afbb86c-68a3-488d-bd6d-d84d79ac7dbf", + "colId": "bf518063-1214-41a0-a602-702e12d98039", + "containerId": "", + "name": "/api/namespaces/{namespace}/pvcs/{pvc}", + "url": "http://localhost:5000/api/namespaces/{namespace}/pvcs/{pvc}", + "method": "DELETE", + "sortNum": 140000, + "created": "2022-09-23T13:47:44.961Z", + "modified": "2022-09-23T13:49:54.989Z", + "headers": [], + "params": [ + { + "name": "namespace", + "value": "{{KF_NAMESPACE}}", + "isPath": true + }, + { + "name": "pvc", + "value": "{{KF_PVC_NAME}}", + "isPath": true + } + ], + "tests": [] + }, + { + "_id": "1147c8d9-3ae9-4865-b1c9-509d617bbd58", + "colId": "bf518063-1214-41a0-a602-702e12d98039", + "containerId": "", + "name": "/api/namespaces/{namespace}/poddefaults", + "url": "http://localhost:5000/api/namespaces/{namespace}/poddefaults", + "method": "GET", + "sortNum": 150000, + "created": "2022-09-23T13:47:44.962Z", + "modified": "2022-09-23T13:47:44.962Z", + "headers": [], + "params": [ + { + "name": "namespace", + "value": "{{KF_NAMESPACE}}", + "isPath": true + } + ], + "tests": [] + } +] \ No newline at end of file