E2E test for jupyter web app

[jlewi]

We should have E2E tests for the Jupyter web app using a framework like selenium.

See kubeflow/testing#288

[issue-label-bot]

Issue-Label Bot is automatically applying the label improvement/enhancement to this issue, with a confidence of 0.95. Please mark this comment with 👍 or 👎 to give our bot feedback!

Links: app homepage, dashboard and code for this bot.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[kimwnasptd]

I can try and take that one.

I'll start with some basic end-to-end cases such as creating and deleting Notebooks and will gradually try to add more advanced ones.

/assign kimwnasptd

[jlewi]

@kimwnasptd any update on this?

[jlewi]

Bump to P0 because this is a requirement for 1.0.

[kimwnasptd]

@jlewi I'm reading the e2e instructions in the testing repo and I have some questions.

1. Does each test (.py file) need to be encapsulated in an argo workflow in order to be triggered from prow?
   • If yes, where should this bundle (.py + argo app) live in the code base?
   • Will the argo workflow also need to be added to the repo's prow_config.yaml to be run as a postsubmit job?
2. The test code will need to have access to a kubeflow cluster with the latest jupyter-web-app image deployed. How will it know this endpoint? I see the testing#273 but I'm not sure of its current state.
3. Will the test code be responsible for setting up the kubeflow cluster or is this automated and a kubeflow endpoint could be provided as an ENV Var (for example) in the container that will run the test?

[jlewi]

Does each test (.py file) need to be encapsulated in an argo workflow in order to be triggered from prow?
If yes, where should this bundle (.py + argo app) live in the code base?

An E2E test will be an Argo workflow that contains steps for building and deploying the application and then testing it. The invocation of the python code (e.g. pytests) to test your application will be steps in this workflow.

• The code for the primary E2E test for kubeflow is defined in kfctl_e2e_workflow.py

• So you can use that as the basis for defining an E2E workflow for the jupyter web app

• If you want to create a presubmit test that you would write an E2E workflow that would have Argo steps for the following

  • Build the jupyter web app
  • Modify the manifests to use your new image
  • Deploy Kubeflow using the updated image
  • Run one or my pytests to verify the jupyter web app is working

• You will like want to define this workflow in the same repository as the source code; i.e. kubeflow/klubeflow

  • Python code should go under py/kubeflow/${REPO} to allow consistent absolute imports
  • So you'd probably put it in py/kubeflow/kubeflow/ci

• Once you have a python file you can add it to the prow file.

• Writing a postsubmit test might be a little easier because you could run the test against the most recently deployed cluster from master. In which case you wouldn't have to worry updating the manifests or deploying the Kubeflow clusters.

  • You can model this on create_e2e_workflow.py which is a test for our examples

    • In this case the test has a step to get credentials to talk to the latest cluster deployed from master

It might be useful to discuss at the community meeting or in an out of band meeting.

[jlewi]

@kimwnasptd any update on this?

[jlewi]

@kimwnasptd any update on this?

[fediazgon]

If @kimwnasptd is Ok with it I can start working on this. I've seen there are a lot of area/jupyter tickets in KF1.0 that could be easily tested if we have a suite in place.

I've already started to refactor https://github.com/jlewi/kubeflow/blob/1690703d261d29ccd7e1d5ff154d79064473d9d8/kubeflow/jupyter/tests/jupyter_test.py#L85

[kimwnasptd]

Just finished the basic structure of the test. It has two test cases for now:

1. Check that the index page can load and doesn't have a popup error
2. Check that the form page can load and doesn't have a popup error

Lets first add these tests to have something to work with and we can then further expand them.

@jlewi IIUC the python test file will be in the kubeflow/kfctl repo, under the py/kubeflow/kfctl/testing/pytests/ directory correct?

[jlewi]

@kimwnasptd You could probably put it in kubeflow/kfctl or kubeflow/kubeflow depending on what is easier. The e2e test will be split across those repos since it depends on code in multiple repos.

[jlewi]

@kimwnasptd Any update on this? Do you have a link to your PR?

@fediazgon I think what you are working on is complementary. https://github.com/jlewi/kubeflow/blob/1690703d261d29ccd7e1d5ff154d79064473d9d8/kubeflow/jupyter/tests/jupyter_test.py#L85 is testing that Jupyter itself is accessible; i.e. its testing that the notebook controller correctly sets up ISTIO reverse proxy routes for notebooks.

I think @kimwnasptd is testing the UI for launching notebooks (i.e. creating Notebook resources).

I think we need both. So if you want to go ahead and work on testing the Jupyter Notebook UI that would be a huge help. Perhaps we should open up a separate issue for that?

[fediazgon]

@kimwnasptd Any update on this? Do you have a link to your PR?

@fediazgon I think what you are working on is complementary. https://github.com/jlewi/kubeflow/blob/1690703d261d29ccd7e1d5ff154d79064473d9d8/kubeflow/jupyter/tests/jupyter_test.py#L85 is testing that Jupyter itself is accessible; i.e. its testing that the notebook controller correctly sets up ISTIO reverse proxy routes for notebooks.

I think @kimwnasptd is testing the UI for launching notebooks (i.e. creating Notebook resources).

I think we need both. So if you want to go ahead and work on testing the Jupyter Notebook UI that would be a huge help. Perhaps we should open up a separate issue for that?

I was working on the Argo workflow that you mentioned in this comment to wrap @kimwnasptd tests. My idea was to add a flag in create_unique_kf_instance to allow creating a deployment but overriding some images. Then, test this deployment in the pytests.

[jlewi]

@fediazgon replied in slack; my suggestion would be to start by running in postsubmits and periodic tests run against our auto-deployed clusters so that we don't have to modify create_unique_kf_instance. I added some pointers in slack. We can continue to discuss there.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[jlewi]

@kimwnasptd What's the status of E2E testing for the jupyter web app? Are we running and reporting the e2e test results?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[jlewi]

@kimwnasptd and @StefanoFioravanzo any update on this?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[jlewi]

@kubeflow/wg-notebook-leads Any update on this?

[thesuperzapper]

I will raise this in the notebook working group slack channel.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[kimwnasptd]

/reopen
/lifecycle frozen

[google-oss-robot]

@kimwnasptd: Reopened this issue.

In response to this:

/reopen
/lifecycle frozen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[juliusvonkohout]

/transfer notebooks

[juliusvonkohout]

/close
as decided by @andreyvelich @tarilabs in the migration document

[google-oss-prow]

@juliusvonkohout: Closing this issue.

In response to this:

/close
as decided by @andreyvelich @tarilabs in the migration document

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.