Tests

How to run functional tests with Jupyter on CI?

They should run as part of a PR and as part of the insiders build.

How to run functional tests with Jupyter locally?

Launch the debug configuration Functional Tests (without VS Code, *.functional.test.ts) Ensure you update the following environment variables accordingly:

• VSC_PYTHON_ROLLING (Remove the X prefix)
• CI_PYTHON_PATH (Remove the X prefix, & set to fully qualified path to python interpreter
• CI_PYTHON_PATH2 (Remove the X prefix, & set to fully qualified path to python interpreter)
  • This is used for 1-2 tests that require two interpreters. Both of which must have Jupyter dependencies installed.

How to debug output of a failed run.

You might end up with something like so:

That image is showing a single failure in functional test group 4 (right now tests are run in groups, each group on a separate machine)

To see what tests actually failed, go down to the 'Functional Test Group 4' output:

That shows that there was a failure in one of the Variable Explorer tests.

Figuring out what to do about the test failure

If the failure isn't apparent just from the stack shown, you might have to look at the log data. Log data is a little confusing though because tests are running in parallel.

There's a way to split it up by test though. Here's the steps:

1. Download the raw log from the failed job

2. Use the 'logParser.py' file to list test output python pythonFiles\vscode_datascience_helpers\tests\logParser.py <downloaded file path> --testoutput

3. In the output of the logParser, there should be an indicator of the test failure (it will be in red):

4. The test failure should have a process id next to it. In the example above this is 3703.

5. Run the logParser again, but with the --split argument instead of the --testoutput argument. It will split the output into separate files based on the process id.

6. Open the appropriate process id file (in this case the one ending with 3703), and use the console logs to debug the failure.