Improve the perf of functional tests running with real jupyter #10242
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| const globalStorage = this.serviceManager.get<Memento>(IMemento, GLOBAL_MEMENTO); | ||
| const localStorage = this.serviceManager.get<Memento>(IMemento, WORKSPACE_MEMENTO); | ||
|
|
||
| // Create a custom persistent state factory that remembers specific things between tests |
There was a problem hiding this comment.
Seems kinda risky, hope this doesn't break anything. Guess we'll find out in our tests.
E.g. setting something into persistent state and checking that flag to change behaviour of code.
DonJayamanne
approved these changes
Feb 21, 2020
Codecov Report
@@ Coverage Diff @@
## master #10242 +/- ##
==========================================
+ Coverage 61.23% 61.23% +<.01%
==========================================
Files 567 567
Lines 30862 30928 +66
Branches 4432 4441 +9
==========================================
+ Hits 18897 18939 +42
- Misses 10982 11001 +19
- Partials 983 988 +5
Continue to review full report at Codecov.
|
| if (!deferred || ignoreCache) { | ||
| deferred = createDeferred<PythonInterpreter[]>(); | ||
| this.promisesPerResource.set(cacheKey, deferred); | ||
| if (Array.isArray(cachedInterpreters)) { |
There was a problem hiding this comment.
Why are these checking for Array here specifically? If there was a refactor where .getCachedInterpreters returned a class wrapping the array list of Interpreters we would want to return that.
IanMatthewHuff
approved these changes
Feb 21, 2020
…as it was before)
|
Kudos, SonarCloud Quality Gate passed!
|
|
@DonJayamanne and @IanMatthewHuff can you guys take another look? Thanks |
DonJayamanne
added a commit
that referenced
this pull request
Feb 24, 2020
* fixing a typo in CONTRIBUTING.md (#10044) * fix for button backgrounds (#10234) * add kernel connection sys info string (#10236) * Fix to return env variables of interpreter that is not current interpreter (#10251) For #10250 * Bug fix * Support jupyter output in the remote scenario too (#10241) * Add some output for remote situations * Add news entry * Skip flaky FS test (#10244) Merely disables test for #10240 * Switch experiments settings scope to machine (#10237) * Switch experiments settings scope to machine * Add news item * Show quickfixes for launch.json (#10245) * Added implementation * Added tests * Moved into providers * Add code actions vscode mock and convert .test.ts into unit.test.ts * Rename folders * Dispose registered services * News entry * Added unit tests * Rename core service * Reduce the number of build agents used by CI. (#10221) * Drop extra jobs from the PR-validation pipeline. This reduces the number of build agents we are using most frequently, without sacrificing much coverage. Note that this relies on 2 things: * sufficient unit test coverage * manual "full CI" runs when there may be OS-specific concerns With this change: * all tests get run on 3.x on linux (including smoke) * for 2.7 only the unit and functional tests are run (and only on linux) * on Windows and OSX, only functional and "single workspace" tests are run The total number of test jobs (and hence agents) drops from 13 to 8. * Drop extra jobs from the PR-merge pipeline. This change relies on the nightly CI run covering the full matrix. The total number of test jobs (and hence agents) drops from 39 to 24. * Limit # of parallel jobs in each job matrix in the PR-merge and nightly pipelines. This reduces the max number of agents used by the PR-merge pipeline to 8 (from 24). For the nightly pipeline it goes down from 84 to 12. Both will take longer to complete, but that shouldn't be a major problem. * Add a "manual build" CI pipeline for faster turnaround. * Do not test internal tools on lower than Python 3.7. * Drop the temporary workaround jobs. * (again) Do not test internal tools on lower than Python 3.7. * Improve the perf of functional tests running with real jupyter (#10242) * Improve the perf of subsequent tests by caching interpreters * Add back the nightly flake * Remove coverage * Add news entry * Use a static map to allow promise to be cleared on new interpreters (as it was before) Co-authored-by: Panpan Lin <panpanli@ie.ibm.com> Co-authored-by: Ian Huff <ian.huff@gmail.com> Co-authored-by: Rich Chiodo <rchiodo@users.noreply.github.com> Co-authored-by: Karthik Nadig <kanadig@microsoft.com> Co-authored-by: Kartik Raj <karraj@microsoft.com> Co-authored-by: Eric Snow <ericsnowcurrently@gmail.com>
DonJayamanne
added a commit
that referenced
this pull request
Feb 24, 2020
* fixing a typo in CONTRIBUTING.md (#10044) * fix for button backgrounds (#10234) * add kernel connection sys info string (#10236) * Fix to return env variables of interpreter that is not current interpreter (#10251) For #10250 * Bug fix * Support jupyter output in the remote scenario too (#10241) * Add some output for remote situations * Add news entry * Skip flaky FS test (#10244) Merely disables test for #10240 * Switch experiments settings scope to machine (#10237) * Switch experiments settings scope to machine * Add news item * Show quickfixes for launch.json (#10245) * Added implementation * Added tests * Moved into providers * Add code actions vscode mock and convert .test.ts into unit.test.ts * Rename folders * Dispose registered services * News entry * Added unit tests * Rename core service * Reduce the number of build agents used by CI. (#10221) * Drop extra jobs from the PR-validation pipeline. This reduces the number of build agents we are using most frequently, without sacrificing much coverage. Note that this relies on 2 things: * sufficient unit test coverage * manual "full CI" runs when there may be OS-specific concerns With this change: * all tests get run on 3.x on linux (including smoke) * for 2.7 only the unit and functional tests are run (and only on linux) * on Windows and OSX, only functional and "single workspace" tests are run The total number of test jobs (and hence agents) drops from 13 to 8. * Drop extra jobs from the PR-merge pipeline. This change relies on the nightly CI run covering the full matrix. The total number of test jobs (and hence agents) drops from 39 to 24. * Limit # of parallel jobs in each job matrix in the PR-merge and nightly pipelines. This reduces the max number of agents used by the PR-merge pipeline to 8 (from 24). For the nightly pipeline it goes down from 84 to 12. Both will take longer to complete, but that shouldn't be a major problem. * Add a "manual build" CI pipeline for faster turnaround. * Do not test internal tools on lower than Python 3.7. * Drop the temporary workaround jobs. * (again) Do not test internal tools on lower than Python 3.7. * Improve the perf of functional tests running with real jupyter (#10242) * Improve the perf of subsequent tests by caching interpreters * Add back the nightly flake * Remove coverage * Add news entry * Use a static map to allow promise to be cleared on new interpreters (as it was before) * Merge release branch into master (#10258) * Davidkutu/ds interactive window focus fix (#9895) * Fixed the focus on the interactive window when pressing ctrl +2 * the new prop 'focusPending' needs to change in order to give focus * added functional test * In test: force focus to buttons to take it away from * finished the functional test * updated changelog * deleted bug#.md * Fix variables on mac and linux (#9897) (#9903) * Fix regexes to work on other platforms * Fix data frame viewer * Fix linter problems Co-authored-by: Don Jayamanne <don.jayamanne@yahoo.com> Co-authored-by: Don Jayamanne <don.jayamanne@yahoo.com> * Fix focusing of cells when navigating cells using up/down arrow (#9885) (#9908) For #9884 * Disable webHostNotebook experiment and fix name of exp (#9894) (#9907) Disable an experiment Fix name of an existing experiment * Davidkutu/ds scrapbook render fix (#9912) * check if we support mimetypes before loading html to avoid crashing * Silently skip rendering of the scapbook mime types * added news file * updated changelog and deleted news file * Disable the terminal activation experiment (#9937) Disable until next release of VSC (for hidden terminals). Disabled because a terminal gets displayed to the user and then hidden (doesn't happen in insiders, we will need to wait for their new API to hide terms) * Notebook should use kernel from metadata (#9936) For #9927 * Notebook should use kernel from metadata * Fix formatting * Port server cache fix to release (#9932) * Port variable restart fix (#9938) * Fix variable explorer when restarting a kernel (#9931) * Make sure to clear variable list on reset * Add news entry * Update changelog * Update changelog * Do not override kernels info in other notebooks (#9958) For #9951, #9950, #9949 * Do not override kernels info in other notebooks * Fixes * Disables the use of a terminal to activate an environment (#9967) * Disables the use of a terminal to activate an environment * Fix tests * Fix linter * Port server name fix to release (#9970) * When starting Jupyter activate conda env (#9972) * If interpreter is provided use that to check if is conda environment * Fix tests For #9973 Currently conda isn't getting activated when auto starting Jupyter server. * Porting K8 fix to release (#9987) * Update version and changelog date for release (#10022) * Ignore FS errors from stat() in some cases. (#10032) (for #9901) The earlier FS code ignored errors in a couple of situations. Recent changes inadvertently stopped ignoring them. This change restores the earlier behavior (and adds logging of the errors). This is a backport of #10021. * Update chang log (#10041) * Install jupyter instead of installing kernel spec (#10080) (#10083) * Install jupyter instead of installing kernel spec For #10071 * Fix for zh-tw loading issue. (#10085) (#10089) * Update version and changelog for point release. (#10091) * Update version and changelog for point release. * Add note to change log. * Port fixes for second point release. (#10146) * Better messaging on notebook fail (#10056) * Install jupyter instead of installing kernel spec (#10080) * Install jupyter instead of installing kernel spec For #10071 * Eliminate variable value when computing data frame info (#10081) * Fix ndarray types to be viewable again (#10093) * Eliminate variable value when computing data frame info * Fix ndarrays to work again Add test to make sure we don't regress this again * Rchiodo/kernel telemetry (#10115) * Add duration to select local/remote kernel * Add notebook language telemetry * Add news entries * #9883 telemetry * News entry * Another spot for kernel spec failure * Add telemetry on product install * Fix install telemetry * Undo launch.json change * Handle other cases * Better way to handle case * Wrong event for jupyter install * Fix unit tests * Clear variables when restarting regardless if visible or not (#10117) * Use different method for checking if kernelspec is available (#10114) * Use different method for checking if kernelspec is available * Fix unit tests * More logging for kernelspec problems (#10132) * More logging for kernelspec problems * Actually capture the exception on the new code * Not actually using output if first exception still there. * Actually only return output on one of the expected calls. * Fix nightly flake * Check our saved jupyter interpreters before allowing any of them to be used as active interpreters (#10113) * Update changelog and package.json * Missing part of changelog * Fix tests and linter problems Co-authored-by: Ian Huff <ian.huff@gmail.com> Co-authored-by: Don Jayamanne <don.jayamanne@yahoo.com> * Cherry pick more fixes for feb point2 (#10183) * Ensure DS auto start code is non-blocking (#10171) * Perf fixes * Add news entry * Include interperter name in message (#10174) For #10071 Fix to ensure interpreter is included in message, here's the actual message seen by the user. Error: Jupyter cannot be started. Error attempting to locate jupyter: 'Kernelspec' module not installed in the selected interpreter ({0}). Note the {0} * Ignore display name when searching interpreters (#10175) Partial fix for #10173 Comparing against path of interpreter alone is sufficient. Figured there's no harm in trying to minimise occurrences of dup kernels (basically just compare against python executable path, as thats sufficient and always accurate) Display name: Can change from version to version of Python extension (i.e. it shouldn't have been used as a unique identifier) Display name can change after extension loads and more information about interpreter is available. * Track cold/warm times to execute notebook cells (#10180) For #10176 * fix SVG mimetype name (#10181) (#10185) * Update change log * Some more ds telemetry (#10218) * Add telemetry to capture perceived time taken to start jupyter and run cell (#10217) * Add telemetry * Only for nb For #10098 For #10212 * Fix change log * Fix linter * Update date in change log * Point release with fix for Issue 10250 (#10254) * Fix to return env variables of interpreter that is not current interpreter (#10251) For #10250 * Bug fix * Update change log * Fix date * Oops * Fix linter * Clean up * More clean up * Some more text fixes Co-authored-by: David Kutugata <dakutuga@microsoft.com> Co-authored-by: Rich Chiodo <rchiodo@users.noreply.github.com> Co-authored-by: Don Jayamanne <don.jayamanne@yahoo.com> Co-authored-by: Ian Huff <ian.huff@gmail.com> Co-authored-by: Eric Snow <ericsnowcurrently@gmail.com> Co-authored-by: Panpan Lin <panpanli@ie.ibm.com> Co-authored-by: Ian Huff <ian.huff@gmail.com> Co-authored-by: Rich Chiodo <rchiodo@users.noreply.github.com> Co-authored-by: Karthik Nadig <kanadig@microsoft.com> Co-authored-by: Kartik Raj <karraj@microsoft.com> Co-authored-by: Eric Snow <ericsnowcurrently@gmail.com> Co-authored-by: David Kutugata <dakutuga@microsoft.com>
10 tasks
DonJayamanne
added a commit
that referenced
this pull request
Feb 25, 2020
* fixing a typo in CONTRIBUTING.md (#10044) * fix for button backgrounds (#10234) * add kernel connection sys info string (#10236) * Fix to return env variables of interpreter that is not current interpreter (#10251) For #10250 * Bug fix * Support jupyter output in the remote scenario too (#10241) * Add some output for remote situations * Add news entry * Skip flaky FS test (#10244) Merely disables test for #10240 * Switch experiments settings scope to machine (#10237) * Switch experiments settings scope to machine * Add news item * Show quickfixes for launch.json (#10245) * Added implementation * Added tests * Moved into providers * Add code actions vscode mock and convert .test.ts into unit.test.ts * Rename folders * Dispose registered services * News entry * Added unit tests * Rename core service * Reduce the number of build agents used by CI. (#10221) * Drop extra jobs from the PR-validation pipeline. This reduces the number of build agents we are using most frequently, without sacrificing much coverage. Note that this relies on 2 things: * sufficient unit test coverage * manual "full CI" runs when there may be OS-specific concerns With this change: * all tests get run on 3.x on linux (including smoke) * for 2.7 only the unit and functional tests are run (and only on linux) * on Windows and OSX, only functional and "single workspace" tests are run The total number of test jobs (and hence agents) drops from 13 to 8. * Drop extra jobs from the PR-merge pipeline. This change relies on the nightly CI run covering the full matrix. The total number of test jobs (and hence agents) drops from 39 to 24. * Limit # of parallel jobs in each job matrix in the PR-merge and nightly pipelines. This reduces the max number of agents used by the PR-merge pipeline to 8 (from 24). For the nightly pipeline it goes down from 84 to 12. Both will take longer to complete, but that shouldn't be a major problem. * Add a "manual build" CI pipeline for faster turnaround. * Do not test internal tools on lower than Python 3.7. * Drop the temporary workaround jobs. * (again) Do not test internal tools on lower than Python 3.7. * Improve the perf of functional tests running with real jupyter (#10242) * Improve the perf of subsequent tests by caching interpreters * Add back the nightly flake * Remove coverage * Add news entry * Use a static map to allow promise to be cleared on new interpreters (as it was before) * Merge release branch into master (#10258) * Davidkutu/ds interactive window focus fix (#9895) * Fixed the focus on the interactive window when pressing ctrl +2 * the new prop 'focusPending' needs to change in order to give focus * added functional test * In test: force focus to buttons to take it away from * finished the functional test * updated changelog * deleted bug#.md * Fix variables on mac and linux (#9897) (#9903) * Fix regexes to work on other platforms * Fix data frame viewer * Fix linter problems Co-authored-by: Don Jayamanne <don.jayamanne@yahoo.com> Co-authored-by: Don Jayamanne <don.jayamanne@yahoo.com> * Fix focusing of cells when navigating cells using up/down arrow (#9885) (#9908) For #9884 * Disable webHostNotebook experiment and fix name of exp (#9894) (#9907) Disable an experiment Fix name of an existing experiment * Davidkutu/ds scrapbook render fix (#9912) * check if we support mimetypes before loading html to avoid crashing * Silently skip rendering of the scapbook mime types * added news file * updated changelog and deleted news file * Disable the terminal activation experiment (#9937) Disable until next release of VSC (for hidden terminals). Disabled because a terminal gets displayed to the user and then hidden (doesn't happen in insiders, we will need to wait for their new API to hide terms) * Notebook should use kernel from metadata (#9936) For #9927 * Notebook should use kernel from metadata * Fix formatting * Port server cache fix to release (#9932) * Port variable restart fix (#9938) * Fix variable explorer when restarting a kernel (#9931) * Make sure to clear variable list on reset * Add news entry * Update changelog * Update changelog * Do not override kernels info in other notebooks (#9958) For #9951, #9950, #9949 * Do not override kernels info in other notebooks * Fixes * Disables the use of a terminal to activate an environment (#9967) * Disables the use of a terminal to activate an environment * Fix tests * Fix linter * Port server name fix to release (#9970) * When starting Jupyter activate conda env (#9972) * If interpreter is provided use that to check if is conda environment * Fix tests For #9973 Currently conda isn't getting activated when auto starting Jupyter server. * Porting K8 fix to release (#9987) * Update version and changelog date for release (#10022) * Ignore FS errors from stat() in some cases. (#10032) (for #9901) The earlier FS code ignored errors in a couple of situations. Recent changes inadvertently stopped ignoring them. This change restores the earlier behavior (and adds logging of the errors). This is a backport of #10021. * Update chang log (#10041) * Install jupyter instead of installing kernel spec (#10080) (#10083) * Install jupyter instead of installing kernel spec For #10071 * Fix for zh-tw loading issue. (#10085) (#10089) * Update version and changelog for point release. (#10091) * Update version and changelog for point release. * Add note to change log. * Port fixes for second point release. (#10146) * Better messaging on notebook fail (#10056) * Install jupyter instead of installing kernel spec (#10080) * Install jupyter instead of installing kernel spec For #10071 * Eliminate variable value when computing data frame info (#10081) * Fix ndarray types to be viewable again (#10093) * Eliminate variable value when computing data frame info * Fix ndarrays to work again Add test to make sure we don't regress this again * Rchiodo/kernel telemetry (#10115) * Add duration to select local/remote kernel * Add notebook language telemetry * Add news entries * #9883 telemetry * News entry * Another spot for kernel spec failure * Add telemetry on product install * Fix install telemetry * Undo launch.json change * Handle other cases * Better way to handle case * Wrong event for jupyter install * Fix unit tests * Clear variables when restarting regardless if visible or not (#10117) * Use different method for checking if kernelspec is available (#10114) * Use different method for checking if kernelspec is available * Fix unit tests * More logging for kernelspec problems (#10132) * More logging for kernelspec problems * Actually capture the exception on the new code * Not actually using output if first exception still there. * Actually only return output on one of the expected calls. * Fix nightly flake * Check our saved jupyter interpreters before allowing any of them to be used as active interpreters (#10113) * Update changelog and package.json * Missing part of changelog * Fix tests and linter problems Co-authored-by: Ian Huff <ian.huff@gmail.com> Co-authored-by: Don Jayamanne <don.jayamanne@yahoo.com> * Cherry pick more fixes for feb point2 (#10183) * Ensure DS auto start code is non-blocking (#10171) * Perf fixes * Add news entry * Include interperter name in message (#10174) For #10071 Fix to ensure interpreter is included in message, here's the actual message seen by the user. Error: Jupyter cannot be started. Error attempting to locate jupyter: 'Kernelspec' module not installed in the selected interpreter ({0}). Note the {0} * Ignore display name when searching interpreters (#10175) Partial fix for #10173 Comparing against path of interpreter alone is sufficient. Figured there's no harm in trying to minimise occurrences of dup kernels (basically just compare against python executable path, as thats sufficient and always accurate) Display name: Can change from version to version of Python extension (i.e. it shouldn't have been used as a unique identifier) Display name can change after extension loads and more information about interpreter is available. * Track cold/warm times to execute notebook cells (#10180) For #10176 * fix SVG mimetype name (#10181) (#10185) * Update change log * Some more ds telemetry (#10218) * Add telemetry to capture perceived time taken to start jupyter and run cell (#10217) * Add telemetry * Only for nb For #10098 For #10212 * Fix change log * Fix linter * Update date in change log * Point release with fix for Issue 10250 (#10254) * Fix to return env variables of interpreter that is not current interpreter (#10251) For #10250 * Bug fix * Update change log * Fix date * Oops * Fix linter * Clean up * More clean up * Some more text fixes Co-authored-by: David Kutugata <dakutuga@microsoft.com> Co-authored-by: Rich Chiodo <rchiodo@users.noreply.github.com> Co-authored-by: Don Jayamanne <don.jayamanne@yahoo.com> Co-authored-by: Ian Huff <ian.huff@gmail.com> Co-authored-by: Eric Snow <ericsnowcurrently@gmail.com> * Disable run/interrupt/restart buttons when switching kernels (#10291) * Support opening spark data frames in the data viewer (#10304) * Support opening spark data frames in the data viewer * Review feedback * Use new Mac OS VM (#10287) * Use new Mac OS VM * News For #10288 * Fix messed up regex to account for empty string form (#10307) * Fix messed up regex to account for empty string form * Fix unit test * More scrollbar fixes and some CSS cleanup (#10300) * Fix scrolling in the interactive window (#10319) Co-authored-by: Panpan Lin <panpanli@ie.ibm.com> Co-authored-by: Ian Huff <ian.huff@gmail.com> Co-authored-by: Rich Chiodo <rchiodo@users.noreply.github.com> Co-authored-by: Karthik Nadig <kanadig@microsoft.com> Co-authored-by: Kartik Raj <karraj@microsoft.com> Co-authored-by: Eric Snow <ericsnowcurrently@gmail.com> Co-authored-by: David Kutugata <dakutuga@microsoft.com>
DonJayamanne
added a commit
that referenced
this pull request
Feb 25, 2020
* master: Fix scrolling in the interactive window (#10319) More scrollbar fixes and some CSS cleanup (#10300) Fix messed up regex to account for empty string form (#10307) Use new Mac OS VM (#10287) Support opening spark data frames in the data viewer (#10304) Disable run/interrupt/restart buttons when switching kernels (#10291) Merge release branch into master (#10258) Improve the perf of functional tests running with real jupyter (#10242) Reduce the number of build agents used by CI. (#10221) Show quickfixes for launch.json (#10245) Switch experiments settings scope to machine (#10237) Skip flaky FS test (#10244) Support jupyter output in the remote scenario too (#10241) Fix to return env variables of interpreter that is not current interpreter (#10251) add kernel connection sys info string (#10236) fix for button backgrounds (#10234) fixing a typo in CONTRIBUTING.md (#10044)
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
For #7997
Rework the caching algorithm for interpreters (so it doesn't run the fetch at all if the cached value is there) and make a custom cache for tests.
This improves the run time for flakey tests from 3:30 hours to 1 hour on windows. Linux and Mac are similar.