Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Faster activation of Python environments such as Conda #8342

Merged
merged 8 commits into from
Nov 24, 2021

Conversation

DonJayamanne
Copy link
Contributor

@DonJayamanne DonJayamanne commented Nov 23, 2021

For #7849

  • Get activated environment variable activation ourselves (& not using Python extension)
  • Note: We still rely on python extension to generate the commands used to activate an environment

Here are some stats:

  • Non-Conda - Our code takes 300-400ms and Python extension takes 1300-1500ms
  • Conda - Our code takes 5s with conda and Python extension takes 10s

So we're definitely saving a few seconds

@@ -10,7 +10,7 @@ export const IEnvironmentActivationService = Symbol('IEnvironmentActivationServi
export interface IEnvironmentActivationService {
getActivatedEnvironmentVariables(
resource: Resource,
interpreter?: PythonEnvironment,
interpreter: PythonEnvironment,
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Its never undefined, hence (all of these optionals make the code lot more messy).
Fortunately most of them aren't optional anymore (simply because we were always passing the values).

@codecov-commenter
Copy link

codecov-commenter commented Nov 23, 2021

Codecov Report

Merging #8342 (8d3cef9) into main (14f28cd) will increase coverage by 0%.
The diff coverage is 75%.

❗ Current head 8d3cef9 differs from pull request most recent head b8b7101. Consider uploading reports for the commit b8b7101 to get more accurate results

@@          Coverage Diff           @@
##            main   #8342    +/-   ##
======================================
  Coverage     71%     71%            
======================================
  Files        373     375     +2     
  Lines      23554   23718   +164     
  Branches    3619    3646    +27     
======================================
+ Hits       16881   17038   +157     
+ Misses      5211    5209     -2     
- Partials    1462    1471     +9     
Impacted Files Coverage Δ
src/client/api/types.ts 100% <ø> (ø)
src/client/common/installer/productInstaller.ts 51% <0%> (+1%) ⬆️
src/client/interpreter/activation/types.ts 100% <ø> (ø)
src/client/telemetry/index.ts 63% <ø> (ø)
src/client/common/process/currentProcess.ts 50% <50%> (ø)
src/client/common/process/interpreterActivation.ts 75% <75%> (ø)
src/client/api/pythonApi.ts 68% <100%> (-1%) ⬇️
src/client/api/serviceRegistry.ts 100% <100%> (ø)
src/client/common/process/proc.ts 70% <100%> (+5%) ⬆️
src/client/common/process/processFactory.ts 90% <100%> (+<1%) ⬆️
... and 12 more

@@ -239,6 +236,7 @@ export class PythonInstaller implements IPythonInstaller {
@inject(IMemento) @named(GLOBAL_MEMENTO) private readonly memento: Memento
) {}

@traceDecorators.verbose('Installing Product', TraceOptions.Arguments | TraceOptions.BeforeCall)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

More logging

@@ -289,34 +287,6 @@ export class PythonInstaller implements IPythonInstaller {
}
}

// eslint-disable-next-line max-classes-per-file
@injectable()
export class EnvironmentActivationService implements IEnvironmentActivationService {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moved to a separate file (logic is more complex now)

@@ -78,7 +78,7 @@ export type PythonApi = {
*/
getActivatedEnvironmentVariables(
resource: Resource,
interpreter?: PythonEnvironment,
interpreter: PythonEnvironment,
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is never undefined, we always pass this around, hence changed this to not be nullable.
Makes the code cleaner

cancel?: CancellationToken,
reInstallAndUpdate?: boolean,
installPipIfRequired?: boolean
): Promise<InstallerResponse> {
// Precondition
if (isResource(interpreterUri)) {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This would never happen with our code path, hence removed this, typings ensure this never happens.
I think this was a carry on from when we moved code from Python repo

});
return;
}
traceVerbose(`Activation Commands received ${activationCommands} for shell ${shellInfo.shell}`);
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Borrowed this logic from Python extension

pythonEnvType: envType,
source: 'jupyter',
failed: true,
reason: 'noActivationCommands'
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Better telemetry, this way when things fall over we know what could be causing issues with kernel startups.
E.g. if we cannot activate a conda enviornment, then we know for certain that this is the cause for kernel startup to fail.

* We've found that doing this in jupyter yields much better results.// Copyright (c) Microsoft Corporation. All rights reserved.
* Stats: In Jupyter activation takes 800ms & the same in Python would take 2.6s, or with a complex Conda (5s vs 9s).
* Note: We cache the activate commands, as this is not something that changes day to day. Its almost a constant.
* Either way, we always fetch the latest from Python extension & update the cache.
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comments added,

@DonJayamanne DonJayamanne changed the title Faster env activation Faster activation of Python environments such as Conda Nov 23, 2021
@DonJayamanne DonJayamanne marked this pull request as ready for review November 24, 2021 00:25
@DonJayamanne DonJayamanne requested a review from a team as a code owner November 24, 2021 00:25
@DonJayamanne DonJayamanne merged commit e737ea1 into main Nov 24, 2021
@DonJayamanne DonJayamanne deleted the fasterEnvActivation branch November 24, 2021 00:25
}
}
if (!envVariablesOurSelves.resolved) {
traceVerbose(`Got env vars with python ext faster ${getDisplayPath(interpreter?.path)}`);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we should add telemetry to track this?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants