Refactor how env variables are loaded and passed into kernels and processes spawned

[DonJayamanne]

We have a lot of places & different ways we load env variables from .env files & merges them with the activated Python envs & then we also get these env variables from the Python extension.

We also have a few issues where these env varaibles are getting cached when they shouldn't or we're not loading the updated .env files.
To get OCAML & a couple of other kernels working we need to load env variables form the shell and merge them again with the .env files.

Before we do that, I think we need to first fix up how env variables are parsed, loaded, merged, cached.

Required for :

• vscode not picking up environment variables (when opened from terminal)
• Kernel issues related to login shells #9682
• Kernel issues related to login shells #9682
• python.envFile is not respected by Python code executed in Notebook and IW #9691 (load env variables from a specific file when dealing with Python kernels)
• Can not reload .env when ZMQ not working #9774
• Environment variables persist after being unset #10392
• Jupyter environment variables load only once ever per kernel. #10755

Other possible issues:

• command not found when I try execute shell script in jupyter cell #9917
• Environment variables persist after being unset #10392
• Changes to .env need a restart of vscode to become active in the Interactive Window #1722
• Environment variables persist after being unset #10392

[DonJayamanne]

Personal notes:

• KernelEnvironmentVariablesService (one method to get kernel env vars)
• IEnvironmentActivationService (one method to get activated Python env vars)
• IEnvironmentVariablesService (helper, parsing .env file and merging vars)
• IEnvironmentVariablesProvider (helper, event when env vars change and ability to get env vars and custom env vars)

PythonApi.getActivatedEnvironmentVariables

• Does this take .env variables into account
• Does this merge process.env into the return value
• Is this cached?
• If user updates .env file, will this be updated?
• Is user updates shell env variables, will this be updated?

KernelEnvironmentVariablesService

• getEnvironmentVariables <Top level>
  • Used in one place to get the environment variables of the kernel when starting raw kernel process.
  • This calls KernelEnvironmentVariablesService.getCustomEnvironmentVariables and IEnvironmentActivationService.getActivatedEnvironmentVariables and merges the two results, also with the process.env.
    TODO: Won't both already have been merged with process.env.
  • Special code for handling PYTHONPATH and PATH|Path variables.
  • Special code for PYTHONNOUSERSITE

EnvironmentVariablesProvider

• getEnvironmentVariables <Top level>

  • Merges the env variables from .env into process.env and returns the merged result.
  • Special code for handling PYTHONPATH and PATH|Path variables.
  • Caches the results in memory and gets cleared when editing .env files.
  • Not cached across vscode sessions.
  • Used in:
    • JupyterPaths.node.ts (reading user env vars)
    • ProcessFactory.node.ts (use this when spawning new processes).
    • PreWarmVariables.node.ts (just calls this method to initialize cache for each workspace folder)
    • EnvironmentActivationService.node.ts <IEnvironmentActivationService>

• getCustomEnvironmentVariables

  • Locates the .env file
  • Parses the .env file using IEnvironmentVariablesService.parseFile
  • Parsed info is returned
  • Used in:
    • KernelEnvironmentVariablesService.getEnvironmentVariables
    • EnvironmentActivationService.getActivatedEnvironmentVariables

EnvironmentActivationService.node.ts

• getActivatedEnvironmentVariables <Top level>
  • Returns the merged variables from .env file and python environment
  • If we fail to activate the Python env then merges the results from getEnvironmentVariables with process.env.
  • TODO: this is pointless as getEnvironmentVariables already merges the results with process.env.
  • We seem to do this in two places within the same file EnvironmentActivationService.node.ts
  • Used in:
    • ModuleInstaller.installModule <IModuleInstaller>,
      • TODO: This also merges the result with process.env variables & a few other things.
    • JupyterInterpreterSubCommandExecutionService.startNotebook <IJupyterSubCommandExecutionService>, and that class also merges the result with process.env variables & a few other things.
    • KernelEnvironmentVariablesService.getEnvironmentVariables already covered above.
    • JupyterKernelService.updateKernelEnvironment <IJupyterKernelService>
      • Creates the env variables for use in kernels spawned using Jupyter.
      • Merges the result with the env variables in the kernelspec.
      • Special code for PYTHONNOUSERSITE
      • Why can't this do the same thing as raw kernels, by using KernelEnvironmentVariablesService.getEnvironmentVariables
    • PythonExecutionFactory.createDaemon
    • PythonExecutionFactory.createActivatedEnvironment
    • PreWarmVariables.node.ts (just calls this method to initialize cache)
• Split into two, one to get env vars from Python ext and one to get ourselves. Right now we have two public methods in there that should be private and its easier to test if separate (at the end of the day they are two separate implementations for the same feature).

Process ctor

• We pass the env variables into Process class for use by all processes spawned using the Process class.
• This is done only in two places:
  • EnvironmentActivationService.node.ts
  • ProcessFactory.node.ts
  • PythonExecutionFactory.node.ts

[DonJayamanne]

Notes:

• When generating kernelspec.json file with env variables, we can have two instances of VSC opened and each with two sepraate folders having two seprate .env files.
  • Either we should re-start Jupyter server when .env files change
  • Or have a way to keep the env variables in kernelspec.json per workspace folder (e.g. separate env_<workspace folder hash> sections in the kernelspec.json file or the like)
  • Or create seprate kernelspec.json files (not best as this could cause other issues as this is cached and used accros vscode sessions and folders)

[connor4312]

Are there any verification steps for this?