Support for Pyspark interactive

[rchiodo]

Create an extensibility point similar to this:

interface IKernelSpec {
    // Fields of a jupyter kernelspec.json
    // Maybe live id for an already running kernel?
}

interface IKernelProvider {
     getKernels(): Promise<IKernelSpec[]> // This is implicitly local. Maybe it needs something to say that?
}

interface IInteractiveWindow {
    executeCode(code: string, file: Uri | undefined = undefined, line: number = 0, silent: boolean = false);
    readonly kernel: IKernelSpec;
}

interface IPythonExtensionApi {
    registerKernelProvider(provider: IKernelProvider) : Disposable;
    openInteractiveWindow(kernelSpec: IKernelSpec, remoteUri?: string, bearerToken?: string): Promise<IInteractiveWindow>
}

This would be used to

• Let an extension register its own set of kernels (works around Pyspark having to set the Jupyter interpreter to get their kernels listed)
• Let an extension open an interactive window connected to a specific kernel and a specific remote URI (Pyspark needs this in order to run their code)
• Let an extension send code to this interactive window (This would likely be useful for other language extensions)
• Let an extension be able to tell what kernel is currently picked in an interactive window (Pyspark needs this to determine if an interactive window still has their kernel enabled)

[rchiodo]

This should resolve a number of issues for @zesluo:

https://github.com/microsoft/vscode-python/issues/10422
https://github.com/microsoft/vscode-python/issues/10246

[DonJayamanne]

Let an extension send code to this interactive window (This would likely be useful for other language extensions)

Wouldn't it be better for other languages to provide the cell locstions instead? This way we can add the code lenses, and execute things ourselves and support all of the shortcuts. That's how I did it in the past. Is easier for other languages to integrate. And we won't have to expose execution of code.

Woulf be good to know their requirements here as well.

[DonJayamanne]

interface IKernelSpec {
// Fields of a jupyter kernelspec.json
// Maybe live id for an already running kernel?
}

Doesn't the kernel model contain information about a running kernel. Is that what you mean by live Id?
I'd prefer to use the existing Jupyter types where possible.

[DonJayamanne]

Probably good to discuss the requirements and the api, on teams

[rchiodo]

Let an extension send code to this interactive window (This would likely be useful for other language extensions)

Wouldn't it be better for other languages to provide the cell locstions instead? This way we can add the code lenses, and execute things ourselves and support all of the shortcuts. That's how I did it in the past. Is easier for other languages to integrate. And we won't have to expose execution of code.

Woulf be good to know their requirements here as well.

Pyspark team doesn't want cells but rather the ability to send arbitrary code.

[zesluo]

Thanks for your great help! @DonJayamanne , @rchiodo, any news? Please feel free to let me know if more information is needed^_^

[rchiodo]

It's on our list of stuff to do for December.

[zesluo]

Hello @rchiodo , thanks for your reply! would you mind help prioritize the work item? Will this be available within two weeks? We are going to need this by the end of this month. Thanks a lot!

[rchiodo]

I've moved it back to triage to get it in front of more people.

[DonJayamanne]

Suggestions:

• We might want to expose an API so other languages can provide the cell ranges
• We might want to expose an API so other languages can provide the details of code blocks (e.g when hitting ctrl+enter, then where is the next line)
• commenting/converting code blocks to comments

Such an api is sufficient for #273 & #273 (comment) (R language)
This allows us to have a:

• Unique command set for sending lines/codeblocks/cells to interactive window for all languages
• Converting cells to markdown & vice versa will work for all languages
• Majority of the commands will just work.
  Else other languages will need to reimplement the same commands/features (lot more work) & it will no longer be consistent.

[greazer]

@zesluo, FYI, we recently fixed issue #411 to re-enable picking a kernel in the interactive window. I think this will unblock you for the end of the month. If not, please let us know!

[zesluo]

Thank you! I will try it then

[zesluo]

@rchiodo , @DonJayamanne we just found that we cannot have the place to change kernel even with previous version of ms-python. And we don't have any code changes. any ideas about it? Thanks in advance!

[DonJayamanne]

Please could you ensure you have the latest version of VSCode and both extensions.
If it didn't work, please could you provide the logs for the console window (like likes an issue in VSCode, I know we had an issue and it was fixed).

with previous version of ms-python

Please install latest version. A change in new version of VSCode broke the old version.

[zesluo]

Hello @DonJayamanne ,

Thanks for your reply! We notice some changes in the new view version.

1. it needs to install Jupyter extension and ms-python together at same time.
2. By using the command :
   , it can now get the interactive window. Thanks for this support!
   The problem is: when we try to change the kernel, it cannot succeed. The logs are attached. Please feel free to let me know if more information is needed. Thanks!
   1126errorlogs.log

[zesluo]

@DonJayamanne, @rchiodo , Feel free to let me know if more information is needed. Thanks^_^

[DonJayamanne]

The problem is: when we try to change the kernel, it cannot succeed. The logs are attached. Please feel free to let me know if more information is needed. Thanks!

@zesluo please could you file a separate issue for the kernel not starting up.

[zesluo]

Sure! Here it is ,#4055. Thanks

[DavidKutu]

@zesluo feel free to reopen this or create a new issue if its not fixed.