No autocomplete for pandas column name in Jupyter notebook using bracket notation ['col']

[natwille1]

Environment data

• VS Code version: 1.64.00
• Jupyter Extension version (available under the Extensions sidebar): v2022.1.1001821375
• Python Extension version (available under the Extensions sidebar): v2022.0.1814523869
• OS (Windows | Mac | Linux distro) and version: Windows 10
• Python and/or Anaconda version: python 3.8.12
• Type of virtual environment used (N/A | venv | virtualenv | conda | ...): conda
• Jupyter server running: Local

Expected behaviour

Autocomplete works for pandas dataframes in jupyter notebooks.

Actual behaviour

Autocomplete does not work. Users have to manually type the column names every time. It even doesn't work when I've manually specified the column name several times. I've tried all the language servers, including Jedi, Pylance, and the default option. No autocomplete.

Steps to reproduce:

[NOTE: Self-contained, minimal reproducing code samples are extremely helpful and will expedite addressing your issue]

1. Create pandas dataframe

df = pd.DataFrame({'test': [1,2], 'test2': [2,3]})

2. Call dataframe variable and check if autocomplete works for the columns

df['test']

Logs

Output for Jupyter in the Output panel (View→Output, change the drop-down the upper-right of the Output panel to Jupyter)

XXX

[IanMatthewHuff]

@natwille1 I repro the same. We have . completion after the column is name is completed but no autocompletion for the name itself when typing it as df['column_name']. We do have auto complete for just df.column_name, but that doesn't look like what you are looking for here:

Just to check do you get this ['column_name'] auto complete in a Jupyter notebook outside of vscode? I would assume this is the case, but in the Jupyter install that I have locally I don't see autocompletions for column names, might just be a misconfiguration that I have.

[natwille1]

Hi Ian, thanks for quick response - I do not get autocomplete for df['test'] in a jupyter notebook, but it appears autocomplete in a juypter notebook isn't working for me in general (another issue). I do get completion when using .column_name in a jupyter notebook in vscode, but it's quite slow (takes a few seconds).. what language server do you use?

I also do not get any autocompletion using either method when using the interactive feature of executing jupyter code cells in a python file in vscode

On a side note, I never had problems with autocompletion for the df['test'] or df.test methods in Atom with Hydrogen.. (also inline, interactive cell execution)

Thanks

[IanMatthewHuff]

I have my language server set to pylance. Pylance should be using the python interpreter that you have selected for the current workspace so there could be an issue here if the kernel you have selected doesn't line up with the currently selected python interpreter. For the interactive inline execution were you looking for completions in the .py file itself? Or in the input box of the interactive window? If the completions of both types are missing in the .py file that would be an issue that we might have to transfer to the pylance team to investigate.

[natwille1]

Okay, my settings for python language server are "Default" which should use pylance given I have the extension installed. How can I check that the kernel and the selected python interpreter match? I use conda and select a specific environment for a specific project in the dropdown menu in the top right hand corner of the jupyter notebook.

For interactive inline execution, I do not see any completions in the jupyter code cell in the .py file, but I do get the df.column_name completion in the input box of the interactive window. Thanks

[ido-greenfeld]

Jumping in here, I do have autocompletion of column names but it's buried in other suggestions that are not related to my dataframe.
This is what I get when I press ctrl+space inside the '' -

[natwille1]

Interesting - I never see this. What could be the factors affecting this behaviour? I can think of language server but I think we're also using pylance? Anything else?

[IanMatthewHuff]

@natwille1 Could you take a look at the Python Language Service section of the output window? When that starts up Pylance there is some logging about the interpreter that is being used, as well as any possible errors. Here is an example from my machine:

[Info  - 12:45:21 PM] (61504) Pylance language server 2022.2.2-pre.1 (pyright 618f54ed) starting
[Info  - 12:45:21 PM] (61504) Server root directory: /Users/ianhuff/.vscode-insiders/extensions/ms-python.vscode-pylance-2022.2.2-pre.1/dist
[Info  - 12:45:21 PM] (61504) No configuration file found.
[Info  - 12:45:21 PM] (61504) No pyproject.toml file found.
[Info  - 12:45:21 PM] (61504) Setting pythonPath for service "DebuggingDemo": "/Users/ianhuff/Documents/DataScience/DebuggingDemo/.debugEnv/bin/python"
[Warn  - 12:45:21 PM] (61504) stubPath /Users/ianhuff/Documents/DataScience/DebuggingDemo/typings is not a valid directory.
[Info  - 12:45:21 PM] (61504) Assuming Python version 3.7
[Info  - 12:45:21 PM] (61504) Assuming Python platform Darwin

[natwille1]

Yep mine looks the same as yours.. :/

[Info - 09:53:47] (20496) Pylance language server 2022.2.3 (pyright 9b0ea9eb) starting
[Info - 09:53:47] (20496) Server root directory: c:\Users\nwillems.vscode\extensions\ms-python.vscode-pylance-2022.2.3\dist
[Info - 09:53:47] (20496) No configuration file found.
[Info - 09:53:47] (20496) No pyproject.toml file found.
[Info - 09:53:47] (20496) Setting pythonPath for service "proms-prediction": "C:\Users\nwillems\Anaconda3\envs\databricks\python.exe"
[Warn - 09:53:47] (20496) stubPath c:\Users\nwillems\dev\proms-prediction\typings is not a valid directory.
[Info - 09:53:48] (20496) Assuming Python version 3.8
[Info - 09:53:48] (20496) Assuming Python platform Windows

[IanMatthewHuff]

@natwille1 My apologies. I think I need to back up a second to make sure that I'm looking at your specific scenario. Do you mind if I back up for a second and restate?

From your comments (all of this is referring to inside of VS Code):

• In a Jupyter notebook you don't see completions for something like df['column_name'] when trying to complete the column name.
• In a Jupyter notebook you do see completions for df.column_name, but it's slow and take a few seconds to come up.
• When executing cell blocks from a .py file in the Interactive Window you don't see any column completions for the df at all either with a "." or with ['. But you do see other completions on the df (like df. and you would see 'abs' and other functions on the df).
• When executing cell blocks from a .py fine in the Interactive Window in the Input box at the bottom of the Interactive Window you see the same . completions as in the notebook file.

Is that the right summary? I've looking at these scenarios and I think that I repro basically the same as you do. I think that I understand why most of the differences are happening, but I agree that there is some confusion about why things are seen in some scenarios and not others.

• Right now we don't provide column name completions with the ['column_name'] syntax. So that's just a missing feature
• In the notebook and the input box intellisense is being provided by both pylance and by the running jupyter server. Jupyter has their own completions and we combine those with the static analysis from pylance. That's why in those scenarios you see .column_name as an option. However in .py file intellisense has only pylance so you see the member functions but not the dynamically added function names. Their might be a way to change this where if an Interactive Window is running we provide the same Jupyter completions to the .py file, but we don't do that now.
• The fact that completions are coming from Jupyter might be related to why they are slow to come back. For those Jupyter specific completions like the column names we are at the mercy of how fast the Jupyter server completion is, we can't control that directly.

@rchiodo

[rchiodo]

I can get completions to work in a jupyter notebook (outside of VS code) and it does support the df.['column_name'] syntax:

(Jupyter brings up completions on the 'tab' key)

So it would seem like we could implement this.

[IanMatthewHuff]

Sounds good I wasn't seeing the [''] syntax in my Jupyter install but it was having a bunch of auto complete issues in general. I'm going to leave this issue open for supporting the bracket syntax, and I'm going to file a new issue to look at if we want to consider adding Jupyter completions into .py files when using the interactive window so we can track those issues separately.

[natwille1]

@natwille1 My apologies. I think I need to back up a second to make sure that I'm looking at your specific scenario. Do you mind if I back up for a second and restate?

From your comments (all of this is referring to inside of VS Code):

• In a Jupyter notebook you don't see completions for something like df['column_name'] when trying to complete the column name.
• In a Jupyter notebook you do see completions for df.column_name, but it's slow and take a few seconds to come up.
• When executing cell blocks from a .py file in the Interactive Window you don't see any column completions for the df at all either with a "." or with ['. But you do see other completions on the df (like df. and you would see 'abs' and other functions on the df).
• When executing cell blocks from a .py fine in the Interactive Window in the Input box at the bottom of the Interactive Window you see the same . completions as in the notebook file.

Is that the right summary? I've looking at these scenarios and I think that I repro basically the same as you do. I think that I understand why most of the differences are happening, but I agree that there is some confusion about why things are seen in some scenarios and not others.

• Right now we don't provide column name completions with the ['column_name'] syntax. So that's just a missing feature
• In the notebook and the input box intellisense is being provided by both pylance and by the running jupyter server. Jupyter has their own completions and we combine those with the static analysis from pylance. That's why in those scenarios you see .column_name as an option. However in .py file intellisense has only pylance so you see the member functions but not the dynamically added function names. Their might be a way to change this where if an Interactive Window is running we provide the same Jupyter completions to the .py file, but we don't do that now.
• The fact that completions are coming from Jupyter might be related to why they are slow to come back. For those Jupyter specific completions like the column names we are at the mercy of how fast the Jupyter server completion is, we can't control that directly.

@rchiodo

Correct description and thanks for outlining why and creating a new issue around this - looking forward to seeing any new support for these features!

[mazin-abdelghany]

In relation to this open issue, Jupyter Notebooks are also able to tab autocomplete files that are in the same working directory as the open notebook.

For example, if the file example.csv is in the same working directory as the the open .ipynb file, pd.read_csv('exam') + tab will autocompletes the file name within the pd.read_csv() call to example.csv.

The Jupyter Notebook extension in VS Code does not replicate this behavior.

[DonJayamanne]

Internal notes for the team to discuss

I think we should revert the change that fixed #7136
That was merely a paper cut from an internal user, and not something that was reported by users, and as a result of that change we now have code completion not appearing when it should. I.e. this is a regression in functionality now, as a result of a fix for a paper cut.

My suggestion is to first unblock the users) then look into a long term proper fix.

[rchiodo]

We could change this so that:

• Don't filter to just file paths in strings
• Make the default trigger for completions to always include string chars

"jupyter.pythonCompletionTriggerCharacters": ".%\"'"

[rchiodo]

Not super happy with this. Jupyter completions suck.

This is the result I get if I 'fix' this and put it back how it was:

As you can see it's listing out every possible match for the letter 'a'. Not just columns. Jupyter isn't filtering anything.

Jupyter notebook is more precise though. It gives me this:

And the difference is in our use of Jedi. We disable this because of perf problems with Jedi in a kernel. If I reenable it, our completions come out the same as Jupyter

(after typing a)

I'm going to add a setting to enable more precise intellisense. With a warning that this makes everything potentially slower.

[rchiodo]

Verification steps:

• Enable this setting:

"jupyter.enableExtendedKernelCompletions": true

• Follow the steps above for creating a dataframe and trying to enter the columns

[tanhakabir]

I have the setting enabled but weirdly I'm getting suggestions for all the folders I have in my GitHub repo:

[rchiodo]

That's by design. Jupyter isn't smart enough to filter them out.

[rchiodo]

The verification would be that the column names do show up in that list. It looks like you had 'test' and 'test2'? They did show up?

[rebornix]

@rchiodo it works well but still reproduces with python.pylanceLspNotebooksEnabled enabled.

[rchiodo]

@rebornix can you explain what you mean by still reproduces? I have that setting enabled and it works for me:

[rebornix]

@rchiodo I was not seeing test and test2 with the sample in #8983 (comment)

[rchiodo]

It works for me? Did you run the cell that generated the DF first?

These completions come from the kernel. You have to create the DF before it will know about the columns.

[stevenlis]

This does not work, or at least is very unstable. Sometimes, it works in the editor but not the interactive window, and sometimes the other way around. Very often... Neither dot nor brackets notion works at all.