Bugfix Qtconsole autocomplete

[TobiasSpohn]

Description

This PR fixes a bug, where the QtConsole and Jupyter-Notebook completer is not working, except for top-level namespace objects such as the individual modules for rpyc==6.0.0. This makes typing in QtConsole qudi.<Tab> possible again (or in jupyter-notebook).

Motivation and Context

When starting qudi, QtConsole connects to the local-namespace-server. This rpyc service has the exposed_get_namespace_dict method that returns a dict of all loaded modules. This should make it possible for the qtconsole to autocomplete the loaded modules and their attributes. For rpyc<6.0.0 this works, one can press to autocomplete e.g. qudi upon startup. Typing qudi.<Tab> also autocompletes by giving the list for all attributes of qudi.

With rpyc 6.0.0 this no longer works. I found out that this was caused by the QudiIPythonKernel.shell.Completer.use_jedi variable. When checking the QudiIPythonKernel class one can see in the __init__ that this has caused some issues in the past and has been fixed already. In fact one can check the config during runtime by typing get_ipython().Completer.config in QtConsole. The use_jedi variable is set to False. However, when checking the runtime parameters of the Completer by typing get_ipython().Completer.use_jedi it is still set to True. In rpyc 5.3.1 it seems to work with use_jedi=True.
Installing rpyc 6.0.0 shows the same output of both commands, however autocomplete is only working for the top level modules as e.g. q<Tab> which can complete to qudi, but qudi.<Tab> does not show any results. When setting get_ipython().Completer.use_jedi = False, autocompletion works as before. This PR sets this variable directly during init of the Kernel. As the comment in the code above my changes suggests, this is not properly documented how to change the config properly for embedded IPython. Maybe we have to work with a properly set up individual IPython profile or using the traitlets module. However, I didn't get it to work quickly so I decided to again go for this workaround.

How Has This Been Tested?

Testing autocomplete in jupyter-notebook and qtconsole with rpyc 5.3.1 and 6.0.0 installed.

Types of changes

• Bug fix
• New feature
• Breaking change (Causes existing functionality to not work as expected)

Checklist:

• My code follows the code style of this project.
• I have documented my changes in /docs/changelog.md.
• My change requires additional/updated documentation.
• I have updated the documentation accordingly.
• I have added/updated the config example for any module docstrings as necessary.
• I have checked that the change does not contain obvious errors
  (syntax, indentation, mutable default values, etc.).
• I have tested my changes using 'Load all modules' on the default dummy configuration.
• All changed Jupyter notebooks have been stripped of their output cells.

[Neverhorst]

Using rpyc 5.3.1 this branch actually introduces this bug for me. Rolling back to current main and everything works as expected. So it seems this fix is not backwards-compatible.

See my output of pip freeze

anyio==4.0.0
argon2-cffi==23.1.0
argon2-cffi-bindings==21.2.0
arrow==1.3.0
asteval==0.9.31
asttokens==2.4.0
async-lru==2.0.4
attrs==23.1.0
Babel==2.13.0
backcall==0.2.0
beautifulsoup4==4.12.2
bleach==6.1.0
bronkhorst-propar==1.1.1
certifi==2023.7.22
cffi==1.16.0
charset-normalizer==3.3.0
colorama==0.4.6
comm==0.1.4
contourpy==1.1.1
cycler==0.12.1
debugpy==1.8.0
decorator==5.1.1
defusedxml==0.7.1
deprecation==2.1.0
entrypoints==0.4
exceptiongroup==1.1.3
executing==2.0.0
fastjsonschema==2.18.1
fonttools==4.43.1
fqdn==1.5.1
future==0.18.3
fysom==2.1.6
gitdb==4.0.11
GitPython==3.1.40
idna==3.4
ipykernel==6.25.2
ipython==8.16.1
ipython-genutils==0.2.0
ipywidgets==8.1.1
isoduration==20.11.0
jedi==0.19.1
Jinja2==3.1.2
json5==0.9.14
jsonpointer==2.4
jsonschema==4.19.1
jsonschema-specifications==2023.7.1
jupyter==1.0.0
jupyter-console==6.6.3
jupyter-events==0.8.0
jupyter-lsp==2.2.0
jupyter_client==8.4.0
jupyter_core==5.4.0
jupyter_server==2.8.0
jupyter_server_terminals==0.4.4
jupyterlab==4.0.7
jupyterlab-pygments==0.2.2
jupyterlab-widgets==3.0.9
jupyterlab_server==2.25.0
jupytext==1.15.2
kiwisolver==1.4.5
lmfit==1.2.2
lxml==4.9.3
markdown-it-py==3.0.0
MarkupSafe==2.1.3
matplotlib==3.8.0
matplotlib-inline==0.1.6
mdit-py-plugins==0.4.0
mdurl==0.1.2
mistune==3.0.2
nbclient==0.8.0
nbconvert==7.9.2
nbformat==5.9.2
nest-asyncio==1.5.8
nidaqmx==0.5.7
nmrglue==0.9
notebook==7.0.6
notebook_shim==0.2.3
numpy==1.26.1
overrides==7.4.0
packaging==23.2
pandocfilters==1.5.0
parso==0.8.3
Phidget22==1.17.20231004
pickleshare==0.7.5
Pillow==10.1.0
platformdirs==3.11.0
plumbum==1.8.2
prometheus-client==0.17.1
prompt-toolkit==3.0.39
protobuf==4.21.12
psutil==5.9.6
pure-eval==0.2.2
pycparser==2.21
pyFirmata==1.1.0
pyftdi==0.55.0
Pygments==2.16.1
pyparsing==3.1.1
pyqtgraph==0.13.3
pyserial==3.5
PySide2==5.15.2.1
python-dateutil==2.8.2
python-json-logger==2.0.7
pyusb==1.2.1
PyVISA==1.13.0
PyVISA-py==0.7.1
pywin32==306
pywinpty==2.0.12
PyYAML==6.0.1
pyzmq==25.1.1
qtconsole==5.4.4
QtPy==2.4.0
-e git+https://github.com/Ulm-IQO/qudi-core.git@e72c655d427419c36e03c4f343f1da9b09e94d2e#egg=qudi_core
-e git+https://github.com/Ulm-IQO/qudi-iqo-modules.git@3039e05d4e1599417a74da7dae6dc61979c7eed1#egg=qudi_iqo_modules
referencing==0.30.2
requests==2.31.0
rfc3339-validator==0.1.4
rfc3986-validator==0.1.1
rpds-py==0.10.6
rpyc==5.3.1
ruamel.yaml==0.17.40
ruamel.yaml.clib==0.2.8
Rx==3.2.0
scipy==1.11.3
Send2Trash==1.8.2
shiboken2==5.15.2.1
six==1.16.0
smmap==5.0.1
sniffio==1.3.0
soupsieve==2.5
stack-data==0.6.3
terminado==0.17.1
tinycss2==1.2.1
toml==0.10.2
tomli==2.0.1
tornado==6.3.3
traitlets==5.11.2
types-python-dateutil==2.8.19.14
typing_extensions==4.8.0
uncertainties==3.1.7
uri-template==1.3.0
urllib3==2.0.7
wcwidth==0.2.8
webcolors==1.13
webencodings==0.5.1
websocket-client==1.6.4
widgetsnbextension==4.0.9
windfreak==0.3.0
zaber-motion==4.5.1
zaber-motion-bindings-windows==4.5.1

[TobiasSpohn]

For me it works this way, what is your pip freeze output?

pip freeze

argon2-cffi==21.3.0
argon2-cffi-bindings==21.2.0
arrow==1.2.3
asteval==0.9.28
asttokens==2.2.1
attrs==22.2.0
backcall==0.2.0
beautifulsoup4==4.11.1
bleach==6.0.0
cffi==1.15.1
colorama==0.4.6
comm==0.1.2
contourpy==1.0.7
cycler==0.11.0
debugpy==1.6.6
decorator==5.1.1
defusedxml==0.7.1
entrypoints==0.4
executing==1.2.0
fastjsonschema==2.16.2
finufft==2.1.0
fonttools==4.38.0
fqdn==1.5.1
future==0.18.3
fysom==2.1.6
gitdb==4.0.10
GitPython==3.1.30
idna==3.4
importlib-metadata==6.0.0
ipykernel==6.21.0
ipympl==0.9.3
ipython==8.9.0
ipython-genutils==0.2.0
ipywidgets==8.0.4
isoduration==20.11.0
jedi==0.18.2
Jinja2==3.1.2
jsonpointer==2.3
jsonschema==4.17.3
jupyter==1.0.0
jupyter-console==6.4.4
jupyter-events==0.6.3
jupyter_client==8.0.2
jupyter_core==5.2.0
jupyter_server==2.1.0
jupyter_server_terminals==0.4.4
jupyterlab-pygments==0.2.2
jupyterlab-widgets==3.0.5
jupyterthemes==0.20.0
jupytext==1.14.4
kiwisolver==1.4.4
lesscpy==0.15.1
lmfit==1.1.0
lxml==4.9.2
markdown-it-py==2.1.0
MarkupSafe==2.1.2
matplotlib==3.6.3
matplotlib-inline==0.1.6
mdit-py-plugins==0.3.3
mdurl==0.1.2
mistune==2.0.4
mpmath==1.3.0
nbclassic==0.5.1
nbclient==0.7.2
nbconvert==7.2.9
nbformat==5.7.3
nest-asyncio==1.5.6
nidaqmx==0.6.5
notebook==6.5.2
notebook_shim==0.2.2
numpy==1.24.1
packaging==23.0
pandocfilters==1.5.0
parso==0.8.3
pickleshare==0.7.5
Pillow==9.4.0
platformdirs==2.6.2
plumbum==1.8.1
ply==3.11
prometheus-client==0.16.0
prompt-toolkit==3.0.36
protobuf==4.21.12
psutil==5.9.4
pure-eval==0.2.2
pycparser==2.21
Pygments==2.14.0
pyparsing==3.0.9
pyqtgraph==0.13.1
pyrsistent==0.19.3
PySide2==5.15.2
PySide6_Addons==6.7.2
PySide6_Essentials==6.7.2
python-dateutil==2.8.2
python-json-logger==2.0.4
PyVISA==1.13.0
pywin32==305
pywinpty==2.0.10
PyYAML==6.0
pyzmq==25.0.0
qtconsole==5.4.0
QtPy==2.3.0
-e git+https://github.com/Ulm-IQO/qudi-core.git@53a86e4d217b4ae72d3cfaa54dc8b8c07e227913#egg=qudi_core
-e git+https://github.com/Ulm-IQO/qudi-iqo-modules.git@3039e05d4e1599417a74da7dae6dc61979c7eed1#egg=qudi_iqo_modules
rfc3339-validator==0.1.4
rfc3986-validator==0.1.1
rpyc==5.3.1
ruamel.yaml==0.17.21
ruamel.yaml.clib==0.2.7
Rx==3.2.0
scipy==1.10.0
Send2Trash==1.8.0
shiboken2==5.15.2
shiboken6==6.7.2
six==1.16.0
smmap==5.0.0
sniffio==1.3.0
soupsieve==2.3.2.post1
stack-data==0.6.2
sympy==1.12
terminado==0.17.1
tinycss2==1.2.1
toml==0.10.2
tornado==6.2
traitlets==5.9.0
typing_extensions==4.4.0
uncertainties==3.1.7
uri-template==1.2.0
wcwidth==0.2.6
webcolors==1.12
webencodings==0.5.1
websocket-client==1.5.0
widgetsnbextension==4.0.5
zaber-motion==2.15.4
zaber-motion-bindings-windows==2.15.4
zipp==3.12.0``` 

</details> 

[TobiasSpohn]

I just reinstalled from scratch:

• py -3.10 -m venv .qudi-env
  - .\.qudi-env\Scripts\activate
• python --version
  • Python 3.10.11
• pip install -e .\qudi-core\
• pip install -e .\qudi-iqo-modules\
• qudi-install-kernel.exe
• pip freeze
• qudi -d
  • works as expected
• pip install --upgrade rpyc==5.3.1
• pip freeze
• qudi-install-kernel.exe
• qudi -d
  • works as expected

The difference between both pip freezes is

rpyc:
    First list: rpyc==5.3.1
    Second list: rpyc==6.0.0

sniffio:
    First list: sniffio==1.3.1
    Second list: sniffio==1.3.1

according to chatgpt.
The first pip freeze gave this:

Details

anyio==4.4.0
argon2-cffi==23.1.0
argon2-cffi-bindings==21.2.0
arrow==1.3.0
asteval==1.0.4
asttokens==2.4.1
async-lru==2.0.4
attrs==24.2.0
babel==2.16.0
beautifulsoup4==4.12.3
bleach==6.1.0
certifi==2024.8.30
cffi==1.17.1
charset-normalizer==3.3.2
click==8.1.7
colorama==0.4.6
comm==0.2.2
contourpy==1.3.0
cycler==0.12.1
debugpy==1.8.5
decorator==5.1.1
defusedxml==0.7.1
deprecation==2.1.0
dill==0.3.8
entrypoints==0.4
exceptiongroup==1.2.2
executing==2.1.0
fastjsonschema==2.20.0
fonttools==4.53.1
fqdn==1.5.1
fysom==2.1.6
gitdb==4.0.11
GitPython==3.1.43
h11==0.14.0
hightime==0.2.2
httpcore==1.0.5
httpx==0.27.2
idna==3.10
ipykernel==6.29.5
ipython==8.27.0
ipywidgets==8.1.5
isoduration==20.11.0
jedi==0.19.1
Jinja2==3.1.4
json5==0.9.25
jsonpointer==3.0.0
jsonschema==4.23.0
jsonschema-specifications==2023.12.1
jupyter==1.1.1
jupyter-console==6.6.3
jupyter-events==0.10.0
jupyter-lsp==2.2.5
jupyter_client==8.6.3
jupyter_core==5.7.2
jupyter_server==2.14.2
jupyter_server_terminals==0.5.3
jupyterlab==4.2.5
jupyterlab_pygments==0.3.0
jupyterlab_server==2.27.3
jupyterlab_widgets==3.0.13
jupytext==1.16.4
kiwisolver==1.4.7
lmfit==1.3.2
lxml==5.3.0
markdown-it-py==3.0.0
MarkupSafe==2.1.5
matplotlib==3.9.2
matplotlib-inline==0.1.7
mdit-py-plugins==0.4.2
mdurl==0.1.2
mistune==3.0.2
nbclient==0.10.0
nbconvert==7.16.4
nbformat==5.10.4
nest-asyncio==1.6.0
nidaqmx==1.0.1
notebook==7.2.2
notebook_shim==0.2.4
numpy==1.26.4
overrides==7.7.0
packaging==24.1
pandocfilters==1.5.1
parso==0.8.4
pillow==10.4.0
platformdirs==4.3.3
plumbum==1.8.3
prometheus_client==0.20.0
prompt_toolkit==3.0.47
psutil==6.0.0
pure_eval==0.2.3
pycparser==2.22
Pygments==2.18.0
pyparsing==3.1.4
pyqtgraph==0.13.7
PySide2==5.15.2.1
python-dateutil==2.9.0.post0
python-decouple==3.8
python-json-logger==2.0.7
PyVISA==1.14.1
pywin32==306
pywinpty==2.0.13
PyYAML==6.0.2
pyzmq==26.2.0
qtconsole==5.6.0
QtPy==2.4.1
-e git+https://github.com/Ulm-IQO/qudi-core.git@53a86e4d217b4ae72d3cfaa54dc8b8c07e227913#egg=qudi_core
-e git+https://github.com/Ulm-IQO/qudi-iqo-modules.git@3039e05d4e1599417a74da7dae6dc61979c7eed1#egg=qudi_iqo_modules
reactivex==4.0.4
referencing==0.35.1
requests==2.32.3
rfc3339-validator==0.1.4
rfc3986-validator==0.1.1
rpds-py==0.20.0
rpyc==6.0.0
ruamel.yaml==0.18.6
ruamel.yaml.clib==0.2.8
scipy==1.14.1
Send2Trash==1.8.3
shiboken2==5.15.2.1
six==1.16.0
smmap==5.0.1
sniffio==1.3.1
soupsieve==2.6
stack-data==0.6.3
terminado==0.18.1
tinycss2==1.3.0
tomli==2.0.1
tornado==6.4.1
traitlets==5.14.3
types-python-dateutil==2.9.0.20240906
typing_extensions==4.12.2
tzdata==2024.1
tzlocal==5.2
uncertainties==3.2.2
uri-template==1.3.0
urllib3==2.2.3
wcwidth==0.2.13
webcolors==24.8.0
webencodings==0.5.1
websocket-client==1.8.0
widgetsnbextension==4.0.13
zaber_motion==6.6.1

I just saw your pip freeze and will check with yours.

[TobiasSpohn]

I put your pip freeze into a requirements.txt, removed the qudi-core specifier, to keep my currently checked out commit and did pip install --upgrade -r requirements.txt and qudi-install-kernel and it still works as expected, autocompletion is done.

Chatgpt says the difference between our freeze is (Before being my current freeze, after is yours)
Before:

qudi-core is installed from commit 53a86e4d217b4ae72d3cfaa54dc8b8c07e227913.

After:

qudi-core is installed from commit e72c655d427419c36e03c4f343f1da9b09e94d2e.

is this pip freeze the working or the not working commit as you are with this commit on current main.
Choosing the same commit hash for core still works for me with these package versions.

[Neverhorst]

Well since this PR does not change the package requirements for qudi-core it should not matter on which branch I did the pip freeze.
With this freeze I can autocomplete qudi.module_manager.<TAB> on main but not on qtconsole_autocomplete. Simply switching branches makes this behaviour appear/disappear.

[TobiasSpohn]

Ah now I see. I did not try qudi.module_manager.<Tab> or any other third level completion before. My changes only make qudi.<Tab> work, even on rpyc 6.0.0. So there seems to be something deeper broken.
The changelog of rpyc 6.0.0 was not really helpful, looking at the source, it seems like they only changed allow_pickle default option for a service to False. However, in qudi we are setting these flags, and even logging them in the debugger

 local-namespace-server: protocol_config is {'allow_all_attrs': True, 'allow_setattr': True, 'allow_delattr': True, 'allow_pickle': True, 'sync_request_timeout': 3600, 'logger': <Logger QUDINAMESPACE/18861 (warning)>}, authenticator is None

So I assumed that these flags were set correctly, but maybe it is something similar like the ipython kernel config that it is not correctly applied.

[Neverhorst]

I mean you can access those attributes by typing instead of tabbing, so it does not seem to be an issue with rpyc access settings but rather the code inspection tools used by IPython. As you have seen for yourself now, IPython is barely documented and this issue unfortunately needs to be explored in source code and tedious trial-and-error. Took me quite some time to find out about the first "fix"...

[TobiasSpohn]

My latest commit fixes the issue. In debug mode one can see that Jedi crashes when trying to evaluate the netref objects. Something has changed in rpyc 6.0.0 that makes jedi no longer work with them and the error message is not very helpful. Turning jedi off and changing the evaluation flag fixes this. The IPython documentation mentions this flag in their documentation. By default limited is chosen, however this seems to exclude functions from netref objects that expose the members of the objects they proxy (I didn't find documentation for this, just by trial and error). By changing to unsafe this is fixed and the netref objects can properly be inspected. Now tab-completion works for deeper levels as well. However, there is still something different:

suppose we want to examine the list of modules that are loaded by the configuration file.
We access this by qudi.module_manager.modules. For me this takes a while to execute , but prints the dictionary. The normal operations can also be done e.g. indexing or looping through the elements. However, if I want to use the .items, .keys or .values functions the whole console is covered in the same TypeError: descriptor '__len__' of 'dict_items' object needs an argument and I don't see an out statement, where the items are returned. However, if I assign the result of these functions to a variable I can correctly iterate through the items.
In current main the same behavior is present, except that .items, .values or .keys functions will return something (although after again filling the console with the above exception).
However, when switching to rpyc 6.0.0 and applying the last commit all error messages are gone, autocompletion works and all functions return the expected result.

[Neverhorst]

Nice. The second part is a well known issue between IPython and rpyc and one of the reasons I tried to restrict code execution during auto-completion. But nice to see that version 6 seems to fix that to some extent. Maybe we should think about setting this version as minimum requirement to reduce compatibility problems.

[TobiasSpohn]

Should I include this requirement in setup.py or merge the pyproject.toml branch before?

[Neverhorst]

Well, considering Issue #114 we should probably test this some more with remote modules.
I would like to merge PR #111 as well before changing the package requirements.

[TobiasSpohn]

I checked Issue #114 and noticed upon experimenting that autocompletion for importing qudi modules (e.g. from qudi.util.network import netobtain) also no longer works in rpyc 6.

Furthermore, autocompletion for a remote module also does no longer work.

While the first error gives some error (in autocompletion debug mode) when trying to type from qudi.<Tab>, the second one does not do anything. I will do some further testing, maybe its fixable. Else we could fix the version to 5.3.1