Discussion: Jedi wrapper module using server-client architecture to deal with multiprocessing, async interface, multiple Python version, cache lock, etc.

[tkf]

From #384 (comment) (discussion about multithreading / multiprocessing) and #340 (comment) (discussion about asynchronous interface), I got this idea of creating a "wrapper" module of Jedi with server-client architecture which launches possibly multiple Jedi processes behind the scene, to solve problems such as:

• Multiple Python versions. Jedi parses Python stdlib and import *.so so it is not possible to complete Python 3 modules while running Jedi in Python 2. However, getting completions for both Python 2 and 3 can be useful when you are writing Python 2/3 compatible module.
• Async interface. Due to the GIL, and intense computation required by Jedi, it is not feasible to have async interface within Jedi.
• Cache locking/sync: Jedi does not lock cache files. Also, when cache is updated in two processes, later one might override the former one.
• Cache update triggered by file modification: When some modules are updated, Jedi has to wait until next access to update the cache. If we can trigger cache update by file modification then Jedi can be faster.

If we have a centralized server, a pool of workers and clients to connect to the workers then we can solve the above problems nicely. I think it is also consistent with a philosophy of keeping Jedi itself minimal and free of external modules.

Client part of this module can expose (blocking) API which is exactly the same as Jedi itself. Then, every editor can just import this wrapper module instead of Jedi to use it, without much change to the editor plugin. If the editor has an async support, then it can use async API of this client module.

Maybe my suggestion over complicates the problem, but I think it is worth discussing it anyway, since some problems are better not solved in Jedi. Delegating these problems to other module can be a good option to consider.

[proofit404]

Is it necessary to include it in jedi source tree? Maybe separate project would be better?

[tkf]

Yes, my idea to do it outside of Jedi project.

[proofit404]

So, what kind of client interaction you'll implement in it?

[tkf]

I am not sure what do you mean by "client interaction", but proposed module is in between IDE and Jedi so I guess you can say it is an "interface layer."

you'll implement

Well, I don't know if I am going to work on now, to be honest. But I want to have mixed Python 2/3 completion and if I am going to do it, I'll do it in Python side as I suggested above, rather at Emacs side (because doing this in Python side helps more people).

On the other hand, I started think that "Probably I should wait a few more years then Python 2 will be gone... I'll not be using it afterward so probably it is not worth trying?" (But maybe Python 4 will start soon later?!) Also other points I mentioned above are sort of OK for my personal usage.

Anyway, I just like the idea. My motivation fluctuates a lot right now.

[proofit404]

I mean "What client/server protocol you prefer?" Json-Rpc or something different?

[tkf]

What I have in mind is ZMQ (+ JSON or something for serializer), like IPython does
http://ipython.org/ipython-doc/stable/development/messaging.html
http://ipython.org/ipython-doc/stable/development/parallel_connections.html

I think it should be more than peer-to-peer communication. For example, you may want many-to-many connection between workers and IDEs/editors. I guess the word "server-client" is a but misleading in this sense but I don't know a good short word.

[dbrgn]

Sounds like a nice idea. ZMQ might be a great choice, even though I've never used it myself so far...

I guess the word "server-client" is a but misleading in this sense but I don't know a good short word.

Service? Background service? Completion service? Completion daemon?

[davidhalter]

While I totally support the idea (if inside of the Jedi code base or not, can be discussed), we have to really define the solution first and discuss all edge cases.

That wrapper module should be to Jedi what tox is to pytest.

How do you propose to get virtualenv support right?
I don't quite understand how you get rid of the locking problem. By having only one server running per Python version? Does your idea also work for Windows clients?

[tkf]

@dbrgn "Background service" sounds the best.

@davidhalter

if inside of the Jedi code base or not, can be discussed

I think it's best to develop outside of Jedi code base. It makes the two part (Jedi and the other module discussed here) "orthogonal".

That wrapper module should be to Jedi what tox is to pytest.

That's a good analogy!

How do you propose to get virtualenv support right?

Let user specify a list of virtual environments (created by virtualenv, venv, anaconda, etc.) then launch Jedi process per virtual environment which talks to Editor clients. LD_LIBRARY_PATH also needed to set appropriately to load .so properly. (It is possible to reduce number of Jedi processes to launch only one process per same pair of Python version and LD_LIBRARY_PATH but I am simplifying for now)

I don't quite understand how you get rid of the locking problem. By having only one server running per Python version?

My idea was to run only one process per machine to manage cache ("cache manager process"). But maybe cache locking can be solved just by doing file system based locking. It is still useful for updating (syncing) memory based cache, though.

Does your idea also work for Windows clients?

I think socket based communication like ZMQ works on Windows too.

[tkf]

To make the project more crazy :-), it can even be language agnostic so that you can write ZMQ workers using a new language and Jedi-like library for that language to have completion/help/goto UI without writing any editor side code.

[proofit404]

@tkf Personally I don't like the language agnostic idea.

[tkf]

@proofit404 I am just brainsotrming here. So it would be helpful if you say the reason why.

[proofit404]

@tkf Programming languages simply to different to has one common api for autocomplete tool. No matter how you can access it - zmq, rest or library function call. Focus on python support will save much time of development process.

[tkf]

@proofit404 I've never write Python specific Emacs Lisp code (i.e., editor side code) for Emacs-Jedi. It does not require python.el at all. I think the editor side code works, for example, for javascript if you write the server side code. I think it is a huge waste of time if you need to write editor side code for all languages.

FYI, IPython's REPL RPC is language-agnostic (of course this is where I got the idea). You can make REPL language-agnostic. Why not completion/help/goto?

[davidhalter]

@tkf Please also watch this thread: davidhalter/jedi-vim#119 (comment) We've just started to discuss the exact same thing.
cc @Valloric

[fjfnaranjo]

I will like to add a little information here about jedi-vim so it can be taken into account when anyone starts working on this.

I have been using this https://github.com/nakamuray/jedi-rpc.vim to fix all my problems related to Python+virtualenv version compatibility problems and package discovery in jedi-vim. All credit goes to @nakamuray , of course.

What this little Vim module does is intercepting the jedi-vim interface and run all jedi calls in the current system's python. In other words: If you launch Vim inside an activated Python virtual environment, you can use all the options in jedi-vim with whatever version of Python you have in the venv. The internal Python inside Vim is not used for anything related to jedi or jedi-vim, just by the jedi-rpc adapter.

What I wanted to say is that to keep supporting this kind of schemes it will be nice if the server is able to be launched as a in-environment process. This will allow it to be sensible to whatever tools and libraries can be found in the environment.

I guess jedi already uses a Python 2+3 approach to cover a many cases as possible. If we keep this jedi will be able to adapt completely to the environments.

We can event add support for https://github.com/pypa/pipenv or .env files.

Your thoughts about this?

[davidhalter]

@fjfnaranjo The virtualenv support is actually kind of finished in the virtualenv branch. There are still possibilities to do what you asked for.

I will try to wrap up that work in the next few days weeks and propose a new API that fixes most of the problems we've had in virtualenvs. The RPC approach might not be bad, but I think just moving builtin evaluation to another library has a lot of advantages.

Generally I don't think that there will be a jedi wrapper module using a server/client architecture in the short future. If anything that will come when Python 2.7/3.3/3.4 are deprecated so asyncio and (async commands) could be used. That's probably speaking about 2021, which means that it's still going to take 3 years - at least.

[fjfnaranjo]

@davidhalter Sorry to hear that :( There are lots of ppl using 2.7 and they will miss a good chance to do all the stuff jedi is going to do (I recently read you talking about refactoring ;) ). But I totally understand your position.

I tend to do half of my stuff in PyDev and the other half directly using Vim. The combination of jedi and jedi-vim is something I am very happy to have available. I don't care waiting for the occasional two or three seconds when I deserve them and it keeps my tools up to the modern expectations.

Let me use this opportunity to thank you about your work. It is useful to lots of people and fill an important hole left by the modern IDEs when you switch to a terminal window.

I will wait for the virtualenv branch and test it in my different environments. If I find a good reason to stick to the RPC approach I may try someday to port the jedi-rpc code to the jedi-vim project so it can live there if you wish to adopt it (is quite simple) and the original author doesn't oppose too much.

[davidhalter]

I'm not sure if the rpc approach is even needed. What you really want is an approach where you're actually able to continue typing (async completions). I think that would be the big break-through. AFAIK YouCompleteMe has already done that, so there's probably not much of a hurry.

[dbrgn]

YCM complete-as-you-type with the Jedi backend (and a server-client model, btw) works really nicely.

[davidhalter]

Most of the issues here are fixed or not really an issue.

Multiple Python versions.

This is very much possible now. Please use https://jedi.readthedocs.io/en/latest/docs/api.html#environments.

Async interface.

Jedi will stay synchronous for now. I don't see the advantage in creating a third-party async framework for Jedi. Jedi is mostly CPU-bound. Using async/await won't help at all. Since Python has the GIL only separate processes really help. For this the easiest way is probably using Microsoft's Language Server Protocol. It might not be feature complete, but it has a lot to offer. There are a few competing implementations. The palantir version is probably the most mature, but I haven't looked into it.

• https://github.com/palantir/python-language-server
• https://github.com/pappasam/jedi-language-server
• https://github.com/sourcegraph/python-langserver (archived)

Cache locking/sync

While there is currently no cache locking (for pickled cache files), because it's not really a problem. The files are not written/read very often and we haven't had any complaints in years. If this is an issue for you, please open a new issue and let us know.

Cache update triggered by file modification

This is something that might be interesting in the future. The ticket for this however is #1059. If anything we will look at this at that point.