[RFC] Allow one to use YCM with (a custom) python 2/3 interpreter

[hotgloupi]

This is a request for comments regarding the best way to allow one to complete compiled extensions for python, possibly using a custom interpreter.

In my understanding, the state of vim + python + YCM is the following:

• Vim must be compiled explicitly with the python interpreter (possibly two)
• One interpreter can be active at a time in vim
• Python 2 is widely preferred in vim plugins
• jedi-vim can complete both python 2 and 3 by parsing py files
• YCM has compiled python extension to interact with vim, and a completion server (ycmd)

Solution 1: Compile Vim + YCM with the targeted interpreter
This would imply that YCM extensions and server code to be python3 compatible, which can be hard to do.

Solution 2: Specify the path to the wanted interpreter and let jedi-vim works in a separate process
This would need inter-process communication and a way to specify a completion specific python interpreter path. Moreover, the PYTHONPATH should be specific to the completion env, and not affect how the embedded vim python searches its module.

Solution X: (I will edit this post if you provide one !)

I've seen attempts on solution 1, but I still don't think that's the best way to go as it ties vim to one interpreter, which might break vim plugins. I would gladly volunteer to implement the second solution but I really need guidance on that.

Please shoot !

[Valloric]

Something like solution 2 is probably the way to go. The approach that works great in ycmd is the following: each semantic engine is a separate HTTP+JSON server that ycmd then talks to. OmniSharpServer is used for C# completion for instance and this works great. Problems with libclang crashing ycmd demonstrate that it should probably be in its own separate server, but that's too much work at this point because of past architectural decisions.

So if you want to work on this, here's how to go about it: wrap the Jedi library API with a Flask server. See how ycmd itself is implemented. For the JSON API, look at ycmd's examples directory which has an example client. Run it, play with it and see how it works. Note that your server should work with both a python2 and a python3 interpreter, since AFAIK that's how Jedi multi-python support works.

Build that server and plugging it into ycmd will be piece of cake.

[hotgloupi]

Could you let this open while we discuss the design ?

[hotgloupi]

The JediCompleter class would have to support start/stop commands as the CS completer does, and spawn an HTTP server (written using flask or bottle ?)
I've seen that the completer accepts a user_options parameters, where does it come from ? And what's the correct way to let the user specify the python executable that should be used to start the completion server (and a custom PYTHON_PATH) ?

Given a python executable, and the fact that jedi and flask are modules available to this interpreter, what would be the best way to start the server ?
Is something like this utils.SafePopen(["/path/to/the/python", "../../third_party/JediServer/bin/main.py"]) the way to go ?

Finally, just to recap, this server is just a third party project that wraps jedi behind an http server, and the JediCompleter class has to be adapted to use it. The communication between this class and the completion server should use the same protocol described to communicate with ycmd itself. Am I right about this ?

[Valloric]

Oops, this issue fell off my radar. Sorry about that.

Could you let this open while we discuss the design ?

I normally leave issues open if there's some work I need to implement and the whole thing is blocked on me. This isn't such an issue.

I've seen that the completer accepts a user_options parameters, where does it come from

Look at ycmd's __main__.py and go from there. The infrastructure involved with going from a request to a Completer object is pretty straightforward; you should be able to figure it out very quickly by reading through the code.

what would be the best way to start the server

cs_completer.py starts an OmniSharp server, so you could take a look at that for inspiration. But plugging the server into ycmd is further down the line; step 1 is building the server in the first place as a separate project & repo.

Finally, just to recap, this server is just a third party project that wraps jedi behind an http server, and the JediCompleter class has to be adapted to use it.

Yep, exactly.

The communication between this class and the completion server should use the same protocol described to communicate with ycmd itself.

That isn't strictly necessary (OmniSharpServer has an entirely different interface and ycmd doesn't have problems talking to it), but it would be a good idea. To see what the ycmd interface is, I recommend reading through and running the example_client.py file that comes with ycmd. Refer to ycmd docs for details.