Help using pygls.lsp.client.BaseLanguageClient

[Moosems]

I want to eventually make simple multipurpose client that makes it easy to interface with Language Servers for a project of mine but I have very little knowledge of how to work with LSP's. Having looked at many alternatives, pygls seems like the best option available to get started but the Clients have very few examples and its not very clear how to get started with them. A basic example for interfacing with ruff-lsp (already installed locally) would be a major help. I saw this discussion but it didn't make much sense to me. Thank you for any time and effort in helping me out!!

[alcarney]

but the Clients have very few examples and its not very clear how to get started with them

Yes... compared to the rest of pygls support for clients is quite new so you have to do almost everything yourself still - which is why we still only have a BaseLanguageClient! 😅

With that out of the way, the following code will spin up and instance of ruff-lsp, start an LSP session and immediately close it.

import asyncio

from lsprotocol import types
from pygls.lsp.client import BaseLanguageClient


async def main():
    client = BaseLanguageClient("example-client", "v1")

    # Start the server running in a subprocess
    await client.start_io("ruff-lsp")

    # Start the LSP session
    response = await client.initialize_async(
        types.InitializeParams(
            capabilities=types.ClientCapabilities(),
            root_uri="file:///path/to/project",
        )
    )
    print(f"Got response: {response}")

    # Tell ruff-lsp to get ready to exit and it acknowledge the request
    await client.shutdown_async(None)

    # Tell ruff-lsp to actually exit
    client.exit(None)

    # Close down the client and server subprocess
    await client.stop()

asyncio.run(main())

(assuming you have ruff-lsp on your $PATH)

This does however, leave you however in a bit of a "draw the rest of the owl" situation! 😅

At the very least you're probably going to want to look into the notifications textDocument/didOpen, textDocument/didChange and textDocument/didClose so that you can keep ruff-lsp informed of any changes the user is making to files as they work.

From there you can find examples of some specific LSP methods in this test folder and I did put together a proof of concept language client built on Textual, which you might find useful

Also if you have any follow up questions feel free to ask! :)

[Moosems]

Hey @alcarney!

Thank you for the response! I'm surprised I didn't notice this earlier! I was messing around a little bit and noticed when I tried using await client.start_io("ruff", "server") instead of ruff-lsp (since they're in the process of moving the lsp to the main ruff package) and noticed that every time I try running it that the process hangs. Any thoughts? Am I doing this wrong? Thanks for your help!

[alcarney]

Yes... compared to the rest of pygls support for clients is quite new so you have to do almost everything yourself still

Unfortunately, this includes error handling! 🤦

I tried the above code with client.start_io("ruff", "server") and it also hangs for me.

Turns out the server process was exiting almost immediately after calling start_io but the example code I gave you does not detect this and so blindly awaits on the result of initialize_async - something that will never come!

Thankfully, if you implement the server_exit method in a subclass of BaseLanguageClient you can add some code to detect the error and exit the process.

import asyncio
import sys

from lsprotocol import types
from pygls.lsp.client import BaseLanguageClient


class LanguageClient(BaseLanguageClient):

    async def server_exit(self, server: asyncio.subprocess.Process):
        """Called when the server process exits."""

        #   0: all good
        # -15: terminated
        if server.returncode not in {0, -15}:
            print(f"Server process exited with code: {server.returncode}")

            if server.stderr is not None:
                stderr = await server.stderr.read()
                print(f"Stderr:\n{stderr.decode('utf8')}")

        sys.exit(server.returncode)


async def main():
    client = LanguageClient("example-client", "v1")

    # Start the server running in a subprocess
    await client.start_io("ruff", "server", "--preview")

    # Start the LSP session
    response = await client.initialize_async(
        types.InitializeParams(
            capabilities=types.ClientCapabilities(),
            root_uri="file:///path/to/project",
        )
    )
    print(f"Got response: {response}")

    # Tell ruff that we have finished our client side setup and are ready to go
    client.initialized(types.InitializedParams())

    # Tell ruff to get ready to exit and wait for it acknowledge the request
    await client.shutdown_async(None)

    # Tell ruff to actually exit
    client.exit(None)

    # Close down the client and server subprocess
    await client.stop()

asyncio.run(main())

Now when I run the script I get a useful error message!

$ python test.py
process exited with code: 2
Stderr:
error: --preview needs to be provided as a command line argument while the server is still unstable.
For example: `ruff server --preview`

Using client.start_io("ruff", "server", "--preview") it started working for me

$ python test.py
Got response: InitializeResult(capabilities=ServerCapabilities(position_encoding='utf-16', text_document_sync=TextDocumentSyncOptions(open_close=True, change=<TextDocumentSyncKind.Incremental: 2>, will_save=False, will_save_wait_until=False, save=None), notebook_document_sync=NotebookDocumentSyncOptions(notebook_selector=[NotebookDocumentSyncOptionsNotebookSelectorType2(cells=[NotebookDocumentSyncOptionsNotebookSelectorType2CellsType(language='python')], notebook=None)], save=False), completion_provider=None, hover_provider=True, signature_help_provider=None, declaration_provider=None, definition_provider=None, type_definition_provider=None, implementation_provider=None, references_provider=None, document_highlight_provider=None, document_symbol_provider=None, code_action_provider=CodeActionOptions(code_action_kinds=['quickfix', 'source.fixAll.ruff', 'source.organizeImports.ruff', 'notebook.source.fixAll.ruff', 'notebook.source.organizeImports.ruff'], resolve_provider=True, work_done_progress=True), code_lens_provider=None, document_link_provider=None, color_provider=None, workspace_symbol_provider=None, document_formatting_provider=True, document_range_formatting_provider=True, document_on_type_formatting_provider=None, rename_provider=None, folding_range_provider=None, selection_range_provider=None, execute_command_provider=ExecuteCommandOptions(commands=['ruff.applyFormat', 'ruff.applyAutofix', 'ruff.applyOrganizeImports', 'ruff.printDebugInformation'], work_done_progress=False), call_hierarchy_provider=None, linked_editing_range_provider=None, semantic_tokens_provider=None, moniker_provider=None, type_hierarchy_provider=None, inline_value_provider=None, inlay_hint_provider=None, diagnostic_provider=DiagnosticOptions(inter_file_dependencies=False, workspace_diagnostics=False, identifier='Ruff', work_done_progress=True), inline_completion_provider=None, workspace=ServerCapabilitiesWorkspaceType(workspace_folders=WorkspaceFoldersServerCapabilities(supported=True, change_notifications=True), file_operations=None), experimental=None), server_info=InitializeResultServerInfoType(name='ruff', version='0.5.1'))

[blankRiot96]

@alcarney Thank you for your wonderful base example on using pygls clients. ruff-lsp, ruff server --preview and pylsp work as you'd expect, or at least throw a useful error now.

But basedpyright and pyright haven't been so kind to me

import asyncio

from lsprotocol import types
from pygls.lsp.client import BaseLanguageClient


class LanguageClient(BaseLanguageClient):

    async def server_exit(self, server: asyncio.subprocess.Process):
        """Called when the server process exits."""

        #   0: all good
        # -15: terminated
        print(server.returncode)
        if server.returncode not in {0, -15}:
            print(f"Server process exited with code: {server.returncode}")

            if server.stderr is not None:
                stderr = await server.stderr.read()
                print(f"Stderr:\n{stderr.decode('utf8')}")

        raise SystemExit(server.returncode)


async def main():
    client = LanguageClient("example-client", "v1")
    await client.start_io("basedpyright-langserver", "--stdio")
    await client.initialize_async(
        types.InitializeParams(
            capabilities=types.ClientCapabilities(),
            root_uri="file:///home/axis/p/axedit",
        )
    )

    response = await client.text_document_completion_async(
        types.CompletionParams(
            types.TextDocumentIdentifier("file:///home/axis/p/axedit/axedit/cli.py"),
            types.Position(0, 1),
        )
    )
    print([item.label for item in response.items])

    await client.shutdown_async(None)
    client.exit(None)
    await client.stop()


asyncio.run(main())

The output

Ignoring notification for unknown method "window/logMessage"
Ignoring notification for unknown method "window/logMessage"
Ignoring notification for unknown method "window/logMessage"

I'm not sure where I'm going wrong, but switching it out with client.start_io("pylsp") or client.start_io("ruff", "server", "--preview") both work in the way you'd expect - with pylsp giving completions and ruff throwing an error because it doesnt support it - but pyrights' are just silent

[blankRiot96]

Related, I found an issue
Apparently an empty object for configuration should be sent for this to work, how does one do that with pygls?
DetachHead/basedpyright#405

edit: figured it out, i think it was:

client.initialized(types.InitializedParams())

[blankRiot96]

Trying it with some updated code now, it still only gives me empty results

import asyncio

from lsprotocol import types
from pygls.lsp.client import BaseLanguageClient


class LanguageClient(BaseLanguageClient):

    async def server_exit(self, server: asyncio.subprocess.Process):
        """Called when the server process exits."""

        #   0: all good
        # -15: terminated
        print(server.returncode)
        if server.returncode not in {0, -15}:
            print(f"Server process exited with code: {server.returncode}")

            if server.stderr is not None:
                stderr = await server.stderr.read()
                print(f"Stderr:\n{stderr.decode('utf8')}")

        raise SystemExit(server.returncode)


async def main():
    client = LanguageClient("example-client", "v1")
    await client.start_io("basedpyright-langserver", "--stdio")
    # await client.start_io("pyright-langserver", "--stdio")
    await client.initialize_async(
        types.InitializeParams(
            capabilities=types.ClientCapabilities(),
            root_uri="file:///home/axis/p/axedit",
        )
    )
    client.initialized(types.InitializedParams())

    response = await client.text_document_completion_async(
        types.CompletionParams(
            types.TextDocumentIdentifier("file:///home/axis/p/axedit/axedit/cli.py"),
            types.Position(0, 2),
        )
    )
    print(f"AUTOCOMPLETE RESULTS = {response}")
    response = await client.text_document_semantic_tokens_full_async(
        types.SemanticTokensParams(
            types.TextDocumentIdentifier("file:///home/axis/p/axedit/axedit/cli.py")
        )
    )
    print(f"SEMANTIC TOKENS RESULT = {response}")

    await client.shutdown_async(None)
    client.exit(None)
    await client.stop()


asyncio.run(main())

Ignoring notification for unknown method "window/logMessage"
Ignoring notification for unknown method "window/logMessage"
Ignoring notification for unknown method "window/logMessage"
Ignoring notification for unknown method "window/logMessage"
Ignoring notification for unknown method "window/logMessage"
Ignoring notification for unknown method "window/logMessage"
Ignoring notification for unknown method "window/logMessage"
Ignoring notification for unknown method "window/logMessage"
Ignoring notification for unknown method "window/logMessage"
AUTOCOMPLETE RESULTS = None
SEMANTIC TOKENS RESULT = SemanticTokens(data=[], result_id='1721972775810')
-15

[alcarney]

To start with we can register a handler for the window/logMessage notification which should start giving us some more information to work with

    client = LanguageClient("example-client", "v1")

    @client.feature(types.WINDOW_LOG_MESSAGE)
    def log_message(ls: LanguageClient, params: types.LogMessageParams):
        print(params.message)

Now when I run your code above I get the following output

$ python test.py 
Pyright language server 1.1.354 starting
Server root directory: file:///nix/store/v75bizr14cwdvfcalcc864w0jid807gw-pyright-1.1.354/lib/node_modules/pyright/dist
Starting service instance "<default>"
No source files found.
AUTOCOMPLETE RESULTS = None
-15

While it's part of the spec root_uri is actually deprecated in favour of workspace_folders, so on a hunch I changed the initialize request to the following

    await client.initialize_async(
        types.InitializeParams(
            capabilities=types.ClientCapabilities(),
            workspace_folders=[
                types.WorkspaceFolder(
                    name="folder",
                    uri="file:///home/alex/Projects/swyddfa/esbonio/develop/lib/esbonio",
                )
            ],
        )
    )

Which seems to be enough to get pyright to do something

Pyright language server 1.1.354 starting
Server root directory: file:///nix/store/v75bizr14cwdvfcalcc864w0jid807gw-pyright-1.1.354/lib/node_modules/pyright/dist
Starting service instance "folder"
Loading pyproject.toml file at /home/alex/Projects/swyddfa/esbonio/develop/lib/esbonio/pyproject.toml
venvPath not specified, so venv settings will be ignored.
Found 57 source files
AUTOCOMPLETE RESULTS = CompletionList(is_incomplete=True, items=[CompletionItem(label='_Feature', label_details=None, kind=<CompletionItemKind.Class: 7>, tags=None, detail=None, documentation=None, deprecated=None, preselect=None, sort_text='09.9999._Feature', filter_text=None, insert_text=None, insert_text_format=None, insert_text_mode=None, text_edit=None, text_edit_text=None, additional_text_edits=None, commit_characters=None, command=None, data={'uri': 'file:///home/alex/Projects/swyddfa/esbonio/develop/lib/esbonio/esbonio/server/server.py', 'position': {'line': 0, 'character': 2}, 'funcParensDisabled': True, 'symbolLabel': '_Feature'}), CompletionItem(label='all_feature_names', label_details=None, kind=<CompletionItemKind.Variable: 6>, tags=None, detail=None, documentation=None, deprecated=None, preselect=None, sort_text='09.9999.all_feature_names', filter_text=None, insert_text=None, insert_text_format=None, insert_text_mode=None, text_edit=None, text_edit_text=None, additional_text_edits=None, commit_characters=None, command=None, data={'uri': 'file:///home/alex/Projects/swyddfa/esbonio/develop/lib/esbonio/esbonio/server/server.py', 'position': {'line': 0, 'character': 2}, 'funcParensDisabled': True, 'symbolLabel': 'all_feature_names'})], item_defaults=None)
-15

Apparently an empty object for configuration should be sent for this to work, how does one do that with pygls?
edit: figured it out, i think it was:

client.initialized(types.InitializedParams())

Actually, that's just informing the server that you've finished your initial client-side setup.
To provide configuration to pyright you're going to want to implement a handler for workspace/configuration requests. See here for an example implementation

Hope that helps!

[blankRiot96]

Thank you for informing me about the workspace URI thing
Yes, This is now my code

    client = LanguageClient("example-client", "v1")
    await client.start_io("basedpyright-langserver", "--stdio")
    await client.initialize_async(
        types.InitializeParams(
            capabilities=types.ClientCapabilities(),
            workspace_folders=[
                types.WorkspaceFolder("folder", "file:///home/axis/p/axedit")
            ],
        )
    )
    client.initialized(types.InitializedParams())
    client.workspace_did_change_configuration(
        types.DidChangeConfigurationParams(types.ConfigurationParams([]))
    )

    @client.feature(types.WINDOW_LOG_MESSAGE)
    def log_message(ls: LanguageClient, params: types.LogMessageParams):
        pass

    file_path = "file:///home/axis/p/axedit/axedit/cli.py"
    client.text_document_did_open(
        types.DidOpenTextDocumentParams(
            text_document=types.TextDocumentItem(
                uri=file_path, language_id="python", version=1, text="pr"
            )
        )
    )

    response = await client.text_document_completion_async(
        types.CompletionParams(
            types.TextDocumentIdentifier(file_path),
            types.Position(0, 2),
        )
    )
    print(f"AUTOCOMPLETE RESULTS = {[item.label for item in response.items]}")
    response = await client.text_document_semantic_tokens_full_async(
        types.SemanticTokensParams(types.TextDocumentIdentifier(file_path))
    )
    print(f"SEMANTIC TOKENS RESULT = {response}")

As you can see I did a textDocument/didChangeConfiguration request
I do get autocomplete to work, but not semantic highlights

Output:

Ignoring notification for unknown method "window/logMessage"
Ignoring notification for unknown method "window/logMessage"
Ignoring notification for unknown method "window/logMessage"
AUTOCOMPLETE RESULTS = ['super', 'property', 'copyright', 'print', 'repr', '__import__', 'StopIteration', 'ImportError', 'LookupError', 'StopAsyncIteration', 'TypeError', 'FloatingPointError', 'ChildProcessError', 'BrokenPipeError', 'InterruptedError', 'PermissionError', 'ProcessLookupError', 'NotImplementedError', 'DeprecationWarning', 'PendingDeprecationWarning', 'ImportWarning', 'BaseExceptionGroup', 'ExceptionGroup', 'import']
Ignoring notification for unknown method "textDocument/publishDiagnostics"
SEMANTIC TOKENS RESULT = SemanticTokens(data=[], result_id='1722091583128')
-15

[blankRiot96]

@alcarney Okay, so I got the semantic highlighting working!

    response = await client.text_document_semantic_tokens_full_async(
        types.SemanticTokensParams(types.TextDocumentIdentifier(file_path))
    )
    print(f"SEMANTIC TOKENS RESULT = {response}")
    # print(list(itertools.batched(response.data, n=5)))

What does the tokens result data actually mean? response.data

SEMANTIC TOKENS RESULT = SemanticTokens(data=[0, 7, 3, 7, 0, 0, 0, 3, 7, 0, 1, 7, 6, 7, 0, 0, 10, 1, 7, 0, 0, 0, 1, 7, 0, 1, 5, 7, 7, 0, 0, 15, 4, 0, 0, 0, 0, 4, 0, 0, 2, 7, 11, 7, 0, 0, 0, 11, 7, 0, 1, 5, 4, 7, 0, 0, 5, 7, 7, 0, 0, 15, 7, 0, 0, 0, 0, 7, 0, 0, 2, 0, 4, 8, 0, 0, 20, 3, 3, 48, 0, 4, 3, 7, 0, 0, 4, 4, 8, 0, 0, 16, 3, 7, 0, 0, 4, 4, 8, 0, 2, 0, 8, 8, 0, 0, 11, 4, 0, 0, 0, 5, 4, 8, 0, 0, 6, 6, 8, 0, 1, 0, 16, 8, 0, 2, 0, 11, 8, 0, 0, 14, 16, 8, 0, 0, 17, 3, 4, 0, 0, 4, 8, 8, 0, 1, 0, 6, 8, 0, 0, 9, 10, 3, 48, 0, 26, 11, 8, 0, 2, 0, 1, 8, 0, 0, 4, 11, 7, 0, 0, 12, 6, 0, 0, 0, 7, 11, 7, 0, 0, 12, 8, 0, 0, 0, 9, 6, 8, 0, 1, 5, 4, 3, 48, 0, 5, 4, 8, 0, 0, 15, 1, 8, 0, 1, 4, 4, 8, 0, 0, 7, 1, 8, 0, 0, 2, 4, 4, 0, 1, 4, 4, 8, 0, 0, 7, 1, 8, 0, 0, 2, 5, 4, 0, 0, 6, 4, 8, 0, 3, 0, 10, 8, 0, 0, 12, 4, 0, 48, 0, 5, 3, 0, 48, 0, 5, 3, 0, 48, 3, 4, 13, 3, 1, 0, 0, 13, 3, 0, 0, 14, 4, 1, 1, 0, 6, 11, 7, 0, 0, 12, 4, 0, 0, 0, 9, 1, 7, 0, 0, 2, 8, 0, 0, 0, 9, 11, 7, 0, 0, 12, 4, 0, 0, 1, 4, 6, 8, 0, 0, 9, 4, 1, 0, 0, 5, 4, 4, 0, 2, 4, 16, 8, 0, 1, 4, 10, 8, 0, 2, 15, 16, 8, 0, 1, 19, 6, 8, 0, 0, 7, 16, 4, 0, 1, 19, 6, 8, 0, 0, 7, 4, 8, 0, 0, 5, 4, 8, 0, 1, 20, 10, 8, 0, 0, 11, 6, 8, 0, 0, 7, 4, 8, 0, 0, 5, 10, 8, 0, 0, 14, 10, 8, 0, 1, 22, 6, 8, 0, 0, 7, 4, 8, 0, 1, 16, 16, 8, 0, 1, 13, 6, 8, 0, 0, 7, 17, 4, 0, 1, 12, 16, 8, 0, 1, 17, 6, 8, 0, 0, 7, 11, 4, 0, 3, 8, 10, 8, 0, 0, 11, 6, 4, 0, 0, 7, 6, 8, 0, 0, 7, 4, 8, 0, 3, 4, 10, 3, 1, 0, 0, 10, 3, 0, 0, 11, 9, 1, 1, 1, 4, 9, 1, 0, 0, 12, 9, 1, 0, 1, 11, 5, 0, 48, 0, 6, 3, 0, 48, 0, 4, 9, 1, 0, 0, 10, 1, 8, 0, 0, 4, 1, 8, 0, 0, 16, 1, 8, 0, 3, 4, 11, 3, 1, 0, 0, 11, 3, 0, 0, 12, 9, 1, 1, 1, 4, 3, 8, 0, 0, 6, 10, 3, 0, 0, 11, 9, 1, 0, 1, 4, 5, 3, 48, 0, 19, 3, 8, 0, 0, 9, 3, 8, 0, 0, 9, 3, 8, 0, 0, 11, 3, 1, 0, 3, 0, 10, 8, 0, 1, 0, 8, 8, 0, 1, 0, 9, 8, 0, 1, 0, 7, 8, 0, 1, 0, 7, 8, 0, 1, 0, 10, 8, 0, 1, 0, 8, 8, 0, 1, 0, 8, 8, 0, 1, 0, 3, 8, 0, 1, 0, 14, 8, 0, 1, 0, 7, 8, 0, 1, 0, 7, 8, 0, 1, 0, 5, 8, 0, 1, 0, 4, 8, 0, 1, 0, 8, 8, 0, 1, 0, 10, 8, 0, 2, 0, 6, 8, 8, 1, 4, 8, 8, 0, 15, 4, 7, 8, 0, 1, 4, 14, 8, 0, 1, 4, 7, 8, 0, 1, 4, 7, 8, 0, 1, 4, 4, 8, 0, 1, 4, 3, 8, 0, 2, 0, 6, 8, 8, 1, 4, 5, 8, 0, 0, 7, 5, 8, 0, 0, 9, 6, 8, 8, 0, 7, 5, 4, 0, 1, 8, 2, 8, 0, 0, 6, 5, 8, 0, 1, 8, 6, 8, 8, 0, 7, 2, 8, 0, 0, 6, 5, 8, 0, 2, 16, 3, 7, 0, 0, 4, 4, 8, 0, 1, 4, 5, 3, 48, 0, 8, 4, 8, 0, 0, 5, 4, 8, 0, 0, 9, 4, 8, 0, 0, 8, 13, 3, 0, 0, 14, 4, 8, 0, 0, 8, 3, 1, 0, 1, 4, 4, 3, 48, 1, 17, 3, 7, 0, 0, 4, 4, 8, 0, 1, 4, 5, 3, 48, 0, 6, 4, 8, 0, 0, 5, 9, 8, 0, 1, 4, 4, 3, 48, 1, 15, 3, 7, 0, 1, 4, 4, 0, 48, 0, 5, 13, 3, 0, 0, 14, 4, 8, 0, 1, 4, 5, 3, 48, 0, 7, 10, 8, 0, 0, 11, 6, 4, 0, 0, 10, 3, 1, 0, 1, 4, 4, 3, 48, 2, 0, 7, 8, 0, 0, 10, 7, 0, 0, 1, 0, 5, 8, 0, 0, 8, 13, 3, 0, 0, 14, 4, 8, 0, 1, 0, 9, 8, 0, 0, 12, 4, 3, 48, 0, 5, 5, 8, 0, 1, 4, 12, 8, 0, 0, 14, 4, 8, 0, 0, 8, 9, 0, 48, 0, 10, 4, 8, 0, 0, 5, 6, 4, 0, 1, 7, 12, 8, 0, 0, 16, 9, 8, 0, 0, 10, 10, 8, 0, 1, 8, 9, 8, 0, 0, 12, 9, 8, 0, 0, 10, 4, 8, 0, 1, 11, 9, 8, 0, 0, 10, 4, 8, 0, 1, 12, 9, 8, 0, 0, 12, 10, 8, 0, 0, 11, 3, 4, 0, 0, 4, 12, 8, 0, 2, 8, 5, 8, 0, 0, 8, 6, 8, 8, 0, 7, 3, 4, 0, 0, 4, 9, 8, 0, 1, 8, 11, 3, 0, 0, 12, 5, 8, 0, 2, 12, 9, 8, 0, 0, 12, 4, 3, 48, 0, 5, 5, 8, 0, 1, 15, 13, 0, 48, 2, 4, 5, 3, 48, 0, 6, 4, 8, 0, 0, 6, 3, 1, 0], result_id='1722108597816')

[blankRiot96]

Nevermind, I figured it out all thanks to the microsoft LSP specification documentation
https://microsoft.github.io/language-server-protocol/specifications/lsp/3.17/specification/#textDocument_semanticTokens