Message is not JSON serializable.

[Ttayu]

I want to use python-lsp-server and python-lsp-ruff with neovim, but I get the following error.
Other plugins such as python-lsp-black work.

LspLog

[ERROR][2023-11-19 23:17:03] .../vim/lsp/rpc.lua:734 "rpc" "/home/Ttayu/.local/share/nvim/mason/bin/pylsp" "stderr" "2023-11-19 23:17:03,317 JST - ERROR - pylsp_jsonrpc.streams - Failed to write message to output file {'jsonrpc': '2.0', 'method': 'textDocument/publishDiagnostics', 'params': {'uri': 'file:///home/Ttayu/sample.py', 'diagnostics': [{'range': {'start': {'line': 1, 'character': 7}, 'end': {'line': 1, 'character': 13}}, 'message': 'shutil imported but unused', 'severity': 1, 'code': 'F401', 'source': 'ruff', 'tags': [1], 'data': Fix(edits=[Edit(content='', location=Location(row=2, column=1), end_location=Location(row=3, column=1))], message='Remove unused import: shutil', applicability='safe')}, {'range': {'start': {'line': 15, 'character': 0}, 'end': {'line': 15, 'character': 14}}, 'message': 'Module level import not at top of file', 'severity': 2, 'code': 'E402', 'source': 'ruff', 'tags': []}, {'range': {'start': {'line': 15, 'character': 7}, 'end': {'line': 15, 'character': 14}}, 'message': 'pathlib imported but unused', 'severity': 1, 'code': 'F401', 'source': 'ruff', 'tags': [1], 'data': Fix(edits=[Edit(content='', location=Location(row=16, column=1), end_location=Location(row=17, column=1))], message='Remove unused import: pathlib', applicability='safe')}, {'source': 'mypy', 'range': {'start': {'line': 9, 'character': 15}, 'end': {'line': 9, 'character': 38}}, 'message': '"filter[Any]" has no attribute "exists"', 'severity': 1, 'code': 'attr-defined'}]}}\nTraceback (most recent call last):\n File "/home/Ttayu/.local/share/nvim/mason/packages/python-lsp-server/venv/lib/python3.10/site-packages/pylsp_jsonrpc/streams.py", line 97, in write\n body = json.dumps(message, **self._json_dumps_args)\nTypeError: Fix(edits=[Edit(content='', location=Location(row=2, column=1), end_location=Location(row=3, column=1))], message='Remove unused import: shutil', applicability='safe') is not JSON serializable\n"

import shutil
from pathlib import Path
import Path

Environment

python-lsp-ruff: 1.6.0
ruff: 0.1.6
OS:

• WSL-Ubuntu-22.04
  Linux 5.15.133.1-microsoft-standard-WSL2
• Ubuntu-22.04.3 LTS
  5.15.0-88-generic

NeoVim: NVIM v0.9.4, v0.10.0-dev-09a17f9

Build type: Release
LuaJIT 2.1.1692716794

pip list

/home/Ttayu/.local/share/nvim/mason/packages/python-lsp-server/venv/bin/pip list

Package               Version
--------------------- --------
astroid               3.0.1
attrs                 23.1.0
autopep8              2.0.4
cattrs                23.2.1
dill                  0.3.7
docstring-to-markdown 0.13
flake8                6.1.0
importlib-metadata    6.8.0
isort                 5.12.0
jedi                  0.19.1
lsprotocol            2023.0.0
mccabe                0.7.0
packaging             23.2
parso                 0.8.3
pip                   23.2.1
platformdirs          4.0.0
pluggy                1.3.0
pycodestyle           2.11.1
pydocstyle            6.3.0
pyflakes              3.1.0
pylint                3.0.2
python-lsp-jsonrpc    1.1.2
python-lsp-ruff       1.6.0
python-lsp-server     1.9.0
pytoolconfig          1.2.6
rope                  1.11.0
ruff                  0.1.6
setuptools            65.5.0
snowballstemmer       2.2.0
tomli                 2.0.1
tomlkit               0.12.3
ujson                 5.8.0
whatthepatch          1.0.5
yapf                  0.40.2
zipp                  3.17.0

[Ttayu]

I have found the cause. This is caused by cattrs.
Install cattrs==23.1.2 instead of cattrs==23.2.1.

This is a dependent library of lsprotoco

python-lsp-ruff==1.6.0
├── lsprotocol [required: >=2022.0.0a1, installed: 2023.0.0]
│   ├── attrs [required: >=21.3.0, installed: 23.1.0]
│   └── cattrs [required: Any, installed: 23.1.2]  <------ *
│       ├── attrs [required: >=20, installed: 23.1.0]
│       ├── exceptiongroup [required: Any, installed: 1.1.3]
│       └── typing-extensions [required: >=4.1.0, installed: 4.8.0]
├── python-lsp-server [required: Any, installed: 1.9.0]
│   ├── docstring-to-markdown [required: Any, installed: 0.13]
│   ├── jedi [required: >=0.17.2,<0.20.0, installed: 0.19.1]
│   │   └── parso [required: >=0.8.3,<0.9.0, installed: 0.8.3]
│   ├── pluggy [required: >=1.0.0, installed: 1.3.0]
│   ├── python-lsp-jsonrpc [required: >=1.1.0,<2.0.0, installed: 1.1.2]
│   │   └── ujson [required: >=3.0.0, installed: 5.8.0]
│   └── ujson [required: >=3.0.0, installed: 5.8.0]
├── ruff [required: >=0.1.0,<0.2.0, installed: 0.1.6]
└── tomli [required: >=1.1.0, installed: 2.0.1]

[jhossbach]

Hey @Ttayu, can you also check the logs of pylsp? You have to call pylsp with e.g. -vvv --log-file=/tmp/lsp.log

[Ttayu]

[ERROR][2023-11-19 23:17:03] .../vim/lsp/rpc.lua:734 "rpc" "/home/Ttayu/.local/share/nvim/mason/bin/pylsp" "stderr" "2023-11-19 23:17:03,317 JST - ERROR - pylsp_jsonrpc.streams - Failed to write message to output file {'jsonrpc': '2.0', 'method': 'textDocument/publishDiagnostics', 'params': {'uri': 'file:///home/Ttayu/sample.py', 'diagnostics': [{'range': {'start': {'line': 1, 'character': 7}, 'end': {'line': 1, 'character': 13}}, 'message': 'shutil imported but unused', 'severity': 1, 'code': 'F401', 'source': 'ruff', 'tags': [1], 'data': Fix(edits=[Edit(content='', location=Location(row=2, column=1), end_location=Location(row=3, column=1))], message='Remove unused import: shutil', applicability='safe')}, {'range': {'start': {'line': 15, 'character': 0}, 'end': {'line': 15, 'character': 14}}, 'message': 'Module level import not at top of file', 'severity': 2, 'code': 'E402', 'source': 'ruff', 'tags': []}, {'range': {'start': {'line': 15, 'character': 7}, 'end': {'line': 15, 'character': 14}}, 'message': 'pathlib imported but unused', 'severity': 1, 'code': 'F401', 'source': 'ruff', 'tags': [1], 'data': Fix(edits=[Edit(content='', location=Location(row=16, column=1), end_location=Location(row=17, column=1))], message='Remove unused import: pathlib', applicability='safe')}, {'source': 'mypy', 'range': {'start': {'line': 9, 'character': 15}, 'end': {'line': 9, 'character': 38}}, 'message': '"filter[Any]" has no attribute "exists"', 'severity': 1, 'code': 'attr-defined'}]}}\nTraceback (most recent call last):\n File "/home/Ttayu/.local/share/nvim/mason/packages/python-lsp-server/venv/lib/python3.10/site-packages/pylsp_jsonrpc/streams.py", line 97, in write\n body = json.dumps(message, **self._json_dumps_args)\nTypeError: Fix(edits=[Edit(content='', location=Location(row=2, column=1), end_location=Location(row=3, column=1))], message='Remove unused import: shutil', applicability='safe') is not JSON serializable\n"

@jhossbach
Is this not enough?
Sorry, I don't know how to do this using the CLI.
simply running this command pylsp -vvv --log-file=./lsp.log did not yield any messages

2023-11-20 18:01:41,551 JST - INFO - pylsp.python_lsp - Starting PythonLSPServer IO language server

[Tinche]

Hi,

I'm the author of cattrs and came here via python-attrs/cattrs#453.

It would appear the unstructure at https://github.com/python-lsp/python-lsp-ruff/blob/ce637aecf46df5a40fee1e5301b0618ff58d2ba5/pylsp_ruff/plugin.py#L138C46-L138C46 isn't unstructuring completely so json encoding blows up. Would be great if someone could fish out a nice example of a problematic input payload.

[Tinche]

I managed to track it down. There's a type in lsprotocol.types, LSPAny, typed as Union[Any, None].

Looks like there's a regression in unstructuring this type, it used to be unstructured as the runtime type, and now is unstructured as Any.

[jhossbach]

Great, thanks for tracking it down!
It looks to me like this should be fixed in lsprotocol, e.g. why is data in the implementation marked as Optional if LSPAny itself is a union of None, Any?
What do you think is the solution @Tinche?

[Tinche]

Eh it's difficult to say since the type is weird.

This is definitely being caused by a change in cattrs but I don't really know whether the current behavior or the old behavior was correct.

It comes down to how cattrs handles unstructuring fields annotated as Any (the old behavior was masking inconsistencies here). Looking at the docs, it's actually not specified so there may be some leeway here but I'm still a little bit concerned about backwards compatibility.

I think the old behavior might be more useful. My plan right now is to restore the old behavior in this particular case (Optional[Any]) and then do a consistent rework of Any handling in 24.1. So I'll probably release a patch update today.

[Tinche]

@Ttayu can you try installing cattrs from https://github.com/python-attrs/cattrs/tree/23.2 and see if it fixes the issue?

If yes, I'll push it out.

[Ttayu]

@Tinche
I've just checked. The error is fixed and python-lsp-ruff is working correctly.

pip install git+https://github.com/python-attrs/cattrs.git@23.2

Thank you for your great contribution.

[Tinche]

23.2.2 has been released.

[jhossbach]

Great! I will create a topic in lsprocotol and ask them to finetune their dependency