pyls remote over tramp not working

[ringprince]

Thank you for the bug report

• I am using the latest version of lsp-mode related packages.
• I checked FAQ and Troubleshooting sections
• You may also try reproduce the issue using clean environment using the following command: M-x lsp-start-plain

Bug description

I am having trouble to set up pyls on a remote machine via tramp.

I've read #1845 , #2514 and #2375 but still can't get it to work.

Either (see below) due to a json issue, pyls-remote remains in 'starting' and the session, thus, has no capabilities.
Or the session starts and has capabilities but these do not return.

Steps to reproduce

For my tests I run

emacs -Q -l ~/.emacs.lsp.minimal '/ssh:<server>:<path/to/remote/pythonfile.py>'

with this ~/.emacs.lsp.minimal

;; -*- mode: Lisp -*-

(require 'package)

(setq debug-on-error t
      no-byte-compile t
      byte-compile-warnings nil
      inhibit-startup-screen t)

(let* ((pkg-list '(use-package lsp-mode lsp-ui
                            ;; modes
                            python-mode)))

  (add-to-list 'package-archives '("gnu" . "http://elpa.gnu.org/packages/") t)
  (add-to-list 'package-archives '("melpa" . "https://melpa.org/packages/") t)

  (package-initialize)
  (package-refresh-contents)

  (mapc (lambda (pkg)
          (unless (package-installed-p pkg)
            (package-install pkg))
          (require pkg))
        pkg-list))


(eval-when-compile
  (require 'use-package))

(use-package tramp
 :ensure nil
 :config
 (add-to-list 'tramp-remote-path 'tramp-own-remote-path))


;; FIRST SUGGESTION
;;
;; from https://github.com/emacs-lsp/lsp-mode/issues/2514#issuecomment-759452037
;;
;; (defun start-file-process-shell-command@around (start-file-process-shell-command name buffer &rest args)
;;   "Start a program in a subprocess.  Return the process object for it.
;;  Similar to `start-process-shell-command', but calls `start-file-process'."
;;   ;; On remote hosts, the local `shell-file-name' might be useless.
;;   (let ((command (mapconcat 'identity args " ")))
;;     (funcall start-file-process-shell-command name buffer command)))
;; (advice-add 'start-file-process-shell-command :around #'start-file-process-shell-command@around)



(use-package lsp-mode
  :ensure t
  :commands (lsp lsp-deferred)
  :init
  (setq lsp-keymap-prefix "C-c l")
  :config
  (setq lisp-log-io t)
  ;; SECOND(a) SUGGESTION
  ;;
  ;; from https://github.com/emacs-lsp/lsp-mode/issues/2375#issuecomment-749132863
  ;;
  ;; (defun lsp-tramp-connection@override (local-command &optional generate-error-file-fn)
  ;;    "Create LSP stdio connection named name.
  ;; LOCAL-COMMAND is either list of strings, string or function which
  ;; returns the command to execute."
  ;;   ;; Force a direct asynchronous process.
  ;;   (add-to-list 'tramp-connection-properties
  ;;                (list (regexp-quote (file-remote-p default-directory))
  ;;                      "direct-async-process" t))
  ;;   (list :connect (lambda (filter sentinel name environment-fn)
  ;;                    (let* ((final-command (lsp-resolve-final-function
  ;; local-command))
  ;;                           (_stderr (or (when generate-error-file-fn
  ;;                                         (funcall generate-error-file-fn name))
  ;;                                       (format "/tmp/%s-%s-stderr" name
  ;;                                               (cl-incf lsp--stderr-index))))
  ;;                           (process-name (generate-new-buffer-name name))
  ;;                           (process-environment
  ;;                            (lsp--compute-process-environment environment-fn))
  ;;                           (proc (make-process
  ;;                                  :name process-name
  ;;                                  :buffer (format "*%s*" process-name)
  ;;                                  :command final-command
  ;;                                  :connection-type 'pipe
  ;;                                  :coding 'no-conversion
  ;;                                  :noquery t
  ;;                                  :filter filter
  ;;                                  :sentinel sentinel
  ;;                                  :file-handler t)))
  ;;                      (cons proc proc)))
  ;;         :test? (lambda () (-> local-command lsp-resolve-final-function
  ;; lsp-server-present?))))
  ;; (advice-add 'lsp-tramp-connection :override #'lsp-tramp-connection@override)
  ;;
  ;;
  ;; SECOND(b) SUGGESTION
  ;;
  ;; from https://github.com/tshu-w/.emacs.d/blob/master/lisp/editor-completion.el#L296-L299
  ;; as suggested in https://github.com/emacs-lsp/lsp-mode/issues/2375#issuecomment-989443667
  ;;
  ;; (defun lsp-tramp-connection@override (local-command &optional generate-error-file-fn)
  ;;    "Create LSP stdio connection named name.
  ;; LOCAL-COMMAND is either list of strings, string or function which
  ;; returns the command to execute."
  ;;    (defvar tramp-connection-properties)
  ;;    (list :connect (lambda (filter sentinel name environment-fn)
  ;;                     ;; Force a direct asynchronous process.
  ;;                     (add-to-list 'tramp-connection-properties
  ;;                                  (list (regexp-quote (file-remote-p default-directory))
  ;;                                        "direct-async-process" t))
  ;;                     (let* ((final-command (lsp-resolve-final-function
  ;;                                            local-command))
  ;;                            (process-name (generate-new-buffer-name name))
  ;;                            (stderr-buf (format "*%s::stderr*" process-name))
  ;;                            (err-buf (generate-new-buffer stderr-buf))
  ;;                            (process-environment
  ;;                             (lsp--compute-process-environment environment-fn))
  ;;                            (proc (make-process
  ;;                                   :name process-name
  ;;                                   :buffer (format "*%s*" process-name)
  ;;                                   :command final-command
  ;;                                   :connection-type 'pipe
  ;;                                   :coding 'no-conversion
  ;;                                   :noquery t
  ;;                                   :filter filter
  ;;                                   :sentinel sentinel
  ;;                                   :stderr err-buf
  ;;                                   :file-handler t)))
  ;;                       (cons proc proc)))
  ;;          :test? (lambda () (-> local-command lsp-resolve-final-function
  ;;                           lsp-server-present?))))
  ;; (advice-add 'lsp-tramp-connection :override #'lsp-tramp-connection@override)
  (lsp-register-client
   (make-lsp-client :new-connection (lsp-tramp-connection "pyls")
                    :major-modes '(python-mode)
                    :remote? t
                    :server-id 'pyls-remote)))



;; THIRD SUGGESTION
;;
;; from https://github.com/emacs-lsp/lsp-mode/issues/1845#issuecomment-699169414
;;
;; (defun lsp--make-message@override (params)
;;   "Create a LSP message from PARAMS, after encoding it to a JSON string."
;;   (let ((body (lsp--json-serialize params)))
;;     (concat "Content-Length: "
;;             (number-to-string (+ 2 (string-bytes body))) ;; dirty fix for pyls remote (https://github.com/emacs-lsp/lsp-mode/issues/1845#issuecomment-699169414)
;;             ;;(number-to-string (1+ (string-bytes body)))
;;             "\r\n\r\n"
;;             body
;;             "\n")))
;; (advice-add 'lsp--make-message :override #'lsp--make-message@override)



(use-package python-mode
  :ensure t
  :custom
  (python-shell-interpreter "python3")
  :hook (python-mode . lsp-deferred))

Without the dirty fix from the THIRD SUGGESTION
(#1845 (comment))
pyls-remote remains in 'starting' and lsp-describe-session reports
no capabilities. Additionally, this is in tmp/pyls-remote-1-stderr:

2022-03-09 05:04:52,260 UTC - ERROR - pyls_jsonrpc.streams - Failed to parse JSON message b'\n\n{"jsonrpc":"2.0","method":"initialize","params":{"processId":null,"rootPath":"/SAN/bioinf_core/work/Uhlig_PD_Dr_Johannes_multicenter_radiomics","clientInfo":{"name":"emacs","version":"GNU Emacs 29.0.50 (build 2, x86_64-pc-linux-gnu, GTK+ Version 3.24.30, cairo version 1.16.0)\\n of 2022-03-08"},"rootUri":"file:///SAN/bioinf_core/work/Uhlig_PD_Dr_Johannes_multicenter_radiomics","capabilities":{"workspace":{"workspaceEdit":{"documentChanges":true,"resourceOperations":["create","rename","delete"]},"applyEdit":true,"symbol":{"symbolKind":{"valueSet":[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26]}},"executeCommand":{"dynamicRegistration":false},"didChangeWatchedFiles":{"dynamicRegistration":true},"workspaceFolders":true,"configuration":true,"codeLens":{"refreshSupport":true},"fileOperations":{"didCreate":false,"willCreate":false,"didRename":false,"willRename":false,"didDelete":false,"willDelete":false}},"textDocument":{"declaration":{"linkSupport":true},"definition":{"linkSupport":true},"implementation":{"linkSupport":true},"typeDefinition":{"linkSupport":true},"synchronization":{"willSave":true,"didSave":true,"willSaveWaitUntil":true},"documentSymbol":{"symbolKind":{"valueSet":[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26]},"hierarchicalDocumentSymbolSupport":true},"formatting":{"dynamicRegistration":true},"rangeFormatting":{"dynamicRegistration":true},"rename":{"dynamicRegistration":true,"prepareSupport":true},"codeAction":{"dynamicRegistration":true,"isPreferredSupport":true,"codeActionLiteralSupport":{"codeActionKind":{"valueSet":["","quickfix","refactor","refactor.extract","refactor.inline","refactor.rewrite","source","source.organizeImports"]}},"resolveSupport":{"properties":["edit","command"]},"dataSupport":true},"completion":{"completionItem":{"snippetSupport":true,"documentationFormat":["markdown","plaintext"],"resolveAdditionalTextEditsSupport":true,"insertReplaceSupport":true,"deprecatedSupport":true,"resolveSupport":{"properties":["documentation","details","additionalTextEdits","command"]},"insertTextModeSupport":{"valueSet":[1,2]}},"contextSupport":true},"signatureHelp":{"signatureInformation":{"parameterInformation":{"labelOffsetSupport":true}}},"documentLink":{"dynamicRegistration":true,"tooltipSupport":true},"hover":{"contentFormat":["markdown","plaintext"]},"foldingRange":{"dynamicRegistration":true},"callHierarchy":{"dynamicRegistration":false},"publishDiagnostics":{"relatedInformation":true,"tagSupport":{"valueSet":[1,2]},"versionSupport":true},"linkedEditingRange":{"dynamicRegistration":true}},"window":{"workDoneProgress":true,"showMessage":null,"showDocument":{"support":true}}},"initializationOptions":null,"workDoneToken":"1"},"id":1'
Traceback (most recent call last):
  File "/path/to/lib/python3.6/site-packages/pyls_jsonrpc/streams.py", line 40, in listen
    message_consumer(json.loads(request_str.decode('utf-8')))
ValueError: Unexpected character in found when decoding object value

With the dirty fix from the THIRD SUGGESTION
(#1845 (comment))
pyls-remote is successfully started and lsp-describe-session reports
some capabilities. Additionally, tmp/pyls-remote-1-stderr is
created, but remains empty. But all of the lsp-related functions
never return (I've even removed the timeout and waited for ~15 minutes).

With the SECOND SUGGESTIONS (a and b) I get

LSP :: The following servers support current file but do not have automatic installation: pyls-remote
You may find the installation instructions at https://emacs-lsp.github.io/lsp-mode/page/languages.
(If you have already installed the server check *lsp-log*).

but (executable-find "pyls" t) successfully finds pyls.

The FIRST SUGGESTIONS does not seem to change any of this behaviour.

Expected behavior

pls-find-references to return (ideally with correct references...)

Which Language Server did you use?

pyls

OS

Linux

Error callstack

No response

Anything else?

Here is my setup.

Client:

• debian (bullseye)
• GNU Emacs 29.0.50 (build 2, x86_64-pc-linux-gnu, GTK+ Version 3.24.30, cairo version 1.16.0) of 2022-03-08
• tramp 2.6.0-pre
• lsp-mode 20220306.1858

Remote Machine:

• CentOS Linux release 7.9.2009 (Core)

[juergenhoetzel]

Possibly related to this egot-fsharp issue/workaround: Make `eglot-fsharp' tramp aware #329