Best way to test servers that implement LanguageServer?

[rockstar]

(This is likely just a documentation issue more than anything, but...)

The way I currently test LanguageServer implementations is to construct the server and then call the direct methods on the server and checking the response. This works fine, until you need to test interactions with the client, e.g. for textDocument/publishDiagnostics. The other oddity that happens as a result of this, is that our language server wraps the client access in an option, because the tests don't have a client when they start up. Now we have production code that has test-specific code in it, which is a huge no-no.

I assume there's better machinery than creating the server raw, but it's not clear exactly what the "happy path" is here. Any help would be appreciated.

[silvanshade]

Just FYI, I think this project is dead.

[rockstar]

Just FYI, I think this project is dead.

That's disappointing. Would that mean lspower development is going to start back up?

[silvanshade]

Just FYI, I think this project is dead.

That's disappointing. Would that mean lspower development is going to start back up?

Probably not. To be honest, I don't think anyone uses it anyway.

[ebkalderon]

I suspect the non-tower story around testing for tower-lsp needs to be improved.

Currently, it should be possible to drive an LspService manually, though it is pretty cumbersome.

use futures::{SinkExt, StreamExt};
use tower_lsp::jsonrpc::{Request, Response};
use tower_lsp::LspService;

let (mut server, mut client) = LspService::new(|client| FooServer { client });

// Use `server.call(...).await` to send a `Request` to the server and receive a `Response`.
// Use `client.next().await` to receive any pending client `Request`s from the server.
// Use `client.send(...).await` to reply to those requests with mock client `Response`s.

This makes use of these trait implementations included with tower-lsp:

impl<S: LanguageServer> Service<Request> for LspService<S> { ... }
impl Stream for ClientSocket { type Item = Request; ... }
impl Sink<Response> for ClientSocket { ... }

It would be nice if we could ship some convenient testing tools to make testing LSP flows easier.

I'm looking to make a maintenance release sometime this week which updates dependencies and makes a few quality-of-life improvements, but I suspect this may require a somewhat larger effort and might be addressed in the medium-term down the line. Let's keep this ticket open to track this.

[ebkalderon]

Related to #229, though that ticket is more about documenting how to test the LspService (as explained in the comment above) rather than the LanguageServer implementations themselves.

[ebkalderon]

Just noticed some relevant information in a prior PR comment I'd like to link here: #344 (comment)

Thanks for looping me in! I don't have a particularly strong stance for or against this addition either. Personally, when I write servers I try to make the underlying state machine easily testable on its own without needing access to tower-lsp functionality at all. Still, I understand that this may be impossible or otherwise satisfactory to everyone's needs.

I wonder why wrapping Mock in the LspService and then retrieving it again is necessary compared to calling the <Mock as LanguageServer> methods directly in tests? I presume this may be because most servers need a Client handle in order to initialize themselves, and there's currently no way to create one (for good reason).

Perhaps we could approach this shortcoming another way by instead shipping a test harness of some kind with tower-lsp or as a separate crate? Something like:

use tower_lsp::test;
 
 #[tokio::test(flavor = "current_thread")]
 async fn test_server() {
     // This sets up a `Client` and `ClientSocket` pair that allow for
     // manual testing of the server.
     //
     // The returned tuple is of type `(Arc<Mock>, ClientSocket)`.
     let (server, mut socket) = test::server(|client| Mock { client });
 
     // Call `LanguageServer` methods directly, examine internal state, etc.
     assert!(server.initialize(...).await.is_ok());
 
     // Let's assume one server method must make a client request.
     let s = server.clone();
     let handle = tokio::spawn(async move { s.initialized(...).await });
 
     // Reply to the request as if we were the client.
     let request = socket.next().await.unwrap();
     socket.send(...).await.unwrap();
 
     // We can still inspect `server`'s state and call other methods
     ///at any time during this.
 
     let response = handle.await.unwrap();
     assert!(response.is_ok());
 }

Any thoughts on this idea? It could be used either as an alternative to or in conjunction with this PR's approach.

[tekumara]

I like the test harness you've proposed above 😍

I too need something that allows both client -> server and server -> client requests to be testable.

At the moment I'm using the following approach to test requests going both ways. It feels a little low-level, and can probably be much improved on (I'm new to Rust!):

    async fn test_did_open_e2e() {
        let initialize = r#"{"jsonrpc":"2.0","method":"initialize","params":{"capabilities":{"textDocumentSync":1}},"id":1}"#;

        let did_open = r#"{
                "jsonrpc": "2.0",
                "method": "textDocument/didOpen",
                "params": {
                  "textDocument": {
                    "uri": "file:///foo.rs",
                    "languageId": "rust",
                    "version": 1,
                    "text": "this is a\ntest fo typos\n"
                  }
                }
              }
              "#;

        let (mut req_client, mut resp_client) = start_server();
        let mut buf = vec![0; 1024];

        req_client.write_all(req(initialize).as_bytes()).await.unwrap();
        let _ = resp_client.read(&mut buf).await.unwrap();

        tracing::info!("{}", did_open);
        req_client.write_all(req(did_open).as_bytes()).await.unwrap();
        let n = resp_client.read(&mut buf).await.unwrap();

        assert_eq!(
            body(&buf[..n]).unwrap(),
            r#"{"jsonrpc":"2.0","method":"textDocument/publishDiagnostics","params":{"diagnostics":[{"message":"`fo` should be `of`, `for`","range":{"end":{"character":7,"line":1},"start":{"character":5,"line":1}},"severity":2,"source":"typos-lsp"}],"uri":"file:///foo.rs","version":1}}"#,
        )
    }

    fn start_server() -> (tokio::io::DuplexStream, tokio::io::DuplexStream) {
        let (req_client, req_server) = tokio::io::duplex(1024);
        let (resp_server, resp_client) = tokio::io::duplex(1024);

        let (service, socket) = LspService::new(|client| Backend { client });

        // start server as concurrent task
        tokio::spawn(Server::new(req_server, resp_server, socket).serve(service));

        (req_client, resp_client)
    }

    fn req(msg: &str) -> String {
        format!("Content-Length: {}\r\n\r\n{}", msg.len(), msg)
    }

    fn body(src: &[u8]) -> Result<&str, anyhow::Error> {
        // parse headers to get headers length
        let mut dst = [httparse::EMPTY_HEADER; 2];

        let (headers_len, _) = match httparse::parse_headers(src, &mut dst)? {
            httparse::Status::Complete(output) => output,
            httparse::Status::Partial => return Err(anyhow::anyhow!("partial headers")),
        };

        // skip headers
        let skipped = &src[headers_len..];

        // return the rest (ie: the body) as &str
        std::str::from_utf8(skipped).map_err(anyhow::Error::from)
    }

[trevor-scheer]

use futures::{SinkExt, StreamExt};
use tower_lsp::jsonrpc::{Request, Response};
use tower_lsp::LspService;

let (mut server, mut client) = LspService::new(|client| FooServer { client });

// Use `server.call(...).await` to send a `Request` to the server and receive a `Response`.
// Use `client.next().await` to receive any pending client `Request`s from the server.
// Use `client.send(...).await` to reply to those requests with mock client `Response`s.

Thanks for this, I was trying to collect all client messages at the end of my test, but the client request futures won't resolve if you don't grab them as they filter in. Pulling them one at a time allows the request flow to progress.

[dalance]

I tried lsp testing based on #355 (comment).

https://github.com/veryl-lang/veryl/blob/master/crates/languageserver/src/tests.rs

In the above test, TestServer implementation may be reusable.
A minimal test code will become like below:

#[tokio::test]
async fn initialize() {
    let mut server = TestServer::new(Backend::new);

    let params = InitializeParams::default();
    let req = Request::build("initialize")
        .params(json!(params))
        .id(1)
        .finish();

    server.send_request(req).await;
    let res = server.recv_response().await;
    assert!(res.is_ok());
}