Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

JSON-RPC stdio server #3695

Merged
merged 1 commit into from
Nov 6, 2022
Merged

JSON-RPC stdio server #3695

merged 1 commit into from
Nov 6, 2022

Conversation

link2xt
Copy link
Collaborator

@link2xt link2xt commented Oct 25, 2022

deltachat-rpc-server is a binary that speaks JSON-RPC protocol (the same as currently used by DeltaChat Desktop to interact with the core) over stdio. Requests are sent over stdin, responses and event notifications are printed on stdout, one JSON dictionary per line. This is similar to how Language Server Protocol works when Visual Studio Code or Emacs talks to clangd or rust-analyzer, actually even simpler.
There is also an example python script in this PR that spawns a deltachat-server and calls a few methods asynchronously (via async-await). The plan is to move it into a separate deltachat-client python package.
Advantages over existing Python bindings are:

  • No deltachat-ffi on the core side (unsafe Rust code) is used, deltachat-server is written completely in safe Rust except unavoidable low-level dependencies. If it crashes, at least it's not our fault. It is hard/impossible to crash it from Python even if you try to.
  • No dependencies on the Python side. We only use built-in json and asyncio libraries to spawn the process and exchange JSONs. It means pypi package can be uploaded to pypi without rebuilding it for each architecture
  • Core is easy to update and statically linked deltachat-rpc-server binary can be distributed separately, you just need to throw it somewhere in the $PATH.
  • Unlike deltachat-ffi blocking calls, this approach is async all the way down. Python is using asyncio and JSON-RPC calls are async rust functions. If a call to library is slow, for example because bot did an expensive database query while processing an incoming message, bot does not block waiting for the reuslt, but processes messages of other users.

@link2xt link2xt force-pushed the link2xt/jsonrpc-bin branch 3 times, most recently from f3f9a5b to 364902a Compare October 25, 2022 19:39
@link2xt
Copy link
Collaborator Author

link2xt commented Oct 25, 2022

Added an example that creates an account, but we need to fix deltachat/yerpc#6 first before JSON-RPC is usable (fix deltachat/yerpc#22 is waiting for review).

@@ -0,0 +1,26 @@
[package]
name = "deltachat-server"
version = "1.98.0"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rust 1.64 allows this to be done using workspace.package. not sure how eagerly we upgrade these days, and in any case probably not something for this PR

deltachat-server/Cargo.toml Outdated Show resolved Hide resolved
response = json.loads(line)
if "id" in response:
response_id = response["id"]
event = self.request_events[response_id]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is it worth handling the potential KeyError that could be raised here?

deltachat-server/rpc-client.py Outdated Show resolved Hide resolved
deltachat-server/rpc-client.py Outdated Show resolved Hide resolved
deltachat-server/rpc-client.py Outdated Show resolved Hide resolved
deltachat-server/rpc-client.py Outdated Show resolved Hide resolved
deltachat-server/src/bin/deltachat-server/main.rs Outdated Show resolved Hide resolved

#[tokio::main(flavor = "multi_thread")]
async fn main() -> Result<()> {
env_logger::Builder::from_env(env_logger::Env::default().default_filter_or("info")).init();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fwiw might be worth using tracing for new things instead of env_logger

deltachat-server/Cargo.toml Outdated Show resolved Hide resolved
@link2xt link2xt force-pushed the link2xt/jsonrpc-bin branch from 5a091fb to 2aa80ba Compare October 26, 2022 21:43
@link2xt link2xt force-pushed the link2xt/jsonrpc-bin branch 2 times, most recently from ae80f1e to bb38e6c Compare October 31, 2022 00:28
@link2xt

This comment was marked as outdated.

@link2xt link2xt force-pushed the link2xt/jsonrpc-bin branch 8 times, most recently from 97abc4f to 57dbdc9 Compare November 5, 2022 18:53
@link2xt link2xt marked this pull request as ready for review November 5, 2022 18:55
@link2xt link2xt force-pushed the link2xt/jsonrpc-bin branch from 57dbdc9 to 7da212d Compare November 5, 2022 20:24
@link2xt link2xt mentioned this pull request Nov 5, 2022
@link2xt
Copy link
Collaborator Author

link2xt commented Nov 5, 2022

This PR is ready for review, JSON-RPC node tests are ported from WebSocket to this stdio server. Python client is moved into #3734.

@link2xt link2xt force-pushed the link2xt/jsonrpc-bin branch 2 times, most recently from 426727f to b839e9c Compare November 5, 2022 21:25
@link2xt link2xt requested review from Simon-Laux and Jikstra November 5, 2022 22:24
Copy link
Member

@flub flub left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh, nice! didn't realise this would completely eliminate the Arc and RwLock, that makes it certainly better.

@link2xt
Copy link
Collaborator Author

link2xt commented Nov 5, 2022

@Simon-Laux @Jikstra
Do you have any comments on JS part of this? Not sure if my code is ok, e.g. when I should call super(), if it's ok to pass stdin/stdout like this etc.

I tried to keep things compatible and avoid breaking WebSocket example, so things are named to avoid conflicts with existing DeltaChat class.

@link2xt link2xt force-pushed the link2xt/jsonrpc-bin branch from c90505b to 4daaac7 Compare November 6, 2022 20:08
It speaks JSON-RPC serialized into JSON Lines over stdio.
@link2xt link2xt force-pushed the link2xt/jsonrpc-bin branch from 4daaac7 to 75ed4fe Compare November 6, 2022 20:34
@link2xt link2xt merged commit 75ed4fe into master Nov 6, 2022
@link2xt link2xt deleted the link2xt/jsonrpc-bin branch November 6, 2022 21:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants