Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rework syncing code, and replace local sync server #2329

Merged
merged 1 commit into from
Jan 18, 2023
Merged

Rework syncing code, and replace local sync server #2329

merged 1 commit into from
Jan 18, 2023

Conversation

dae
Copy link
Member

@dae dae commented Jan 18, 2023

This PR replaces the existing Python-driven sync server with a new one in Rust. The new server supports both collection and media syncing, and is compatible with both the new protocol mentioned below, and older clients. A setting has been added to the preferences screen to point Anki to a local server, and a similar setting is likely to come to AnkiMobile soon.

Documentation is available here: https://docs.ankiweb.net/sync-server.html

In addition to the new server and refactoring, this PR also makes changes to the sync protocol. The existing sync protocol places payloads and metadata inside a multipart POST body, which causes a few headaches:

  • Legacy clients build the request in a non-deterministic order, meaning the entire request needs to be scanned to extract the metadata.
  • Reqwest's multipart API directly writes the multipart body, without exposing the resulting stream to us, making it harder to track the progress of the transfer. We've been relying on a patched version of reqwest for timeouts, which is a pain to keep up to date.

To address these issues, the metadata is now sent in a HTTP header, with the data payload sent directly in the body. Instead of the slower gzip, we now use zstd. The old timeout handling code has been replaced with a new implementation that wraps the request and response body streams to track progress, allowing us to drop the git dependencies for reqwest, hyper-timeout and tokio-io-timeout.

The main other change to the protocol is that one-way syncs no longer need to downgrade the collection to schema 11 prior to sending.

This PR replaces the existing Python-driven sync server with a new one in Rust.
The new server supports both collection and media syncing, and is compatible
with both the new protocol mentioned below, and older clients. A setting has
been added to the preferences screen to point Anki to a local server, and a
similar setting is likely to come to AnkiMobile soon.

Documentation is available here: <https://docs.ankiweb.net/sync-server.html>

In addition to the new server and refactoring, this PR also makes changes to the
sync protocol. The existing sync protocol places payloads and metadata inside a
multipart POST body, which causes a few headaches:

- Legacy clients build the request in a non-deterministic order, meaning the
entire request needs to be scanned to extract the metadata.
- Reqwest's multipart API directly writes the multipart body, without exposing
the resulting stream to us, making it harder to track the progress of the
transfer. We've been relying on a patched version of reqwest for timeouts,
which is a pain to keep up to date.

To address these issues, the metadata is now sent in a HTTP header, with the
data payload sent directly in the body. Instead of the slower gzip, we now
use zstd. The old timeout handling code has been replaced with a new implementation
that wraps the request and response body streams to track progress, allowing us
to drop the git dependencies for reqwest, hyper-timeout and tokio-io-timeout.

The main other change to the protocol is that one-way syncs no longer need to
downgrade the collection to schema 11 prior to sending.
@dae
Copy link
Member Author

dae commented Jan 18, 2023

These changes have been through a fair bit of local testing, but if you encounter any issues, please let me know.

@tatsumoto-ren
Copy link
Contributor

tatsumoto-ren commented Jan 19, 2023

Is there a particular reason why Anki should use Rust instead of a different language? (like c++ for example? how about go?)

Back in the day there was no Rust code in this repository. Of course at this point you could say that it's already too late to go back and rewrite everything from Rust to another language, but if you could go back in time, would you still make the same decision?

@dae
Copy link
Member Author

dae commented Jan 19, 2023

C++ is a very complicated language with a lot of footguns. It still may make sense in some industries like game development, but I think the industry is moving away from it as a whole, as it's difficult to work with, and too easy to introduce security issues.

I used Go for parts of AnkiWeb's backend for a number of years. It brought some nice things to the table like fast compilation, automatic formatting and good networking support, but I found the language cumbersome to work with as a whole - it repeats the billion dollar mistake with nullable references, has awkward error handling, up until recently had no support for generics, and has some unfortunate decisions like making your compile fail if you comment out a line that makes a variable unused. An article that points out some of the issues: https://fasterthanli.me/articles/i-want-off-mr-golangs-wild-ride

Overall, I'm very happy with the Rust integration. It's a well-designed language that tends to lead to robust code, and is easy to maintain as the size of the codebase increases - something that dynamically-typed languages aren't so good at. The lack of garbage collection makes it easy to integrate into the various clients, and performance is excellent. It's not a perfect language, and there's some syntax sugar I miss from Swift/Kotlin/Python, but I think it was (and is) the best of the available options for this use case, and I've been happy to see it continue to improve/become more popular over the last few years.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants