Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(wallet): Add a vertex history streamer to the wallet websocket API #1082

Merged
merged 1 commit into from
Jul 24, 2024

Conversation

msbrogli
Copy link
Member

@msbrogli msbrogli commented Jul 11, 2024

Motivation

Wallets might need to download their transaction history. The full node currently offers a polling API but this is too slow to both large wallets and high latency environments. This PR implements a streaming API that speeds up the download for wallets.

Streaming protocol

Client sends either request:history:xpub or request:history:manual.
Server open the streaming and follows the documented protocol (see the streamer docustring).

Performance

Testing with a wallet that has 1,060 addresses and 1,187 transactions with 480ms of latency between the wallet and the full node.

Download time

  • Polling API: 17,764ms
  • XPub Streamer: 3,284ms
  • Manual Streamer: 2,893ms

Total time (including a post-processing that occurs after the download):

  • Polling API: 27,336ms
  • XPub Streamer: 13,189ms
  • Manual Streamer: 13,551ms

Notice that the post-processing time is the same for all APIs (~10 seconds).

Acceptance Criteria

  1. Refactor HathorAdminWebsocketFactory.handle_message(), moving it to HathorAdminWebsocketProtocol.
  2. Refactor HathorAdminWebsocketFactory._handle_ping(), moving it to HathorAdminWebsocketProtocol.
  3. Add HathorAdminWebsocketFactory.subscribe_address(), which subscribes the connection to receive real time updates for all vertices related to an address.
  4. Add the websocket message request:history:xpub, which opens a streaming generating addresses from the xpub and stopping when the GAP limit is reached.
  5. Add the websocket message request:history:manual, which opens a streaming from a list of addresses and stopping when the GAP limit is reached or the list of addresses ran out.
  6. Add the websocket message request:history:stop, which stops the current streamer.
  7. Only one streaming at a time per websocket connection.
  8. Set the maximum list of pending addresses in request:history:manual to 5,000 addresses per connection.
  9. Add a capabilities message when a new websocket connection is opened.
  10. Add a --disable-ws-history-streaming argument to the run_node cli command.
  11. Add a disable_history_streaming() to the factory that disables history streaming in all connections too.
  12. Add a disable_history_streaming() to the protocol that stops the streaming if one is opened.

Checklist

  • If you are requesting a merge into master, confirm this code is production-ready and can be included in future releases as soon as it gets merged

@msbrogli msbrogli requested a review from jansegre as a code owner July 11, 2024 19:41
@msbrogli msbrogli self-assigned this Jul 11, 2024
@msbrogli msbrogli force-pushed the feat/ws-history-streamer branch 3 times, most recently from b9937c8 to 6cd3fea Compare July 11, 2024 20:53
@msbrogli msbrogli force-pushed the feat/ws-history-streamer branch from 6cd3fea to 4f71c51 Compare July 12, 2024 05:58
@msbrogli msbrogli force-pushed the feat/ws-history-streamer branch 7 times, most recently from a8a77d0 to b4f9209 Compare July 12, 2024 07:21
@msbrogli msbrogli force-pushed the feat/ws-history-streamer branch from b4f9209 to 8ed7341 Compare July 12, 2024 07:54
@msbrogli msbrogli force-pushed the feat/ws-history-streamer branch from 8ed7341 to ce31121 Compare July 12, 2024 08:44
@msbrogli msbrogli force-pushed the feat/ws-history-streamer branch 5 times, most recently from 38be174 to 2cb449e Compare July 12, 2024 16:26
@msbrogli msbrogli force-pushed the feat/ws-history-streamer branch 4 times, most recently from 887c9f4 to 82c9605 Compare July 16, 2024 17:16
@msbrogli msbrogli force-pushed the feat/ws-history-streamer branch 3 times, most recently from 238ec39 to 6a09d06 Compare July 18, 2024 19:42
@msbrogli msbrogli force-pushed the feat/ws-history-streamer branch 8 times, most recently from f73ae2a to 5383196 Compare July 19, 2024 18:12
jansegre
jansegre previously approved these changes Jul 20, 2024
hathor/websocket/protocol.py Show resolved Hide resolved
hathor/websocket/streamer.py Outdated Show resolved Hide resolved
r4mmer
r4mmer previously approved these changes Jul 22, 2024
@msbrogli msbrogli force-pushed the feat/ws-history-streamer branch from 5383196 to f42d005 Compare July 22, 2024 20:01
@msbrogli msbrogli dismissed stale reviews from jansegre and r4mmer via 4116945 July 22, 2024 20:12
@msbrogli msbrogli force-pushed the feat/ws-history-streamer branch 5 times, most recently from 52ae6e5 to be56084 Compare July 23, 2024 21:45
@msbrogli msbrogli force-pushed the feat/ws-history-streamer branch from 2f7ee1e to 631f2ca Compare July 23, 2024 22:52
@msbrogli msbrogli merged commit 631f2ca into master Jul 24, 2024
12 checks passed
@msbrogli msbrogli deleted the feat/ws-history-streamer branch July 24, 2024 14:33
This was referenced Jul 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

4 participants