Refactor syncing and introduce ChainSource
#365
Conversation
tnull
force-pushed
the
2024-10-chain-source-refactor
branch
2 times, most recently
from
a704f7c
to
1ddb475
Compare
src/chain/mod.rs
Outdated
|
|
||
| let wallet_sync_timeout_fut = tokio::time::timeout( | ||
| Duration::from_secs(BDK_WALLET_SYNC_TIMEOUT_SECS), | ||
| esplora_client.full_scan( |
There was a problem hiding this comment.
Unrelated to this PR, but why is full_scan used over sync?
There was a problem hiding this comment.
On first initialization when we import from a possibly pre-existing seed, we may not have the descriptor index data available. Therefore we first do a full_scan that keeps checking historical data until the stop_gap is reached (this is the approach we previously always employed, although back then the API was called Wallet::sync).
Then, for subsequent syncs we switch to the new incremental sync approach that only retrieves information on previously revealed addresses, which should be more efficient.
That said, in local testing the latter seems to currently be a bit slower than the former. While this is likely a function of the specifics of the regtest network, I'm very curious how much more efficient the new incremental approach works out to be in practice.
| onchain_wallet: Arc<Wallet>, | ||
| onchain_wallet_sync_status: Mutex<WalletSyncStatus>, |
There was a problem hiding this comment.
Will these be common to any ChainSource? If so, should we pass them to sync_onchain_wallet instead? Similarly for the other fields added later. Alternatively, maybe there should be a ChainSync holding these common fields which delegates to ChainSource.
There was a problem hiding this comment.
Mhh, so yes, these fields will likely be common, but I think I'll leave it here as is for now.
We might need to slightly adapt these structures once more in the upcoming bitcoind RPC PR, as there for example we want to use ChainPoller to poll and update onchain wallet and Lightning wallet in one go, i.e., not by calling sync_onchain_wallet/sync_lightning_wallet independently. Happy to do some final cleanup at the end of it then though once it's clear how exactly the wiring will work out.
tnull
force-pushed
the
2024-10-chain-source-refactor
branch
3 times, most recently
from
b90363f
to
996c8eb
Compare
|
Now addressed the open TODO mentioned above: moved Esplora-specific sync options to a new |
tnull
force-pushed
the
2024-10-chain-source-refactor
branch
2 times, most recently
from
daf7cdd
to
1df47fe
Compare
We will be adding some wallet persistence/serialization related types and in a separate module down the line, so here we perepare for it by already moving the wallet code to a module directory.
... we update LDK, lightning-liquidity, BDK, rust-bitcoin, rust-esplora-client, rust-electrum-client, etc.
.. which will allow us to switch between different chain sources.
.. which also gives us the opportunity to simplify and DRY up the logic between background and manual syncing.
Previously, we persisted some of the `latest_` fields exposed via `NodeStatus`. Here, we now refactor this via a persisted `NodeMetrics` struct which allows to persist more fields across restarts. In particular, we now persist the latest time we sync the on-chain wallet, resulting in only doing a full scan on first initialization, and doing incremental syncing afterwards. As both of these operations are really really lightweight, we don't bother to migrate the old persisted timestamps for RGS updates and node announcement broadcasts over to the new data format.
.. to further de-clutter the top-level docs.
.. to also expose it via the `config` module rather than at the top-level docs.
.. as other upcoming chain sources might not have the same config options such as syncing intervals, or at least not with the same semantics.
tnull
force-pushed
the
2024-10-chain-source-refactor
branch
from
1df47fe
to
d9df8ac
Compare
|
Rebased after #366 landed. |
Based on #358
Closes #362
We refactor our chain syncing and introduce a new
ChainSourceenum, which is a prerequisite for adding new chain data sources in a follow-up.To this end, we first move the syncing logic to
ChainSourceand then introduce incremental syncing for Esplora which is always used except for initial syncs (now permanently persisted via a newNodeMetricsstruct).TODO:
EsploraConfigobject, whose defaults can be overridden in theBuilder::set_chain_source_esploramethod.