[MTG-972] synchronizer optimized #333
Conversation
… end or sync, core collection authority refactored - was potentially broken, now as straightforward as possible
| authority | ||
| .as_ref() | ||
| .map(|auth| auth.authority.to_string()) | ||
| .unwrap_or("".to_string()), |
| let config: ApiConfig = setup_config("API_"); | ||
| init_logger(&config.get_log_level()); | ||
| // init_logger(&config.get_log_level()); |
There was a problem hiding this comment.
Please, delete it if it will not be used
| } | ||
|
|
||
| #[tokio::main(flavor = "multi_thread")] | ||
| pub async fn main() -> Result<(), IngesterError> { |
There was a problem hiding this comment.
unwrap() in this main() looks unnecessary especially while the main() returns Result
| @@ -294,7 +296,7 @@ pub async fn get_by_ids< | |||
| }, | |||
| ); | |||
| } | |||
| Err(e) => {} | |||
| Err(_) => {} | |||
There was a problem hiding this comment.
The parameter is unused, indeed. However, completely swallowing the error looks dubious. Maybe should it be logged at least?
| }); | ||
| } | ||
|
|
||
| for (start, end) in fungible_shards.iter() { |
There was a problem hiding this comment.
Seems like fungible shards are waiting for nft shards to be spawned, I don't think it's necessary. Nevertheless, it will not save much time in comparison with the work that should be done.
| total_assets as f64 / duration.as_secs_f64() | ||
| ); | ||
|
|
||
| while let Some(task) = fungible_tasks.join_next().await { |
There was a problem hiding this comment.
Seems like fungible task will be awaited for results only when nfts are awaited. However, they might be done first. I mean, they will be executed in parallel but we always receive fungible results after the nft result while the order is not defined afaik.
…hout some data points
…-for-synchronizer [MTG-1018] Synchronizer bug fix for potential out of order updates
moved shards count to constant reduced the number of nft shards to 48 fmt
This change includes several fixes and optimizations as well as an additional tool to dump the DB into "indexable" csv.
The key change is related to bucketing/sharding of assets processing. The keyspace is divided into buckets/shards with each shard having its own iterator over assets. As of now 64 shares have proven itself as a good starting point for assets. And 8-16 shards for token_accounts. Each shard has its own iterator and is run in a separate blocking thread.
The results are as follows:
Dump (total 568G) created in < 1h 15mins:
Dump and load took slightly less then 5 hours in total: