Use indexation cache to satisfy "coins to spend" queries #2463

rafal-ch · 2024-11-28T13:43:02Z

Closes #2391

This PR includes all changes from the Part 1 PR, making it deprecated.

Description

Changes in this PR:

The new `CoinsToSpend` index

This is the database that stores all coins to spend sorted by the amounts (i.e. largest-by-value coins first)
The key consists of several parts
- Retryable flag - to distinguish between retryable messages and other coins
- Address (owner)
- AssetID
- Amount - as "big-endian" bytes to leverage the RocksDB key sorting capabilities
- Foreign Key - this are bytes of the key from either the Messages or Coins on-chain databases
  - for messages this is a 32-bytes Nonce
  - for coins this is a 34-bytes UtxoId
The value is an instance of IndexedCoinType enum, so we know which on-chain database to query when returning the actual coins
This index is updated when executor events are processed
When querying for "coins to spend" the following algorithm is applied:
- First, we get as many "big" coins as required to satisfy double the amount from the query (respecting max and excluded params)
- If we have enough coins, but there are still some "slots" in the query left (because we selected less coins than max) we fill the remaining slots with a random number of "dust" coins
- If it happens that the value of selected "dust coins" is able to cover the value of some of the already selected "big coins", we remove the latter from the response
- If at any step we encounter a problem (reading from database, integer conversions, etc.) we bail with an appropriate error

Changes to `CoinsQueryError` type

The MaxCoinsReached variant has been removed because in the new algorithm we never query for more coins than the specified max, hence, without additional effort, we are not able to tell whether the query could be satisfied if user provided bigger max
The InsufficientCoins has been renamed to InsufficientCoinsForTheMax and it now contains the additional max field

Off-chain database metadata

The metadata for off-chain database now contain the additional IndexationKind - CoinsToSpend

Refactoring

The indexation.rs module was split into separate files, each per indexation type + errors + some utils.

Other

Integration tests have to be updated to not expect the exact number of coins to spend in the response (currently, due to randomness, we're not deterministic in this regard)
The number of excluded ids in the coinsToSpend GraphQL query is now limited to the maximum number of inputs allowed in transaction.

Before requesting review

I have reviewed the code myself
I have created follow-up issues caused by this PR and linked them here

Follow-up issues

…to_spend_cache_part_2

netrome

Nice stuff so far. I have only done a relatively shallow sweep and will get back again during the week for a deeper look. I have a few nits and suggestions so far, but nothing major.

I'm not a fan of the lack of explicit error handling in crates/fuel-core/src/graphql_api/storage/coins.rs but other than that I think the code looks generally really well-written.

crates/fuel-core/src/coins_query.rs

crates/fuel-core/src/graphql_api.rs

crates/fuel-core/src/graphql_api/indexation/balances.rs

crates/fuel-core/src/graphql_api/storage/coins.rs

…to_spend_cache_part_2

Closes #2474 ## Description This PR changes the following: 1. Adds new test in `tests/tests/relayer.rs` - to ensure that only **non**-retryable messages contribute to the balance returned from `balance()`, `balances()` and `coins_to_spend()`[^1] endpoints 2. Modifies the `balance()` test in `tests/tests/balances.rs` to also include some retryable message which is expected **not** to contribute to the total balance. * for `balances()` and `coins_to_spend()` it was already the case 1<sup>st</sup> test checks the behavior of the actual blockchain operations while the 2<sup>nd</sup> tests the proper initialization of the balances index at genesis. ## Checklist - [X] Breaking changes are clearly marked as such in the PR description and changelog - [X] New behavior is reflected in tests ### Before requesting review - [X] I have reviewed the code myself [^1]: currently, `coins_to_spend()` is not using indexation, this will change when [this PR](#2463) is merged. --------- Co-authored-by: Green Baneling <XgreenX9999@gmail.com>

xgreenx · 2025-01-02T23:33:24Z

crates/fuel-core/src/coins_query.rs

+    excluded_ids: &ExcludedCoinIds<'_>,
+    batch_size: usize,
+) -> Result<Vec<CoinsToSpendIndexEntry>, CoinsQueryError> {
+    const TOTAL_AMOUNT_ADJUSTMENT_FACTOR: u64 = 2;


It would be nice to put a comment why we decided to do it like this=)

xgreenx · 2025-01-02T23:40:57Z

crates/fuel-core/src/graphql_api/indexation/coins_to_spend.rs

+        // TODO[RC]: No clone() required for coins? Double check the types used,
+        // maybe we want `MessageCoin` (which is Copy) for messages?
+        // 1) Currently we use the same types as embedded in the executor `Event`.
+        // 2) `MessageCoin` will refuse to construct itself from a `Message` if the data is empty
+        //      impl TryFrom<Message> for MessageCoin { ... if !data.is_empty() ... }
+        // Actually it shouldn't matter from the indexation perspective, as we just need
+        // to read data from the type and don't care about which data type we took it from.


It is okay to use clone in tests=) I don't think that we need to add more specific type

netrome

Apologies for taking longer than expected. While most of this looks good to me, I have a few questions of some things that puzzle me. Moreover, there are two concrete things I'd like to see updated before approving:

First, we should update the description and name of the random_improve function. Second, I'd like to see an integration test that shows how this algorithm behaves end to end.

netrome · 2025-01-05T23:38:25Z

crates/fuel-core/src/coins_query.rs

Commenting on this file since Github doesn't allow me to comment on some unmodified lines for some reason.

The random_improve function has a TODO comment which references this change. While we still seem to keep the function around for when indexation is unavailable, the comment and name seems to assume the function is to be removed. It should be updated to reflect that the method is used if indexation is not available.

netrome · 2025-01-05T23:39:23Z

crates/fuel-core/src/coins_query.rs

+    excluded_ids: &ExcludedCoinIds<'_>,
+    batch_size: usize,
+) -> Result<Vec<CoinsToSpendIndexEntry>, CoinsQueryError> {
+    const TOTAL_AMOUNT_ADJUSTMENT_FACTOR: u64 = 2;


netrome · 2025-01-05T23:50:37Z

crates/fuel-core/src/coins_query.rs

+    let big_coins_stream = futures::stream::iter(big_coins_iter).yield_each(batch_size);
+    let dust_coins_stream = futures::stream::iter(dust_coins_iter).yield_each(batch_size);


I would expect these to typically be quite short iterators, so do we really need to convert them into futures? The default batch size is 100, and I'd expect it to be quite uncommon to exceed that number of coins for a single transaction. Especially since the max number of inputs in ignition is 255 if I recall correctly.

netrome · 2025-01-05T23:56:42Z

crates/fuel-core/src/graphql_api/indexation/coins_to_spend.rs

+// Indicates that a message is retryable.
+pub(crate) const RETRYABLE_BYTE: [u8; 1] = [0x00];


What does it mean for a message to be retryable? I can see that it means that the message contains data, but what is it that we can retry in this case that we cannot with empty messages?

netrome · 2025-01-06T00:08:06Z

tests/tests/coins.rs

Could we add an integration test that demonstrates the expected behavior of the algorithm? I.e. a test that sets up a wallet with a few big and small coins, makes a query that should be satisfied by adding X big coins and Y small ones for a few sensible scenarios.

rafal-ch added 30 commits October 14, 2024 10:27

Add basic balances functionality

4a70f52

Add support for querying all balances for user

7d472fb

Adding balances_indexation_progress to DB metadata

4296fd4

Attempt at migrating the metadata to store the indexation progress

77d0f16

Merge remote-tracking branch 'upstream/master' into 1965_balances

525e6f9

Hack the replace_forced() and commit_changes_forced in

8eba5d7

Introduce ForcedCommitDatabase

c083f45

Update dependencies

6b76c37

DB metadata can track multiple indexation progresses

f246cd0

into_genesis() attemt

7472db7

Add some TODOs with ideas for the future

b4d2e0f

Use double_key! macro to define the balances key

d38fdb3

Add basic_storage_tests! for Balances

292acab

Merge remote-tracking branch 'upstream/master' into 1965_balances

651bcc8

Balances DB stores separate information for coins and messages

8342c8f

Fix the recursive call

9a9f120

Init indexation progresses with 0 upon metadata migration

6aa9325

Remove debug prints

b153db4

Store incoming balance in the new Balances DB

512a8a3

Read balance from the new Balances database

adf9e2a

Update coin balance, don't overwrite

344ca90

Use more detailed IndexationStatus, not just block height

f73dbed

Add processing of MessageImported

80da09b

Simplify processing of coins and message amounts

ad5216d

Extract increase_balance()

c784e44

Store coin and message balances separately

5762665

Clean up column naming

5595985

Add test for coin balances

38dd8d6

Support both coins and messages in the new balance system

530bcaf

Merge remote-tracking branch 'upstream/master' into 1965_balances

5604f30

rafal-ch added 13 commits December 12, 2024 15:20

Improve error handling in coins to spend query

bef1548

Add test cases for errors in indexed_coins_to_spend

ba883d2

Remove unnecessary variable

0602da1

Swap functions to reduce diff size

9d52683

into_coin_id() does not have to be async

385aea7

setup() function in tests now accepts consensus parameters

1645c14

Simplify vec initialization in into_coin_id()

f1420e3

Update comment

db1e97f

Move the byte conversion from into_coin_id() to the key itself

bdc8ce2

Move the byte conversion from is_excluded() to the key itself

8e42b87

Merge remote-tracking branch 'upstream/master' into rafal_2391_coins_…

e841d66

…to_spend_cache_part_2

Make fields of CoinsToSpendIndexIter pub

2533060

Return proper error from select_coins_to_spend()

6740e7d

rafal-ch mentioned this pull request Dec 13, 2024

Convert the CoinsToSpendIndexKey from Vec<u8> to typed struct, similarly to OwnedTransactionIndexKey #2498

Open

Mention follow-up issue

03a0662

rafal-ch mentioned this pull request Dec 13, 2024

The OffChainDatabase::coins_to_spend_index() function should return error if indexation is not available #2499

Open

Mention follow-up issue

1049fea

rafal-ch requested a review from xgreenx December 13, 2024 12:54

Merge remote-tracking branch 'upstream/master' into rafal_2391_coins_…

9c87c8d

…to_spend_cache_part_2

netrome self-requested a review December 16, 2024 15:14

netrome reviewed Dec 16, 2024

View reviewed changes

rafal-ch added 5 commits December 17, 2024 15:43

Mention follow-up issue in the comment

65f701d

Replace as u128 with u128::from() where applicable

5eb9bb0

Getters on CoinsToSpendIndexKey will no longer panic

792dc33

Remove comment which is no longer relevant

4cc75bf

Merge remote-tracking branch 'upstream/master' into rafal_2391_coins_…

f776cb4

…to_spend_cache_part_2

xgreenx mentioned this pull request Jan 2, 2025

Use own codec for CoinsToSpendIndexKey #2529

Open

1 task

xgreenx reviewed Jan 2, 2025

View reviewed changes

netrome requested changes Jan 6, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use indexation cache to satisfy "coins to spend" queries #2463

Use indexation cache to satisfy "coins to spend" queries #2463

rafal-ch commented Nov 28, 2024 •

edited

Loading

netrome left a comment •

edited

Loading

xgreenx Jan 2, 2025

netrome Jan 5, 2025

xgreenx Jan 2, 2025

netrome left a comment

netrome Jan 5, 2025

netrome Jan 5, 2025

netrome Jan 5, 2025

netrome Jan 5, 2025

netrome Jan 6, 2025

		let big_coins_stream = futures::stream::iter(big_coins_iter).yield_each(batch_size);
		let dust_coins_stream = futures::stream::iter(dust_coins_iter).yield_each(batch_size);

		// Indicates that a message is retryable.
		pub(crate) const RETRYABLE_BYTE: [u8; 1] = [0x00];

Use indexation cache to satisfy "coins to spend" queries #2463

Are you sure you want to change the base?

Use indexation cache to satisfy "coins to spend" queries #2463

Conversation

rafal-ch commented Nov 28, 2024 • edited Loading

Description

The new CoinsToSpend index

Changes to CoinsQueryError type

Off-chain database metadata

Refactoring

Other

Before requesting review

Follow-up issues

netrome left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

netrome left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rafal-ch commented Nov 28, 2024 •

edited

Loading

The new `CoinsToSpend` index

Changes to `CoinsQueryError` type

netrome left a comment •

edited

Loading