Prune owned coin idx when inputs are spent #1055

Voxelot · 2023-03-08T08:06:23Z

Remove owned coin index entries when coins are spent to improve the performance of random-improve based coin selection.

Already released v0.17.4 hotfix from this branch, ready to merge into mainline now.

crates/fuel-core/src/executor.rs

xgreenx

Could you also update all internal functions to indicate that we return Unspent coins?
Like unspent_owned_coins_ids instead of owned_coins_ids.

Also, in the code, we will fetch coins and try to filter them based on the status, but now we can do it without filtering.

In the main PR for the next release(0.18.0), we also need to update the endpoint to unspent_coins instead of coins.

P.S. We also need to inform SDK and the application team because it could affect them.

crates/fuel-core/src/executor.rs

xgreenx

We can remove the status field from CompressedCoin now=)

crates/fuel-core/src/executor.rs

crates/storage/src/tables.rs

xgreenx

Removing this field will break the database for beta 3=) Let's remove it in the main PR

…ernally

# Conflicts: # crates/fuel-core/src/executor.rs # crates/types/src/services/executor.rs

crates/fuel-core/src/database/coin.rs

xgreenx · 2023-03-11T18:53:11Z

crates/fuel-core/src/executor.rs

-                        *asset_id,
-                        (*maturity).into(),
-                    )?;
-                    coin.status = CoinStatus::Spent;


Could you also remove the status fields from CompressedCoin, please?=)

And in some places, we still try to fetch MessageStatus, but we know that all messages from our database are unspent. Could you also update those places, please?

Co-authored-by: Green Baneling <XgreenX9999@gmail.com>

Voxelot · 2023-03-16T00:30:56Z

Removed spent status from messages and coins. I'm hesitant to rename all the apis to unspent_, since in most utxo based chains it's expected that any utxos returned from the state set are unspent.

xgreenx · 2023-03-16T00:36:09Z

After removing of Spent status setting, we don't need to rename with unspent_ prefix=)

xgreenx · 2023-03-16T00:41:12Z

crates/client/assets/schema.sdl

@@ -111,10 +111,6 @@ type Coin {
 	assetId: AssetId!
 	maturity: U64!
 	"""
-	The spendable status of the coin
-	"""
-	status: CoinStatus!


From an API perspective, we still need to return the status of the coins and messages. Because we plan to move this API to the indexer and they can fetch this information. It is why I said to remove it only from "Compressed" types=)

Hmm maybe, although no one was using this part of the API and so it's possible the indexer may not need to track spent coins either. I'd rather rip the bandaid fully off now, and add the status back later if there's an actual requirement for it.

Also the indexer uses a completely different system for managing graphql schemas, so I don't think it'll be a simple migration of existing logic in fuel-core.

@Voxelot

The change bumps the version to `0.18.0` and exposes `sync_max_get_header` and `sync_max_get_txns` in the helm chart. # Release 0.18.0 ## Overview A new release brings: - Optimization for the execution based on the performance from beta 3 and on internal benchmarks. Improved metrics gathering. - Stabilization and better test coverage of the `fuel-vm`. We removed almost all unsafe code and added test cases for each opcode. Fixed some edge cases with memory in the `fuel-vm`. - Fully integrated Merkle trees and filled all malleable fields in the transactions. - Added retryable messages, removed redundant fields from it, and updated the API to support a new commitment schema. ## What's Changed ### Breaking - All unsafe functions were replaced with safe analog in the `fuel-crypto` - FuelLabs/fuel-vm#346 - `$hp` holds the address of the last available byte in a heap, while previously it was `$hp - 1` - FuelLabs/fuel-vm#377 - Each variant in the `fuel_tx::Input` enum now has its own type - FuelLabs/fuel-vm#364 - Message nonce is unified and now `Bytes32` everywhere - FuelLabs/fuel-vm#394 - Removed the `message_id` field from all places - FuelLabs/fuel-vm#397, FuelLabs/fuel-vm#373, - Unified the block height in all places with the introduction of a new `BlockHeigh` - FuelLabs/fuel-vm#410 - Make SMO instruction take data ptr as an argument - FuelLabs/fuel-vm#404 - Now the chain id affects the signature and predicate's owner and should be passed into the `sign` function - FuelLabs/fuel-vm#406 - Updated the `produce_blocks` endpoint to accept the start time and the number of blocks. All new blocks will use the previous timestamp as a base - #1059 - The `fuel-core` stores only unspent coins and messages, so all API that previously returned spent coins is affected - Prune owned coin idx when inputs are spent by @Voxelot in #1055 - The message proof API has been changed to be compatible with a new version - #1071 - The `fuel-core` now has retryable messages and coin messages. Retryable messages can only be consumed during successful transaction execution. The coin message acts as common coins. `resouces_to_spend` API was replaced with `coins_to_spend` that returns a new `CoinType` type. - #1067 ## All changes * adding fuel-core service monitor to helm chart by @rfuelsh in #1037 * Adding specific app selector by @rfuelsh in #1039 * use sticky sessions for GQL requests to sentries by @Voxelot in #1042 * Add ingress service name to helm chart by @rfuelsh in #1043 * Added test to verify the iteration over owned transactions by @xgreenx in #1041 * Change sentry lb to use clusterIP by @Voxelot in #1045 * Fixed Tempfile dependency by @ControlCplusControlV in #1048 * Write contract code in raw by @freesig in #994 * attempt to use buildjet runners by @Voxelot in #1017 * Set tx pointer during block production by @Voxelot in #1054 * Used `BASE_AMOUNT` for test with bob to pay for fee by @xgreenx in #1057 * Prepare GraphQL Crate for extraction by @ControlCplusControlV in #1012 * Support large contract deployments over p2p by @Voxelot in #1062 * fix yaml and add linting job by @Voxelot in #1063 * Actualized the ABI for message to be compatible with last version of the Solidity contracts. by @xgreenx in #1060 * feat: Contract state and assets merkle data storage by @bvrooman in #1008 * Take into account the previous block timestamp during block production by @xgreenx in #1059 * Prune owned coin idx when inputs are spent by @Voxelot in #1055 * Retrayable messages fuel-core part by @xgreenx in #1067 * chore: Additional Tests for Contract States and Balances by @bvrooman in #1073 * Rust 1.68.1 & Sparse Registry by @Voxelot in #1077 * Nginx tuneup by @Voxelot in #1080 * chore: Update balance and state Merkleization by @bvrooman in #1084 * Support sticky session in the client by @xgreenx in #1079 * Added e2e test to run 1000 `dry_run` by @xgreenx in #1091 * Use docker.io auth credentials to overcome rate limiting by @Voxelot in #1092 * honeycomb tracing subscription by @Voxelot in #1083 * Update withdrawal proof API to support proving from a block header committed to L1 by @xgreenx in #1071 * Upstream v0.17.8 by @Voxelot in #1102 * use chainid for transactions and predicates by @Voxelot in #1107 * fix the dry-run e2e test to actually perform dry runs by @Voxelot in #1111 * peer reputation with separate app & gossipsub score by @leviathanbeak in #1028 * Adding sentry data synchronization by @rfuelsh in #1115 * Capture metrics for graphql api by @Voxelot in #1113 * Update cargo audit and add to commit by @freesig in #1152 * add task params by @leviathanbeak in #1159 * Took into account that block creation could take some time by @xgreenx in #1162 * Applying optimizations from `0.17.11` release by @xgreenx in #1158 **Full Changelog**: v0.17.3...v0.18.0

@Voxelot

The change bumps the version to `0.18.0` and exposes `sync_max_get_header` and `sync_max_get_txns` in the helm chart. # Release 0.18.0 ## Overview A new release brings: - Optimization for the execution based on the performance from beta 3 and on internal benchmarks. Improved metrics gathering. - Stabilization and better test coverage of the `fuel-vm`. We removed almost all unsafe code and added test cases for each opcode. Fixed some edge cases with memory in the `fuel-vm`. - Fully integrated Merkle trees and filled all malleable fields in the transactions. - Added retryable messages, removed redundant fields from it, and updated the API to support a new commitment schema. ## What's Changed ### Breaking - All unsafe functions were replaced with safe analog in the `fuel-crypto` - FuelLabs/fuel-vm#346 - `$hp` holds the address of the last available byte in a heap, while previously it was `$hp - 1` - FuelLabs/fuel-vm#377 - Each variant in the `fuel_tx::Input` enum now has its own type - FuelLabs/fuel-vm#364 - Message nonce is unified and now `Bytes32` everywhere - FuelLabs/fuel-vm#394 - Removed the `message_id` field from all places - FuelLabs/fuel-vm#397, FuelLabs/fuel-vm#373, - Unified the block height in all places with the introduction of a new `BlockHeigh` - FuelLabs/fuel-vm#410 - Make SMO instruction take data ptr as an argument - FuelLabs/fuel-vm#404 - Now the chain id affects the signature and predicate's owner and should be passed into the `sign` function - FuelLabs/fuel-vm#406 - Updated the `produce_blocks` endpoint to accept the start time and the number of blocks. All new blocks will use the previous timestamp as a base - FuelLabs/fuel-core#1059 - The `fuel-core` stores only unspent coins and messages, so all API that previously returned spent coins is affected - Prune owned coin idx when inputs are spent by @Voxelot in FuelLabs/fuel-core#1055 - The message proof API has been changed to be compatible with a new version - FuelLabs/fuel-core#1071 - The `fuel-core` now has retryable messages and coin messages. Retryable messages can only be consumed during successful transaction execution. The coin message acts as common coins. `resouces_to_spend` API was replaced with `coins_to_spend` that returns a new `CoinType` type. - FuelLabs/fuel-core#1067 ## All changes * adding fuel-core service monitor to helm chart by @rfuelsh in FuelLabs/fuel-core#1037 * Adding specific app selector by @rfuelsh in FuelLabs/fuel-core#1039 * use sticky sessions for GQL requests to sentries by @Voxelot in FuelLabs/fuel-core#1042 * Add ingress service name to helm chart by @rfuelsh in FuelLabs/fuel-core#1043 * Added test to verify the iteration over owned transactions by @xgreenx in FuelLabs/fuel-core#1041 * Change sentry lb to use clusterIP by @Voxelot in FuelLabs/fuel-core#1045 * Fixed Tempfile dependency by @ControlCplusControlV in FuelLabs/fuel-core#1048 * Write contract code in raw by @freesig in FuelLabs/fuel-core#994 * attempt to use buildjet runners by @Voxelot in FuelLabs/fuel-core#1017 * Set tx pointer during block production by @Voxelot in FuelLabs/fuel-core#1054 * Used `BASE_AMOUNT` for test with bob to pay for fee by @xgreenx in FuelLabs/fuel-core#1057 * Prepare GraphQL Crate for extraction by @ControlCplusControlV in FuelLabs/fuel-core#1012 * Support large contract deployments over p2p by @Voxelot in FuelLabs/fuel-core#1062 * fix yaml and add linting job by @Voxelot in FuelLabs/fuel-core#1063 * Actualized the ABI for message to be compatible with last version of the Solidity contracts. by @xgreenx in FuelLabs/fuel-core#1060 * feat: Contract state and assets merkle data storage by @bvrooman in FuelLabs/fuel-core#1008 * Take into account the previous block timestamp during block production by @xgreenx in FuelLabs/fuel-core#1059 * Prune owned coin idx when inputs are spent by @Voxelot in FuelLabs/fuel-core#1055 * Retrayable messages fuel-core part by @xgreenx in FuelLabs/fuel-core#1067 * chore: Additional Tests for Contract States and Balances by @bvrooman in FuelLabs/fuel-core#1073 * Rust 1.68.1 & Sparse Registry by @Voxelot in FuelLabs/fuel-core#1077 * Nginx tuneup by @Voxelot in FuelLabs/fuel-core#1080 * chore: Update balance and state Merkleization by @bvrooman in FuelLabs/fuel-core#1084 * Support sticky session in the client by @xgreenx in FuelLabs/fuel-core#1079 * Added e2e test to run 1000 `dry_run` by @xgreenx in FuelLabs/fuel-core#1091 * Use docker.io auth credentials to overcome rate limiting by @Voxelot in FuelLabs/fuel-core#1092 * honeycomb tracing subscription by @Voxelot in FuelLabs/fuel-core#1083 * Update withdrawal proof API to support proving from a block header committed to L1 by @xgreenx in FuelLabs/fuel-core#1071 * Upstream v0.17.8 by @Voxelot in FuelLabs/fuel-core#1102 * use chainid for transactions and predicates by @Voxelot in FuelLabs/fuel-core#1107 * fix the dry-run e2e test to actually perform dry runs by @Voxelot in FuelLabs/fuel-core#1111 * peer reputation with separate app & gossipsub score by @leviathanbeak in FuelLabs/fuel-core#1028 * Adding sentry data synchronization by @rfuelsh in FuelLabs/fuel-core#1115 * Capture metrics for graphql api by @Voxelot in FuelLabs/fuel-core#1113 * Update cargo audit and add to commit by @freesig in FuelLabs/fuel-core#1152 * add task params by @leviathanbeak in FuelLabs/fuel-core#1159 * Took into account that block creation could take some time by @xgreenx in FuelLabs/fuel-core#1162 * Applying optimizations from `0.17.11` release by @xgreenx in FuelLabs/fuel-core#1158 **Full Changelog**: FuelLabs/fuel-core@v0.17.3...v0.18.0

Voxelot added 2 commits March 8, 2023 00:00

prune utxoid from owned coin idx when input is spent

df8cc23

bump versions

aad41a0

Voxelot requested a review from a team March 8, 2023 08:06

Voxelot marked this pull request as draft March 8, 2023 08:06

xgreenx reviewed Mar 8, 2023

View reviewed changes

crates/fuel-core/src/executor.rs Outdated Show resolved Hide resolved

xgreenx reviewed Mar 8, 2023

View reviewed changes

crates/fuel-core/src/executor.rs Outdated Show resolved Hide resolved

Voxelot added 6 commits March 8, 2023 10:41

use table instead of direct db access for OwnedCoins

220edb7

add cookie store to client for session stickyness

8b5fa2c

add cookie based sessions to ingress

b7714a2

cleanup lockfile

b02583f

prune coins and messages while spending them

ca4505f

add comment about cookie store

6baacd8

xgreenx reviewed Mar 8, 2023

View reviewed changes

crates/fuel-core/src/executor.rs Outdated Show resolved Hide resolved

crates/storage/src/tables.rs Outdated Show resolved Hide resolved

xgreenx previously approved these changes Mar 8, 2023

View reviewed changes

add extra safety check to ensure messages aren't already spent

41967f8

Voxelot dismissed xgreenx’s stale review via 41967f8 March 9, 2023 00:16

xgreenx previously approved these changes Mar 9, 2023

View reviewed changes

move OwnedCoins table definition to fuel-core since it isn't used ext…

ccadd8b

…ernally

Voxelot dismissed xgreenx’s stale review via ccadd8b March 9, 2023 00:20

Voxelot mentioned this pull request Mar 10, 2023

add sticky session annotations to ingress #1058

Closed

Merge branch 'master' into Voxelot/beta-3-coin-select-perf

c194cc7

# Conflicts: # crates/fuel-core/src/executor.rs # crates/types/src/services/executor.rs

Voxelot marked this pull request as ready for review March 11, 2023 03:03

xgreenx reviewed Mar 11, 2023

View reviewed changes

xgreenx requested a review from a team March 13, 2023 16:09

xgreenx assigned Voxelot Mar 13, 2023

Voxelot and others added 5 commits March 13, 2023 09:34

Update crates/fuel-core/src/database/coin.rs

4de141c

Co-authored-by: Green Baneling <XgreenX9999@gmail.com>

fmt

052683b

review feedback

c63d8af

merge master

aeda596

remove spent status from messages

53f6905

xgreenx reviewed Mar 16, 2023

View reviewed changes

Voxelot requested a review from xgreenx March 16, 2023 23:33

Voxelot added 3 commits March 16, 2023 16:33

Merge branch 'master' into Voxelot/beta-3-coin-select-perf

51ec7eb

fix ci and remove usages of unspent_

a64749a

Merge branch 'master' into Voxelot/beta-3-coin-select-perf

b3a0f2d

xgreenx approved these changes Mar 17, 2023

View reviewed changes

Voxelot merged commit 13aa644 into master Mar 17, 2023

Voxelot deleted the Voxelot/beta-3-coin-select-perf branch March 17, 2023 00:53

xgreenx mentioned this pull request Apr 27, 2023

Release 0.18.0 #1165

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Prune owned coin idx when inputs are spent #1055

Prune owned coin idx when inputs are spent #1055

Voxelot commented Mar 8, 2023 •

edited

Loading

xgreenx left a comment •

edited

Loading

xgreenx left a comment

xgreenx left a comment

xgreenx Mar 11, 2023

xgreenx Mar 13, 2023

Voxelot commented Mar 16, 2023

xgreenx commented Mar 16, 2023

xgreenx Mar 16, 2023

Voxelot Mar 16, 2023 •

edited

Loading

Prune owned coin idx when inputs are spent #1055

Prune owned coin idx when inputs are spent #1055

Conversation

Voxelot commented Mar 8, 2023 • edited Loading

xgreenx left a comment • edited Loading

Choose a reason for hiding this comment

xgreenx left a comment

Choose a reason for hiding this comment

xgreenx left a comment

Choose a reason for hiding this comment

xgreenx Mar 11, 2023

Choose a reason for hiding this comment

xgreenx Mar 13, 2023

Choose a reason for hiding this comment

Voxelot commented Mar 16, 2023

xgreenx commented Mar 16, 2023

xgreenx Mar 16, 2023

Choose a reason for hiding this comment

Voxelot Mar 16, 2023 • edited Loading

Choose a reason for hiding this comment

Voxelot commented Mar 8, 2023 •

edited

Loading

xgreenx left a comment •

edited

Loading

Voxelot Mar 16, 2023 •

edited

Loading