Block producer selects da height to never exceed u64::MAX - 1 transactions from L1 #2189

acerone85 · 2024-09-11T17:08:00Z

Linked Issues/PRs

Part of #2114

Description

When producing a block, we want to make sure that we never include more than 65_536 transactions.
To this end, we select the DA height for including Forced transactions so that no more than 65_535 transactions will ever be included.

Changes:

A new function for the RelayerPort to get the number of forced transactions at a given level
Refactor the function select_new_da_height to return early if more than 65_535 transactions will be included.
Test the changes to the select_new_da_height function.

Checklist

Breaking changes are clearly marked as such in the PR description and changelog
New behavior is reflected in tests
The specification matches the implemented behavior (link update PR if changes are needed)

Before requesting review

I have reviewed the code myself
I have created follow-up issues caused by this PR and linked them here

After merging, notify other teams

[Add or remove entries as needed]

Rust SDK
Sway compiler
Platform documentation (for out-of-organization contributors, the person merging the PR will do this)
Someone else?

crates/services/producer/src/block_producer.rs

xgreenx · 2024-09-13T13:53:44Z

crates/services/producer/src/block_producer.rs

        }

        if highest == previous_da_height {
-            return Ok(highest);
+            return Ok(highest)


It would be nice to avoid unnecessary changes

Done, still trying to figure out the formatter configuration to avoid these. See 5b48295

xgreenx · 2024-09-13T13:55:18Z

crates/services/producer/src/block_producer.rs

            } else {
-                new_best = DaBlockHeight(height);
+                let transactions_number = self
+                    .relayer
+                    .get_transactions_number_for_block(&DaBlockHeight(height))
+                    .await?;
+                total_transactions =
+                    total_transactions.saturating_add(transactions_number);
+                if total_transactions > transactions_limit {
+                    break;
+                } else {
+                    new_best = DaBlockHeight(height);
+                }
            }


It seems like we can simplify it to be

if gas_check { break; } if tx_number_check { break; } new_best = DaBlockHeight(height);

Done, simplified it a bit more in 6b5008d

xgreenx · 2024-09-13T13:57:04Z

crates/services/producer/src/block_producer.rs

@@ -404,7 +409,17 @@ where
            if total_cost > gas_limit {
                break;
            } else {
-                new_best = DaBlockHeight(height);
+                let transactions_number = self


get_cost_for_block and get_transactions_number_for_block do the same lookup into the database. I think it will be better to have one function that returns both values for performance reasons.

Yep, I have changed it in 5b48295 and changed all the adapters implementations accordingly

MitchTurner

LGTM. I'll wait to approve until after someone else looks.

MitchTurner · 2024-09-17T17:15:40Z

crates/services/executor/src/executor.rs

@@ -956,7 +956,7 @@ where
            return Err(ForcedTransactionFailure::InsufficientMaxGas {
                claimed_max_gas,
                actual_max_gas,
-            });


Nit: remove this ";"

xgreenx

LGTM!=)

xgreenx · 2024-09-18T13:46:43Z

CHANGELOG.md

@@ -9,6 +9,7 @@ and this project adheres to [Semantic Versioning](http://semver.org/).
 ### Added
 - [2135](https://github.com/FuelLabs/fuel-core/pull/2135): Added metrics logging for number of blocks served over the p2p req/res protocol.
 - [2151](https://github.com/FuelLabs/fuel-core/pull/2151): Added limitations on gas used during dry_run in API.
+- [2189](https://github.com/FuelLabs/fuel-core/pull/2151): Select next DA height to never include more than u16::MAX -1 transactions from L1.


We need to move this into unreleased section because of release=)

xgreenx · 2024-09-18T13:47:22Z

crates/fuel-core/src/service/adapters/producer.rs

+    async fn get_cost_and_transactions_number_for_block(
+        &self,
+        height: &DaBlockHeight,
+    ) -> anyhow::Result<(u64, u64)> {


I would prefer more structure with fields instead of tuple

xgreenx · 2024-09-18T13:52:43Z

crates/services/producer/src/block_producer/tests.rs

+        .map(|(height, gas_cost)| (DaBlockHeight(height), gas_cost));
+
+        let ctx = TestContextBuilder::new()
+            .with_latest_block_height((prev_da_height + 4u64).into())


Suggested change

.with_latest_block_height((prev_da_height + 4u64).into())

.with_latest_da_block_height_from_relayer((prev_da_height + 4u64).into())

…ed (#2182) ## Linked Issues/PRs  Part of #2114 ## Description - Add a new argument `transactions_limit: u16` to the function `TxSource::next()`. - Changes the logic of the block production to never exceed `u16::MAX` transactions (included FTI transactions and mint transactions) when selecting L2 transacitons. The number of transactions can still exceed `u16::MAX` when selecting transactions from L1. This is addressed in #2189. - Changes the implementation of all TxSource adapters to fetch a number of transactions below or equal to the `transaction_limit`. - Introduces a new WASM host function to peek the next transactions size for a number of transactions below a specified limit. Breaks forward compatibility of the genesis transition function. See https://github.com/FuelLabs/fuel-core/blob/8eeae5d6730d34ae06390b7797478bb98fd07987/version-compatibility/forkless-upgrade/README.md ## TODO: - [x] Check that requirements are met - [x] Implementation for the WasmTxSource is missing (only the argument has been added) - [ ] Integration Tests ## Checklist - [ x] Breaking changes are clearly marked as such in the PR description and changelog - [x] New behavior is reflected in tests - [ ] [The specification](https://github.com/FuelLabs/fuel-specs/) matches the implemented behavior (link update PR if changes are needed) ### Before requesting review - [x] I have reviewed the code myself - [ ] I have created follow-up issues caused by this PR and linked them here ### After merging, notify other teams [Add or remove entries as needed] - [ ] [Rust SDK](https://github.com/FuelLabs/fuels-rs/) - [ ] [Sway compiler](https://github.com/FuelLabs/sway/) - [ ] [Platform documentation](https://github.com/FuelLabs/devrel-requests/issues/new?assignees=&labels=new+request&projects=&template=NEW-REQUEST.yml&title=%5BRequest%5D%3A+) (for out-of-organization contributors, the person merging the PR will do this) - [ ] Someone else?

crates/services/executor/src/executor.rs

crates/services/producer/src/block_producer.rs

…ed (#2182) ## Linked Issues/PRs  Part of #2114 ## Description - Add a new argument `transactions_limit: u16` to the function `TxSource::next()`. - Changes the logic of the block production to never exceed `u16::MAX` transactions (included FTI transactions and mint transactions) when selecting L2 transacitons. The number of transactions can still exceed `u16::MAX` when selecting transactions from L1. This is addressed in #2189. - Changes the implementation of all TxSource adapters to fetch a number of transactions below or equal to the `transaction_limit`. - Introduces a new WASM host function to peek the next transactions size for a number of transactions below a specified limit. Breaks forward compatibility of the genesis transition function. See https://github.com/FuelLabs/fuel-core/blob/8eeae5d6730d34ae06390b7797478bb98fd07987/version-compatibility/forkless-upgrade/README.md ## TODO: - [x] Check that requirements are met - [x] Implementation for the WasmTxSource is missing (only the argument has been added) - [ ] Integration Tests ## Checklist - [ x] Breaking changes are clearly marked as such in the PR description and changelog - [x] New behavior is reflected in tests - [ ] [The specification](https://github.com/FuelLabs/fuel-specs/) matches the implemented behavior (link update PR if changes are needed) ### Before requesting review - [x] I have reviewed the code myself - [ ] I have created follow-up issues caused by this PR and linked them here ### After merging, notify other teams [Add or remove entries as needed] - [ ] [Rust SDK](https://github.com/FuelLabs/fuels-rs/) - [ ] [Sway compiler](https://github.com/FuelLabs/sway/) - [ ] [Platform documentation](https://github.com/FuelLabs/devrel-requests/issues/new?assignees=&labels=new+request&projects=&template=NEW-REQUEST.yml&title=%5BRequest%5D%3A+) (for out-of-organization contributors, the person merging the PR will do this) - [ ] Someone else?

…tions from L1 (#2189) ## Linked Issues/PRs  Part of #2114 ## Description When producing a block, we want to make sure that we never include more than 65_536 transactions. To this end, we select the DA height for including Forced transactions so that no more than 65_535 transactions will ever be included. Changes: - A new function for the RelayerPort to get the number of forced transactions at a given level - Refactor the function `select_new_da_height` to return early if more than 65_535 transactions will be included. - Test the changes to the `select_new_da_height` function. - ## Checklist - [x] Breaking changes are clearly marked as such in the PR description and changelog - [x] New behavior is reflected in tests - [ ] [The specification](https://github.com/FuelLabs/fuel-specs/) matches the implemented behavior (link update PR if changes are needed) ### Before requesting review - [x] I have reviewed the code myself - [ ] I have created follow-up issues caused by this PR and linked them here ### After merging, notify other teams [Add or remove entries as needed] - [ ] [Rust SDK](https://github.com/FuelLabs/fuels-rs/) - [ ] [Sway compiler](https://github.com/FuelLabs/sway/) - [ ] [Platform documentation](https://github.com/FuelLabs/devrel-requests/issues/new?assignees=&labels=new+request&projects=&template=NEW-REQUEST.yml&title=%5BRequest%5D%3A+) (for out-of-organization contributors, the person merging the PR will do this) - [ ] Someone else? --------- Co-authored-by: Green Baneling <XgreenX9999@gmail.com>

@rymnc

## Version v0.37.0 ### Added - [1609](#1609): Add DA compression support. Compressed blocks are stored in the offchain database when blocks are produced, and can be fetched using the GraphQL API. - [2290](#2290): Added a new CLI argument `--graphql-max-directives`. The default value is `10`. - [2195](#2195): Added enforcement of the limit on the size of the L2 transactions per block according to the `block_transaction_size_limit` parameter. - [2131](#2131): Add flow in TxPool in order to ask to newly connected peers to share their transaction pool - [2182](#2151): Limit number of transactions that can be fetched via TxSource::next - [2189](#2151): Select next DA height to never include more than u16::MAX -1 transactions from L1. - [2162](#2162): Pool structure with dependencies, etc.. for the next transaction pool module. Also adds insertion/verification process in PoolV2 and tests refactoring - [2265](#2265): Integrate Block Committer API for DA Block Costs. - [2280](#2280): Allow comma separated relayer addresses in cli - [2299](#2299): Support blobs in the predicates. - [2300](#2300): Added new function to `fuel-core-client` for checking whether a blob exists. ### Changed #### Breaking - [2299](#2299): Anyone who wants to participate in the transaction broadcasting via p2p must upgrade to support new predicates on the TxPool level. - [2299](#2299): Upgraded `fuel-vm` to `0.58.0`. More information in the [release](https://github.com/FuelLabs/fuel-vm/releases/tag/v0.58.0). - [2276](#2276): Changed how complexity for blocks is calculated. The default complexity now is 80_000. All queries that somehow touch the block header now are more expensive. - [2290](#2290): Added a new GraphQL limit on number of `directives`. The default value is `10`. - [2206](#2206): Use timestamp of last block when dry running transactions. - [2153](#2153): Updated default gas costs for the local testnet configuration to match `fuel-core 0.35.0`. ## What's Changed * fix: use core-test.fuellabs.net for dnsaddr resolution by @rymnc in #2214 * Removed state transition bytecode from the local testnet by @xgreenx in #2215 * Send whole transaction pool upon subscription to gossip by @AurelienFT in #2131 * Update default gas costs based on 0.35.0 benchmarks by @xgreenx in #2153 * feat: Use timestamp of last block when dry running transactions by @netrome in #2206 * fix(dnsaddr_resolution): use fqdn separator to prevent suffixing by dns resolvers by @rymnc in #2222 * TransactionSource: specify maximum number of transactions to be fetched by @acerone85 in #2182 * Implement worst case scenario for price algorithm v1 by @rafal-ch in #2219 * chore(gas_price_service): define port for L2 data by @rymnc in #2224 * Block producer selects da height to never exceed u64::MAX - 1 transactions from L1 by @acerone85 in #2189 * Weekly `cargo update` by @github-actions in #2236 * Use fees to calculate DA reward and avoid issues with Gwei/Wei conversions by @MitchTurner in #2229 * Protect against passing `i128::MIN` to `abs()` which causes overflow by @rafal-ch in #2241 * Acquire `da_finalization_period` from the command line by @rafal-ch in #2240 * Executor: test Tx_count limit with incorrect tx source by @acerone85 in #2242 * Minor updates to docs + a few typos fixed by @rafal-ch in #2250 * chore(gas_price_service): move algorithm_updater to fuel-core-gas-price-service by @rymnc in #2246 * Use single heavy input in the `transaction_throughput.rs` benchmarks by @xgreenx in #2205 * Enforce the block size limit by @rafal-ch in #2195 * feat: build ARM and AMD in parallel by @mchristopher in #2130 * Weekly `cargo update` by @github-actions in #2268 * chore(gas_price_service): split into v0 and v1 and squash FuelGasPriceUpdater type into GasPriceService by @rymnc in #2256 * feat(gas_price_service): update block committer da source with established contract by @rymnc in #2265 * Use bytes from `unrecorded_blocks` rather from the block from DA by @MitchTurner in #2252 * TxPool v2 General architecture by @AurelienFT in #2162 * Add value delimiter and tests args by @AurelienFT in #2280 * fix(da_block_costs): remove Arc<Mutex<>> on shared_state and expose channel by @rymnc in #2278 * fix(combined_database): syncing auxiliary databases on startup with custom behaviour by @rymnc in #2272 * fix: Manually encode Authorization header for eventsource_client by @Br1ght0ne in #2284 * Address `async-graphql` vulnerability by @MitchTurner in #2290 * Update the WASM compatibility tests for `0.36` release by @rafal-ch in #2271 * DA compression by @Dentosal in #1609 * Use different port for every version compatibility test by @rafal-ch in #2301 * Fix block query complexity by @xgreenx in #2297 * Support blobs in predicates by @Voxelot in #2299 **Full Changelog**: v0.36.0...v0.37.0

acerone85 added 2 commits September 11, 2024 17:34

Select DA Height takes into account transactions limit

ad1d2b1

Test that DA height is not advanced to go over the transaction limit

4c527aa

acerone85 requested review from xgreenx, Dentosal and MitchTurner as code owners September 11, 2024 17:08

acerone85 mentioned this pull request Sep 12, 2024

TransactionSource: specify maximum number of transactions to be fetched #2182

Merged

11 tasks

Changelog and formatting

87efdde

acerone85 self-assigned this Sep 12, 2024

acerone85 changed the title ~~Block producer tx limit 1~~ Block producer selects da height to never exceed u64::MAX - 1 transactions from L1 Sep 12, 2024

acerone85 commented Sep 12, 2024

View reviewed changes

crates/services/producer/src/block_producer.rs Show resolved Hide resolved

acerone85 assigned xgreenx and unassigned acerone85 Sep 12, 2024

acerone85 force-pushed the block_producer_tx_limit_1 branch from a68ca3a to 17a7144 Compare September 13, 2024 13:22

Fix trailing semicolons

cec811a

acerone85 force-pushed the block_producer_tx_limit_1 branch from 17a7144 to cec811a Compare September 13, 2024 13:24

xgreenx reviewed Sep 13, 2024

View reviewed changes

acerone85 added 2 commits September 13, 2024 15:00

Typo

f355d25

Address code review

5b48295

acerone85 force-pushed the block_producer_tx_limit_1 branch from 1186675 to 5b48295 Compare September 13, 2024 15:40

Fixup

6b5008d

acerone85 requested a review from xgreenx September 16, 2024 11:15

MitchTurner reviewed Sep 17, 2024

View reviewed changes

Merge branch 'master' into block_producer_tx_limit_1

f45d770

xgreenx reviewed Sep 18, 2024

View reviewed changes

acerone85 added 4 commits September 20, 2024 10:36

Merge branch 'master' into block_producer_tx_limit_1

ca016df

Update changelog entry

4a5c9e9

Use struct for relayer response

03f4160

Rename test context builder function

7949399

xgreenx previously approved these changes Sep 20, 2024

View reviewed changes

crates/services/executor/src/executor.rs Outdated Show resolved Hide resolved

Update crates/services/executor/src/executor.rs

9a91abe

xgreenx dismissed their stale review via 9a91abe September 20, 2024 10:53

Merge branch 'master' into block_producer_tx_limit_1

3252f59

xgreenx approved these changes Sep 20, 2024

View reviewed changes

Merge branch 'master' into block_producer_tx_limit_1

7767d32

rafal-ch reviewed Sep 20, 2024

View reviewed changes

crates/services/producer/src/block_producer.rs Show resolved Hide resolved

MitchTurner approved these changes Sep 20, 2024

View reviewed changes

acerone85 mentioned this pull request Sep 20, 2024

Introduce Limiter abstraction for Executor parameters #2234

Closed

acerone85 merged commit ca6a705 into master Sep 20, 2024
36 checks passed

acerone85 deleted the block_producer_tx_limit_1 branch September 20, 2024 17:18

xgreenx mentioned this pull request Oct 5, 2024

Release v0.37.0 #2302

Merged

xgreenx mentioned this pull request Oct 17, 2024

The block production should take into account the available number of transactions #2019

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Block producer selects da height to never exceed u64::MAX - 1 transactions from L1 #2189

Block producer selects da height to never exceed u64::MAX - 1 transactions from L1 #2189

acerone85 commented Sep 11, 2024 •

edited

Loading

xgreenx Sep 13, 2024

acerone85 Sep 13, 2024

xgreenx Sep 13, 2024

acerone85 Sep 13, 2024

xgreenx Sep 13, 2024

acerone85 Sep 13, 2024

MitchTurner left a comment

MitchTurner Sep 17, 2024

xgreenx left a comment

xgreenx Sep 18, 2024

xgreenx Sep 18, 2024

xgreenx Sep 18, 2024

	.with_latest_block_height((prev_da_height + 4u64).into())
	.with_latest_da_block_height_from_relayer((prev_da_height + 4u64).into())

Block producer selects da height to never exceed u64::MAX - 1 transactions from L1 #2189

Block producer selects da height to never exceed u64::MAX - 1 transactions from L1 #2189

Conversation

acerone85 commented Sep 11, 2024 • edited Loading

Linked Issues/PRs

Description

Checklist

Before requesting review

After merging, notify other teams

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MitchTurner left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xgreenx left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

acerone85 commented Sep 11, 2024 •

edited

Loading