Connect the block importer with outside world #915

xgreenx · 2023-01-18T15:03:20Z

Closes Importer/CM integration #887
Closes [CM] defer commitment of executed blocks to the importer #884

What is done:

Removed CHAIN_HEIGHT_KEY, because now we have a genesis block to track this information.
Unified all database API that works with BlockHeight to work with references. Previously in some places, it was the reference, in some not.
Added manual_produce_block into PoA and used it in the GraphQL to have the one source of the block production. There are no unit tests(help is appreciated).
Connected PoA and BlockImporter via ports. Added BlockImporterAdapter.
Added the FuelBlockRoots column with the corresponding table. It is used to track BMT MMR roots for fuel blocks. The implementation adds new DatabaseColumn and IntoDatabaseKey traits to automate the storage trait implementation for some tables.
Moved implementation of database ports into the corresponding file in the fuel_core::service::adapter.
Removed the Database generic from PoA at all. Now we must pass the last_height while creating the Task. Updated tests to work with it.
Block producer now accepts the block_time along with block_height.

- Added `debug_assert_eq` to verify that the block header id is an identifier of the block. And that it uses the last hash of the application header. No unit tests right now.

Co-authored-by: Tom <tomrgowan@gmail.com>

…track this information. - Added `manual_produce_block` into `PoA` and used it in the `GraphQL` to have the one source of the block production. There are no unit tests(help is appreciated). - Connected `PoA` and `BlockImporter` via ports. Added `BlockImporterAdapter`. - Added the `FuelBlockRoots` column with the corresponding table. It is used to track BMT MMR roots for fuel blocks. The implementation adds new `DatabaseColumn` and `IntoDatabaseKey` traits to automate the storage trait implementation for some tables. - Moved implementation of database ports into the corresponding file in the `fuel_core::service::adapter`. - Removed the `Database` generic from `PoA` at all. Now we must pass the `last_height` while creating the `Task`. Updated tests to work with it. - Block producer now accepts the `block_time` along with `block_height`.

ControlCplusControlV · 2023-01-22T20:30:47Z

.github/workflows/ci.yml

@@ -95,6 +95,8 @@ jobs:
        include:
          - command: clippy
            args: --all-targets --all-features
+          - command: check
+            args: --all-targets


Is there anything caught by this that isn't already caught?

Yep it catches running cargo check --all-targets at the workspace root which nothing else does. It has a different feature set the all-features or running the command at each crate level like the make command does.

Actually, I guess we can remove it, because cargo test --all-features --workspace do the same)

Would prefer to remove it then since CI times are already pretty long

crates/fuel-core/src/database.rs

crates/fuel-core/src/database/storage.rs

ControlCplusControlV · 2023-01-22T20:44:50Z

crates/fuel-core/src/graphql_api/service.rs

 // TODO: When the port of BlockProducer will exist we need to replace it with
 //  `Box<dyn BlockProducerPort>
-pub type BlockProducer = Arc<fuel_core_producer::Producer<crate::database::Database>>;
+pub type BlockProducer = crate::service::adapters::BlockProducerAdapter;


The surface area is really small to change this to Box<dyn BlockProducerPort> and have it in line with the rest of the crate, there is already reference work in #909 you could cannabilize

To use Box here requires the definition of the port. I don't want to add a new code and mix the work that is already handled by a mentioned PR=)

Ok, will refactor that PR off of this one when this gets merged then

ControlCplusControlV · 2023-01-22T20:47:20Z

crates/fuel-core/src/service/genesis.rs

 };
 use itertools::Itertools;

 /// Loads state from the chain config into database
-pub(crate) fn initialize_state(
+pub(crate) fn maybe_initialize_state(


What's the primary reason for the rename? The function returns a result so is the maybe needed?

Returning the result means nothing=) The Err only shows that we got an error during initialization, while Ok maybe be "We changed something" or "We changed nothing"

Oh I see, so it only maybe will have an impact

crates/fuel-core/src/service/genesis.rs

crates/services/producer/src/ports.rs

crates/fuel-core/src/database/block.rs

freesig

Just a few issues around errors instead of options

crates/chain-config/src/config/state.rs

freesig · 2023-01-23T00:22:17Z

crates/fuel-core/src/database/storage.rs

+}
+
+/// The table has a corresponding column in the database.
+trait DatabaseColumn {


@Voxelot Is this the pattern you were talking about in last weeks sync?

crates/fuel-core/src/database/storage.rs

crates/fuel-core/src/service/adapters/producer.rs

# Conflicts: # crates/fuel-core/src/service/config.rs # crates/fuel-core/src/service/sub_services.rs

freesig

Ok looks good

ControlCplusControlV · 2023-01-23T02:10:47Z

Bumping the CI check issue, would prefer to remove that before merging so CI times don't get too long if --workspace is already covering that case

# Conflicts: # crates/fuel-core/src/service/config.rs # crates/fuel-core/src/service/sub_services.rs # crates/services/consensus_module/poa/src/service.rs # crates/services/consensus_module/poa/src/verifier.rs

xgreenx force-pushed the featue/connect-block-importer branch from 64df92f to 646f3c5 Compare January 18, 2023 15:38

xgreenx changed the base branch from feature/block-importer to featue/block-verifier January 18, 2023 15:38

xgreenx self-assigned this Jan 18, 2023

xgreenx mentioned this pull request Jan 18, 2023

Implementation of the block importer #903

Merged

xgreenx requested a review from a team January 18, 2023 17:13

ControlCplusControlV added the fuel-block-importer label Jan 18, 2023

xgreenx force-pushed the featue/block-verifier branch from 6f48202 to 64facf2 Compare January 19, 2023 00:38

xgreenx force-pushed the featue/connect-block-importer branch from 88f203b to 1400dcd Compare January 19, 2023 00:52

xgreenx mentioned this pull request Jan 19, 2023

[WIP] feat: Persistent BMT data for blocks #913

Closed

xgreenx and others added 14 commits January 19, 2023 23:50

- Added the main logic of the block importer.

393b5a7

- Added `debug_assert_eq` to verify that the block header id is an identifier of the block. And that it uses the last hash of the application header. No unit tests right now.

Tested commit_result

a6d835e

Make clippy happy

0b71b7f

Added tests for verify_and_execute_block part of the importer

d7d7a6c

Added test for lock for verify_and_execute_block

d60971a

Update crates/services/importer/src/importer.rs

aa720f3

Co-authored-by: Tom <tomrgowan@gmail.com>

Make fmt happy

7e3ba96

Added verifeir for the block

3be7239

Fixed merge conflicts

f90e026

Fix test to use Validator role

7d21d6b

add unit tests to verify genesis

d9cade3

add unit tests to verify poa block

d67e729

Linked TODO to the issue

5a16bc0

Added one more test for geensis

838a716

xgreenx force-pushed the featue/block-verifier branch from 1c50045 to 838a716 Compare January 19, 2023 23:58

xgreenx added 3 commits January 20, 2023 00:44

Checked that there are no panic.

637f02e

Fix compilation error

ab8c5fd

xgreenx force-pushed the featue/connect-block-importer branch from d4b9fbb to ab8c5fd Compare January 20, 2023 00:44

freesig added 2 commits January 20, 2023 13:07

test manual block production

016a389

check workspace

42c890e

xgreenx added 3 commits January 21, 2023 02:08

Merge branch 'master' into featue/block-verifier

21cf872

Replace poa_config with into

a0ef9a3

Push new snapshots

a195c56

ControlCplusControlV reviewed Jan 22, 2023

View reviewed changes

ControlCplusControlV requested a review from a team January 22, 2023 21:03

freesig mentioned this pull request Jan 23, 2023

Implement IntoDatabaseKey for all keys and use type Type = MultiKey for tuples. #923

Closed

freesig suggested changes Jan 23, 2023

View reviewed changes

Applied suggestions from the PR

cb30cd8

xgreenx requested a review from freesig January 23, 2023 01:36

xgreenx added 2 commits January 23, 2023 01:40

Merge branch 'featue/block-verifier' into featue/connect-block-importer

8de4be2

# Conflicts: # crates/fuel-core/src/service/config.rs # crates/fuel-core/src/service/sub_services.rs

Merge with verifier

872d461

freesig previously approved these changes Jan 23, 2023

View reviewed changes

xgreenx and others added 2 commits January 23, 2023 01:54

Removed "humantime-serde"

3966ad3

Merge branch 'featue/block-verifier' into featue/connect-block-importer

028627e

ControlCplusControlV requested a review from a team January 23, 2023 02:09

ControlCplusControlV mentioned this pull request Jan 23, 2023

Prepare GraphQL For Future Extraction #909

Merged

Voxelot and others added 7 commits January 22, 2023 18:14

upgrade clap in fuel-core cli, improve cli interface for consensus args

f692b3d

fix lockfile

eda254e

cargo lint

13bab2c

add readme instructions for disabling block production

457102f

Merge branch 'featue/block-verifier' into featue/connect-block-importer

9552a24

Make clippy happy

11f9f6d

Merge branch 'featue/block-verifier' into featue/connect-block-importer

09dfafe

Base automatically changed from featue/block-verifier to master January 23, 2023 04:20

Voxelot dismissed freesig’s stale review via 09dfafe January 23, 2023 04:20

Merge branch 'master' into featue/connect-block-importer

f687d69

# Conflicts: # crates/fuel-core/src/service/config.rs # crates/fuel-core/src/service/sub_services.rs # crates/services/consensus_module/poa/src/service.rs # crates/services/consensus_module/poa/src/verifier.rs

freesig approved these changes Jan 23, 2023

View reviewed changes

freesig merged commit 9037d3a into master Jan 23, 2023

freesig deleted the featue/connect-block-importer branch January 23, 2023 06:40

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Connect the block importer with outside world #915

Connect the block importer with outside world #915

xgreenx commented Jan 18, 2023 •

edited

Loading

ControlCplusControlV Jan 22, 2023

freesig Jan 23, 2023

xgreenx Jan 23, 2023

ControlCplusControlV Jan 23, 2023

ControlCplusControlV Jan 22, 2023

xgreenx Jan 23, 2023

ControlCplusControlV Jan 23, 2023

ControlCplusControlV Jan 22, 2023

xgreenx Jan 23, 2023

ControlCplusControlV Jan 23, 2023

freesig left a comment

freesig Jan 23, 2023

freesig left a comment

ControlCplusControlV commented Jan 23, 2023

Connect the block importer with outside world #915

Connect the block importer with outside world #915

Conversation

xgreenx commented Jan 18, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

freesig left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

freesig left a comment

Choose a reason for hiding this comment

ControlCplusControlV commented Jan 23, 2023

xgreenx commented Jan 18, 2023 •

edited

Loading