Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Forkless state transition with upgradable WASM executor #1716

Merged
merged 77 commits into from
Mar 26, 2024
Merged
Changes from 1 commit
Commits
Show all changes
77 commits
Select commit Hold shift + click to select a range
3783579
Extraction of the `FuelBlockSecondaryKeyBlockHeights` table to an off…
xgreenx Feb 15, 2024
3c2d424
Updated CHANGELOG.md
xgreenx Feb 15, 2024
7e8689c
Merge branch 'master' into feature/blocks-second-key-extraction
xgreenx Feb 17, 2024
1d95e63
Merge with `master`
xgreenx Feb 17, 2024
95dfa58
Fix CI
xgreenx Feb 17, 2024
111efbd
Apply comments from the PR
xgreenx Feb 18, 2024
cabca80
Merge branch 'master' into feature/blocks-second-key-extraction
xgreenx Feb 18, 2024
0e20633
Merge branch 'master' into feature/blocks-second-key-extraction
xgreenx Feb 18, 2024
4e8ebc6
Merge branch 'master' into feature/blocks-second-key-extraction
Voxelot Feb 20, 2024
ad09341
Moved `StorageTransaction` to the `fuel-core-storage` crate
xgreenx Feb 21, 2024
b619743
Merge branch 'feature/blocks-second-key-extraction' into feature/comm…
xgreenx Feb 21, 2024
71dbc41
Merge branch 'master' into feature/blocks-second-key-extraction
xgreenx Feb 21, 2024
593ad52
Apply comments from the review
xgreenx Feb 21, 2024
97a96d5
Renamed `FuelBlockSecondaryKeyBlockHeights`
xgreenx Feb 21, 2024
3ce8c17
Remove contstrain
xgreenx Feb 21, 2024
802578e
Merge branch 'master' into feature/blocks-second-key-extraction
xgreenx Feb 21, 2024
5cc6c33
Merge branch 'feature/blocks-second-key-extraction' into feature/comm…
xgreenx Feb 22, 2024
0137658
Merge conflicts
xgreenx Feb 22, 2024
cc86bb6
Merge branch 'master' into feature/commit-changes
xgreenx Feb 22, 2024
5b9bd78
Generalized the implementation of the iterator functionality.
xgreenx Feb 22, 2024
7b1141a
Implemented tracking of the height during committing of the changes.
xgreenx Feb 23, 2024
305e567
Small nits
xgreenx Feb 23, 2024
4716c15
Make clippy happy
xgreenx Feb 23, 2024
518a842
Updated CHANGELOG.md
xgreenx Feb 23, 2024
253d188
Merge branch 'master' into feature/commit-changes
xgreenx Feb 27, 2024
f44b903
Removed TODOs
xgreenx Feb 27, 2024
00ef712
Apply comments from the PR
xgreenx Feb 27, 2024
0c27ff9
Apply comments from the PR
xgreenx Feb 28, 2024
fb5bc50
Merge branch 'master' into feature/commit-changes
xgreenx Feb 28, 2024
0c63393
Forkless state transition with upgradable WASM executor
xgreenx Mar 1, 2024
25b9f57
Merge branch 'master' into feature/commit-changes
xgreenx Mar 6, 2024
309a8bf
Merge with `master` before regenesis
xgreenx Mar 6, 2024
2e64af8
Merge branch 'master' into feature/commit-changes
xgreenx Mar 7, 2024
d682dd8
Merged with the `master`
xgreenx Mar 7, 2024
9344069
Address comemnts form the PR
xgreenx Mar 7, 2024
154328c
Merge branch 'master' into feature/commit-changes
xgreenx Mar 11, 2024
f8b9c07
Merge branch 'master' into feature/commit-changes
xgreenx Mar 11, 2024
d0b95df
Merge branch 'master' into feature/commit-changes
xgreenx Mar 11, 2024
ec92011
Merged `master`
xgreenx Mar 11, 2024
ffecd1d
Fixed tests
xgreenx Mar 11, 2024
dd42659
Some configuration for performance
xgreenx Mar 12, 2024
87e9524
Merge branch 'feature/commit-changes' into feature/forkless-state-tra…
xgreenx Mar 12, 2024
8b3be42
Applied comments from the PR
xgreenx Mar 13, 2024
3899b2b
Merge branch 'feature/commit-changes' into feature/forkless-state-tra…
xgreenx Mar 13, 2024
e8295ea
Merged with parent branch
xgreenx Mar 13, 2024
e12cb77
Remove patch section
xgreenx Mar 13, 2024
944a273
Make CI happy
xgreenx Mar 13, 2024
106df96
Make CI happy
xgreenx Mar 13, 2024
d92380a
Added `wasm32-unknown-unknown` and removed `protobuf`, since libp2p u…
xgreenx Mar 13, 2024
551baf6
Updated the implementation to show the relationship to the semaphore.
xgreenx Mar 13, 2024
47d6df8
Merge branch 'feature/commit-changes' into feature/forkless-state-tra…
xgreenx Mar 13, 2024
f909992
Small nits
xgreenx Mar 13, 2024
e21e58e
Hidded the wasm executor udnder the feature flag.
xgreenx Mar 13, 2024
dd45a89
Merge branch 'feature/commit-changes' into feature/forkless-state-tra…
xgreenx Mar 13, 2024
a94af64
Merge branch 'master' into feature/forkless-state-transition
xgreenx Mar 13, 2024
0916f64
Reverted changes in the cargo lock file.
xgreenx Mar 14, 2024
c37b5bb
Make CI happy
xgreenx Mar 14, 2024
15b921a
Self review
xgreenx Mar 14, 2024
e584453
Added comments, fixed clippy, udpated chagnelog
xgreenx Mar 15, 2024
98bbe6b
Merge branch 'master' into feature/forkless-state-transition
xgreenx Mar 16, 2024
b7da355
Merge branch 'master' into feature/forkless-state-transition
xgreenx Mar 18, 2024
b8ac0e8
Merge branch 'master' into feature/forkless-state-transition
xgreenx Mar 18, 2024
91a7c59
Merged with `master`
xgreenx Mar 18, 2024
e789e4e
Merge branch 'master' into feature/forkless-state-transition
xgreenx Mar 18, 2024
6999c47
Merge branch 'master' into feature/forkless-state-transition
xgreenx Mar 19, 2024
243ac6c
Applied suggestions from the PR.
xgreenx Mar 19, 2024
967289f
Merge branch 'master' into feature/forkless-state-transition
xgreenx Mar 19, 2024
2698382
Merge branch 'master' into feature/forkless-state-transition
xgreenx Mar 20, 2024
a0d99e1
Addressed comments. Make clippy happy
xgreenx Mar 21, 2024
95128e3
Rerun for any changes
xgreenx Mar 21, 2024
8ce8ac0
Up Rust in CI to 1.77
xgreenx Mar 21, 2024
09d3d96
Merge branch 'master' into feature/forkless-state-transition
xgreenx Mar 22, 2024
6328d2d
Revert "Up Rust in CI to 1.77"
xgreenx Mar 22, 2024
06d1826
Addressed comments in the PR
xgreenx Mar 22, 2024
072ec0e
Merge remote-tracking branch 'origin/feature/forkless-state-transitio…
xgreenx Mar 22, 2024
a53dd24
Fixed compilation
xgreenx Mar 22, 2024
325afec
Merge branch 'master' into feature/forkless-state-transition
Voxelot Mar 22, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Merge branch 'master' into feature/commit-changes
# Conflicts:
#	crates/fuel-core/src/database/state.rs
#	crates/fuel-core/src/database/storage.rs
#	crates/fuel-core/src/executor.rs
#	crates/fuel-core/src/graphql_api/worker_service.rs
#	crates/fuel-core/src/schema/dap.rs
#	crates/fuel-core/src/service/genesis/off_chain.rs
#	crates/services/executor/src/executor.rs
#	crates/services/executor/src/ports.rs
#	crates/storage/src/structured_storage.rs
#	crates/storage/src/structured_storage/contracts.rs
xgreenx committed Mar 6, 2024
commit 25b9f57a644181ed2646cda29dff386fea0cc7c3
6 changes: 3 additions & 3 deletions crates/fuel-core/src/database/state.rs
Original file line number Diff line number Diff line change
@@ -19,7 +19,7 @@ pub trait StateInitializer {
slots: S,
) -> Result<(), StorageError>
where
S: Iterator<Item = (Bytes32, Bytes32)>;
S: Iterator<Item = (Bytes32, Vec<u8>)>;
}

impl<S> StateInitializer for S
@@ -32,15 +32,15 @@ where
slots: I,
) -> Result<(), StorageError>
where
I: Iterator<Item = (Bytes32, Bytes32)>,
I: Iterator<Item = (Bytes32, Vec<u8>)>,
{
let slots = slots
.map(|(key, value)| (ContractsStateKey::new(contract_id, &key), value))
.collect_vec();
#[allow(clippy::map_identity)]
<_ as StorageBatchMutate<ContractsState>>::init_storage(
self,
&mut slots.iter().map(|(key, value)| (key, value)),
&mut slots.iter().map(|(key, value)| (key, value.as_slice())),
)
}
}
25 changes: 25 additions & 0 deletions crates/fuel-core/src/database/storage.rs
Original file line number Diff line number Diff line change
@@ -129,6 +129,31 @@ where
}
}

#[cfg(feature = "test-helpers")]
impl<M, Description> StorageWrite<M> for Database<Description>
where
Description: DatabaseDescription,
M: Mappable,
StructuredStorage<DataSource<Description>>:
StorageWrite<M, Error = StorageError> + UseStructuredImplementation<M>,
{
fn write(&mut self, key: &M::Key, buf: &[u8]) -> Result<usize, Self::Error> {
<_ as StorageWrite<M>>::write(&mut self.data, key, buf)
}

fn replace(
&mut self,
key: &M::Key,
buf: &[u8],
) -> Result<(usize, Option<Vec<u8>>), Self::Error> {
<_ as StorageWrite<M>>::replace(&mut self.data, key, buf)
}

fn take(&mut self, key: &M::Key) -> Result<Option<Vec<u8>>, Self::Error> {
<_ as StorageWrite<M>>::take(&mut self.data, key)
}
}

#[cfg(feature = "test-helpers")]
impl<Description, M> StorageBatchMutate<M> for Database<Description>
where
1 change: 1 addition & 0 deletions crates/fuel-core/src/executor.rs
Original file line number Diff line number Diff line change
@@ -2120,6 +2120,7 @@ mod tests {
)
.unwrap()
.into_result();
assert!(skipped_transactions.is_empty());

let verifier = create_executor(db, Default::default());
let verify_result = verifier.execute_without_commit(
182 changes: 64 additions & 118 deletions crates/fuel-core/src/graphql_api/worker_service.rs
Original file line number Diff line number Diff line change
@@ -77,16 +77,17 @@ pub struct Task<TxPool, D> {

impl<TxPool, D> Task<TxPool, D>
where
TxPool: ports::worker::TxPool,
D: ports::worker::Transactional,
{
fn process_block(&mut self, result: SharedImportResult) -> anyhow::Result<()> {
let block = &result.sealed_block.entity;
let mut transaction = self.database.transaction();
// save the status for every transaction using the finalized block id
Self::persist_transaction_status(&result, &mut transaction)?;
persist_transaction_status(&result, &mut transaction)?;

// save the associated owner for each transaction in the block
Self::index_tx_owners_for_block(block, &mut transaction)?;
index_tx_owners_for_block(block, &mut transaction)?;

let height = block.header().height();
let block_id = block.id();
@@ -118,58 +119,57 @@ where
}
}

/// Process the executor events and update the indexes for the messages and coins.
pub fn process_executor_events<'a, Iter, T>(
events: Iter,
block_st_transaction: &mut T,
) -> anyhow::Result<()>
where
Iter: Iterator<Item = Cow<'a, Event>>,
T: OffChainDatabase,
{
for event in events {
match event.deref() {
Event::MessageImported(message) => {
block_st_transaction
.storage_as_mut::<OwnedMessageIds>()
.insert(
&OwnedMessageKey::new(message.recipient(), message.nonce()),
&(),
)?;
}
Event::MessageConsumed(message) => {
block_st_transaction
.storage_as_mut::<OwnedMessageIds>()
.remove(&OwnedMessageKey::new(
message.recipient(),
message.nonce(),
))?;
}
Event::CoinCreated(coin) => {
let coin_by_owner = owner_coin_id_key(&coin.owner, &coin.utxo_id);
block_st_transaction
.storage_as_mut::<OwnedCoins>()
.insert(&coin_by_owner, &())?;
}
Event::CoinConsumed(coin) => {
let key = owner_coin_id_key(&coin.owner, &coin.utxo_id);
block_st_transaction
.storage_as_mut::<OwnedCoins>()
.remove(&key)?;
}
/// Process the executor events and update the indexes for the messages and coins.
pub fn process_executor_events<'a, Iter, T>(
events: Iter,
block_st_transaction: &mut T,
) -> anyhow::Result<()>
where
Iter: Iterator<Item = Cow<'a, Event>>,
T: OffChainDatabase,
{
for event in events {
match event.deref() {
Event::MessageImported(message) => {
block_st_transaction
.storage_as_mut::<OwnedMessageIds>()
.insert(
&OwnedMessageKey::new(message.recipient(), message.nonce()),
&(),
)?;
}
Event::MessageConsumed(message) => {
block_st_transaction
.storage_as_mut::<OwnedMessageIds>()
.remove(&OwnedMessageKey::new(
message.recipient(),
message.nonce(),
))?;
}
Event::CoinCreated(coin) => {
let coin_by_owner = owner_coin_id_key(&coin.owner, &coin.utxo_id);
block_st_transaction
.storage_as_mut::<OwnedCoins>()
.insert(&coin_by_owner, &())?;
}
Event::CoinConsumed(coin) => {
let key = owner_coin_id_key(&coin.owner, &coin.utxo_id);
block_st_transaction
.storage_as_mut::<OwnedCoins>()
.remove(&key)?;
}
}
}
Ok(())
}

/// Associate all transactions within a block to their respective UTXO owners
fn index_tx_owners_for_block<D>(
fn index_tx_owners_for_block<T>(
block: &Block,
block_st_transaction: &mut D,
block_st_transaction: &mut T,
) -> anyhow::Result<()>
where
D: ports::worker::OffChainDatabase,
T: OffChainDatabase,
{
for (tx_idx, tx) in block.transactions().iter().enumerate() {
let block_height = *block.header().height();
@@ -205,16 +205,16 @@ where
}

/// Index the tx id by owner for all of the inputs and outputs
fn persist_owners_index<D>(
fn persist_owners_index<T>(
block_height: BlockHeight,
inputs: &[Input],
outputs: &[Output],
tx_id: &Bytes32,
tx_idx: u16,
db: &mut D,
db: &mut T,
) -> StorageResult<()>
where
D: ports::worker::OffChainDatabase,
T: OffChainDatabase,
{
let mut owners = vec![];
for input in inputs {
@@ -225,90 +225,34 @@ where
}
}

/// Associate all transactions within a block to their respective UTXO owners
fn index_tx_owners_for_block<T>(
block: &Block,
block_st_transaction: &mut T,
) -> anyhow::Result<()>
where
T: OffChainDatabase,
{
for (tx_idx, tx) in block.transactions().iter().enumerate() {
let block_height = *block.header().height();
let inputs;
let outputs;
let tx_idx = u16::try_from(tx_idx).map_err(|e| {
anyhow::anyhow!("The block has more than `u16::MAX` transactions, {}", e)
})?;
let tx_id = tx.cached_id().expect(
"The imported block should contains only transactions with cached id",
);
match tx {
Transaction::Script(tx) => {
inputs = tx.inputs().as_slice();
outputs = tx.outputs().as_slice();
}
Transaction::Create(tx) => {
inputs = tx.inputs().as_slice();
outputs = tx.outputs().as_slice();
}
Transaction::Mint(_) => continue,
for output in outputs {
match output {
Output::Coin { to, .. }
| Output::Change { to, .. }
| Output::Variable { to, .. } => {
owners.push(to);
}
Self::persist_owners_index(
block_height,
inputs,
outputs,
&tx_id,
tx_idx,
block_st_transaction,
)?;
Output::Contract(_) | Output::ContractCreated { .. } => {}
}
}

/// Index the tx id by owner for all of the inputs and outputs
fn persist_owners_index<T>(
block_height: BlockHeight,
inputs: &[Input],
outputs: &[Output],
tx_id: &Bytes32,
tx_idx: u16,
db: &mut T,
) -> StorageResult<()>
where
T: OffChainDatabase,
{
let mut owners = vec![];
for input in inputs {
if let Input::CoinSigned(CoinSigned { owner, .. })
| Input::CoinPredicate(CoinPredicate { owner, .. }) = input
{
owners.push(owner);
}
}
// dedupe owners from inputs and outputs prior to indexing
owners.sort();
owners.dedup();

for owner in owners {
db.record_tx_id_owner(owner, block_height, tx_idx, tx_id)?;
}

fn persist_transaction_status<T>(
import_result: &ImportResult,
db: &mut T,
) -> StorageResult<()>
where
T: OffChainDatabase,
{
for TransactionExecutionStatus { id, result } in import_result.tx_status.iter() {
let status = from_executor_to_status(
&import_result.sealed_block.entity,
result.clone(),
);
Ok(())
}

fn persist_transaction_status<D>(
fn persist_transaction_status<T>(
import_result: &ImportResult,
db: &mut D,
db: &mut T,
) -> StorageResult<()>
where
D: ports::worker::OffChainDatabase,
T: OffChainDatabase,
{
for TransactionExecutionStatus { id, result } in import_result.tx_status.iter() {
let status =
@@ -328,6 +272,7 @@ where
#[async_trait::async_trait]
impl<TxPool, D> RunnableService for Task<TxPool, D>
where
TxPool: ports::worker::TxPool,
D: ports::worker::Transactional,
{
const NAME: &'static str = "GraphQL_Off_Chain_Worker";
@@ -363,6 +308,7 @@ where
#[async_trait::async_trait]
impl<TxPool, D> RunnableTask for Task<TxPool, D>
where
TxPool: ports::worker::TxPool,
D: ports::worker::Transactional,
{
async fn run(&mut self, watcher: &mut StateWatcher) -> anyhow::Result<bool> {
5 changes: 5 additions & 0 deletions crates/fuel-core/src/schema/dap.rs
Original file line number Diff line number Diff line change
@@ -419,11 +419,16 @@ impl DapMutation {
.get_mut(&id)
.ok_or_else(|| async_graphql::Error::new("VM not found"))?;

let params = locked.params.clone();

let checked_tx = tx
.into_checked_basic(vm.as_ref().block_height()?, &params)
.map_err(|err| anyhow::anyhow!("{:?}", err))?
.into();

let gas_costs = params.gas_costs();
let fee_params = params.fee_params();

match checked_tx {
CheckedTransaction::Script(script) => {
let ready_tx = script
4 changes: 2 additions & 2 deletions crates/fuel-core/src/service/genesis/off_chain.rs
Original file line number Diff line number Diff line change
@@ -31,7 +31,7 @@ pub fn execute_genesis_block(
Cow::Owned(Event::MessageImported(message))
});

worker_service::Task::<Database<OffChain>>::process_executor_events(
worker_service::process_executor_events(
messages_events,
&mut database_transaction,
)?;
@@ -44,7 +44,7 @@ pub fn execute_genesis_block(
Cow::Owned(Event::CoinCreated(coin))
});

worker_service::Task::<Database<OffChain>>::process_executor_events(
worker_service::process_executor_events(
coin_events,
&mut database_transaction,
)?;
327 changes: 139 additions & 188 deletions crates/services/executor/src/executor.rs

Large diffs are not rendered by default.

25 changes: 1 addition & 24 deletions crates/storage/src/structured_storage/contracts.rs
Original file line number Diff line number Diff line change
@@ -7,11 +7,7 @@ use crate::{
raw::Raw,
},
column::Column,
kv_store::KeyValueInspect,
structured_storage::{
StructuredStorage,
TableWithBlueprint,
},
structured_storage::TableWithBlueprint,
tables::{
ContractsInfo,
ContractsLatestUtxo,
@@ -32,25 +28,6 @@ impl TableWithBlueprint for ContractsRawCode {
}
}

impl<S> StorageRead<ContractsRawCode> for StructuredStorage<S>
where
S: KeyValueInspect<Column = Column>,
{
fn read(
&self,
key: &ContractId,
buf: &mut [u8],
) -> Result<Option<usize>, Self::Error> {
self.inner.read(key.as_ref(), Column::ContractsRawCode, buf)
}

fn read_alloc(&self, key: &ContractId) -> Result<Option<Vec<u8>>, Self::Error> {
self.inner
.get(key.as_ref(), Column::ContractsRawCode)
.map(|value| value.map(|value| value.deref().clone()))
}
}

impl TableWithBlueprint for ContractsInfo {
type Blueprint = Plain<Raw, Postcard>;
type Column = Column;
You are viewing a condensed version of this merge commit. You can view the full changes here.