Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Applying optimizations from 0.17.11 release #1158

Merged
merged 7 commits into from
Apr 27, 2023
Merged

Applying optimizations from 0.17.11 release #1158

merged 7 commits into from
Apr 27, 2023

Conversation

xgreenx
Copy link
Collaborator

@xgreenx xgreenx commented Apr 25, 2023 •
edited
Loading

Closes #1137

The PR duplicates most of the changes from the #1157 but for the master branch.
It adds two e2e tests, but they fail because of new changes in the fuel-vm(the change to the PC register). It is not critical, but replacing them with the correct scripts later would be nice.

  • One test uploads the contract with a huge large state and executes it. The test takes a lot of time to execute and highlights that we have performance problems with the state calculation: The state_root calculation for the Create transaction takes an eternity #1143
  • Another test is more of a base for spamming the testnets. We can put any hex-encoded transaction and dry run it on the network. It contains a simple transfer transaction right now.

The change improves the fuel-core snapshot command to accept the arguments, and you can either snap the whole state of the blockchain via the everyone command, or the state only of the contract. In the follow-up PRs we can snapping of the coins and messages too.

Optimizations:

  • Use a lru cache to store the uncompressed data in the DB. The size is configurable via CLI max-database-cache-size.
  • Added a DryRun execution type that skips the coinbase Mint transaction. In the future, we can skip some non-actual steps too.
  • Updated the PrometheusExtension to not use the lock.
  • Removed usage of the tracing extension in the graph QL because we have PrometheusExtension to gather the statistic about requests. Update cargo audit and add to commit #1152 removes the honeycomb, so I didn't duplicate it here.
  • Replaced the tokio::task::spawn_blocking + Semaphore by tokio_rayon::spawn_fifo.
  • Replaced the concatenated column_key with the array for separate hash maps per column in the MemoryStore and MemoryTransactionView.
  • Replaced the Vec<u8> in the KV API by the Arc<Vec<u8>>, because in most cases, we use it only for deserialization, and the memory database can continue to store the type. It helps reduce the number of clones.
  • Used BTreeMap in the MemoryStore, because it allows to have a more optimized iter_all method. Without this change, a new e2e test fails with a timeout.
  • Dry run doesn't wait for relayer anymore.

xgreenx added 3 commits April 25, 2023 12:22
@xgreenx
Rocksdb database optimisations (#1141)
d32746a 
- Replaced combined `column_key` with an array of hash maps.
- Added RocksDB LRU cache for uncompressed rows. It speeds up the
loading of objects from the database two times.
<img width="1190" alt="image"
src="https://user-images.githubusercontent.com/18346821/233048562-428ee4e5-39fa-4e97-8918-b764824814ba.png">
<img width="1190" alt="image"
src="https://user-images.githubusercontent.com/18346821/233048615-4f4adb88-7736-471d-b5a9-048e24fcd81a.png">

- Optimized the deserialization of the contracts via the
`OptimizedContract` wrapper type. Instead of 53% now it takes only 10%
of the time.
<img width="1195" alt="image"
src="https://user-images.githubusercontent.com/18346821/233048817-85ae413f-888a-4522-9dde-287814514d5a.png">

- Replaced all values `Vec<u8>` with `Arc<Vec<u8>>` to avoid cloning
because we use it only for deserialization, and we can't consume the
actual `Vec<u8>`. It is helpful for `MemoryStore` and
`MemoryTransactionView`.
- Added a new e2e test for the contract with a large state.
- Updated the `snapshot`command to do a snapshot only of the contract's
state.
- Updated the `MemoryStore` to use `BTreeMap` instead of `HashMap`. It
significantly improved the performance of the `iter_all` without
`RocksDB`. Without this optimization, a new e2e failed for
`--no-default-features` build with a timeout(but worked with `RocksDB`).
- Added a new e2e test that allows dry run any hex-encoded transaction.
- Simplified the `PrometheusExtension`.
- Use a new `fuel-vm 0.26.3` with disabled logging for `instruction` function.
- Use `tokio_rayon::spawn_fifo` to run dry runs instead of manual semaphore and `tokio::spawn_blocking`.
- Removed 5 second delay for dry runs caused by the relayer.
@xgreenx
Bump version to 0.17.11
0abf776
@xgreenx
Resolved conflicts with 0.17.11
5b34f88
@xgreenx xgreenx requested a review from a team April 25, 2023 14:46
@xgreenx xgreenx self-assigned this Apr 25, 2023
@freesig
Copy link
Contributor

freesig commented Apr 26, 2023

This seems like the same PR that I already reviewed. If so maybe it can just be merged?

@xgreenx
Copy link
Collaborator Author

xgreenx commented Apr 26, 2023

It is almost the same, but tuned for the master version=) The difference only in KeyValueStore.

If you think that it is ready to land master then approve it=D

xgreenx added 2 commits April 26, 2023 12:03
@xgreenx
Merge branch 'master' into feature/optimizations-from-testnet
08ca706 
# Conflicts:
#	Cargo.lock
#	bin/fuel-core/src/cli/snapshot.rs
#	crates/fuel-core/src/graphql_api/service.rs
@xgreenx
Update cargo lock
d2193b8
Voxelot
Voxelot reviewed Apr 26, 2023
View reviewed changes
crates/fuel-core/src/executor.rs
)?;
self.apply_coinbase(coinbase_tx, block, execution_data, block_db_transaction)?;
if execution_kind != ExecutionKind::DryRun {
coinbase_tx = self.check_coinbase(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we still need to do this check in production mode?

Copy link
Collaborator Author

@xgreenx xgreenx Apr 26, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need only to apply coinbase, but not to check. But while the check_coinbase function does simple check, I think it is better to keep it to be sure that everything is okay(and makes the code consistante)=)

Voxelot
Voxelot reviewed Apr 26, 2023
View reviewed changes
crates/fuel-core/src/executor.rs Outdated
*block.transactions_mut() = vec![script.into()];

let ExecutionResult { block, .. } = producer
.execute_and_commit(ExecutionBlock::DryRun(block.into()))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should we add any special logic here to ensure that dry run's won't actually commit? Or even better, enforce it at a type level?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We use execute_and_commit only for tests, so I marked it with the corresponding compilation feature. Inside the method will panic if someone passes the ExecutionBlock ::DryRun

Voxelot
Voxelot reviewed Apr 26, 2023
View reviewed changes
crates/services/producer/src/block_producer.rs
.select_new_da_height(previous_block_info.da_height)
.await?;
let mut block_header = self._new_header(height, block_time)?;
let new_da_height = self.select_new_da_height(block_header.da_height).await?;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe add some comments around this part of the code, to make it more clear that dry runs should reuse a previous da_height while new blocks should fetch da_height from the relayer. Since this was a big performance issue in the past, I think it's worth making the distinction very clear and explain why.

@xgreenx xgreenx enabled auto-merge (squash) April 27, 2023 09:57
@xgreenx xgreenx merged commit 416ce2f into master Apr 27, 2023
@xgreenx xgreenx deleted the feature/optimizations-from-testnet branch April 27, 2023 10:44
@xgreenx xgreenx mentioned this pull request Apr 27, 2023
Merged
xgreenx added a commit that referenced this pull request Apr 27, 2023
@xgreenx
Release 0.18.0 (#1165)
19f12ea 
The change bumps the version to `0.18.0` and exposes
`sync_max_get_header` and `sync_max_get_txns` in the helm chart.

# Release 0.18.0

## Overview

A new release brings:
- Optimization for the execution based on the performance from beta 3
and on internal benchmarks. Improved metrics gathering.
- Stabilization and better test coverage of the `fuel-vm`. We removed
almost all unsafe code and added test cases for each opcode. Fixed some
edge cases with memory in the `fuel-vm`.
- Fully integrated Merkle trees and filled all malleable fields in the
transactions.
- Added retryable messages, removed redundant fields from it, and
updated the API to support a new commitment schema.

## What's Changed

### Breaking

- All unsafe functions were replaced with safe analog in the
`fuel-crypto` - FuelLabs/fuel-vm#346
- `$hp` holds the address of the last available byte in a heap, while
previously it was `$hp - 1` -
FuelLabs/fuel-vm#377
- Each variant in the `fuel_tx::Input` enum now has its own type -
FuelLabs/fuel-vm#364
- Message nonce is unified and now `Bytes32` everywhere -
FuelLabs/fuel-vm#394
- Removed the `message_id` field from all places -
FuelLabs/fuel-vm#397,
FuelLabs/fuel-vm#373,
- Unified the block height in all places with the introduction of a new
`BlockHeigh` - FuelLabs/fuel-vm#410
- Make SMO instruction take data ptr as an argument -
FuelLabs/fuel-vm#404
- Now the chain id affects the signature and predicate's owner and
should be passed into the `sign` function -
FuelLabs/fuel-vm#406
- Updated the `produce_blocks` endpoint to accept the start time and the
number of blocks. All new blocks will use the previous timestamp as a
base - #1059
- The `fuel-core` stores only unspent coins and messages, so all API
that previously returned spent coins is affected - Prune owned coin idx
when inputs are spent by @Voxelot in
#1055
- The message proof API has been changed to be compatible with a new
version - #1071
- The `fuel-core` now has retryable messages and coin messages.
Retryable messages can only be consumed during successful transaction
execution. The coin message acts as common coins. `resouces_to_spend`
API was replaced with `coins_to_spend` that returns a new `CoinType`
type. - #1067

## All changes
* adding fuel-core service monitor to helm chart by @rfuelsh in
#1037
* Adding specific app selector by @rfuelsh in
#1039
* use sticky sessions for GQL requests to sentries by @Voxelot in
#1042
* Add ingress service name to helm chart by @rfuelsh in
#1043
* Added test to verify the iteration over owned transactions by @xgreenx
in #1041
* Change sentry lb to use clusterIP by @Voxelot in
#1045
* Fixed Tempfile dependency by @ControlCplusControlV in
#1048
* Write contract code in raw by @freesig in
#994
* attempt to use buildjet runners by @Voxelot in
#1017
* Set tx pointer during block production by @Voxelot in
#1054
* Used `BASE_AMOUNT` for test with bob to pay for fee by @xgreenx in
#1057
* Prepare GraphQL Crate for extraction by @ControlCplusControlV in
#1012
* Support large contract deployments over p2p by @Voxelot in
#1062
* fix yaml and add linting job by @Voxelot in
#1063
* Actualized the ABI for message to be compatible with last version of
the Solidity contracts. by @xgreenx in
#1060
* feat: Contract state and assets merkle data storage by @bvrooman in
#1008
* Take into account the previous block timestamp during block production
by @xgreenx in #1059
* Prune owned coin idx when inputs are spent by @Voxelot in
#1055
* Retrayable messages fuel-core part by @xgreenx in
#1067
* chore: Additional Tests for Contract States and Balances by @bvrooman
in #1073
* Rust 1.68.1 & Sparse Registry by @Voxelot in
#1077
* Nginx tuneup by @Voxelot in
#1080
* chore: Update balance and state Merkleization by @bvrooman in
#1084
* Support sticky session in the client by @xgreenx in
#1079
* Added e2e test to run 1000 `dry_run` by @xgreenx in
#1091
* Use docker.io auth credentials to overcome rate limiting by @Voxelot
in #1092
* honeycomb tracing subscription by @Voxelot in
#1083
* Update withdrawal proof API to support proving from a block header
committed to L1 by @xgreenx in
#1071
* Upstream v0.17.8 by @Voxelot in
#1102
* use chainid for transactions and predicates by @Voxelot in
#1107
* fix the dry-run e2e test to actually perform dry runs by @Voxelot in
#1111
* peer reputation with separate app & gossipsub score by @leviathanbeak
in #1028
* Adding sentry data synchronization by @rfuelsh in
#1115
* Capture metrics for graphql api by @Voxelot in
#1113
* Update cargo audit and add to commit by @freesig in
#1152
* add task params by @leviathanbeak in
#1159
* Took into account that block creation could take some time by @xgreenx
in #1162
* Applying optimizations from `0.17.11` release by @xgreenx in
#1158


**Full Changelog**:
v0.17.3...v0.18.0
crypto523 pushed a commit to crypto523/fuel-core that referenced this pull request Oct 7, 2024
Release 0.18.0 (#1165)
2d2a6a3 
The change bumps the version to `0.18.0` and exposes
`sync_max_get_header` and `sync_max_get_txns` in the helm chart.

# Release 0.18.0

## Overview

A new release brings:
- Optimization for the execution based on the performance from beta 3
and on internal benchmarks. Improved metrics gathering.
- Stabilization and better test coverage of the `fuel-vm`. We removed
almost all unsafe code and added test cases for each opcode. Fixed some
edge cases with memory in the `fuel-vm`.
- Fully integrated Merkle trees and filled all malleable fields in the
transactions.
- Added retryable messages, removed redundant fields from it, and
updated the API to support a new commitment schema.

## What's Changed

### Breaking

- All unsafe functions were replaced with safe analog in the
`fuel-crypto` - FuelLabs/fuel-vm#346
- `$hp` holds the address of the last available byte in a heap, while
previously it was `$hp - 1` -
FuelLabs/fuel-vm#377
- Each variant in the `fuel_tx::Input` enum now has its own type -
FuelLabs/fuel-vm#364
- Message nonce is unified and now `Bytes32` everywhere -
FuelLabs/fuel-vm#394
- Removed the `message_id` field from all places -
FuelLabs/fuel-vm#397,
FuelLabs/fuel-vm#373,
- Unified the block height in all places with the introduction of a new
`BlockHeigh` - FuelLabs/fuel-vm#410
- Make SMO instruction take data ptr as an argument -
FuelLabs/fuel-vm#404
- Now the chain id affects the signature and predicate's owner and
should be passed into the `sign` function -
FuelLabs/fuel-vm#406
- Updated the `produce_blocks` endpoint to accept the start time and the
number of blocks. All new blocks will use the previous timestamp as a
base - FuelLabs/fuel-core#1059
- The `fuel-core` stores only unspent coins and messages, so all API
that previously returned spent coins is affected - Prune owned coin idx
when inputs are spent by @Voxelot in
FuelLabs/fuel-core#1055
- The message proof API has been changed to be compatible with a new
version - FuelLabs/fuel-core#1071
- The `fuel-core` now has retryable messages and coin messages.
Retryable messages can only be consumed during successful transaction
execution. The coin message acts as common coins. `resouces_to_spend`
API was replaced with `coins_to_spend` that returns a new `CoinType`
type. - FuelLabs/fuel-core#1067

## All changes
* adding fuel-core service monitor to helm chart by @rfuelsh in
FuelLabs/fuel-core#1037
* Adding specific app selector by @rfuelsh in
FuelLabs/fuel-core#1039
* use sticky sessions for GQL requests to sentries by @Voxelot in
FuelLabs/fuel-core#1042
* Add ingress service name to helm chart by @rfuelsh in
FuelLabs/fuel-core#1043
* Added test to verify the iteration over owned transactions by @xgreenx
in FuelLabs/fuel-core#1041
* Change sentry lb to use clusterIP by @Voxelot in
FuelLabs/fuel-core#1045
* Fixed Tempfile dependency by @ControlCplusControlV in
FuelLabs/fuel-core#1048
* Write contract code in raw by @freesig in
FuelLabs/fuel-core#994
* attempt to use buildjet runners by @Voxelot in
FuelLabs/fuel-core#1017
* Set tx pointer during block production by @Voxelot in
FuelLabs/fuel-core#1054
* Used `BASE_AMOUNT` for test with bob to pay for fee by @xgreenx in
FuelLabs/fuel-core#1057
* Prepare GraphQL Crate for extraction by @ControlCplusControlV in
FuelLabs/fuel-core#1012
* Support large contract deployments over p2p by @Voxelot in
FuelLabs/fuel-core#1062
* fix yaml and add linting job by @Voxelot in
FuelLabs/fuel-core#1063
* Actualized the ABI for message to be compatible with last version of
the Solidity contracts. by @xgreenx in
FuelLabs/fuel-core#1060
* feat: Contract state and assets merkle data storage by @bvrooman in
FuelLabs/fuel-core#1008
* Take into account the previous block timestamp during block production
by @xgreenx in FuelLabs/fuel-core#1059
* Prune owned coin idx when inputs are spent by @Voxelot in
FuelLabs/fuel-core#1055
* Retrayable messages fuel-core part by @xgreenx in
FuelLabs/fuel-core#1067
* chore: Additional Tests for Contract States and Balances by @bvrooman
in FuelLabs/fuel-core#1073
* Rust 1.68.1 & Sparse Registry by @Voxelot in
FuelLabs/fuel-core#1077
* Nginx tuneup by @Voxelot in
FuelLabs/fuel-core#1080
* chore: Update balance and state Merkleization by @bvrooman in
FuelLabs/fuel-core#1084
* Support sticky session in the client by @xgreenx in
FuelLabs/fuel-core#1079
* Added e2e test to run 1000 `dry_run` by @xgreenx in
FuelLabs/fuel-core#1091
* Use docker.io auth credentials to overcome rate limiting by @Voxelot
in FuelLabs/fuel-core#1092
* honeycomb tracing subscription by @Voxelot in
FuelLabs/fuel-core#1083
* Update withdrawal proof API to support proving from a block header
committed to L1 by @xgreenx in
FuelLabs/fuel-core#1071
* Upstream v0.17.8 by @Voxelot in
FuelLabs/fuel-core#1102
* use chainid for transactions and predicates by @Voxelot in
FuelLabs/fuel-core#1107
* fix the dry-run e2e test to actually perform dry runs by @Voxelot in
FuelLabs/fuel-core#1111
* peer reputation with separate app & gossipsub score by @leviathanbeak
in FuelLabs/fuel-core#1028
* Adding sentry data synchronization by @rfuelsh in
FuelLabs/fuel-core#1115
* Capture metrics for graphql api by @Voxelot in
FuelLabs/fuel-core#1113
* Update cargo audit and add to commit by @freesig in
FuelLabs/fuel-core#1152
* add task params by @leviathanbeak in
FuelLabs/fuel-core#1159
* Took into account that block creation could take some time by @xgreenx
in FuelLabs/fuel-core#1162
* Applying optimizations from `0.17.11` release by @xgreenx in
FuelLabs/fuel-core#1158


**Full Changelog**:
FuelLabs/fuel-core@v0.17.3...v0.18.0
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Voxelot Voxelot Voxelot approved these changes

Assignees

@xgreenx xgreenx

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Database caching for uncompressed data
3 participants
@xgreenx @freesig @Voxelot