Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Stream reader for chain state #1519

Merged
merged 273 commits into from
Feb 21, 2024
Merged
Show file tree
Hide file tree
Changes from 250 commits
Commits
Show all changes
273 commits
Select commit Hold shift + click to select a range
93dbffc
fmt
Salka1988 Nov 7, 2023
d85517a
fix tests
MujkicA Nov 7, 2023
e807c2e
use constants
MujkicA Nov 7, 2023
26d9321
fmt
MujkicA Nov 7, 2023
9e9b962
rename chain parameters
MujkicA Nov 7, 2023
412e625
remove unneeded file
MujkicA Nov 7, 2023
0cbfd0a
finish renaming
MujkicA Nov 7, 2023
2cf7782
add std flag
MujkicA Nov 7, 2023
0fc7ef7
add state importer and consolidate db type
MujkicA Nov 8, 2023
2a8554a
fmt and changelog
MujkicA Nov 8, 2023
ac697c0
put importer behidn feature flag
MujkicA Nov 8, 2023
f18fe90
tracing behind rocksdb flag
MujkicA Nov 8, 2023
a37ed56
json support
segfault-magnet Nov 8, 2023
f841430
json writer finished
segfault-magnet Nov 8, 2023
7d7d5c2
Merge branch 'feature/stream-genesis-state' of github.com:FuelLabs/fu…
segfault-magnet Nov 9, 2023
19831bd
wip
segfault-magnet Nov 9, 2023
f3ce913
const for testnet wallet secrets
MujkicA Nov 9, 2023
7060fc5
trait tweaking
segfault-magnet Nov 9, 2023
9f4b9ba
Merge branch 'feature/split-state-from-config' into feature/stream-ge…
MujkicA Nov 9, 2023
671c434
first parquet test passing
segfault-magnet Nov 9, 2023
3c5309b
add type to json batch reader
segfault-magnet Nov 9, 2023
6123126
read trait
MujkicA Nov 13, 2023
72ec9de
Merge branch 'feature/stream-genesis-state' of github.com:FuelLabs/fu…
MujkicA Nov 13, 2023
024852c
finalize reading interface
MujkicA Nov 15, 2023
c566d00
add rev comments
MujkicA Nov 15, 2023
817b020
insta snapshot
MujkicA Nov 15, 2023
b7b42fa
progress
segfault-magnet Nov 17, 2023
78e9af4
interface
MujkicA Nov 17, 2023
bff5bd5
test passing
segfault-magnet Nov 17, 2023
176c7d9
replace ChainConfig with StateConfig
segfault-magnet Nov 20, 2023
6eecf13
Merge branch 'feature/stream-genesis-state' of github.com:FuelLabs/fu…
segfault-magnet Nov 20, 2023
8ca5c27
moved methods from trait into Iterator
segfault-magnet Nov 20, 2023
341fa13
in memory test added
segfault-magnet Nov 20, 2023
6c12d57
core is compiling
segfault-magnet Nov 20, 2023
9e16a32
renaming
segfault-magnet Nov 20, 2023
91c658a
reformat
segfault-magnet Nov 20, 2023
9096b9d
fix nth impl of iterator
segfault-magnet Nov 22, 2023
ef792e6
fix create_config_file()
MujkicA Nov 22, 2023
e40da6f
used enum instead of dyn dispatch
segfault-magnet Nov 22, 2023
051c855
no dyn dysatch remains
segfault-magnet Nov 23, 2023
7bf6a53
nits
segfault-magnet Nov 23, 2023
b34a631
stream state from db when creating a snapshot
segfault-magnet Nov 24, 2023
5461564
nits
segfault-magnet Nov 24, 2023
65da28d
sort toml
segfault-magnet Nov 24, 2023
1d6dc91
mangled cargo toml
segfault-magnet Nov 24, 2023
a318ccf
struct too private
segfault-magnet Nov 24, 2023
f6414d0
add state stream reading
MujkicA Nov 27, 2023
2e60394
Merge branch 'feature/split-state-from-config' into feature/stream-ge…
segfault-magnet Nov 29, 2023
bd0794f
Merge remote-tracking branch 'origin/feature/stream-genesis-state' in…
segfault-magnet Nov 29, 2023
6873d26
Merge remote-tracking branch 'origin/feature/snapshot_generation' int…
segfault-magnet Nov 29, 2023
5857fb2
change to BoxedIter, RPITIT not supported in stable rust currently
segfault-magnet Nov 29, 2023
f0a0dd9
Merge remote-tracking branch 'origin/feature/snapshot_generation' int…
segfault-magnet Nov 29, 2023
156ec3a
rev comments
MujkicA Nov 30, 2023
18e0906
remove local_testnet as arg
MujkicA Nov 30, 2023
620d25c
rename in p2p test helpers
MujkicA Nov 30, 2023
1781f3a
Apply suggestions from code review
segfault-magnet Dec 1, 2023
409fa44
Merge remote-tracking branch 'origin/feature/split-state-from-config'…
segfault-magnet Dec 1, 2023
b3f2098
Merge remote-tracking branch 'origin/feature/snapshot_generation' int…
segfault-magnet Dec 1, 2023
ed563b2
Merge remote-tracking branch 'origin/feature/stream-genesis-state' in…
segfault-magnet Dec 1, 2023
c95cf62
rename Encoder to StateWriter, StateStreamer to StateReader
segfault-magnet Dec 1, 2023
b864634
reorganize
segfault-magnet Dec 5, 2023
09c3908
sort contract ids during regenesis
MujkicA Dec 5, 2023
79059d9
merge v0.21.0
MujkicA Dec 5, 2023
c9757ad
fix tests
MujkicA Dec 5, 2023
93114e0
merge
MujkicA Dec 5, 2023
a9ef147
fmt
MujkicA Dec 5, 2023
341de48
Merge branch 'feature/split-state-from-config' into feature/stream-ge…
MujkicA Dec 5, 2023
0cfeeca
merge
MujkicA Dec 5, 2023
fd38703
remove p2p from default
MujkicA Dec 5, 2023
0c28b31
update error message for tx coin height
segfault-magnet Dec 6, 2023
d611c53
write block height to chain config when generating snapshot
segfault-magnet Dec 6, 2023
1f3ddef
optimize state root calculation for coins and messages
segfault-magnet Dec 6, 2023
ab51197
batch insert state entries
MujkicA Dec 6, 2023
5b33aa1
Update crates/chain-config/src/config/state.rs
MujkicA Dec 7, 2023
0e25780
Update crates/chain-config/src/config/chain.rs
MujkicA Dec 7, 2023
f90bbb5
add checked add and clippy allow
MujkicA Dec 7, 2023
6764f5d
Merge branch 'feature/split-state-from-config' of github.com:FuelLabs…
MujkicA Dec 7, 2023
da73ede
fix poa test
MujkicA Dec 7, 2023
289ec10
Update bin/fuel-core/src/cli/snapshot.rs
MujkicA Dec 9, 2023
a3f3ead
Merge branch 'feature/stream-genesis-state' into feature/snapshot_rea…
segfault-magnet Dec 10, 2023
c92fd9f
group contract states and balances by contract when reading/writing
segfault-magnet Dec 11, 2023
d8028d1
batch insert balance
MujkicA Dec 12, 2023
bcfaf16
accept batches of mixed contracts
MujkicA Dec 12, 2023
6b45be7
Revert "group contract states and balances by contract when reading/w…
segfault-magnet Dec 12, 2023
b44f36d
json and parquet tests separated, naming suggestion applied
segfault-magnet Dec 12, 2023
5100094
added SeedableRng and reverted commented out deny(warnings)
segfault-magnet Dec 12, 2023
7f793e8
split up tests per entity
segfault-magnet Dec 13, 2023
557f39f
test that parquet groups are skipped optimally
segfault-magnet Dec 13, 2023
9a0139d
wip
segfault-magnet Dec 14, 2023
3a66bb8
test for state presence in db
segfault-magnet Dec 14, 2023
6056e33
test multiple contract states in one batch -- metadata is calculated and
segfault-magnet Dec 14, 2023
52f8a67
wip need to finish test that shuffled input will result in correct root
segfault-magnet Dec 14, 2023
7e94d30
finish shuffle test, refactoring pending
segfault-magnet Dec 15, 2023
375283a
balances
MujkicA Dec 15, 2023
a6ca4ab
refactor
segfault-magnet Dec 17, 2023
4e05d6c
Merge branch 'feature/snapshot_reading' of github.com:FuelLabs/fuel-c…
segfault-magnet Dec 17, 2023
7c8e031
DRYed up the implementation
segfault-magnet Dec 17, 2023
07ffa65
add utils -- forgot to commit it
segfault-magnet Dec 17, 2023
1333eb8
tests for writing in expected files
segfault-magnet Dec 18, 2023
195f2a2
Merge remote-tracking branch 'origin/feature/split-state-from-config'…
segfault-magnet Dec 18, 2023
42d81c5
Merge remote-tracking branch 'origin/feature/stream-genesis-state' in…
segfault-magnet Dec 18, 2023
5613886
fix name change in json file
segfault-magnet Dec 18, 2023
39e50cc
Merge remote-tracking branch 'origin/feature/regenesis-support' into …
segfault-magnet Dec 18, 2023
df5c2d7
fix deployment tests
MujkicA Dec 18, 2023
e4d49dc
remove pretty dep
MujkicA Dec 18, 2023
b675f6e
Merge branch 'feature/split-state-from-config' into feature/stream-ge…
MujkicA Dec 18, 2023
867e403
merge
MujkicA Dec 18, 2023
dc9de4f
use new merkle update approach
MujkicA Dec 19, 2023
9b281b3
remove bad formatting
MujkicA Dec 19, 2023
3ed7aee
more format fixes
MujkicA Dec 19, 2023
f8c0d99
one more format fix
MujkicA Dec 19, 2023
cdc6968
Merge branch 'feature/split-state-from-config' into feature/stream-ge…
MujkicA Dec 19, 2023
4ff7da6
Merge branch 'feature/stream-genesis-state' into feature/snapshot_rea…
MujkicA Dec 19, 2023
09a8172
changelog
MujkicA Dec 19, 2023
684227b
changelog
MujkicA Dec 19, 2023
cd0ad2d
Merge branch 'feature/stream-genesis-state' into feature/snapshot_rea…
MujkicA Dec 19, 2023
2ab14fb
expect when consuming batches
MujkicA Dec 19, 2023
b11442d
initial resumability impl
MujkicA Dec 20, 2023
6310848
genesis_progress file
MujkicA Dec 20, 2023
37a66d0
pr comment: increase ident
segfault-magnet Dec 21, 2023
ccbe034
remove magic group size number
segfault-magnet Dec 21, 2023
279c12d
rename flag
segfault-magnet Dec 21, 2023
c99f745
resumability pt2
MujkicA Dec 24, 2023
b6eaed9
Merge branch 'feature/snapshot_reading' of github.com:FuelLabs/fuel-c…
MujkicA Dec 24, 2023
b1c02d3
add configurable compression
segfault-magnet Dec 24, 2023
4bf4eeb
Revert "remove magic group size number"
segfault-magnet Dec 24, 2023
f8e7787
Revert "rename flag"
segfault-magnet Dec 24, 2023
9b44002
Apply suggestions from code review
segfault-magnet Dec 24, 2023
629b701
Merge branch 'feature/regenesis-support' into feature/stream-genesis-…
segfault-magnet Dec 24, 2023
34f20da
inline variables
segfault-magnet Dec 24, 2023
7b65052
remove suffix
segfault-magnet Dec 24, 2023
87062fb
formatting
segfault-magnet Dec 24, 2023
ea93fb5
parquet hidden behind a flag
segfault-magnet Dec 25, 2023
0b68be2
fix tests for all combinations of features
segfault-magnet Dec 25, 2023
66127f1
issue with string lifetime in serde Deserialize is fixed
segfault-magnet Dec 25, 2023
11ca936
add back tests
segfault-magnet Dec 25, 2023
8655d10
Merge branch 'feature/stream-genesis-state' into feature/snapshot_rea…
segfault-magnet Dec 26, 2023
f23246e
remove redundant tests
segfault-magnet Dec 26, 2023
ad01122
Merge branch 'feature/stream-genesis-state' into feature/snapshot_rea…
segfault-magnet Dec 26, 2023
0f4a5e9
remove unused fn
segfault-magnet Dec 26, 2023
548c436
Merge branch 'feature/stream-genesis-state' into feature/snapshot_rea…
segfault-magnet Dec 26, 2023
35c7c90
fix columns, make genesis async spawn tasks
segfault-magnet Dec 27, 2023
cde1964
use constant for stat config file
segfault-magnet Dec 27, 2023
af71db0
fix renaming issue
segfault-magnet Dec 27, 2023
049b5c8
test all compressions
segfault-magnet Dec 27, 2023
cf70370
Merge branch 'feature/stream-genesis-state' into feature/snapshot_rea…
segfault-magnet Dec 27, 2023
c895092
nits
segfault-magnet Dec 27, 2023
cad0094
debugging compression levels
segfault-magnet Dec 27, 2023
e00c6e3
Revert "debugging compression levels"
segfault-magnet Dec 27, 2023
9629f5f
change compression algo to zstd to avoid the buggy flate2 gzip backend
segfault-magnet Dec 27, 2023
cb023a1
add height to chain config
segfault-magnet Dec 28, 2023
f649fbf
Merge branch 'feature/stream-genesis-state' into feature/snapshot_rea…
segfault-magnet Dec 28, 2023
e84bcf4
fix anyhow err
segfault-magnet Dec 28, 2023
26c5e1d
change contract snapshot to output in the same format as 'everything'
segfault-magnet Dec 29, 2023
3f86d4a
snapshot testing wip
segfault-magnet Jan 1, 2024
9fd29ca
fix p2p state updating logic
segfault-magnet Jan 2, 2024
4c735fe
fix deployment config check tests
segfault-magnet Jan 2, 2024
124707a
fix imports for all feature combinations
segfault-magnet Jan 2, 2024
e728f97
sort toml files
segfault-magnet Jan 2, 2024
031e684
progress on the genesis runner
segfault-magnet Jan 2, 2024
adf4c68
finished genesis runner
segfault-magnet Jan 4, 2024
b15efec
progress is now taken from read group. simplify runner interface
segfault-magnet Jan 4, 2024
3dacafa
refactoring
segfault-magnet Jan 4, 2024
6c0ce7d
add checkpoints for contracts root updating
segfault-magnet Jan 7, 2024
20fdc19
add checkpoints for contracts root updating
segfault-magnet Jan 7, 2024
9a749ef
Merge branch 'feature/snapshot_reading' of github.com:FuelLabs/fuel-c…
segfault-magnet Jan 8, 2024
6a797b4
Remove hack with imports
xgreenx Jan 8, 2024
d6b62ec
Update crates/fuel-core/src/service.rs
segfault-magnet Jan 8, 2024
ddce7bb
use autoimpl for &T and &mut T implementations of ChainStateDb
segfault-magnet Jan 8, 2024
42d597b
don't use magic values
segfault-magnet Jan 8, 2024
77e6d94
use postcard/canonical encoding inside parquet entities
segfault-magnet Jan 9, 2024
6d971c9
add json human readability tests
segfault-magnet Jan 9, 2024
4c909d8
add tests for the HexType and HexNumber encodings
segfault-magnet Jan 9, 2024
c2a4ddf
fix benches
MujkicA Jan 10, 2024
2d9e02b
Merge branch 'feature/stream-genesis-state' into feature/snapshot_rea…
segfault-magnet Jan 10, 2024
22dd1df
ensure all tasks are terminated on shutdown
MujkicA Jan 11, 2024
5271b45
sort cargo
MujkicA Jan 11, 2024
2130043
update tests
MujkicA Jan 11, 2024
f5eba1f
snapshots
MujkicA Jan 11, 2024
63b2465
implement snapshot metadata
segfault-magnet Jan 11, 2024
0b00f3e
improve worker cancelling and add tests
MujkicA Jan 12, 2024
da890c3
delegate hex encoding (where possible) to fuel types serde impl
segfault-magnet Jan 13, 2024
7661412
Merge branch 'feature/stream-genesis-state' into feature/snapshot_rea…
segfault-magnet Jan 13, 2024
e08ecfc
snapshot metadata tests
segfault-magnet Jan 14, 2024
9038e9e
sort cargo files
segfault-magnet Jan 14, 2024
7a48a40
Merge branch 'feature/stream-genesis-state' into feature/snapshot_rea…
segfault-magnet Jan 14, 2024
073eb73
refactor workers
MujkicA Jan 14, 2024
5bcb4e9
update tests
segfault-magnet Jan 15, 2024
47b9752
Merge branch 'feature/snapshot_reading' of github.com:FuelLabs/fuel-c…
segfault-magnet Jan 15, 2024
ed6bc6c
cleanup db after genesis
MujkicA Jan 16, 2024
69f4372
non skipping serialize
segfault-magnet Jan 16, 2024
47def96
Merge branch 'feature/snapshot_reading' of github.com:FuelLabs/fuel-c…
segfault-magnet Jan 16, 2024
60ac6d7
nit
segfault-magnet Jan 17, 2024
67ebfd2
move tests from codec
segfault-magnet Jan 17, 2024
a40bc7e
remove todos
MujkicA Jan 17, 2024
bfebba0
fix coin test
MujkicA Jan 17, 2024
f887e91
forward error on workers
MujkicA Jan 17, 2024
b349271
put snapshot metadata behind std flag
MujkicA Jan 20, 2024
0f09ba6
Merge branch 'feature/stream-genesis-state' into feature/snapshot_rea…
MujkicA Jan 20, 2024
313d1f3
fix test for flags
MujkicA Jan 20, 2024
2f711fd
apply review comments pt1
MujkicA Jan 23, 2024
a35cca7
remove semicolons
MujkicA Jan 24, 2024
da3543c
merge
MujkicA Jan 29, 2024
b453a67
clean up after merge
MujkicA Feb 1, 2024
c245c58
more fixerd
MujkicA Feb 1, 2024
ccef09d
fmt
MujkicA Feb 1, 2024
f4608bf
nits
segfault-magnet Feb 2, 2024
e563e0d
sort cargo
segfault-magnet Feb 3, 2024
496e53f
remove rogue file
MujkicA Feb 5, 2024
7136b16
partially merge
MujkicA Feb 5, 2024
74dfcbe
fix genesis tables
MujkicA Feb 6, 2024
30fd1e6
fix genesis cleanup
MujkicA Feb 6, 2024
22be17b
fix balance and state updating
MujkicA Feb 7, 2024
2ff617b
fix merge mishap
segfault-magnet Feb 7, 2024
d433b83
feature powerset lint
segfault-magnet Feb 7, 2024
90ed16a
work around optional dev deps not supported by cargo
segfault-magnet Feb 7, 2024
5fa4942
add cargo toml format check
segfault-magnet Feb 7, 2024
6cde15c
fix install line
segfault-magnet Feb 7, 2024
caa1df9
add prettierrc
segfault-magnet Feb 7, 2024
bc1d3a2
temporary package.json
segfault-magnet Feb 7, 2024
c85d466
use npx
segfault-magnet Feb 7, 2024
9983627
add formatting suggestions
segfault-magnet Feb 7, 2024
58822ed
fix diff
segfault-magnet Feb 7, 2024
9470025
format toml files
segfault-magnet Feb 7, 2024
257afb7
use alloc on itertools
segfault-magnet Feb 7, 2024
d845645
clippy suggestions
segfault-magnet Feb 7, 2024
3c00401
snapshot cli optional commands work
MujkicA Feb 8, 2024
062d4b4
resolve TODOs
MujkicA Feb 8, 2024
1db86d0
remove semicolon diffs
MujkicA Feb 8, 2024
10996f5
sort cargo
MujkicA Feb 8, 2024
2833b34
add genesis cleanup to block commit db tx
MujkicA Feb 8, 2024
a6d8bd3
add existence check before pruning db
segfault-magnet Feb 9, 2024
ccfad4d
bumped rust version, dependency failed to build otherwise
segfault-magnet Feb 9, 2024
a0c6b91
move block height from chain config to state
segfault-magnet Feb 11, 2024
3ecb62e
fix merge mishap
segfault-magnet Feb 11, 2024
ad3a685
bump rust version because of clap derive 4.5
segfault-magnet Feb 11, 2024
295728a
more rust bumps
segfault-magnet Feb 11, 2024
89f6279
add block height to test state data
segfault-magnet Feb 11, 2024
0e54c66
open as combined database when snapshotting
segfault-magnet Feb 11, 2024
5527c9a
fix prune to delegate to rocksdb
segfault-magnet Feb 11, 2024
549163b
add parquet to production features
segfault-magnet Feb 14, 2024
2cbe02f
Small changes during review
xgreenx Feb 14, 2024
643dc86
DataSource uses inner batch delete
MujkicA Feb 15, 2024
8ab1c65
test rocks_db delete_all
MujkicA Feb 15, 2024
27803eb
fmt
MujkicA Feb 15, 2024
550e4b4
unify handler and process traits
MujkicA Feb 16, 2024
5e4d8ca
rename helper for loaded contracts
MujkicA Feb 16, 2024
4446031
remove prettier toml from ci
MujkicA Feb 16, 2024
5b5799d
read contract ids from existing table
MujkicA Feb 19, 2024
14c8ac7
use contract latest utxo table instead of contract info
MujkicA Feb 21, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 14 additions & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ concurrency:

env:
CARGO_TERM_COLOR: always
RUST_VERSION: 1.73.0
RUST_VERSION: 1.74.0
NIGHTLY_RUST_VERSION: nightly-2023-10-29
RUSTFLAGS: -D warnings
REGISTRY: ghcr.io
Expand Down Expand Up @@ -57,6 +57,19 @@ jobs:
run: cargo binstall --no-confirm cargo-sort
- name: Run Cargo.toml sort check
run: cargo sort -w --check
- name: Install Prettier and TOML Plugin
MujkicA marked this conversation as resolved.
Show resolved Hide resolved
run: |
# prettier has issues with global installation and plugin discovery. The temporary `package.json` createrd by `npm init -y` is a workaround.
npm init -y > /dev/null 2>&1
npm install prettier prettier-plugin-toml
- name: Check format of Cargo.toml files
run: |
if ! npx prettier --check "**/Cargo.toml"; then
npx prettier --write "**/Cargo.toml" > /dev/null 2>&1
echo "Changes required:"
git --no-pager diff
exit 1
fi
- uses: FuelLabs/.github/.github/actions/slack-notify-template@master
if: always() && github.ref == 'refs/heads/master'
with:
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/dependencies.yml
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ defaults:
env:
# So cargo doesn't complain about unstable features
RUSTC_BOOTSTRAP: 1
RUST_VERSION: 1.73.0
RUST_VERSION: 1.74.0
PR_TITLE: Weekly `cargo update`
PR_MESSAGE: |
Automation to keep dependencies in `Cargo.lock` current.
Expand Down
3 changes: 3 additions & 0 deletions .prettierrc
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
{
"plugins": ["prettier-plugin-toml"]
}
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -193,6 +193,7 @@ FuelVM received a lot of safety and stability improvements:
- Refactored errors, returning more detailed errors to the user, simplifying debugging.

### Added
- [#1519](https://github.com/FuelLabs/fuel-core/pull/1519): Stream chain state from file
MujkicA marked this conversation as resolved.
Show resolved Hide resolved
- [#1474](https://github.com/FuelLabs/fuel-core/pull/1474): Chain state encoding for streaming support.
- [#1459](https://github.com/FuelLabs/fuel-core/pull/1459): Split initial state from chain config.
- [#1503](https://github.com/FuelLabs/fuel-core/pull/1503): Add `gtf` opcode sanity check.
Expand Down
Loading
Loading