Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Convert TrieBackendEssence::{keys, pairs} to return an iterator #29

Open
koute opened this issue Dec 14, 2022 · 0 comments
Open

Convert TrieBackendEssence::{keys, pairs} to return an iterator #29

koute opened this issue Dec 14, 2022 · 0 comments
Labels
I4-refactor Code needs refactoring. I9-optimisation An enhancement to provide better overall performance in terms of time-to-completion for a task.

Comments

@koute
Copy link
Contributor

koute commented Dec 14, 2022

The TrieBackendEssence::keys and TrieBackendEssence::pairs (and in consequence TrieBackendStorage::keys and TrieBackendStorage::pairs) grab all of the data in one go and return a Vec. If the amount of data is big this is really bad for latency and it wastes a lot of memory. We should either remove them as they're essentially practically broken as-is, or make those into iterators which would just stream the data (possibly in chunks, but definitely not preloading everything into memory).
@koute koute added I8-footprint I9-optimisation An enhancement to provide better overall performance in terms of time-to-completion for a task. labels Dec 14, 2022
@the-right-joyce the-right-joyce transferred this issue from paritytech/substrate Aug 24, 2023
@the-right-joyce the-right-joyce added I4-refactor Code needs refactoring. and removed I7-refactor labels Aug 25, 2023
Neopallium pushed a commit to PolymeshAssociation/polkadot-sdk that referenced this issue Dec 8, 2023
@quinndiggitypolymath @Neopallium
max_total_wal_size (paritytech#29)
b06dd02 
* adds static `db_config.max_total_wal_size`, pinned to `40 GB` to match current documentation @ https://github.com/PolymeshAssociation/polymesh-tools/blob/7d8146deb459ffca3de6cd5da270928a529f233c/docs/operator/README.md#node-resource-requirements : `It is recommended that you reserve an additional 40GB of disk space for the WAL.`

* adds cli flag `--db-max-total-wal-size` to allow specifying `db_config.max_total_wal_size`

* adds cli flag `--db-max-total-wal-size` to allow specifying `db_config.max_total_wal_size`

* adds cli flag `--db-max-total-wal-size` to allow specifying `db_config.max_total_wal_size`

Co-authored-by: Quinn Diggity <git@quinn.dev>

Max wal size (paritytech#30)

* Use `Option<u64>` for max_total_wal_size.

Signed-off-by: Robert G. Jakabosky <rjakabosky+neopallium@neoawareness.com>
jonathanudd pushed a commit to jonathanudd/polkadot-sdk that referenced this issue Apr 10, 2024
@dependabot-preview @bkchr
Bump derive_more from 0.99.2 to 0.99.3 (paritytech#29)
4b20a78 
Bumps [derive_more](https://github.com/JelteF/derive_more) from 0.99.2 to 0.99.3.
- [Release notes](https://github.com/JelteF/derive_more/releases)
- [Changelog](https://github.com/JelteF/derive_more/blob/master/CHANGELOG.md)
- [Commits](JelteF/derive_more@v0.99.2...v0.99.3)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
liuchengxu added a commit to subcoin-project/polkadot-sdk that referenced this issue Sep 20, 2024
@liuchengxu
Introduce --verify-script flag and optimize subcoin-network (parityte…
e6379c5 
…ch#29)

* Introduce --verify-script flag

* Update usage

* Disconnect peer if peer latency is higher than 10s

* Periodically evict the slowest peer for potential better new peers

* Expose 8333 in Dockerfile and add Syncing the Bitcoin Network docs

* Update installation.md
Neopallium pushed a commit to PolymeshAssociation/polkadot-sdk that referenced this issue Sep 30, 2024
@quinndiggitypolymath @Neopallium
max_total_wal_size (paritytech#29)
2cf3399 
* adds static `db_config.max_total_wal_size`, pinned to `40 GB` to match current documentation @ https://github.com/PolymeshAssociation/polymesh-tools/blob/7d8146deb459ffca3de6cd5da270928a529f233c/docs/operator/README.md#node-resource-requirements : `It is recommended that you reserve an additional 40GB of disk space for the WAL.`

* adds cli flag `--db-max-total-wal-size` to allow specifying `db_config.max_total_wal_size`

* adds cli flag `--db-max-total-wal-size` to allow specifying `db_config.max_total_wal_size`

* adds cli flag `--db-max-total-wal-size` to allow specifying `db_config.max_total_wal_size`

Co-authored-by: Quinn Diggity <git@quinn.dev>

Max wal size (paritytech#30)

* Use `Option<u64>` for max_total_wal_size.

Signed-off-by: Robert G. Jakabosky <rjakabosky+neopallium@neoawareness.com>
Neopallium pushed a commit to PolymeshAssociation/polkadot-sdk that referenced this issue Sep 30, 2024
@quinndiggitypolymath @Neopallium
max_total_wal_size (paritytech#29)
d7a22de 
* adds static `db_config.max_total_wal_size`, pinned to `40 GB` to match current documentation @ https://github.com/PolymeshAssociation/polymesh-tools/blob/7d8146deb459ffca3de6cd5da270928a529f233c/docs/operator/README.md#node-resource-requirements : `It is recommended that you reserve an additional 40GB of disk space for the WAL.`

* adds cli flag `--db-max-total-wal-size` to allow specifying `db_config.max_total_wal_size`

* adds cli flag `--db-max-total-wal-size` to allow specifying `db_config.max_total_wal_size`

* adds cli flag `--db-max-total-wal-size` to allow specifying `db_config.max_total_wal_size`

Co-authored-by: Quinn Diggity <git@quinn.dev>

Max wal size (paritytech#30)

* Use `Option<u64>` for max_total_wal_size.

Signed-off-by: Robert G. Jakabosky <rjakabosky+neopallium@neoawareness.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
I4-refactor Code needs refactoring. I9-optimisation An enhancement to provide better overall performance in terms of time-to-completion for a task.
Projects
SDK Node
Status: backlog
Milestone
No milestone
Development

No branches or pull requests
2 participants
@koute @the-right-joyce