Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Modify utxo_by_outpoint to utxo_by_out_loc in the finalized state #3952

Closed
9 tasks done
Tracked by #3134
teor2345 opened this issue Mar 24, 2022 · 1 comment
Closed
9 tasks done
Tracked by #3134

Modify utxo_by_outpoint to utxo_by_out_loc in the finalized state #3952

teor2345 opened this issue Mar 24, 2022 · 1 comment
Assignees
Labels
A-state Area: State / database changes C-enhancement Category: This is an improvement lightwalletd any work associated with lightwalletd

Comments

@teor2345
Copy link
Contributor

teor2345 commented Mar 24, 2022

Motivation

We want to answer transparent address RPCs quickly. So we need to make existing indexes in the finalized state smaller and more efficient.

We also need to return some RPCs in chain order.

Designs

UTXOs are stored in utxo_by_out_loc by OutputLocation, rather than OutPoint. (This makes keys 28 bytes smaller.)

OutPoints can be looked up using tx_loc_by_hash, and reconstructed using hash_by_tx_loc.

The Utxo type can be constructed from the OutputLocation and Output data:

  • height: OutputLocation.height, and
  • is_coinbase: OutputLocation.transaction_index == 0 (coinbase transactions are always the first transaction in a block).

Types

  • OutputIndex: 24 bits, big-endian, unsigned (max ~223,000 transfers in the 2 MB block limit)
    See Height serialization for an example implementation.
  • OutputLocation: TransactionLocation \|\| OutputIndex
  • AddressLocation: the first OutputLocation used by a transparent::Address. Always has the same value for each address, even if the first output is spent.

https://github.com/ZcashFoundation/zebra/blob/main/book/src/dev/rfcs/0005-state-updates.md#rocksdb-data-structures

Tasks

Implementation:

  • Modify the column family in the finalized state
    • Change OutputLocation to use OutputIndex rather than OutPoint
    • Store Output values rather than Utxo values (and delete Utxo serialization)
    • Re-construct the Utxo from the Output value and OutputLocation key
    • Store the AddressLocation value after the Output value
  • Increment the database version

Testing:

  • Update raw data snapshot tests
  • Update struct data snapshot tests

CI:

  • If you added new Rust module files, make sure the paths in the regenerate state job are up to date
@teor2345
Copy link
Contributor Author

Closed by PRs #3993 / #3999

@mpguerra mpguerra removed the S-needs-triage Status: A bug report needs triage label Sep 28, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-state Area: State / database changes C-enhancement Category: This is an improvement lightwalletd any work associated with lightwalletd
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants