[storage] Introduce `Archive` #152

patrick-ogrady · 2024-10-31T18:30:54Z

Summary by CodeRabbit

New Features
- Introduced a new Archive module for a write-once key-value store optimized for high throughput and low-latency reads.
- Added benchmarking capabilities for HashMap operations to evaluate performance.
- New dependencies added to enhance functionality, including prometheus-client and rand.
Bug Fixes
- Enhanced error handling in the journal module for various scenarios.
Documentation
- Comprehensive documentation added for the journal module and the Archive module, including usage and error handling.
Tests
- Expanded test coverage for new features and error handling scenarios.

TODO:

coderabbitai · 2024-10-31T18:31:02Z

Important

Review skipped

Auto reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

📝 Walkthrough

Walkthrough

The pull request introduces modifications across several Cargo.toml files and Rust source files. Key changes include the addition of the criterion dependency in the main workspace, updates to the commonware-cryptography package to reference the workspace version of criterion, and the introduction of a new development dependency in the commonware-runtime package. Additionally, significant enhancements to the Blob struct in the runtime module improve concurrency and error handling. New benchmarking files are added to the commonware-storage package, along with a new Archive module for a write-once key-value store.

Changes

File	Change Summary
Cargo.toml	Added dependency `criterion = "0.5.1"`; reformatted `clap` dependency.
cryptography/Cargo.toml	Updated `criterion` in `[dev-dependencies]` from version `0.5.1` to workspace reference.
runtime/Cargo.toml	Added `[dev-dependencies]` section with `tracing-subscriber = { workspace = true }`.
runtime/src/deterministic.rs	Updated `Blob` struct to use `Arc<RwLock<Vec<u8>>>` for `content`; modified methods for thread safety; added `truncate` method.
runtime/src/lib.rs	Added `InsufficientLength` variant to `Error` enum; updated method signatures in `Storage` and `Blob` traits for concurrency.
runtime/src/tokio.rs	Changed `file` field in `Blob` to `Arc<AsyncMutex<(fs::File, u64)>>`; updated methods for safe concurrent file operations.
storage/Cargo.toml	Added dependencies `prometheus-client` and `rand`; added three benchmarks under `[[bench]]`.
storage/src/archive/benches/hashmap_iteration.rs	Introduced benchmarking for iterating over `HashMap` using `Criterion`.
storage/src/archive/benches/hashmap_load.rs	Introduced benchmarking for loading data into `HashMap`.
storage/src/archive/benches/hashmap_load_fixed.rs	Introduced benchmarking for loading fixed-size data into `HashMap`.
storage/src/archive/mod.rs	Added `Archive` module for a write-once key-value store; defined `Config`, `Translator`, and `Error` types.
storage/src/archive/storage.rs	Implemented `Archive` struct for managing records; added methods for data manipulation and error handling.
storage/src/archive/translator.rs	Added key translation functionality with `TwoCap`, `FourCap`, and `EightCap` structs.
storage/src/journal/mod.rs	Enhanced documentation and error handling; updated `Config` struct and `Error` enum.
storage/src/journal/storage.rs	Added metrics tracking to `Journal`; updated methods for improved functionality and error handling.
storage/src/lib.rs	Marked `commonware-storage` as ALPHA; declared `archive` module as public.

Possibly related PRs

[runtime] Add support for Storage and Blob #148: Modifications to Cargo.toml and new methods related to blob storage management, relevant to the dependency updates in this PR.
[storage] Introduce Journal #149: Introduction of a Journal for managing data storage, aligning with the broader context of enhancing storage capabilities across the project.

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

❤️ Share

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

Review comments: Directly reply to a review comment made by CodeRabbit. Example:
- I pushed a fix in commit <commit_id>, please review it.
- Generate unit testing code for this file.
- Open a follow-up GitHub issue for this discussion.
Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
- @coderabbitai generate unit testing code for this file.
- @coderabbitai modularize this function.
PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
- @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
- @coderabbitai read src/utils.ts and generate unit testing code.
- @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
- @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

@coderabbitai pause to pause the reviews on a PR.
@coderabbitai resume to resume the paused reviews.
@coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
@coderabbitai full review to do a full review from scratch and review all the files again.
@coderabbitai summary to regenerate the summary of the PR.
@coderabbitai resolve resolve all the CodeRabbit review comments.
@coderabbitai configuration to show the current CodeRabbit configuration for the repository.
@coderabbitai help to get help.

Other keywords and placeholders

Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (`.coderabbit.yaml`)

You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
Please see the configuration documentation for more information.
If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

Visit our Documentation for detailed information on how to use CodeRabbit.
Join our Discord Community to get help, request features, and share feedback.
Follow us on X/Twitter for updates and announcements.

…#153)

codecov · 2024-11-04T19:18:15Z

Codecov Report

Attention: Patch coverage is 92.77523% with 126 lines in your changes missing coverage. Please review.

Project coverage is 92.50%. Comparing base (6505afa) to head (b628851).
Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
storage/src/archive/mod.rs	91.91%	58 Missing ⚠️
storage/src/journal/mod.rs	81.81%	32 Missing ⚠️
storage/src/archive/storage.rs	93.39%	21 Missing ⚠️
storage/src/journal/storage.rs	94.71%	11 Missing ⚠️
runtime/src/deterministic.rs	94.59%	4 Missing ⚠️

@@            Coverage Diff            @@
##             main     #152     +/-   ##
=========================================
  Coverage   92.49%   92.50%             
=========================================
  Files          51       54      +3     
  Lines        9039    10577   +1538     
=========================================
+ Hits         8361     9784   +1423     
- Misses        678      793    +115

Files with missing lines	Coverage Δ
runtime/src/lib.rs	`97.73% <100.00%> (+0.70%)`	⬆️
runtime/src/tokio.rs	`96.27% <100.00%> (+0.61%)`	⬆️
storage/src/archive/translator.rs	`100.00% <100.00%> (ø)`
runtime/src/deterministic.rs	`98.10% <94.59%> (-0.15%)`	⬇️
storage/src/journal/storage.rs	`93.15% <94.71%> (-0.79%)`	⬇️
storage/src/archive/storage.rs	`93.39% <93.39%> (ø)`
storage/src/journal/mod.rs	`85.74% <81.81%> (-1.31%)`	⬇️
storage/src/archive/mod.rs	`91.91% <91.91%> (ø)`

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6505afa...b628851. Read the comment docs.

patrick-ogrady added 24 commits October 31, 2024 07:04

spike on archive init

a3f001c

implement put

6bb208e

implement get

c911935

first pass at prune

9405c66

pruning compiles

d231b7f

simpler prune

27cacea

use capped_key

2224bfd

add more comments

8424383

add bench for map iteration

57a8f04

add benchmark for map iteration

b041086

add benchmark for hashmap load

5e2ad41

add load benchmark

8601f5e

add benchmarks

a98620f

allow caller to specify arbitrary cap function and fixed size keys

5093b96

add custom cap func

fc02306

add explanation to index

c80e8bd

cleanup cap definition

5de87d6

add warning

6db989b

add tip for hashing

0a513b1

provide access to self during transform

8b5258a

move translator to own file

3dabf9b

add comment on integrity check

ccf69d1

add warning about translator

ea363f1

add read_exact to read_at

cee38d0

patrick-ogrady added 5 commits November 1, 2024 08:55

[runtime] Migrate Storage/Blob Operations to Immutable References (…

a70b52d

…#153)

spike on initial tests

f90bcd6

cleanup init

6645853

add pruning test

37baf55

get failing test

344c0ce

commonwarexyz unlocked this conversation Nov 4, 2024

patrick-ogrady added 13 commits November 4, 2024 06:16

more granular metrics

39ea792

[storage/archive] Remove Index padding (#161)

2d13338

cleanup MockIndex

07e6254

reuse vec for key benchmark

b81a0a1

use translator macro

112c595

add docs to cap

9a86de4

add more logs to check

f3d6388

add comment explaining why comparators are different

3347497

ensure we remove pending write tracking on prune

23ba8ef

add more context to InsufficientLength

7a7026a

elaborate on why read_at doesn't return bytes

d83403a

add test for key lookup corruption

43c1a2c

add note about cloning

e516c2d

coderabbitai bot previously approved these changes Nov 4, 2024

View reviewed changes

clone content before acquiring partition lock

423b7ac

patrick-ogrady dismissed coderabbitai[bot]’s stale review via 423b7ac November 4, 2024 18:35

patrick-ogrady added 2 commits November 4, 2024 10:48

add compression

ac3fff3

update zstd version

7ebe32e

patrick-ogrady marked this pull request as ready for review November 4, 2024 18:52

patrick-ogrady added 4 commits November 4, 2024 11:01

fix loop

68e5a70

add compression for archive example

7c32723

make compression field public

223ccf4

test determinism

b628851

patrick-ogrady merged commit 82e9d67 into main Nov 4, 2024
4 checks passed

patrick-ogrady deleted the archive branch November 4, 2024 19:15

patrick-ogrady mentioned this pull request Nov 4, 2024

[consensus] Add support for commonware-consensus::authority #111

Closed

27 tasks

This was referenced Nov 6, 2024

[storage] Introduce Metadata #170

Merged

[storage] Add index to Archive #176

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[storage] Introduce `Archive` #152

[storage] Introduce `Archive` #152

patrick-ogrady commented Oct 31, 2024 •

edited

Loading

coderabbitai bot commented Oct 31, 2024 •

edited

Loading

Review skipped

Walkthrough

Changes

Possibly related PRs

Chat

CodeRabbit Commands (Invoked using PR comments)

Other keywords and placeholders

CodeRabbit Configuration File (`.coderabbit.yaml`)

Documentation and Community

codecov bot commented Nov 4, 2024

[storage] Introduce Archive #152

[storage] Introduce Archive #152

Conversation

patrick-ogrady commented Oct 31, 2024 • edited Loading

Summary by CodeRabbit

TODO:

coderabbitai bot commented Oct 31, 2024 • edited Loading

Review skipped

Walkthrough

Changes

Possibly related PRs

Chat

CodeRabbit Commands (Invoked using PR comments)

Other keywords and placeholders

CodeRabbit Configuration File (.coderabbit.yaml)

Documentation and Community

codecov bot commented Nov 4, 2024

Codecov Report

[storage] Introduce `Archive` #152

[storage] Introduce `Archive` #152

patrick-ogrady commented Oct 31, 2024 •

edited

Loading

coderabbitai bot commented Oct 31, 2024 •

edited

Loading

CodeRabbit Configuration File (`.coderabbit.yaml`)