Add support for Cid-indexed data in forest-tool api generate-test-snapshot #5597

elmattic · 2025-04-23T13:32:52Z

Summary of changes

Changes introduced in this pull request:

Add support for writing Cid-indexed data in forest-tool api generate-test-snapshot
Avoid CDN caching in rpc_regression_tests
Add shell script to easily upload rcpsnap files to DO

./scripts/tests/upload_rcpsnaps.sh temp
temp/filecoin_chaingetevents_1745400136273439.rpcsnap.json : 74.99%   (  3.46 MiB =>   2.59 MiB, temp/filecoin_chaingetevents_1745400136273439.rpcsnap.json.zst) 
✅ Uploaded: temp/filecoin_chaingetevents_1745400136273439.rpcsnap.json.zst
temp/filecoin_chaingetevents_1745400136273611.rpcsnap.json : 74.81%   (  3.54 MiB =>   2.65 MiB, temp/filecoin_chaingetevents_1745400136273611.rpcsnap.json.zst) 
✅ Uploaded: temp/filecoin_chaingetevents_1745400136273611.rpcsnap.json.zst
temp/filecoin_chaingetevents_1745400136273670.rpcsnap.json : 74.42%   (  3.64 MiB =>   2.71 MiB, temp/filecoin_chaingetevents_1745400136273670.rpcsnap.json.zst) 
✅ Uploaded: temp/filecoin_chaingetevents_1745400136273670.rpcsnap.json.zst
temp/filecoin_chaingetevents_1745400136273739.rpcsnap.json : 74.10%   (  3.79 MiB =>   2.81 MiB, temp/filecoin_chaingetevents_1745400136273739.rpcsnap.json.zst) 
✅ Uploaded: temp/filecoin_chaingetevents_1745400136273739.rpcsnap.json.zst
temp/filecoin_chaingetevents_1745400136273790.rpcsnap.json : 73.66%   (  3.95 MiB =>   2.91 MiB, temp/filecoin_chaingetevents_1745400136273790.rpcsnap.json.zst) 
✅ Uploaded: temp/filecoin_chaingetevents_1745400136273790.rpcsnap.json.zst
temp/filecoin_chaingetevents_1745400136273834.rpcsnap.json : 73.08%   (  4.04 MiB =>   2.95 MiB, temp/filecoin_chaingetevents_1745400136273834.rpcsnap.json.zst) 
✅ Uploaded: temp/filecoin_chaingetevents_1745400136273834.rpcsnap.json.zst
temp/filecoin_chaingetevents_1745400136273881.rpcsnap.json : 73.12%   (  4.04 MiB =>   2.95 MiB, temp/filecoin_chaingetevents_1745400136273881.rpcsnap.json.zst) 
✅ Uploaded: temp/filecoin_chaingetevents_1745400136273881.rpcsnap.json.zst
temp/filecoin_chaingetevents_1745400136273923.rpcsnap.json : 72.69%   (  4.12 MiB =>   2.99 MiB, temp/filecoin_chaingetevents_1745400136273923.rpcsnap.json.zst) 
✅ Uploaded: temp/filecoin_chaingetevents_1745400136273923.rpcsnap.json.zst
temp/filecoin_chaingetevents_1745400136273969.rpcsnap.json : 72.24%   (  4.26 MiB =>   3.08 MiB, temp/filecoin_chaingetevents_1745400136273969.rpcsnap.json.zst) 
✅ Uploaded: temp/filecoin_chaingetevents_1745400136273969.rpcsnap.json.zst
temp/filecoin_chaingetevents_1745400136274014.rpcsnap.json : 72.17%   (  4.26 MiB =>   3.07 MiB, temp/filecoin_chaingetevents_1745400136274014.rpcsnap.json.zst) 
✅ Uploaded: temp/filecoin_chaingetevents_1745400136274014.rpcsnap.json.zst
🧪 Do you want to run 'cargo test --lib -- --test rpc_regression_tests --nocapture'? [y/N] y
▶ Running tests...
   Compiling forest-filecoin v0.26.0 (/Users/guillaume/Code/dark-forest)
    Finished `test` profile [unoptimized] target(s) in 29.54s
     Running unittests src/lib.rs (target/debug/deps/forest-852e1bcfd35dd13c)

running 2 tests
test tool::subcommands::api_cmd::test_snapshot::tests::rpc_regression_tests_print_uncovered ... ok
Testing filecoin_beacongetentry_1741270283524366.rpcsnap.json.zst ...  succeeded, took 682ms 813us 333ns.
Testing filecoin_chaingetblock_1736937164811210.rpcsnap.json.zst ...  succeeded, took 7ms 796us 458ns.
Testing filecoin_chaingetblockmessages_1736937164799678.rpcsnap.json.zst ...  succeeded, took 5ms 803us 542ns.
Testing filecoin_chaingetevents_1745400136273439.rpcsnap.json.zst ...  succeeded, took 22s 971ms 800us 83ns.
Testing filecoin_chaingetevents_1745400136273670.rpcsnap.json.zst ...  succeeded, took 83ms 517us.
Testing filecoin_chaingetevents_1745400136273739.rpcsnap.json.zst ...  succeeded, took 85ms 146us 834ns.
Testing filecoin_chaingetevents_1745400136273790.rpcsnap.json.zst ...  succeeded, took 103ms 65us 792ns.
Testing filecoin_chaingetevents_1745400136273611.rpcsnap.json.zst ...  succeeded, took 72ms 524us 792ns.
Testing filecoin_chaingetevents_1745400136273881.rpcsnap.json.zst ...  succeeded, took 93ms 428us 583ns.
Testing filecoin_chaingetevents_1745400136273923.rpcsnap.json.zst ...  succeeded, took 74ms 324us 833ns.
Testing filecoin_chaingetgenesis_1736937286915866.rpcsnap.json.zst ...  succeeded, took 6ms 605us 667ns.
Testing filecoin_chaingetmessage_1741270616352800.rpcsnap.json.zst ...  succeeded, took 7ms 997us.
Testing filecoin_chaingetevents_1745400136274014.rpcsnap.json.zst ...  succeeded, took 111ms 873us 625ns.
Testing filecoin_chaingetmessagesintipset_1741270616353560.rpcsnap.json.zst ...  succeeded, took 3ms 932us
...

test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 517 filtered out; finished in 121.97s

Reference issue to close (if applicable)

Closes #5585

Change checklist

I have performed a self-review of my own code,
I have made corresponding changes to the documentation. All new code adheres to the team's documentation standards,
I have added tests that prove my fix is effective or that my feature works (if possible),
I have made sure the CHANGELOG is up-to-date. All user-facing changes should be reflected in this document.

LesnyRumcajs · 2025-04-29T10:19:21Z

src/tool/subcommands/api_cmd/test_snapshot.rs

+                );
+                let mut url = Url::parse(&base_url).ok()?;
+                url.query_pairs_mut()
+                    .append_pair("cachebust", &Utc::now().timestamp().to_string());


Let's document what is cachebust.

sudo-shashank · 2025-04-29T12:08:43Z

docs/docs/developers/guides/rpc_test_snapshot.md

- run `cargo test --lib -- --test rpc_regression_tests --nocapture`
+- Manual Method
+
+  1.  Upload the test snapshots (`.zst` format is recommended) to the Digital Ocean space `forest-snapshots/rpc_test`


shall we mention to upload only .zst format to the DO space for these test snapshots instead of recommending?

Good idea, in practice, we always compress the files before uploading, and the script handles that automagically.

sudo-shashank

LGTM👍🏻

LesnyRumcajs · 2025-04-29T15:16:51Z

scripts/tests/upload_rcpsnaps.sh

+    BUCKET_URL="s3://${SPACE_NAME}/${DEST_PATH}"
+
+    if zstd -f "$FILE_PATH" -o "$COMPRESSED_FILE"; then
+        if s3cmd --quiet --no-progress put "${COMPRESSED_FILE}" "${BUCKET_URL}" --acl-public --mime-type="application/json" --add-header="Cache-Control: no-cache, no-store, must-revalidate"; then


s3cmd is not a standard tool. It'd be good UX to have a check at the beginning of the script if it's in PATH and if not, nag the user.

LesnyRumcajs · 2025-04-29T15:18:10Z

scripts/tests/upload_rcpsnaps.sh

+    DEST_PATH="${DEST_DIR}/${FILE_NAME}.zst"
+    BUCKET_URL="s3://${SPACE_NAME}/${DEST_PATH}"
+
+    if zstd -f "$FILE_PATH" -o "$COMPRESSED_FILE"; then


In general, can we simplify things via set -euo pipefail? I think error handling would be better this way.

LesnyRumcajs · 2025-04-30T09:36:29Z

src/tool/subcommands/api_cmd/test_snapshots.txt

+filecoin_chaingetevents_1745400136273439.rpcsnap.json.zst
+filecoin_chaingetevents_1745400136273611.rpcsnap.json.zst
+filecoin_chaingetevents_1745400136273670.rpcsnap.json.zst
+filecoin_chaingetevents_1745400136273739.rpcsnap.json.zst
+filecoin_chaingetevents_1745400136273790.rpcsnap.json.zst
+filecoin_chaingetevents_1745400136273834.rpcsnap.json.zst
+filecoin_chaingetevents_1745400136273881.rpcsnap.json.zst
+filecoin_chaingetevents_1745400136273923.rpcsnap.json.zst
+filecoin_chaingetevents_1745400136273969.rpcsnap.json.zst
+filecoin_chaingetevents_1745400136274014.rpcsnap.json.zst


Do we need all of those snapshots? If yes, how are they different?

We don't really need them all. I can do some trimming. ✂️

LesnyRumcajs · 2025-04-30T09:36:56Z

It'd be great to have a review from @hanabi1224 once he's back.

elmattic added 11 commits April 23, 2025 15:32

Add support for filecoin data

a067199

Comment-out region

8a21e52

Fix caching issues

2b391b8

Merge branch 'main' into elmattic/rcpsnap-index-cids

853aaf2

Fix lint issues

ad821c4

Fix lint issue

71a4f91

Add dummy param

9b7d62d

Appease Clippy

f84de88

Remove dup lines

017a933

Make mappings optional

a8c6bf7

Refactor

0f1cabd

elmattic changed the title ~~Add support for filecoin data~~ Add support for Cid-indexed data in forest-tool api generate-test-snapshot Apr 28, 2025

Update documentation

c77fb7d

github-actions bot deployed to preview April 29, 2025 07:51 View deployment

elmattic added 2 commits April 29, 2025 10:01

Fix spellcheck

2e5bdc4

Swap bullets and numbers

c68f986

github-actions bot deployed to preview April 29, 2025 08:08 View deployment

Update comment

de7b333

elmattic marked this pull request as ready for review April 29, 2025 08:11

elmattic requested a review from a team as a code owner April 29, 2025 08:11

elmattic requested review from LesnyRumcajs and sudo-shashank and removed request for a team April 29, 2025 08:11

github-actions bot deployed to preview April 29, 2025 08:12 View deployment

Fix dictionary

cc9643a

github-actions bot deployed to preview April 29, 2025 08:20 View deployment

Fix dictionary

6d8e9bb

github-actions bot deployed to preview April 29, 2025 08:32 View deployment

Add multiaddress

68ffe50

github-actions bot deployed to preview April 29, 2025 08:46 View deployment

Fix with prettier

3e5550f

github-actions bot deployed to preview April 29, 2025 08:54 View deployment

LesnyRumcajs reviewed Apr 29, 2025

View reviewed changes

sudo-shashank reviewed Apr 29, 2025

View reviewed changes

sudo-shashank approved these changes Apr 29, 2025

View reviewed changes

elmattic added 2 commits April 29, 2025 14:42

Add comment

3975da0

Improve documentation

5b1b59e

github-actions bot deployed to preview April 29, 2025 12:47 View deployment

LesnyRumcajs reviewed Apr 29, 2025

View reviewed changes

elmattic added 2 commits April 30, 2025 10:45

Address review comments

cea06d0

Merge branch 'main' into elmattic/rcpsnap-index-cids

ac82351

github-actions bot deployed to preview April 30, 2025 08:47 View deployment

github-actions bot deployed to preview April 30, 2025 08:48 View deployment

LesnyRumcajs reviewed Apr 30, 2025

View reviewed changes

LesnyRumcajs requested a review from hanabi1224 April 30, 2025 09:37

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for Cid-indexed data in forest-tool api generate-test-snapshot #5597

Add support for Cid-indexed data in forest-tool api generate-test-snapshot #5597

elmattic commented Apr 23, 2025 •

edited

Loading

LesnyRumcajs Apr 29, 2025

elmattic Apr 29, 2025

sudo-shashank Apr 29, 2025

elmattic Apr 29, 2025

sudo-shashank left a comment •

edited

Loading

LesnyRumcajs Apr 29, 2025

LesnyRumcajs Apr 29, 2025

LesnyRumcajs Apr 30, 2025

elmattic Apr 30, 2025 •

edited

Loading

LesnyRumcajs commented Apr 30, 2025

Add support for Cid-indexed data in forest-tool api generate-test-snapshot #5597

Are you sure you want to change the base?

Add support for Cid-indexed data in forest-tool api generate-test-snapshot #5597

Conversation

elmattic commented Apr 23, 2025 • edited Loading

Summary of changes

Reference issue to close (if applicable)

Other information and links

Change checklist

LesnyRumcajs Apr 29, 2025

Choose a reason for hiding this comment

elmattic Apr 29, 2025

Choose a reason for hiding this comment

sudo-shashank Apr 29, 2025

Choose a reason for hiding this comment

elmattic Apr 29, 2025

Choose a reason for hiding this comment

sudo-shashank left a comment • edited Loading

Choose a reason for hiding this comment

LesnyRumcajs Apr 29, 2025

Choose a reason for hiding this comment

LesnyRumcajs Apr 29, 2025

Choose a reason for hiding this comment

LesnyRumcajs Apr 30, 2025

Choose a reason for hiding this comment

elmattic Apr 30, 2025 • edited Loading

Choose a reason for hiding this comment

LesnyRumcajs commented Apr 30, 2025

elmattic commented Apr 23, 2025 •

edited

Loading

sudo-shashank left a comment •

edited

Loading

elmattic Apr 30, 2025 •

edited

Loading