[Fix] Fix indexer FN and localnet. #14727

grao1991 · 2024-09-23T23:09:03Z

Description

This is to fix the issue from the config combination of (sharding, internal index, table info).

Tested major 3 path and looks good.

How Has This Been Tested?

tested locally and it looks good.

3 cases:

no table info or sharding

# only
api:
    enabled: true
    address: 127.0.0.1:8084

table info without sharding

indexer_table_info:
    parser_task_count: 10
    parser_batch_size: 100
    table_info_service_mode: IndexingOnly

indexer_grpc:
    enabled: true
    address: 0.0.0.0:50051
    processor_task_count: 10
    processor_batch_size: 100
    output_batch_size: 100

table info with sharding

indexer_db_config:
    enable_event: true
    enable_statekeys: true
    enable_transaction: true
storage:
    rocksdb_configs:
        enable_storage_sharding: true

indexer_table_info:
    parser_task_count: 10
    parser_batch_size: 100
    table_info_service_mode: IndexingOnly

Localnet looks good too:

Key Areas to Review

Type of Change

Which Components or Systems Does This Change Impact?

Checklist

I have read and followed the CONTRIBUTING doc
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
I identified and added all stakeholders and component owners affected by this change as reviewers
I tested both happy and unhappy path of the functionality
I have made corresponding changes to the documentation

trunk-io · 2024-09-23T23:09:06Z

⏱️ 4h 8m total CI duration on this PR

Job	Cumulative Duration	Recent Runs
execution-performance / single-node-performance	46m	🟩 🟩
forge-compat-test / forge	40m	🟩 🟩
forge-framework-upgrade-test / forge	31m	🟩 🟩
forge-e2e-test / forge	28m	🟩 🟩
general-lints	11m	🟩 🟩 🟩 🟩 🟩 (+1 more)
rust-cargo-deny	10m	🟩 🟩 🟩 🟩 🟩 (+1 more)
rust-move-tests	10m	🟩
rust-move-tests	9m	🟩
execution-performance / test-target-determinator	8m	🟩 🟩
test-target-determinator	8m	🟩 🟩
check	7m	🟩 🟩
rust-move-tests	7m	⬜
check-dynamic-deps	6m	🟩 🟩 🟩 🟩 🟩 (+2 more)
rust-doc-tests	5m	🟩
rust-doc-tests	5m	🟩
rust-move-tests	4m	🟥
semgrep/ci	3m	🟩 🟩 🟩 🟩 🟩 (+2 more)
rust-move-tests	2m	🟩
rust-move-tests	2m	🟩
file_change_determinator	1m	🟩 🟩 🟩 🟩 🟩 (+2 more)
file_change_determinator	1m	🟩 🟩 ⬜ 🟩 🟩 (+2 more)
Backport PR	40s	🟥 🟩
permission-check	22s	🟩 🟩 🟩 🟩 🟩 (+2 more)
permission-check	20s	🟩 🟩 🟩 🟩 🟩 (+2 more)
file_change_determinator	18s	🟩 🟩
permission-check	17s	🟩 🟩 🟩 🟩 🟩 (+2 more)
permission-check	16s	🟩 🟩 🟩 🟩 🟩 (+2 more)
permission-check	6s	🟩 🟩
rust-move-tests	6s	⬜
permission-check	5s	🟩 🟩
determine-docker-build-metadata	3s	🟩 🟩
run-tests-main-branch	1s	⬜

_{settings ⋅ feedback ⋅ docs ⋅ learn more about trunk.io}

msmouse · 2024-09-23T23:38:41Z

api/src/context.rs

- self.get_latest_storage_ledger_info()
+ if let Some(indexer_reader) = self.indexer_reader.as_ref() {
+ if indexer_reader.is_internal_indexer_enabled() {
+ return self.get_latest_internal_indexer_ledger_version_and_ledger_info();


while you are at this, can you rename the function being called?

* [Fix] Only use version from internal indexer when it is enabled. * fix the localnet support. --------- Co-authored-by: Larry Liu <larry@aptoslabs.com> (cherry picked from commit e74b307)

github-actions · 2024-09-24T00:28:51Z

💚 All backports created successfully

Status	Branch	Result
✅	aptos-release-v1.20

Questions ?

Please refer to the Backport tool documentation and see the Github Action logs for details

github-actions · 2024-09-24T00:56:29Z

✅ Forge suite `realistic_env_max_load` success on `245a04c5b8bf1dff53f9f088041aac9d9eb3c8b1`

two traffics test: inner traffic : committed: 14353.32 txn/s, latency: 2768.08 ms, (p50: 2700 ms, p70: 2700, p90: 3000 ms, p99: 3300 ms), latency samples: 5457460
two traffics test : committed: 99.96 txn/s, latency: 1599.30 ms, (p50: 1500 ms, p70: 1600, p90: 1700 ms, p99: 7300 ms), latency samples: 1680
Latency breakdown for phase 0: ["QsBatchToPos: max: 0.240, avg: 0.225", "QsPosToProposal: max: 1.170, avg: 1.087", "ConsensusProposalToOrdered: max: 0.311, avg: 0.294", "ConsensusOrderedToCommit: max: 0.444, avg: 0.413", "ConsensusProposalToCommit: max: 0.739, avg: 0.707"]
Max non-epoch-change gap was: 1 rounds at version 27122 (avg 0.00) [limit 4], 1.79s no progress at version 27122 (avg 0.20s) [limit 15].
Max epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 8.53s no progress at version 2142048 (avg 8.53s) [limit 15].
Test Ok

github-actions · 2024-09-24T00:58:21Z

✅ Forge suite `framework_upgrade` success on `25a081116546670e62ca927ba90478de78557056` ==> `245a04c5b8bf1dff53f9f088041aac9d9eb3c8b1`

Compatibility test results for 25a081116546670e62ca927ba90478de78557056 ==> 245a04c5b8bf1dff53f9f088041aac9d9eb3c8b1 (PR)
Upgrade the nodes to version: 245a04c5b8bf1dff53f9f088041aac9d9eb3c8b1
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1185.57 txn/s, submitted: 1190.04 txn/s, failed submission: 4.48 txn/s, expired: 4.48 txn/s, latency: 2557.49 ms, (p50: 2400 ms, p70: 2700, p90: 3900 ms, p99: 4800 ms), latency samples: 105920
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1231.47 txn/s, submitted: 1233.48 txn/s, failed submission: 2.01 txn/s, expired: 2.01 txn/s, latency: 2434.14 ms, (p50: 2100 ms, p70: 2400, p90: 3900 ms, p99: 5500 ms), latency samples: 110020
5. check swarm health
Compatibility test for 25a081116546670e62ca927ba90478de78557056 ==> 245a04c5b8bf1dff53f9f088041aac9d9eb3c8b1 passed
Upgrade the remaining nodes to version: 245a04c5b8bf1dff53f9f088041aac9d9eb3c8b1
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1207.12 txn/s, submitted: 1209.19 txn/s, failed submission: 2.07 txn/s, expired: 2.07 txn/s, latency: 2527.50 ms, (p50: 2400 ms, p70: 2900, p90: 3700 ms, p99: 5400 ms), latency samples: 105180
Test Ok

github-actions · 2024-09-24T01:02:40Z

✅ Forge suite `compat` success on `25a081116546670e62ca927ba90478de78557056` ==> `245a04c5b8bf1dff53f9f088041aac9d9eb3c8b1`

Compatibility test results for 25a081116546670e62ca927ba90478de78557056 ==> 245a04c5b8bf1dff53f9f088041aac9d9eb3c8b1 (PR)
1. Check liveness of validators at old version: 25a081116546670e62ca927ba90478de78557056
compatibility::simple-validator-upgrade::liveness-check : committed: 14609.58 txn/s, latency: 2341.16 ms, (p50: 2100 ms, p70: 2200, p90: 2400 ms, p99: 6600 ms), latency samples: 469100
2. Upgrading first Validator to new version: 245a04c5b8bf1dff53f9f088041aac9d9eb3c8b1
compatibility::simple-validator-upgrade::single-validator-upgrading : committed: 7723.91 txn/s, latency: 3568.28 ms, (p50: 3900 ms, p70: 4300, p90: 4700 ms, p99: 5000 ms), latency samples: 137840
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 7806.77 txn/s, latency: 4082.91 ms, (p50: 4200 ms, p70: 4300, p90: 6100 ms, p99: 6300 ms), latency samples: 259740
3. Upgrading rest of first batch to new version: 245a04c5b8bf1dff53f9f088041aac9d9eb3c8b1
compatibility::simple-validator-upgrade::half-validator-upgrading : committed: 7579.21 txn/s, latency: 3718.12 ms, (p50: 4200 ms, p70: 4300, p90: 4400 ms, p99: 4600 ms), latency samples: 143020
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 7054.03 txn/s, latency: 4556.78 ms, (p50: 4400 ms, p70: 4500, p90: 7000 ms, p99: 7400 ms), latency samples: 255080
4. upgrading second batch to new version: 245a04c5b8bf1dff53f9f088041aac9d9eb3c8b1
compatibility::simple-validator-upgrade::rest-validator-upgrading : committed: 11495.17 txn/s, latency: 2385.24 ms, (p50: 2300 ms, p70: 2500, p90: 3700 ms, p99: 4000 ms), latency samples: 197020
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 11379.54 txn/s, latency: 2715.99 ms, (p50: 2500 ms, p70: 2600, p90: 5000 ms, p99: 6600 ms), latency samples: 369320
5. check swarm health
Compatibility test for 25a081116546670e62ca927ba90478de78557056 ==> 245a04c5b8bf1dff53f9f088041aac9d9eb3c8b1 passed
Test Ok

grao1991 marked this pull request as ready for review September 23, 2024 23:13

grao1991 requested review from banool, gregnazario and 0xmaayan as code owners September 23, 2024 23:13

grao1991 requested review from areshand, msmouse and larry-aptos September 23, 2024 23:13

grao1991 force-pushed the grao_fix_ledger branch from 79d5721 to adc7399 Compare September 23, 2024 23:27

grao1991 requested a review from lightmark as a code owner September 23, 2024 23:27

grao1991 changed the title ~~[Fix] Only use version from internal indexer when it is enabled.~~ [Fix] Fix indexer FN and localnet. Sep 23, 2024

areshand approved these changes Sep 23, 2024

View reviewed changes

msmouse approved these changes Sep 23, 2024

View reviewed changes

grao1991 and others added 2 commits September 23, 2024 23:45

[Fix] Only use version from internal indexer when it is enabled.

db42884

fix the localnet support.

245a04c

grao1991 force-pushed the grao_fix_ledger branch from 763f1e5 to 245a04c Compare September 23, 2024 23:45

grao1991 enabled auto-merge (squash) September 23, 2024 23:46

This comment has been minimized.

Sign in to view

grao1991 merged commit e74b307 into main Sep 24, 2024
81 of 92 checks passed

grao1991 deleted the grao_fix_ledger branch September 24, 2024 00:21

grao1991 added the v1.20 label Sep 24, 2024

github-actions bot mentioned this pull request Sep 24, 2024

[cp][aptos-release-v1.20] [Fix] Fix indexer FN and localnet. #14730

Merged

This comment has been minimized.

Sign in to view

larry-aptos mentioned this pull request Sep 24, 2024

Release CLI 4.2.3. #14731

Merged

22 tasks

junkil-park mentioned this pull request Sep 26, 2024

Localnet terminates when enabling the storage sharding #14759

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Fix] Fix indexer FN and localnet. #14727

[Fix] Fix indexer FN and localnet. #14727

grao1991 commented Sep 23, 2024 •

edited by larry-aptos

Loading

trunk-io bot commented Sep 23, 2024 •

edited

Loading

msmouse Sep 23, 2024

grao1991 Sep 23, 2024

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

github-actions bot commented Sep 24, 2024

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

github-actions bot commented Sep 24, 2024

github-actions bot commented Sep 24, 2024

github-actions bot commented Sep 24, 2024

[Fix] Fix indexer FN and localnet. #14727

[Fix] Fix indexer FN and localnet. #14727

Conversation

grao1991 commented Sep 23, 2024 • edited by larry-aptos Loading

Description

How Has This Been Tested?

Key Areas to Review

Type of Change

Which Components or Systems Does This Change Impact?

Checklist

trunk-io bot commented Sep 23, 2024 • edited Loading

msmouse Sep 23, 2024

Choose a reason for hiding this comment

grao1991 Sep 23, 2024

Choose a reason for hiding this comment

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

github-actions bot commented Sep 24, 2024

💚 All backports created successfully

Questions ?

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

github-actions bot commented Sep 24, 2024

✅ Forge suite realistic_env_max_load success on 245a04c5b8bf1dff53f9f088041aac9d9eb3c8b1

github-actions bot commented Sep 24, 2024

✅ Forge suite framework_upgrade success on 25a081116546670e62ca927ba90478de78557056 ==> 245a04c5b8bf1dff53f9f088041aac9d9eb3c8b1

github-actions bot commented Sep 24, 2024

✅ Forge suite compat success on 25a081116546670e62ca927ba90478de78557056 ==> 245a04c5b8bf1dff53f9f088041aac9d9eb3c8b1

grao1991 commented Sep 23, 2024 •

edited by larry-aptos

Loading

trunk-io bot commented Sep 23, 2024 •

edited

Loading

✅ Forge suite `realistic_env_max_load` success on `245a04c5b8bf1dff53f9f088041aac9d9eb3c8b1`

✅ Forge suite `framework_upgrade` success on `25a081116546670e62ca927ba90478de78557056` ==> `245a04c5b8bf1dff53f9f088041aac9d9eb3c8b1`

✅ Forge suite `compat` success on `25a081116546670e62ca927ba90478de78557056` ==> `245a04c5b8bf1dff53f9f088041aac9d9eb3c8b1`