Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Digest item must match that calculated #4808

Closed
2 tasks done
mustermeiszer opened this issue Jun 17, 2024 · 20 comments
Closed
2 tasks done

Digest item must match that calculated #4808

mustermeiszer opened this issue Jun 17, 2024 · 20 comments
Labels
I2-bug The node fails to follow expected behavior. I10-unconfirmed Issue might be valid, but it's not yet known.

Comments

@mustermeiszer
Copy link

mustermeiszer commented Jun 17, 2024

Is there an existing issue?

  • I have searched the existing issues

Experiencing problems? Have you tried our Stack Exchange first?

  • This is not a support question.

Description of bug

Testnet chains are halting and not producing any blocks anymore.

We recently upgrade the development environments to release-polkadot-v1.7.2. Asyncronous backing is NOT activated.
At first the chains were producing blocks just fine. But after submitting a certain transaction the block production stopped.

  • collator tries to create collation → not advertising as collator gets banned
  • relay-chain is banning the collator after digest mismatch (see logs)

Relevant Logs

  • Relay-chain
Jun 17 07:57:54.527 ERROR runtime: panicked at /home/builder/cargo/git/checkouts/polkadot-sdk-cff69157b985ed76/c00ca35/substrate/frame/executive/src/lib.rs:671:13:
Digest item must match that calculated.    
2024-06-17 07:57:54.540  WARN tokio-runtime-worker parachain::pvf: execution worker concluded, error occurred: candidate validation: invalid: worker reported: execute: execute error: AbortedDueToTrap(MessageWithBacktrace { message: "wasm trap: wasm `unreachable` instruction executed", backtrace: Some(Backtrace { backtrace_string: "error while executing at wasm backtrace:\n    0: 0x9cbaff - <unknown>!rust_begin_unwind\n    1: 0x1b4ed - <unknown>!core::panicking::panic_fmt::he437d3dc536e58b1\n    2: 0x519432 - <unknown>!frame_executive::Executive<System,Block,Context,UnsignedValidator,AllPalletsWithSystem,COnRuntimeUpgrade>::execute_block::h9e43765471704d26\n    3: 0x3169b8 - <unknown>!<cumulus_pallet_aura_ext::BlockExecutor<T,I> as frame_support::traits::misc::ExecuteBlock<Block>>::execute_block::hf2d5844e3030d2e3\n    4: 0x665d33 - <unknown>!environmental::local_key::LocalKey<T>::with::hbfb85ba184cae654\n    5: 0x9037a0 - <unknown>!cumulus_pallet_parachain_system::validate_block::implementation::validate_block::hd50e0791e879bcec\n    6: 0x77dfc6 - <unknown>!validate_block\n    7: 0xa9f8e0 - <unknown>!<wasm function 12034>" }) }) artifact_id=ArtifactId { code_hash: 0xf82e1d5e0d4f44824e27e0a40f1836eff887e32460e72563f7604434edfbb1c3, executor_params_hash: 0x03170a2e7597b7b7e3d84c05391d139a62b157e78786d8c082f29dcf4c111314 } worker=Worker(1v1) worker_rip=false
2024-06-17 07:57:54.540  INFO tokio-runtime-worker parachain::candidate-validation: Failed to validate candidate para_id=Id(2000) error=Invalid(WorkerReportedInvalid("execute: execute error: AbortedDueToTrap(MessageWithBacktrace { message: \"wasm trap: wasm `unreachable` instruction executed\", backtrace: Some(Backtrace { backtrace_string: \"error while executing at wasm backtrace:\\n    0: 0x9cbaff - <unknown>!rust_begin_unwind\\n    1: 0x1b4ed - <unknown>!core::panicking::panic_fmt::he437d3dc536e58b1\\n    2: 0x519432 - <unknown>!frame_executive::Executive<System,Block,Context,UnsignedValidator,AllPalletsWithSystem,COnRuntimeUpgrade>::execute_block::h9e43765471704d26\\n    3: 0x3169b8 - <unknown>!<cumulus_pallet_aura_ext::BlockExecutor<T,I> as frame_support::traits::misc::ExecuteBlock<Block>>::execute_block::hf2d5844e3030d2e3\\n    4: 0x665d33 - <unknown>!environmental::local_key::LocalKey<T>::with::hbfb85ba184cae654\\n    5: 0x9037a0 - <unknown>!cumulus_pallet_parachain_system::validate_block::implementation::validate_block::hd50e0791e879bcec\\n    6: 0x77dfc6 - <unknown>!validate_block\\n    7: 0xa9f8e0 - <unknown>!<wasm function 12034>\" }) })"))
2024-06-17 07:57:54.543  WARN tokio-runtime-worker parachain::candidate-backing: Validation yielded an invalid candidate candidate_hash=0x58b75b29f3c2119abcc0c78b0f6239a12ef4930dc21ebb56d2c80634da936370 reason=ExecutionError("execute: execute error: AbortedDueToTrap(MessageWithBacktrace { message: \"wasm trap: wasm `unreachable` instruction executed\", backtrace: Some(Backtrace { backtrace_string: \"error while executing at wasm backtrace:\\n    0: 0x9cbaff - <unknown>!rust_begin_unwind\\n    1: 0x1b4ed - <unknown>!core::panicking::panic_fmt::he437d3dc536e58b1\\n    2: 0x519432 - <unknown>!frame_executive::Executive<System,Block,Context,UnsignedValidator,AllPalletsWithSystem,COnRuntimeUpgrade>::execute_block::h9e43765471704d26\\n    3: 0x3169b8 - <unknown>!<cumulus_pallet_aura_ext::BlockExecutor<T,I> as frame_support::traits::misc::ExecuteBlock<Block>>::execute_block::hf2d5844e3030d2e3\\n    4: 0x665d33 - <unknown>!environmental::local_key::LocalKey<T>::with::hbfb85ba184cae654\\n    5: 0x9037a0 - <unknown>!cumulus_pallet_parachain_system::validate_block::implementation::validate_block::hd50e0791e879bcec\\n    6: 0x77dfc6 - <unknown>!validate_block\\n    7: 0xa9f8e0 - <unknown>!<wasm function 12034>\" }) })") traceID=117924102977252537788624498838072342945
2024-06-17 07:57:54.543  WARN tokio-runtime-worker peerset: Report 12D3KooWFBJJtsggfTSuBdzUkug9E5pv8fkjYvCJiqtXPCLL3cEQ: -2147483648 to -2147483648. Reason: A collator was reported by another subsystem. Banned, disconnecting.    
2024-06-17 07:57:57.649  INFO tokio-runtime-worker substrate: 💤 Idle (3 peers), best: #2314512 (0x67a1…c88d), finalized #2314509 (0x0106…8a3b), ⬇ 5.1kiB/s ⬆ 2.5kiB/s    
  • Collator
ERROR 2024-06-17T08:24:54.090488502Z [resource.labels.containerName: centrifugechaincollator-0] 2024-06-17 08:24:54.081 WARN tokio-runtime-worker peerset: [Relaychain] Trying to remove unknown reserved node 12D3KooWJiPVADquvujXz1fn5eWvg7vW9wdxKmTVnnM1Lqbk8Trd from SetId(3).
ERROR 2024-06-17T08:24:54.190575467Z [resource.labels.containerName: centrifugechaincollator-0] 2024-06-17 08:24:54.158 INFO tokio-runtime-worker substrate: [Relaychain] 💤 Idle (3 peers), best: #2314765 (0x73bf…e15d), finalized #2314762 (0xbc52…dfc2), ⬇ 1.4kiB/s ⬆ 5.0kiB/s
ERROR 2024-06-17T08:24:54.190632784Z [resource.labels.containerName: centrifugechaincollator-0] 2024-06-17 08:24:54.185 INFO tokio-runtime-worker substrate: [🌀Parachain] 💤 Idle (1 peers), best: #1136685 (0xb7fb…4bc1), finalized #1136684 (0x4cd4…3304), ⬇ 98 B/s ⬆ 0.1kiB/s
ERROR 2024-06-17T08:24:58.099469378Z [resource.labels.containerName: centrifugechaincollator-0] 2024-06-17 08:24:58.066 WARN tokio-runtime-worker peerset: [Relaychain] Trying to remove unknown reserved node 12D3KooWFX4id8BGKkeGRnjTi4LQG3r9jcbUyaRF5rPYobSoYDnR from SetId(3).
ERROR 2024-06-17T08:24:59.202008281Z [resource.labels.containerName: centrifugechaincollator-0] 2024-06-17 08:24:59.159 INFO tokio-runtime-worker substrate: [Relaychain] 💤 Idle (4 peers), best: #2314765 (0x73bf…e15d), finalized #2314763 (0x1fec…1227), ⬇ 1.1kiB/s ⬆ 1.1kiB/s
ERROR 2024-06-17T08:24:59.202053763Z [resource.labels.containerName: centrifugechaincollator-0] 2024-06-17 08:24:59.186 INFO tokio-runtime-worker substrate: [🌀Parachain] 💤 Idle (1 peers), best: #1136685 (0xb7fb…4bc1), finalized #1136684 (0x4cd4…3304), ⬇ 0.2kiB/s ⬆ 0.2kiB/s
ERROR 2024-06-17T08:25:00.053820149Z [resource.labels.containerName: centrifugechaincollator-0] 2024-06-17 08:25:00.039 INFO tokio-runtime-worker substrate: [Relaychain] ✨ Imported #2314766 (0x5c35…542e)
ERROR 2024-06-17T08:25:00.053857993Z [resource.labels.containerName: centrifugechaincollator-0] 2024-06-17 08:25:00.041 INFO tokio-runtime-worker sc_basic_authorship::basic_authorship: [🌀Parachain] 🙌 Starting consensus session on top of parent 0x4cd4dd8ea216ae861a48b52b0998bc59baf97c877ee8571b4da9e0699ec13304
ERROR 2024-06-17T08:25:00.053863806Z [resource.labels.containerName: centrifugechaincollator-0] 2024-06-17 08:25:00.045 INFO tokio-runtime-worker sc_basic_authorship::basic_authorship: [🌀Parachain] 🎁 Prepared block for proposing at 1136685 (3 ms) [hash: 0x3ebba35e6bf1e67f8aa4f3ac5118935c8a9efaff58762f8acaa032b89b49a90d; parent_hash: 0x4cd4…3304; extrinsics (4): [0x2c3c…d111, 0xdc9d…4ca9, 0xdd6d…9c21, 0xa18a…4622]
ERROR 2024-06-17T08:25:00.053868226Z [resource.labels.containerName: centrifugechaincollator-0] 2024-06-17 08:25:00.048 INFO tokio-runtime-worker aura::cumulus: [🌀Parachain] 🔖 Pre-sealed block for proposal at 1136685. Hash now 0x310aeb0978a942b2c86d81e297812153ffeac009ac622c0b0a24c0c0369aab8f, previously 0x3ebba35e6bf1e67f8aa4f3ac5118935c8a9efaff58762f8acaa032b89b49a90d.
ERROR 2024-06-17T08:25:00.053872946Z [resource.labels.containerName: centrifugechaincollator-0] 2024-06-17 08:25:00.049 INFO tokio-runtime-worker substrate: [🌀Parachain] ✨ Imported #1136685 (0x310a…ab8f)
ERROR 2024-06-17T08:25:00.053878738Z [resource.labels.containerName: centrifugechaincollator-0] 2024-06-17 08:25:00.050 INFO tokio-runtime-worker aura::cumulus: [🌀Parachain] PoV size { header: 0.26171875kb, extrinsics: 4.1123046875kb, storage_proof: 19.853515625kb }
ERROR 2024-06-17T08:25:00.053884038Z [resource.labels.containerName: centrifugechaincollator-0] 2024-06-17 08:25:00.050 INFO tokio-runtime-worker aura::cumulus: [🌀Parachain] Compressed PoV size: 19.3857421875kb
ERROR 2024-06-17T08:25:04.162744505Z [resource.labels.containerName: centrifugechaincollator-0] 2024-06-17 08:25:04.159 INFO tokio-runtime-worker substrate: [Relaychain] 💤 Idle (3 peers), best: #2314766 (0x5c35…542e), finalized #2314763 (0x1fec…1227), ⬇ 3.3kiB/s ⬆ 2.3kiB/s
ERROR 2024-06-17T08:25:04.212882045Z [resource.labels.containerName: centrifugechaincollator-0] 2024-06-17 08:25:04.186 INFO tokio-runtime-worker substrate: [🌀Parachain] 💤 Idle (1 peers), best: #1136685 (0xb7fb…4bc1), finalized #1136684 (0x4cd4…3304), ⬇ 0.5kiB/s ⬆ 0.5kiB/s
ERROR 2024-06-17T08:25:06.066952203Z [resource.labels.containerName: centrifugechaincollator-0] 2024-06-17 08:25:06.024 INFO tokio-runtime-worker substrate: [Relaychain] ✨ Imported #2314767 (0x67ad…fd07)
ERROR 2024-06-17T08:25:06.066999297Z [resource.labels.containerName: centrifugechaincollator-0] 2024-06-17 08:25:06.025 WARN tokio-runtime-worker parachain::collator-protocol: [Relaychain] Collation wasn't advertised to any validator. candidate_hash=0xe14326c354f02da504a2021b6384411113d72cc89ff45b56cf81b96671cfbc57 pov_hash=0x78aa5ae5e2ddd7d95c7824fe2c51d5f9bdf0111391403e19c4c3e84d58270c1b traceID=299424969148471632471728902263712006417
ERROR 2024-06-17T08:25:06.067026081Z [resource.labels.containerName: centrifugechaincollator-0] 2024-06-17 08:25:06.027 INFO tokio-runtime-worker sc_basic_authorship::basic_authorship: [🌀Parachain] 🙌 Starting consensus session on top of parent 0x4cd4dd8ea216ae861a48b52b0998bc59baf97c877ee8571b4da9e0699ec13304
ERROR 2024-06-17T08:25:06.067029746Z [resource.labels.containerName: centrifugechaincollator-0] 2024-06-17 08:25:06.031 INFO tokio-runtime-worker sc_basic_authorship::basic_authorship: [🌀Parachain] 🎁 Prepared block for proposing at 1136685 (4 ms) [hash: 0xdca791ad9dd4d04098be567008e13def624cf4cdbe5fa5e16365443bfd31bab8; parent_hash: 0x4cd4…3304; extrinsics (4): [0xa224…5195, 0xf468…16f1, 0xdd6d…9c21, 0xa18a…4622]
ERROR 2024-06-17T08:25:06.067033655Z [resource.labels.containerName: centrifugechaincollator-0] 2024-06-17 08:25:06.034 INFO tokio-runtime-worker aura::cumulus: [🌀Parachain] 🔖 Pre-sealed block for proposal at 1136685. Hash now 0x120da7ef80b82728316133963e34da3c90e0da4454b87d8be76a8e6836eaefd4, previously 0xdca791ad9dd4d04098be567008e13def624cf4cdbe5fa5e16365443bfd31bab8.
ERROR 2024-06-17T08:25:06.067037095Z [resource.labels.containerName: centrifugechaincollator-0] 2024-06-17 08:25:06.036 INFO tokio-runtime-worker substrate: [🌀Parachain] ✨ Imported #1136685 (0x120d…efd4)
ERROR 2024-06-17T08:25:06.067041018Z [resource.labels.containerName: centrifugechaincollator-0] 2024-06-17 08:25:06.036 INFO tokio-runtime-worker aura::cumulus: [🌀Parachain] PoV size { header: 0.26171875kb, extrinsics: 4.1123046875kb, storage_proof: 19.853515625kb }
ERROR 2024-06-17T08:25:06.067044294Z [resource.labels.containerName: centrifugechaincollator-0] 2024-06-17 08:25:06.036 INFO tokio-runtime-worker aura::cumulus: [🌀Parachain] Compressed PoV size: 19.37890625kb
ERROR 2024-06-17T08:25:08.372380645Z [resource.labels.containerName: centrifugechaincollator-0] 2024-06-17 08:25:08.370 INFO tokio-runtime-worker grandpa: [Relaychain] 👴 Applying authority set change scheduled at block #2314765
ERROR 2024-06-17T08:25:08.372424126Z [resource.labels.containerName: centrifugechaincollator-0] 2024-06-17 08:25:08.371 INFO tokio-runtime-worker grandpa: [Relaychain] 👴 Applying GRANDPA set change to new set [(Public(88dc3417d5058ec4b4503e0c12ea1a0a89be200fe98922423d4334014fa6b0ee (5FA9nQDV...)), 1), (Public(d17c2d7823ebf260fd138f2d7e27d114c0145d968b5ff5006125f2414fadae69 (5GoNkf6W...)), 1)]
ERROR 2024-06-17T08:25:09.174186386Z [resource.labels.containerName: centrifugechaincollator-0] 2024-06-17 08:25:09.160 INFO tokio-runtime-worker substrate: [Relaychain] 💤 Idle (4 peers), best: #2314767 (0x67ad…fd07), finalized #2314765 (0x73bf…e15d), ⬇ 2.1kiB/s ⬆ 5.8kiB/s
ERROR 2024-06-17T08:25:09.224299942Z [resource.labels.containerName: centrifugechaincollator-0] 2024-06-17 08:25:09.187 INFO tokio-runtime-worker substrate: [🌀Parachain] 💤 Idle (1 peers), best: #1136685 (0xb7fb…4bc1), finalized #1136684 (0x4cd4…3304), ⬇ 58 B/s ⬆ 0.1kiB/s
ERROR 2024-06-17T08:25:10.076100871Z [resource.labels.containerName: centrifugechaincollator-0] 2024-06-17 08:25:10.025 WARN tokio-runtime-worker peerset: [Relaychain] Trying to remove unknown reserved node 12D3KooWFX4id8BGKkeGRnjTi4LQG3r9jcbUyaRF5rPYobSoYDnR from SetId(3).
ERROR 2024-06-17T08:25:12.080493628Z [resource.labels.containerName: centrifugechaincollator-0] 2024-06-17 08:25:12.036 INFO tokio-runtime-worker substrate: [Relaychain] ✨ Imported #2314768 (0xea6c…1033)
ERROR 2024-06-17T08:25:12.080545060Z [resource.labels.containerName: centrifugechaincollator-0] 2024-06-17 08:25:12.039 INFO tokio-runtime-worker sc_basic_authorship::basic_authorship: [🌀Parachain] 🙌 Starting consensus session on top of parent 0x4cd4dd8ea216ae861a48b52b0998bc59baf97c877ee8571b4da9e0699ec13304
ERROR 2024-06-17T08:25:12.080550820Z [resource.labels.containerName: centrifugechaincollator-0] 2024-06-17 08:25:12.043 INFO tokio-runtime-worker sc_basic_authorship::basic_authorship: [🌀Parachain] 🎁 Prepared block for proposing at 1136685 (3 ms) [hash: 0xc1e46f89cb98143f252f674f00b36c0d2a4141c0bb786af334bdd82c61ba2330; parent_hash: 0x4cd4…3304; extrinsics (4): [0xfa1c…c00a, 0x21aa…4e28, 0xdd6d…9c21, 0xa18a…4622]
ERROR 2024-06-17T08:25:12.080555040Z [resource.labels.containerName: centrifugechaincollator-0] 2024-06-17 08:25:12.045 INFO tokio-runtime-worker aura::cumulus: [🌀Parachain] 🔖 Pre-sealed block for proposal at 1136685. Hash now 0xd6b374869086225997d5051109bf4512de66df54957b5809e3c943003d66d98a, previously 0xc1e46f89cb98143f252f674f00b36c0d2a4141c0bb786af334bdd82c61ba2330.
ERROR 2024-06-17T08:25:12.080558801Z [resource.labels.containerName: centrifugechaincollator-0] 2024-06-17 08:25:12.047 INFO tokio-runtime-worker substrate: [🌀Parachain] ✨ Imported #1136685 (0xd6b3…d98a)
ERROR 2024-06-17T08:25:12.080582675Z [resource.labels.containerName: centrifugechaincollator-0] 2024-06-17 08:25:12.047 INFO tokio-runtime-worker aura::cumulus: [🌀Parachain] PoV size { header: 0.26171875kb, extrinsics: 4.1123046875kb, storage_proof: 19.853515625kb }
ERROR 2024-06-17T08:25:12.080586960Z [resource.labels.containerName: centrifugechaincollator-0] 2024-06-17 08:25:12.047 INFO tokio-runtime-worker aura::cumulus: [🌀Parachain] Compressed PoV size: 19.3857421875kb
ERROR 2024-06-17T08:25:14.185536151Z [resource.labels.containerName: centrifugechaincollator-0] 2024-06-17 08:25:14.160 INFO tokio-runtime-worker substrate: [Relaychain] 💤 Idle (3 peers), best: #2314768 (0xea6c…1033), finalized #2314765 (0x73bf…e15d), ⬇ 1.4kiB/s ⬆ 0.8kiB/s
ERROR 2024-06-17T08:25:14.235715546Z [resource.labels.containerName: centrifugechaincollator-0] 2024-06-17 08:25:14.187 INFO tokio-runtime-worker substrate: [🌀Parachain] 💤 Idle (0 peers), best: #1136685 (0xb7fb…4bc1), finalized #1136684 (0x4cd4…3304), ⬇ 0.1kiB/s ⬆ 0.1kiB/s
ERROR 2024-06-17T08:25:18.044736431Z [resource.labels.containerName: centrifugechaincollator-0] 2024-06-17 08:25:18.025 INFO tokio-runtime-worker substrate: [Relaychain] ✨ Imported #2314769 (0xaf51…c186)
ERROR 2024-06-17T08:25:18.044789120Z [resource.labels.containerName: centrifugechaincollator-0] 2024-06-17 08:25:18.027 INFO tokio-runtime-worker sc_basic_authorship::basic_authorship: [🌀Parachain] 🙌 Starting consensus session on top of parent 0x4cd4dd8ea216ae861a48b52b0998bc59baf97c877ee8571b4da9e0699ec13304
ERROR 2024-06-17T08:25:18.044798697Z [resource.labels.containerName: centrifugechaincollator-0] 2024-06-17 08:25:18.031 INFO tokio-runtime-worker sc_basic_authorship::basic_authorship: [🌀Parachain] 🎁 Prepared block for proposing at 1136685 (4 ms) [hash: 0xf025f04221544577cd7aefd93aac18359f4d1b1bb0abc09e2d94ac9fdb60963f; parent_hash: 0x4cd4…3304; extrinsics (4): [0x8a77…1b58, 0x3b2c…7bf9, 0xdd6d…9c21, 0xa18a…4622]

Versions

Relay-chain:

  • docker : parity/polkadot:v1.7.2

Parachain:

  • polkadot-sdk : releases-polkadot-v1.7.2

Possibly Related Issues

Possibly related to:

Steps to reproduce

Needs: Docker, rust-toolchain

  • git clone git@github.com:centrifuge/centrifuge-chain.git
  • git checkout fix/release-0.11.0/v1
  • ./scripts/init.sh start-relay-chain
  • ./scripts/init.sh start-parachain purge
  • ./scripts/init.sh onboard-parachain

The above will start 2 relay-chain nodes and one parachain node. The parachain node is afterwards automatically onboarded.

In order to test the diggest issue you must:

  • Submit the following extrinsic on the parachain 0x6f00020000000000000000000000000000000000000000000000000000000000000000
    • through polkadot-js: connect to https://polkadot.js.org/apps/?rpc=ws%3A%2F%2F127.0.0.1%3A11936#/explorer
  • Look at both relay-chain logs, they will show the above Digest Mismatch error

To close down the setup

  • quit the terminal window that is running the parachain
  • run ./scripts/init.sh stop-relay-chain
@mustermeiszer mustermeiszer added I10-unconfirmed Issue might be valid, but it's not yet known. I2-bug The node fails to follow expected behavior. labels Jun 17, 2024
@bkchr
Copy link
Member

bkchr commented Jun 17, 2024

But after submitting a certain transaction the block production stopped.

What kind of transaction? Can you please link the code?

@mustermeiszer
Copy link
Author

We think it is some recursion issue, but I would have thought that this would break the node differently... ^^

@bkchr
Copy link
Member

bkchr commented Jun 17, 2024

We think it is some recursion issue

Can you be more specific?

Looking at your code, you don't deposit any digest item anywhere? Ahh, but you are using pallet-evm that is putting a digest into the header AFAIR?

@mustermeiszer
Copy link
Author

Looking at your code, you don't deposit any digest item anywhere? Ahh, but you are using pallet-evm that is putting a digest into the header AFAIR?

That is correct. We are using that. The weird thing is, that it fails, even when no evm transactions are present...

@mustermeiszer
Copy link
Author

Can you be more specific?

Not really. My current approach is to comment out certain code-paths, restart the chain, submit and see if it hangs.

@bkchr
Copy link
Member

bkchr commented Jun 17, 2024

Hmm, really weird. Looking over the code, I don't really get what is going on. It looks good. Especially weird that this is triggered by a transaction.

If you comment out this line it works?

@mustermeiszer
Copy link
Author

If you comment out [this line](https://github.com/centrifuge/centrifuge- chain/blob/4363f8bc3afd0ed04ff562587ef069de3fb7eb4e/pallets/oracle-feed/src/lib.rs#L128) it works?

Yes. It is also just happening on the relay-chain. The parachain happily produces the block without problems.

@bkchr
Copy link
Member

bkchr commented Jun 18, 2024

The parachain happily produces the block without problems.

The block author is not re-running its own block. Do other parachain nodes are able to import the block?

@mustermeiszer
Copy link
Author

Will they try without the relay-chain approving it? Haven't checked yet. Can try in the dev environments. Any logs to log out for?

I boiled it down to this code-path. Works without it, fails with it. It is rather a portion that is running there.

@bkchr
Copy link
Member

bkchr commented Jun 18, 2024

So if you comment out the withdraw it works?

Will they try without the relay-chain approving it? Haven't checked yet. Can try in the dev environments. Any logs to log out for?

With Aura yes, the other nodes should import it. Just check if the other nodes manage to import the block.

How can I reproduce this? Can you tell me exactly what to launch etc? Then I could also look at it.

@mustermeiszer
Copy link
Author

mustermeiszer commented Jun 18, 2024

So if you comment out the withdraw it works?

No, just the fee.value::<Self>() and replace it with a BalanceOf::<T>::default().

@mustermeiszer
Copy link
Author

How can I reproduce this? Can you tell me exactly what to launch etc? Then I could also look at it.

Give me 5 minutes I will adapt the issue for reproduction. Need a new branch as our launch script is out to date.

@mustermeiszer
Copy link
Author

@bkchr updated. The above will build our node locally. I then simply comment out certain lines and re-do the procedure. We have integration test for that code-path, so it is really weird.

@mustermeiszer
Copy link
Author

No, just the fee.value::() and replace it with a BalanceOf::::default().

I have to correct myself. The withdraw seems to be the real issue. Looking at the implementation it is short-circuiting for 0 values, which is provided by the BalanceOf::<T>::default(). Providing non-zero values, results in the errror.

@mustermeiszer
Copy link
Author

In the non-local testnetworks the behaviour is a bit different.

  • Tx is pending the whole time
    image
  • Blocks are being produced from the Collator
  • Blocks are not valid by the validator
  • Blocks are ANYWAYS imported by the Collator

At least this is my current understanding. Here are the log files. Both as csv and as json. The CSVs side by side can be seen here

dev-centrifuge-collator.csv
dev-centrifuge-collator.json
dev-centrifuge-fullnode.csv
dev-centrifuge-fullnode.json
dev-polkadot-validator-0.csv
dev-polkadot-validator-0.json
dev-polkadot-validator-1.csv
dev-polkadot-validator-1.json

@bkchr
Copy link
Member

bkchr commented Jun 18, 2024

@mustermeiszer just to keep you updated. I could reproduce this locally with your instructions. I already found out that it is related to the frontier digest. Looks like the state root is different. I'm continuing to find out on what is going on.

@mustermeiszer
Copy link
Author

Thanks for the update!! Let me know if you need anything else.

@bkchr
Copy link
Member

bkchr commented Jun 20, 2024

centrifuge/centrifuge-chain#1881 the fix for your issue.

Underlying issue

	pub const fn size_of_feed<T: Config>() -> u32 {
		sp_std::mem::size_of::<(T::OracleKey, T::OracleValue, MomentOf<T>)>() as u32
	}

The function is using mem::size_of to calculate the size of some types. The problem is that the size is different in native versus wasm:

  • native: 96bytes
  • wasm: 88bytes

The problem with Centrifuge was now that the block production was running in native while the validation is running in wasm. The validation was finally failing because the storage root calculated by Frontier was different than the one calculated at block production. As Frontier is putting the ETH block hash into a digest, the validation failed at comparing the digests.

CC @mustermeiszer

@bkchr bkchr closed this as completed Jun 20, 2024
@mustermeiszer
Copy link
Author

The validation was finally failing because the storage root calculated by Frontier was different than the one calculated at block production. As Frontier is putting the ETH block hash into a digest, the validation failed at comparing the digests.

Does that mean, that without Frontier the block production would be able to work just fine? How would that be possible - shouldn't the storage root then still be different between PoV and what the relay computes?

@bkchr
Copy link
Member

bkchr commented Jun 20, 2024

Does that mean, that without Frontier the block production would be able to work just fine?

No. Then it would have failed at comparing the storage_root. However, as the digests are compared first, it failed there first.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
I2-bug The node fails to follow expected behavior. I10-unconfirmed Issue might be valid, but it's not yet known.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants