Parity DB will not sync on external SSD

[Yonatan19981]

Before filing a new issue, please provide the following information.

If you think that your issue is an exploitable security vulnerability, please mail your bugreport to security@parity.io instead; your submission might be eligible for our Bug Bounty.
You can find mode info on the reporting process in SECURITY.md

• OpenEthereum version: 3.0.0
• Operating system: MacOS
• Installation: built from source
• Fully synchronized: no
• Network: _ethereum
• Restarted: no / yes
  -Chain: foundation

Your issue description goes here below. Try to include actual vs. expected behavior and steps to reproduce the issue.

After trying to sync parity db to an external thunderbolt 3 SSD it got stuck with this error. I restarted my mac, deleted the DB and reinstalled it but it is still reoccurring. The expected behaviour is syncing but the results are as follows. The reason I use an external SSD is because I only have 256 GB in my mac , the DB is bigger and I want to use tracing so I cannot use light mode.

I looked online but could not find a solution to this problem. I saw similar problems occurred in 2016 but the solution was it was fixed with updates. For me this is still happening.

Maybe if someone can try installing the db on an external SSD it would reproduce.

2020-06-01 23:28:38 Starting OpenEthereum/v3.0.0-stable-b3d54f026-20200527/x86_64-apple-darwin/rustc1.43.1
2020-06-01 23:28:38 Keys path /Volumes/VTK512GB/pairtyDB/keys/ethereum
2020-06-01 23:28:38 DB path /Volumes/VTK512GB/pairtyDB/chains/ethereum/db/906a34e69aec8c0d
2020-06-01 23:28:38 State DB configuration: fast +Trace
2020-06-01 23:28:38 Operating mode: active
2020-06-01 23:28:39 Configured for Ethereum using Ethash engine
2020-06-01 23:28:39 Updater is deprecated and may be removed in a future release. Please see #11696 for details:
https://github.com/openethereum/openethereum/issues/11696
2020-06-01 23:28:39 Public node URL: enode://21115f73474bbfcc4fd2914a857b8bf98f2b24bd53ab25a811588e0d547040c1f0e4580039d281da461171b5391963f56e4ae7bdb73bd27de17d9ed46daf8a6a@10.0.0.12:30303
2020-06-01 23:28:40 Updated conversion rate to Ξ1 = US$238.95 (19928456 wei/gas)
2020-06-01 23:28:41 Block import failed for #715476 (0xa156…e601)
Error: Execution(Internal("Internal error: Database missing expected key: 0x003da4c959c4cc985dda64a0b93469065ada673e94ec8d0732a27c0f95f799f7"))
2020-06-01 23:28:41
Bad block detected: Execution error: Transaction execution error (Internal error: Database missing expected key: 0x003da4c959c4cc985dda64a0b93469065ada673e94ec8d0732a27c0f95f799f7).
RLP: f90324f90218a0e9c11a9b69565a68cf5b8ab9267c2bdba2608c91c71b473a423bd5ab552050c4a01dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d4934794d8ddb0070aee305fc05be67ecab7e2979e940404a06ecdb3ec8696230ecc3bb9ee7c22c915f6b269efb9c654f1a901c4e99e1168c9a041fbb3755330741e23d0bc4951f968c22c331b9065157642edd3c3b4bf0ca752a0e1b31ee4fa87d1aabe1332b03899dc5977ec7bdb50f00d9c44b9558c5dc8b627b90100000000000000000000000000000000000000000000000000000000000000400000000010000000000000000000000000000080000000000000000000000000000000000000000000000000000000008020000020000000000000000000000000010000000000000000000000000000000000000000000000000000000000000000000000000000200000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000020000000000000000000000000000000080000000000000000000000000000000000000001000000000000000000000000000000000080000000000000008607dd8796a6cf830aead4832fefd883047aba845675437e98d783010400844765746887676f312e352e31856c696e7578a030a66dbf63a5b215d4f711f0a330bcaf0f2d6266079148dfdd48d483cd6acc2988c22dd312874415aaf90105f87083010295850ba43b740083015f9094ff501b27c581c367b563767b50d32b7ffebdbdda8841b0c3a494822000801ba0a0e55de425d1de28ec3ac1ec8dfecd6ca26c8869045ba312ad29f42cac3e3c42a0040abec6f726018419ed05652404f4d459d7f276bd28b9ff4ff83d4032b6bca3f89180850ba43b7400830428b294c7696b27830dd8aa4823a1cba8440c27c36adec488095b167ede26a6d4a4cc445611444f574e4c4f41440000000000000000000000000000000000000000000000001ba0cc5e32d7eeef9b1b99c719036a761ed8bdc8fe113c2c38bffa5f3cb5fefefd40a0bd7f328a1a2573586de0c717a9dfbe1709d840202b465071f8250b795112627dc0
Header: Header { parent_hash: 0xe9c11a9b69565a68cf5b8ab9267c2bdba2608c91c71b473a423bd5ab552050c4, timestamp: 1450525566, number: 715476, author: 0xd8ddb0070aee305fc05be67ecab7e2979e940404, transactions_root: 0x41fbb3755330741e23d0bc4951f968c22c331b9065157642edd3c3b4bf0ca752, uncles_hash: 0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347, extra_data: [215, 131, 1, 4, 0, 132, 71, 101, 116, 104, 135, 103, 111, 49, 46, 53, 46, 49, 133, 108, 105, 110, 117, 120], state_root: 0x6ecdb3ec8696230ecc3bb9ee7c22c915f6b269efb9c654f1a901c4e99e1168c9, receipts_root: 0xe1b31ee4fa87d1aabe1332b03899dc5977ec7bdb50f00d9c44b9558c5dc8b627, log_bloom: 0x00000000000000000000000000000000000000000000000000000000000040000000001000000000000000000000000000008000000000000000000000000000000000000000000000000000000000802000002000000000000000000000000001000000000000000000000000000000000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000002000000000000000000000000000000008000000000000000000000000000000000000000100000000000000000000000000000000008000000000000000, gas_used: 293562, gas_limit: 3141592, difficulty: 8648043964111, seal: [[160, 48, 166, 109, 191, 99, 165, 178, 21, 212, 247, 17, 240, 163, 48, 188, 175, 15, 45, 98, 102, 7, 145, 72, 223, 221, 72, 212, 131, 205, 106, 204, 41], [136, 194, 45, 211, 18, 135, 68, 21, 170]], hash: Some(0xa15639f25e05386f48fbef69448e46f2d37d57285e7701db3c7f2b5460e2e601) }
Uncles:
Transactions:[Tx 0] UnverifiedTransaction { unsigned: Transaction { nonce: 66197, gas_price: 50000000000, gas: 90000, action: Call(0xff501b27c581c367b563767b50d32b7ffebdbdda), value: 4733498320000000000, data: [] }, v: 27, r: 72775311764417853185963420223551836199930346942737545205471688397466720156738, s: 1828236560883950471314771790354185897897851158160360667159554156623283272867, hash: 0xcd007c2dfbc1f4597cb611c2628161d951104d36b4b8d3f6204699a105537acd }
[Tx 1] UnverifiedTransaction { unsigned: Transaction { nonce: 0, gas_price: 50000000000, gas: 272562, action: Call(0xc7696b27830dd8aa4823a1cba8440c27c36adec4), value: 674157303370786516, data: [204, 68, 86, 17, 68, 79, 87, 78, 76, 79, 65, 68, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] }, v: 27, r: 92438255643944277883781664585057694541088553031863668790116271200415015435584, s: 85711866770007707660884960618492304231938316778969927919115172588211112403581, hash: 0xe4cb06a92b40fe28a8403c85b36b8ae4ff17d6ab49eb0d3e61991304e52dc7ac }

*Edit: after trying one more time, I got the same error,

2020-06-01 23:50:25 Verifier #1 WARN client Block import failed for #348901 (0xccde…5c6c)
Error: Execution(Internal("Internal error: Database missing expected key: 0x00748b086271312fd6ac1e0662c60ae7404b6e521afb9b7ad6da4cbb12127f88"))
2020-06-01 23:50:25 Verifier #1 ERROR client

The command line after that looks like this:

0 µs

2020-06-01 23:51:25 IO Worker #2 INFO import Syncing #348900 0x2eec…a5d4 0.00 blk/s 0.0 tx/s 0.0 Mgas/s 0+ 0 Qed #372493 2/25 peers 126 MiB chain 129 MiB db 0 bytes queue 64 MiB sync RPC: 0 conn, 0 req/s, 0 µs
2020-06-01 23:51:30 IO Worker #0 INFO import Syncing #348900 0x2eec…a5d4 0.00 blk/s 0.0 tx/s 0.0 Mgas/s 0+ 0 Qed #372493 5/25 peers 126 MiB chain 129 MiB db 0 bytes queue 67 MiB sync RPC: 0 conn, 0 req/s, 0 µs
2020-06-01 23:51:35 IO Worker #0 INFO import Syncing #348900 0x2eec…a5d4 0.00 blk/s 0.0 tx/s 0.0 Mgas/s 0+ 0 Qed #372493 2/25 peers 126 MiB chain 129 MiB db 0 bytes queue 70 MiB sync RPC: 0 conn, 0 req/s, 0 µs
2020-06-01 23:51:40 IO Worker #2 INFO import Syncing #348900 0x2eec…a5d4 0.00 blk/s 0.0 tx/s 0.0 Mgas/s 0+ 0 Qed #372492 1/25 peers 126 MiB chain 129 MiB db 0 bytes queue 52 MiB sync RPC: 0 conn, 0 req/s, 0 µs
2020-06-01 23:51:45 IO Worker #1 INFO import Syncing #348900 0x2eec…a5d4 0.00 blk/s 0.0 tx/s 0.0 Mgas/s 0+ 0 Qed #372491 2/25 peers 126 MiB chain 129 MiB db 0 bytes queue 52 MiB sync RPC: 0 conn, 0 req/s, 0 µs
2020-06-01 23:51:50 IO Worker #2 INFO import Syncing #348900 0x2eec…a5d4 0.00 blk/s 0.0 tx/s 0.0 Mgas/s 0+ 0 Qed #372491 2/25 peers 126 MiB chain 129 MiB db 0 bytes queue 52 MiB sync RPC: 0 conn, 0 req/s, 0 µs
2020-06-01 23:51:55 IO Worker #2 INFO import Syncing #348900 0x2eec…a5d4 0.00 blk/s 0.0 tx/s 0.0 Mgas/s 0+ 0 Qed #372491 3/25 peers 126 MiB chain 129 MiB db 0 bytes queue 52 MiB sync RPC: 0 conn, 0 req/s, 0 µs
2020-06-01 23:52:00 IO Worker #3 INFO import Syncing #348900 0x2eec…a5d4 0.00 blk/s 0.0 tx/s 0.0 Mgas/s 0+ 0 Qed #372491 2/25 peers 126 MiB chain 129 MiB db 0 bytes queue 52 MiB sync RPC: 0 conn, 0 req/s, 0 µs
2020-06-01 23:52:05 IO Worker #3 INFO import Syncing #348900 0x2eec…a5d4 0.00 blk/s 0.0 tx/s 0.0 Mgas/s 0+ 0 Qed #372491 2/25 peers 126 MiB chain 129 MiB db 0 bytes queue 52 MiB sync RPC: 0 conn, 0 req/s, 0 µs
2020-06-01 23:52:10 IO Worker #1 INFO import Syncing #348900 0x2eec…a5d4 0.00 blk/s 0.0 tx/s 0.0 Mgas/s 0+ 0 Qed #372491 3/25 peers 126 MiB chain 129 MiB db 0 bytes queue 53 MiB sync RPC: 0 conn, 0 req/s, 0 µs

I tried leaving it on all night last time it happened but it is still stuck on 0 blk/s

Thanks

[adria0]

@YONATAN10111, this seems a database corruption, that could happen when IO is abruptly detached or when killing the OE process. Openethereum code does not contain any special logic for external volumes.

Could you try with 2.7.2, please? Parity is super-intensive with disk IO, if the problem persists maybe makes sense to run a low-level check to the SSD.

[Yonatan19981]

@adria0 First of all thank you for replying. I am now trying to install only the db in the external volume and the keys and other data on the internal SSD to see if it works. If it will not work I will try what you offered and will let you know the results.

[Yonatan19981]

@adria0 do you have any data on the needed read/write speed for the sync process? It can be a good indicator whether you external SSD can sync or not if this is really the problem. If the problem repeats itself, I will do a speed test for both my internal and my external SSD the show the results for other people to compare and see if they can sync it

[Yonatan19981]

Ok after trying to sync, a little after block 1,000,000 I get this message:

0: backtrace::backtrace::trace
1: backtrace::capture::Backtrace::new
2: panic_hook::gen_panic_msg
3: panic_hook::set_with::{{closure}}
4: std::panicking::rust_panic_with_hook
5: rust_begin_unwind
6: core::panicking::panic_fmt
7: core::option::expect_failed
8: ethcore_blockchain::blockchain::BlockChain::new
9: ethcore::client::client::Client::new
10: ethcore_service::service::ClientService::start
11: openethereum::run::execute_impl
12: openethereum::main_direct
13: openethereum::main
14: std::rt::lang_start::{{closure}}
15: main

Thread 'main' panicked at 'Genesis hash should always exist', ethcore/blockchain/src/blockchain.rs:179

[dvdplm]

FWIW I use a mac and a Samsung X5 external tb3 ssd and it works very well.

[Yonatan19981]

Tried with 2.6.8 still same problem. Will try with 2.7.2

[Yonatan19981]

Also checked external SSD health and read/write speed. Health is good and read speed is above 2GB/s and write speed is over 600Mb/s