Core dump on long running, fully synced node

[tjayrush]

Before filing a new issue, please provide the following information.

I'm running:

• Which Parity version?: v1.11.3
• Which operating system?: Linux - Ubuntu
• How installed?: via installer
• Are you fully synchronized?: yes
• Which network are you connected to?: ethereum mainnet
• Did you try to restart the node?: yes

(Got message: 'DB has been previously marked as corrpted, attempting repair."

Your issue description goes here below. Try to include actual vs. expected behavior and steps to reproduce the issue.

Long running node (many months, fully synced). I write software that very heavily hits against the node's RPC (many requests per second). The system has worked well up until now.

--------------------------------------------------- This report ----------------
2018-06-20 00:53:46  Syncing #5820471 0x6da4…b2de     0 blk/s   94 tx/s   4 Mgas/s      0+  177 Qed  #5820647    0/25 peers   0 bytes chain 18 MiB db 21 MiB queue 163 KiB sync  RPC:  0 conn, 2632 req/s,  56 µs
2018-06-20 00:53:46  Syncing #5820471 0x6da4…b2de     0 blk/s    0 tx/s   0 Mgas/s      0+  177 Qed  #5820647    0/25 peers   0 bytes chain 18 MiB db 21 MiB queue 163 KiB sync  RPC:  0 conn, 2632 req/s,  56 µs
2018-06-20 00:53:46  Syncing #5820471 0x6da4…b2de     0 blk/s    0 tx/s   0 Mgas/s      0+  177 Qed  #5820647    0/25 peers   2 KiB chain 18 MiB db 21 MiB queue 163 KiB sync  RPC:  0 conn, 2632 req/s,  56 µs
2018-06-20 00:54:39  DB corrupted: Corruption: block checksum mismatch: expected 693719468, got 3602628252  in /media/jrush/6cb7072d-c89f-4377-ab4c-927e15bc721e/ethereum/db/906a34e69aec8c0d/archive/db/5834514.sst offset 6323112 size 16282. Repair will be triggered on next restart

====================

stack backtrace:
   0:     0x55f90f4781fc - <no info>

Thread 'IO Worker #3' panicked at 'DB flush failed.: Error(Msg("Corruption: block checksum mismatch: expected 693719468, got 3602628252  in /media/jrush/6cb7072d-c89f-4377-ab4c-927e15bc721e/ethereum/db/906a34e69aec8c0d/archive/db/5834514.sst offset 6323112 size 16282"), State { next_error: None, backtrace: None })', libcore/result.rs:945

This is a bug. Please report it at:

    https://github.com/paritytech/parity/issues/new

Aborted (core dumped)

[Tbaut]

This looks like #7748 and many others. The cause of this error could not be found. A full resync could each time resolve the issue.
Checking your RAM and SSD for error could also help.

[tjayrush]

DB corrupted: Invalid argument: You have to open all column families. Column families not opened: col3, col0, attempting repair

Also -- on the issues you referenced it says run memtest and system health. Can you explain to me how to do that?

I rebooted the machine.

[Tbaut]

it says run memtest and system health. Can you explain to me how to do that?

• Memtest: https://askubuntu.com/questions/591488/how-do-i-run-memtest86
• Disk test: https://askubuntu.com/questions/59064/how-to-run-a-checkdisk