[ripple node is stopped many times] (1.9.4+ba3c0e51455a88d76d90b996f20c0f102ac3f5a0.DEBUG)

[past2017]

2022-Nov-11 08:16:22.234032320 UTC LoadMonitor:WRN Job: processLedgerData run: 1986ms wait: 0ms
2022-Nov-11 08:16:23.513537418 UTC LoadMonitor:WRN Job: processLedgerData run: 1153ms wait: 0ms
2022-Nov-11 08:16:41.381873448 UTC LoadMonitor:WRN Job: gotFetchPack run: 2492ms wait: 0ms
2022-Nov-11 08:16:41.575358272 UTC LoadMonitor:WRN Job: processLedgerData run: 1150ms wait: 0ms
2022-Nov-11 08:16:52.974906477 UTC LoadMonitor:WRN Job: processLedgerData run: 1008ms wait: 0ms
2022-Nov-11 08:16:53.051118374 UTC LoadMonitor:WRN Job: processLedgerData run: 1289ms wait: 0ms
2022-Nov-11 08:16:53.716542567 UTC LoadMonitor:WRN Job: gotFetchPack run: 741ms wait: 523ms
2022-Nov-11 08:16:56.235694966 UTC LoadMonitor:WRN Job: processLedgerData run: 1139ms wait: 0ms
2022-Nov-11 08:16:56.414271764 UTC LoadMonitor:WRN Job: processLedgerData run: 1173ms wait: 0ms
2022-Nov-11 08:17:00.844353959 UTC LoadMonitor:WRN Job: processLedgerData run: 646ms wait: 473ms
2022-Nov-11 08:17:00.918079269 UTC LoadMonitor:WRN Job: gotFetchPack run: 1352ms wait: 154ms
2022-Nov-11 08:17:11.624420645 UTC LoadMonitor:WRN Job: gotFetchPack run: 2469ms wait: 0ms
2022-Nov-11 08:17:15.716968443 UTC LoadMonitor:WRN Job: processLedgerData run: 1091ms wait: 0ms
2022-Nov-11 08:17:15.893754960 UTC LoadMonitor:WRN Job: processLedgerData run: 1387ms wait: 0ms
2022-Nov-11 08:17:15.894065428 UTC LoadMonitor:WRN Job: processLedgerData run: 0ms wait: 1240ms
2022-Nov-11 08:17:15.925435462 UTC LoadMonitor:WRN Job: gotFetchPack run: 1077ms wait: 222ms
2022-Nov-11 08:17:16.026089813 UTC LoadMonitor:WRN Job: processLedgerData run: 309ms wait: 1080ms
2022-Nov-11 08:17:16.594215302 UTC LoadMonitor:WRN Job: processLedgerData run: 700ms wait: 1210ms
2022-Nov-11 08:17:24.669114169 UTC LoadMonitor:WRN Job: gotFetchPack run: 1165ms wait: 2ms
2022-Nov-11 08:17:25.288168050 UTC LoadMonitor:WRN Job: processLedgerData run: 1082ms wait: 0ms
xrp3: /home/rippled/src/ripple/app/ledger/Ledger.cpp:436: virtual std::shared_ptr ripple::Ledger::read(const ripple::Keylet&) const: Assertion `false' failed.

I have 2 ripple node, all of them stop with above log

[ximinez]

Since you're running a debug build (asserts are only checked in debug builds), would it be possible to run your nodes inside of gdb? That will allow you to relatively easily capture a stack trace and other information if this happens again. I usually do the same thing where I run rippled. Here's how I do it.

gdb -q -ex set pagination off -ex run -ex generate-core-file -ex set logging enabled on -ex thread apply all bt full -ex set logging enabled off -ex quit --args [path to rippled]

That should generate a core file and a gdb.txt file containing any stack trace from the crash. If you get one, please post it here, or post a link to a gist.

[sokiaoba]

We're facing a similar issue. We're running 3 mainnet nodes and occasionally all of them crash simultaneously. Until recent few weeks, this problem hasn't occurred. (we don't save logs for rebooted nodes)