No space left on deviceWhile appending to file

[TomaszWaszczyk]

I got the stacktrace after running:

./polkadot --pruning=archive --name "thirdwave-network-sentry"

Version: 0.7.32-fa1536c9-x86_64-linux-gnu

   0: sp_panic_handler::set::{{closure}}
   1: std::panicking::rust_panic_with_hook
             at src/libstd/panicking.rs:474
   2: rust_begin_unwind
             at src/libstd/panicking.rs:378
   3: std::panicking::begin_panic_fmt
             at src/libstd/panicking.rs:332
   4: <sp_database::kvdb::DbAdapter<D> as sp_database::Database<H>>::commit
   5: sc_client_db::Backend<Block>::try_commit_operation
   6: <sc_client_db::Backend<Block> as sc_client_api::backend::Backend<Block>>::commit_operation
   7: <&sc_service::client::client::Client<B,E,Block,RA> as sp_consensus::block_import::BlockImport<Block>>::import_block
   8: <sc_finality_grandpa::import::GrandpaBlockImport<BE,Block,Client,SC> as sp_consensus::block_import::BlockImport<Block>>::import_block
   9: <sc_consensus_babe::BabeBlockImport<Block,Client,Inner> as sp_consensus::block_import::BlockImport<Block>>::import_block
  10: sp_consensus::import_queue::import_single_block
  11: futures_util::future::future::chain::Chain<Fut1,Fut2,Data>::poll
  12: <futures_util::future::poll_fn::PollFn<F> as core::future::future::Future>::poll
  13: <futures_util::future::select::Select<A,B> as core::future::future::Future>::poll
  14: <std::future::GenFuture<T> as core::future::future::Future>::poll
  15: std::thread::local::LocalKey<T>::with
  16: futures_executor::local_pool::block_on
  17: tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut
  18: tokio::runtime::task::core::Core<T,S>::poll
  19: <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
  20: std::panicking::try::do_call
  21: __rust_maybe_catch_panic
             at src/libpanic_unwind/lib.rs:86
  22: tokio::runtime::task::raw::poll
  23: tokio::runtime::blocking::pool::Inner::run
  24: tokio::runtime::context::enter
  25: std::sys_common::backtrace::__rust_begin_short_backtrace
  26: std::panicking::try::do_call
  27: __rust_maybe_catch_panic
             at src/libpanic_unwind/lib.rs:86
  28: core::ops::function::FnOnce::call_once{{vtable.shim}}
  29: <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once
             at /rustc/4fb7144ed159f94491249e86d5bbd033b5d60550/src/liballoc/boxed.rs:1017
  30: <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once
             at /rustc/4fb7144ed159f94491249e86d5bbd033b5d60550/src/liballoc/boxed.rs:1017
      std::sys_common::thread::start_thread
             at src/libstd/sys_common/thread.rs:13
      std::sys::unix::thread::Thread::new::thread_start
             at src/libstd/sys/unix/thread.rs:80
  31: start_thread
  32: clone


Thread 'tokio-runtime-worker' panicked at 'Critical database eror: Custom { kind: Other, error: Error { message: "IO error: No space left on deviceWhile appending to file: /home/pantarhei/.local/share/polkadot/chains/ksmcc3/db/017028.log: No space left on device" } }', <::std::macros::panic macros>:5

This is a bug. Please report it at:

        https://github.com/paritytech/polkadot/issues/new

More info:

pantarhei@polkadot-sentry-node:~$ uname -r
5.3.0-40-generic

and

pantarhei@polkadot-sentry-node:~$ cat /proc/meminfo
MemTotal:        1004608 kB
MemFree:          628920 kB
MemAvailable:     734724 kB
Buffers:            5904 kB
Cached:           214088 kB
SwapCached:            0 kB
Active:            84168 kB
Inactive:         170388 kB
Active(anon):      43036 kB
Inactive(anon):     1768 kB
Active(file):      41132 kB
Inactive(file):   168620 kB
Unevictable:       18412 kB
Mlocked:           18412 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:               200 kB
Writeback:             0 kB
AnonPages:         53032 kB
Mapped:            29240 kB
Shmem:              2128 kB
KReclaimable:      46924 kB
Slab:              75328 kB
SReclaimable:      46924 kB
SUnreclaim:        28404 kB
KernelStack:        1596 kB
PageTables:         1348 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:      502304 kB
Committed_AS:     205144 kB
VmallocTotal:   34359738367 kB
VmallocUsed:        8672 kB
VmallocChunk:          0 kB
Percpu:              580 kB
HardwareCorrupted:     0 kB
AnonHugePages:         0 kB
ShmemHugePages:        0 kB
ShmemPmdMapped:        0 kB
CmaTotal:              0 kB
CmaFree:               0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:               0 kB
DirectMap4k:       94064 kB
DirectMap2M:      954368 kB

The machine has 25GB storage and 1GB of RAM memory. Ubuntu 19.
PS. I will try to run having 2GB of RAM and will back.

[NikVolf]

You have no disk space left, this is explicitly stated: IO error: No space left on deviceWhile appending to file

[TomaszWaszczyk]

I understand that, but I have space on storage. Will check it having 2GB of RAM

[tomploem]

@TomaszWaszczyk Did you see a change with upgrading the node specs? I'm running a 8GB and saw the issue today.

[TomaszWaszczyk]

@endro-00 Yes, the issue occurs, does not matter how many GB of RAM I had, for me helped addition of 2 cores of CPUs.