Kusama node crashed with error

[Ryabina-io]

Problem: Kusama node fell down with an error message and a request to create a new issue.
Node version: polkadot 0.8.22-c6ee8675-x86_64-linux-gnu
Run with these parameters:

polkadot \
    --no-mdns \
    --chain kusama \
    --pruning 1000 \
    --unsafe-pruning \
    --validator \
    --no-private-ipv4 \
    --wasm-execution Compiled \
    --rpc-methods=Unsafe

Not possible to restart with the same parameters. Helped delete flag --wasm-execution Compiled

Logs:

Aug 14 08:19:31: 2020-08-14 08:19:31 ✍️ Starting parachain attestation session (pa
Aug 14 08:19:31: 2020-08-14 08:19:31 💤 Idle (3 peers), best: #3598672 (0x14bb…84
Aug 14 08:19:36: ====================
Aug 14 08:19:36: Version: 0.8.22-c6ee8675-x86_64-linux-gnu
Aug 14 08:19:36:    0: sp_panic_handler::set::{{closure}}
Aug 14 08:19:36:    1: std::panicking::rust_panic_with_hook
Aug 14 08:19:36:              at rustc/71f8d0c8f1060bbe74100f29cc6f2da63d697c28/l
Aug 14 08:19:36:    2: rust_begin_unwind
Aug 14 08:19:36:              at rustc/71f8d0c8f1060bbe74100f29cc6f2da63d697c28/l
Aug 14 08:19:36:    3: core::panicking::panic_fmt
Aug 14 08:19:36:              at rustc/71f8d0c8f1060bbe74100f29cc6f2da63d697c28/l
Aug 14 08:19:36:    4: core::option::expect_none_failed
Aug 14 08:19:36:              at rustc/71f8d0c8f1060bbe74100f29cc6f2da63d697c28/l
Aug 14 08:19:36:    5: std::thread::spawn
Aug 14 08:19:36:    6: smol::blocking::BlockingExecutor::grow_pool
Aug 14 08:19:36:    7: smol::blocking::BlockingExecutor::schedule
Aug 14 08:19:36:    8: async_task::raw::RawTask<F,R,S,T>::schedule
Aug 14 08:19:36:    9: <core::future::from_generator::GenFuture<T> as core::futur
Aug 14 08:19:36:   10: <futures_util::future::future::map::Map<Fut,F> as core::fu
Aug 14 08:19:36:   11: <futures_util::future::either::Either<A,B> as core::future
Aug 14 08:19:36:   12: <libp2p_core::either::EitherFuture<AFuture,BFuture> as cor
Aug 14 08:19:36:   13: <libp2p_core::transport::and_then::AndThenFuture<TFut,TMap
Aug 14 08:19:36:   14: <libp2p_core::transport::map_err::MapErrDial<T,F> as core:
Aug 14 08:19:36:   15: <libp2p_core::transport::map::MapFuture<T,F> as core::futu
Aug 14 08:19:36:   16: <libp2p_core::transport::map_err::MapErrDial<T,F> as core:
Aug 14 08:19:36:   17: <futures_util::future::future::map::Map<Fut,F> as core::fu
Aug 14 08:19:36:   18: <futures_util::future::future::map::Map<Fut,F> as core::fu
Aug 14 08:19:36:   19: <futures_util::future::try_future::try_flatten::TryFlatten
Aug 14 08:19:36:   20: <libp2p_core::connection::manager::task::Task<F,M,H,I,O,E,
Aug 14 08:19:36:   21: <sc_service::task_manager::prometheus_future::PrometheusFu
Aug 14 08:19:36:   22: <futures_util::future::select::Select<A,B> as core::future
Aug 14 08:19:36:   23: <core::future::from_generator::GenFuture<T> as core::futur
Aug 14 08:19:36:   24: tokio::runtime::task::core::Core<T,S>::poll
Aug 14 08:19:36:   25: <std::panic::AssertUnwindSafe<F> as core::ops::function::F
Aug 14 08:19:36:   26: tokio::runtime::task::harness::Harness<T,S>::poll
Aug 14 08:19:36:   27: std::thread::local::LocalKey<T>::with
Aug 14 08:19:36:   28: tokio::runtime::thread_pool::worker::Context::run_task
Aug 14 08:19:36:   29: tokio::runtime::thread_pool::worker::Context::run
Aug 14 08:19:36:   30: tokio::macros::scoped_tls::ScopedKey<T>::set
Aug 14 08:19:36:   31: tokio::runtime::thread_pool::worker::run
Aug 14 08:19:36:   32: tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut
Aug 14 08:19:36:   33: tokio::runtime::task::core::Core<T,S>::poll
Aug 14 08:19:36:   34: <std::panic::AssertUnwindSafe<F> as core::ops::function::F
Aug 14 08:19:36:   35: tokio::runtime::task::harness::Harness<T,S>::poll
Aug 14 08:19:36:   36: tokio::runtime::blocking::pool::Inner::run
Aug 14 08:19:36:   37: tokio::runtime::context::enter
Aug 14 08:19:36:   38: std::sys_common::backtrace::__rust_begin_short_backtrace
Aug 14 08:19:36:   39: core::ops::function::FnOnce::call_once{{vtable.shim}}
Aug 14 08:19:36:   40: <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::
Aug 14 08:19:36:              at rustc/71f8d0c8f1060bbe74100f29cc6f2da63d697c28/l
Aug 14 08:19:36:       <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::
Aug 14 08:19:36:              at rustc/71f8d0c8f1060bbe74100f29cc6f2da63d697c28/l
Aug 14 08:19:36:       std::sys::unix::thread::Thread::new::thread_start
Aug 14 08:19:36:              at rustc/71f8d0c8f1060bbe74100f29cc6f2da63d697c28/l
Aug 14 08:19:36:   41: start_thread
Aug 14 08:19:36:   42: __clone
Aug 14 08:19:36: Thread 'tokio-runtime-worker' panicked at 'failed to spawn threa
Aug 14 08:19:36: This is a bug. Please report it at:
Aug 14 08:19:36:         https://github.com/paritytech/polkadot/issues/new

[bkchr]

Hey, could you give the full log? There are characters missing on the right side!

[Ryabina-io]

2020-08-14 08:19:31 ✨ Imported #3598672 (0x14bb…849e)
2020-08-14 08:19:31 ✍️ Starting parachain attestation session (parent: 0x14bb…849e) with active duty LocalDuty { validation: Relay, index: 259 }
2020-08-14 08:19:31 💤 Idle (3 peers), best: #3598672 (0x14bb…849e), finalized #3598668 (0x71b1…0c12), ⬇ 107.0kiB/s ⬆ 105.9kiB/s
====================
Version: 0.8.22-c6ee8675-x86_64-linux-gnu
   0: sp_panic_handler::set::{{closure}}
   1: std::panicking::rust_panic_with_hook
             at rustc/71f8d0c8f1060bbe74100f29cc6f2da63d697c28/library/std/src/panicking.rs:530
   2: rust_begin_unwind
             at rustc/71f8d0c8f1060bbe74100f29cc6f2da63d697c28/library/std/src/panicking.rs:437
   3: core::panicking::panic_fmt
             at rustc/71f8d0c8f1060bbe74100f29cc6f2da63d697c28/library/core/src/panicking.rs:85
   4: core::option::expect_none_failed
             at rustc/71f8d0c8f1060bbe74100f29cc6f2da63d697c28/library/core/src/option.rs:1274
   5: std::thread::spawn
   6: smol::blocking::BlockingExecutor::grow_pool
   7: smol::blocking::BlockingExecutor::schedule
   8: async_task::raw::RawTask<F,R,S,T>::schedule
   9: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
  10: <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll
  11: <futures_util::future::either::Either<A,B> as core::future::future::Future>::poll
  12: <libp2p_core::either::EitherFuture<AFuture,BFuture> as core::future::future::Future>::poll
  13: <libp2p_core::transport::and_then::AndThenFuture<TFut,TMap,TMapOut> as core::future::future::Future>::poll
  14: <libp2p_core::transport::map_err::MapErrDial<T,F> as core::future::future::Future>::poll
  15: <libp2p_core::transport::map::MapFuture<T,F> as core::future::future::Future>::poll
  16: <libp2p_core::transport::map_err::MapErrDial<T,F> as core::future::future::Future>::poll
  17: <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll
  18: <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll
  19: <futures_util::future::try_future::try_flatten::TryFlatten<Fut,<Fut as futures_core::future::TryFuture>::Ok> as core::future::future::Future>::poll
  20: <libp2p_core::connection::manager::task::Task<F,M,H,I,O,E,C> as core::future::future::Future>::poll
  21: <sc_service::task_manager::prometheus_future::PrometheusFuture<T> as core::future::future::Future>::poll
  22: <futures_util::future::select::Select<A,B> as core::future::future::Future>::poll
  23: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
  24: tokio::runtime::task::core::Core<T,S>::poll
  25: <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
  26: tokio::runtime::task::harness::Harness<T,S>::poll
 27: std::thread::local::LocalKey<T>::with
  28: tokio::runtime::thread_pool::worker::Context::run_task
  29: tokio::runtime::thread_pool::worker::Context::run
  30: tokio::macros::scoped_tls::ScopedKey<T>::set
  31: tokio::runtime::thread_pool::worker::run
  32: tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut
  33: tokio::runtime::task::core::Core<T,S>::poll
  34: <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
  35: tokio::runtime::task::harness::Harness<T,S>::poll
  36: tokio::runtime::blocking::pool::Inner::run
  37: tokio::runtime::context::enter
  38: std::sys_common::backtrace::__rust_begin_short_backtrace
  39: core::ops::function::FnOnce::call_once{{vtable.shim}}
  40: <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once
             at rustc/71f8d0c8f1060bbe74100f29cc6f2da63d697c28/library/alloc/src/boxed.rs:1025
      <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once
             at rustc/71f8d0c8f1060bbe74100f29cc6f2da63d697c28/library/alloc/src/boxed.rs:1025
      std::sys::unix::thread::Thread::new::thread_start
             at rustc/71f8d0c8f1060bbe74100f29cc6f2da63d697c28/library/std/src/sys/unix/thread.rs:87
  41: start_thread
  42: __clone
Thread 'tokio-runtime-worker' panicked at 'failed to spawn thread: Os { code: 11, kind: WouldBlock, message: "Resource temporarily unavailable" }', /rustc/71f8d0c8f1060bbe74100f29cc6f2da63d697c28/library/std/src/thread/mod.rs:619
This is a bug. Please report it at:
        https://github.com/paritytech/polkadot/issues/new

[tomaka]

According to rust-lang/rust#46345, this means you've reached the limit to the number of threads.
What is intriguing is that we are "only" around 50 to 60 threads if I remember correctly.