Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: drop bug of wal on mq, can't not drop runtime #433

Merged
merged 2 commits into from
Nov 30, 2022

Conversation

Rachelint
Copy link
Contributor

@Rachelint Rachelint commented Nov 29, 2022

Which issue does this PR close?

Closes #

Rationale for this change

Now, I found drop wal on mq may need to panic because dropping runtime in drop().
You can reproduce it using following cmd(need a kafka cluster):

RUST_BACKTRACE=1 cargo test --workspace --package wal --lib -- read_write::test_kafka_wal --ignored

The error message:

---- tests::read_write::test_kafka_wal stdout ----
thread 'tests::read_write::test_kafka_wal' panicked at 'Cannot start a runtime from within a runtime. This happens because a function (like `block_on`) attempted to block the current thread while the thread is being used to drive asynchronous tasks.', /Users/kamiu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.20.1/src/runtime/thread_pool/mod.rs:89:25
stack backtrace:
   0: std::panicking::begin_panic
             at /rustc/d394408fb38c4de61f765a3ed5189d2731a1da91/library/std/src/panicking.rs:616:12
   1: tokio::runtime::enter::enter
             at /Users/kamiu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.20.1/src/runtime/enter.rs:40:9
   2: tokio::runtime::thread_pool::ThreadPool::block_on
             at /Users/kamiu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.20.1/src/runtime/thread_pool/mod.rs:89:25
   3: tokio::runtime::Runtime::block_on
             at /Users/kamiu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.20.1/src/runtime/mod.rs:484:43
   4: common_util::runtime::Runtime::block_on
             at /Users/kamiu/Desktop/github/ceresdb/common_util/src/runtime/mod.rs:82:9
   5: <wal::message_queue_impl::namespace::Namespace<M> as core::ops::drop::Drop>::drop
             at ./src/message_queue_impl/namespace.rs:245:9
   6: core::ptr::drop_in_place<wal::message_queue_impl::namespace::Namespace<message_queue::kafka::kafka_impl::KafkaImpl>>
             at /rustc/d394408fb38c4de61f765a3ed5189d2731a1da91/library/core/src/ptr/mod.rs:487:1
   7: core::ptr::drop_in_place<wal::message_queue_impl::wal::MessageQueueImpl<message_queue::kafka::kafka_impl::KafkaImpl>>
             at /rustc/d394408fb38c4de61f765a3ed5189d2731a1da91/library/core/src/ptr/mod.rs:487:1
   8: core::ptr::drop_in_place<dyn wal::manager::WalManager>
             at /rustc/d394408fb38c4de61f765a3ed5189d2731a1da91/library/core/src/ptr/mod.rs:487:1
   9: alloc::sync::Arc<T>::drop_slow
             at /rustc/d394408fb38c4de61f765a3ed5189d2731a1da91/library/alloc/src/sync.rs:1105:18
  10: <alloc::sync::Arc<T> as core::ops::drop::Drop>::drop
             at /rustc/d394408fb38c4de61f765a3ed5189d2731a1da91/library/alloc/src/sync.rs:1702:13
  11: core::ptr::drop_in_place<alloc::sync::Arc<dyn wal::manager::WalManager>>
             at /rustc/d394408fb38c4de61f765a3ed5189d2731a1da91/library/core/src/ptr/mod.rs:487:1
  12: wal::tests::read_write::simple_read_write::{{closure}}
             at ./src/tests/read_write.rs:216:1
  13: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/d394408fb38c4de61f765a3ed5189d2731a1da91/library/core/src/future/mod.rs:91:19
  14: tokio::park::thread::CachedParkThread::block_on::{{closure}}
             at /Users/kamiu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.20.1/src/park/thread.rs:263:54
  15: tokio::coop::with_budget::{{closure}}
             at /Users/kamiu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.20.1/src/coop.rs:102:9
  16: std::thread::local::LocalKey<T>::try_with
             at /rustc/d394408fb38c4de61f765a3ed5189d2731a1da91/library/std/src/thread/local.rs:445:16
  17: std::thread::local::LocalKey<T>::with
             at /rustc/d394408fb38c4de61f765a3ed5189d2731a1da91/library/std/src/thread/local.rs:421:9
  18: tokio::coop::with_budget
             at /Users/kamiu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.20.1/src/coop.rs:95:5
  19: tokio::coop::budget
             at /Users/kamiu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.20.1/src/coop.rs:72:5
  20: tokio::park::thread::CachedParkThread::block_on
             at /Users/kamiu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.20.1/src/park/thread.rs:263:31
  21: tokio::runtime::enter::Enter::block_on
             at /Users/kamiu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.20.1/src/runtime/enter.rs:152:13
  22: tokio::runtime::thread_pool::ThreadPool::block_on
             at /Users/kamiu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.20.1/src/runtime/thread_pool/mod.rs:90:9
  23: tokio::runtime::Runtime::block_on
             at /Users/kamiu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.20.1/src/runtime/mod.rs:484:43
  24: common_util::runtime::Runtime::block_on
             at /Users/kamiu/Desktop/github/ceresdb/common_util/src/runtime/mod.rs:82:9
  25: wal::tests::read_write::test_simple_read_write_default_batch
             at ./src/tests/read_write.rs:79:5
  26: wal::tests::read_write::test_all
             at ./src/tests/read_write.rs:49:5
  27: wal::tests::read_write::test_kafka_wal
             at ./src/tests/read_write.rs:45:5
  28: wal::tests::read_write::test_kafka_wal::{{closure}}
             at ./src/tests/read_write.rs:42:1
  29: core::ops::function::FnOnce::call_once
             at /rustc/d394408fb38c4de61f765a3ed5189d2731a1da91/library/core/src/ops/function.rs:248:5
  30: core::ops::function::FnOnce::call_once
             at /rustc/d394408fb38c4de61f765a3ed5189d2731a1da91/library/core/src/ops/function.rs:248:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

What changes are included in this PR?

Fix above bug.

Are there any user-facing changes?

User should manually call close_gracefully now.

How does this change test

Test manually.

@Rachelint Rachelint changed the title fix: drop bug of wal on mq, can't not drop runtime. fix: drop bug of wal on mq, can't not drop runtime Nov 29, 2022
@jiacai2050
Copy link
Contributor

Now, I found drop wal on mq may need to panic because dropping runtime in drop().

Could you paste panic stacktrace here?

Fix above bug.

How? Please explain in more details.

@Rachelint
Copy link
Contributor Author

Now, I found drop wal on mq may need to panic because dropping runtime in drop().

Could you paste panic stacktrace here?

Fix above bug.

How? Please explain in more details.

I have added the reproducing way and the detail error.

Copy link
Contributor

@jiacai2050 jiacai2050 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@jiacai2050 jiacai2050 merged commit 21c803b into apache:main Nov 30, 2022
Rachelint added a commit to Rachelint/ceresdb that referenced this pull request Nov 30, 2022
* fix drop bug of wal on mq, can't not drop runtime.

* address CR.
chunshao90 pushed a commit to chunshao90/ceresdb that referenced this pull request May 15, 2023
* fix drop bug of wal on mq, can't not drop runtime.

* address CR.
@Rachelint Rachelint deleted the fix-wal-on-mq branch May 27, 2023 12:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants