Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Crash with "Filesystem changed" #15310

Closed
achimnol opened this issue May 3, 2022 · 4 comments · Fixed by #15320
Closed

Crash with "Filesystem changed" #15310

achimnol opened this issue May 3, 2022 · 4 comments · Fixed by #15320
Labels
bug

Comments

@achimnol
Copy link

achimnol commented May 3, 2022 •
edited
Loading

Describe the bug

  • git clone https://github.com/lablup/backend.ai
  • cd backend.ai
  • git reset --hard be2e624fb6d989bc54daea839288f40ada9e8374
    • Note that this particular commit adds a :dist dependency to a pex_binary() target.
      Without it, pants works fine.
    • lablup/backend.ai@be2e624
  • Configure "running pants from sources" with custom patches below. I've put the script as ./pants-local which runs pants from a separately cloned pants working copy.
  • RUST_BACKTRACE=1 ./pants run src/ai/backend/cli:cli
  • The error shows up.
thread 'tokio-runtime-worker' panicked at 'called `Result::unwrap()` on an `Err` value: PyErr { type: <class 'TypeError'>, value: TypeError("format_exception() got an unexpected keyword argument 'etype'"), traceback: Some(<traceback object at 0xffff9e93bd00>) }', src/python.rs:432:8
stack backtrace:
   0: rust_begin_unwind
             at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/std/src/panicking.rs:584:5
   1: core::panicking::panic_fmt
             at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/core/src/panicking.rs:143:14
   2: core::result::unwrap_failed
             at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/core/src/result.rs:1749:5
   3: core::result::Result<T,E>::unwrap
             at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/core/src/result.rs:1065:23
   4: engine::python::Failure::from_py_err_with_gil
             at /home/joongi/workspace/pants/src/rust/engine/src/python.rs:427:7
   5: engine::externs::generator_send::{{closure}}
             at /home/joongi/workspace/pants/src/rust/engine/src/externs/mod.rs:201:23
   6: core::result::Result<T,E>::map_err
             at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/core/src/result.rs:842:27
   7: engine::externs::generator_send
             at /home/joongi/workspace/pants/src/rust/engine/src/externs/mod.rs:199:18
   8: engine::nodes::Task::generate::{{closure}}::{{closure}}
             at /home/joongi/workspace/pants/src/rust/engine/src/nodes.rs:1097:44
   9: pyo3::marker::Python::with_gil
             at /home/joongi/.cargo/registry/src/github.com-1ecc6299db9ec823/pyo3-0.16.2/src/marker.rs:314:9
  10: engine::nodes::Task::generate::{{closure}}
             at /home/joongi/workspace/pants/src/rust/engine/src/nodes.rs:1097:22
  11: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/core/src/future/mod.rs:84:19
  12: engine::nodes::maybe_side_effecting::{{closure}}
             at /home/joongi/workspace/pants/src/rust/engine/src/nodes.rs:73:6
  13: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/core/src/future/mod.rs:84:19
  14: <engine::nodes::Task as engine::nodes::WrappedNode>::run_wrapped_node::__run_wrapped_node::{{closure}}
             at /home/joongi/workspace/pants/src/rust/engine/src/nodes.rs:1180:8
  15: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/core/src/future/mod.rs:84:19
  16: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/core/src/future/future.rs:123:9
  17: <F as futures_core::future::TryFuture>::try_poll
             at /home/joongi/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-core-0.3.21/src/future.rs:82:9
  18: <futures_util::future::try_future::into_future::IntoFuture<Fut> as core::future::future::Future>::poll
             at /home/joongi/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.21/src/future/try_future/into_future.rs:34:9
  19: <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll
             at /home/joongi/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.21/src/future/future/map.rs:55:37
  20: <futures_util::future::future::Map<Fut,F> as core::future::future::Future>::poll
             at /home/joongi/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.21/src/lib.rs:91:13
  21: <futures_util::future::try_future::MapOk<Fut,F> as core::future::future::Future>::poll
             at /home/joongi/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.21/src/lib.rs:91:13
  22: <engine::nodes::NodeKey as graph::node::Node>::run::__run::{{closure}}::{{closure}}::{{closure}}
             at /home/joongi/workspace/pants/src/rust/engine/src/nodes.rs:1475:41
  23: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/core/src/future/mod.rs:84:19
  24: <engine::nodes::NodeKey as graph::node::Node>::run::__run::{{closure}}::{{closure}}
             at /home/joongi/workspace/pants/src/rust/engine/workunit_store/src/lib.rs:871:8
  25: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/core/src/future/mod.rs:84:19
  26: <tokio::task::task_local::TaskLocalFuture<T,F> as core::future::future::Future>::poll::{{closure}}
             at /home/joongi/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.16.1/src/task/task_local.rs:280:28
  27: tokio::task::task_local::TaskLocalFuture<T,F>::with_task
             at /home/joongi/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.16.1/src/task/task_local.rs:272:9
  28: <tokio::task::task_local::TaskLocalFuture<T,F> as core::future::future::Future>::poll
             at /home/joongi/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.16.1/src/task/task_local.rs:280:9
  29: workunit_store::scope_task_workunit_store_handle::{{closure}}
             at /home/joongi/workspace/pants/src/rust/engine/workunit_store/src/lib.rs:974:37
  30: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/core/src/future/mod.rs:84:19
  31: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/core/src/future/future.rs:123:9
  32: <engine::nodes::NodeKey as graph::node::Node>::run::__run::{{closure}}
             at /home/joongi/workspace/pants/src/rust/engine/src/nodes.rs:1395:5
  33: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/core/src/future/mod.rs:84:19
  34: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/core/src/future/future.rs:123:9
  35: graph::entry::Entry<N>::spawn_node_execution::{{closure}}
             at /home/joongi/workspace/pants/src/rust/engine/graph/src/entry.rs:347:44
  36: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/core/src/future/mod.rs:84:19
  37: graph::entry::Entry<N>::spawn_node_execution::{{closure}}::{{closure}}
             at /home/joongi/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.16.1/src/macros/select.rs:505:49
  38: <tokio::future::poll_fn::PollFn<F> as core::future::future::Future>::poll
             at /home/joongi/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.16.1/src/future/poll_fn.rs:38:9
  39: graph::entry::Entry<N>::spawn_node_execution::{{closure}}
             at /home/joongi/workspace/pants/src/rust/engine/graph/src/entry.rs:354:23
  40: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/core/src/future/mod.rs:84:19
  41: <tokio::task::task_local::TaskLocalFuture<T,F> as core::future::future::Future>::poll::{{closure}}
             at /home/joongi/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.16.1/src/task/task_local.rs:280:28
  42: tokio::task::task_local::TaskLocalFuture<T,F>::with_task
             at /home/joongi/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.16.1/src/task/task_local.rs:272:9
  43: <tokio::task::task_local::TaskLocalFuture<T,F> as core::future::future::Future>::poll
             at /home/joongi/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.16.1/src/task/task_local.rs:280:9
  44: workunit_store::scope_task_workunit_store_handle::{{closure}}
             at /home/joongi/workspace/pants/src/rust/engine/workunit_store/src/lib.rs:974:37
  45: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/core/src/future/mod.rs:84:19
  46: task_executor::Executor::future_with_correct_context::{{closure}}
             at /home/joongi/workspace/pants/src/rust/engine/task_executor/src/lib.rs:184:86
  47: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/core/src/future/mod.rs:84:19
  48: <tokio::task::task_local::TaskLocalFuture<T,F> as core::future::future::Future>::poll::{{closure}}
             at /home/joongi/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.16.1/src/task/task_local.rs:280:28
  49: tokio::task::task_local::TaskLocalFuture<T,F>::with_task
             at /home/joongi/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.16.1/src/task/task_local.rs:272:9
  50: <tokio::task::task_local::TaskLocalFuture<T,F> as core::future::future::Future>::poll
             at /home/joongi/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.16.1/src/task/task_local.rs:280:9
  51: stdio::scope_task_destination::{{closure}}
             at /home/joongi/workspace/pants/src/rust/engine/stdio/src/lib.rs:466:41
  52: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/core/src/future/mod.rs:84:19
  53: tokio::runtime::task::core::CoreStage<T>::poll::{{closure}}
             at /home/joongi/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.16.1/src/runtime/task/core.rs:161:17
  54: tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut
             at /home/joongi/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.16.1/src/loom/std/unsafe_cell.rs:14:9
  55: tokio::runtime::task::core::CoreStage<T>::poll
             at /home/joongi/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.16.1/src/runtime/task/core.rs:151:13
  56: tokio::runtime::task::harness::poll_future::{{closure}}
             at /home/joongi/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.16.1/src/runtime/task/harness.rs:461:19
  57: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/core/src/panic/unwind_safe.rs:271:9
  58: std::panicking::try::do_call
             at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/std/src/panicking.rs:492:40
  59: std::panicking::try
             at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/std/src/panicking.rs:456:19
  60: std::panic::catch_unwind
             at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/std/src/panic.rs:137:14
  61: tokio::runtime::task::harness::poll_future
             at /home/joongi/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.16.1/src/runtime/task/harness.rs:449:18
  62: tokio::runtime::task::harness::Harness<T,S>::poll_inner
             at /home/joongi/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.16.1/src/runtime/task/harness.rs:98:27
  63: tokio::runtime::task::harness::Harness<T,S>::poll
             at /home/joongi/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.16.1/src/runtime/task/harness.rs:53:15
  64: tokio::runtime::task::raw::poll
             at /home/joongi/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.16.1/src/runtime/task/raw.rs:113:5
  65: tokio::runtime::task::raw::RawTask::poll
             at /home/joongi/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.16.1/src/runtime/task/raw.rs:70:18
  66: tokio::runtime::task::LocalNotified<S>::run
             at /home/joongi/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.16.1/src/runtime/task/mod.rs:343:9
  67: tokio::runtime::thread_pool::worker::Context::run_task::{{closure}}
             at /home/joongi/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.16.1/src/runtime/thread_pool/worker.rs:448:21
  68: tokio::coop::with_budget::{{closure}}
             at /home/joongi/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.16.1/src/coop.rs:102:9
  69: std::thread::local::LocalKey<T>::try_with
             at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/std/src/thread/local.rs:413:16
  70: std::thread::local::LocalKey<T>::with
             at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/std/src/thread/local.rs:389:9
  71: tokio::coop::with_budget
             at /home/joongi/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.16.1/src/coop.rs:95:5
  72: tokio::coop::budget
             at /home/joongi/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.16.1/src/coop.rs:72:5
  73: tokio::runtime::thread_pool::worker::Context::run_task
             at /home/joongi/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.16.1/src/runtime/thread_pool/worker.rs:424:9
  74: tokio::runtime::thread_pool::worker::Context::run
             at /home/joongi/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.16.1/src/runtime/thread_pool/worker.rs:398:24
  75: tokio::runtime::thread_pool::worker::run::{{closure}}
             at /home/joongi/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.16.1/src/runtime/thread_pool/worker.rs:376:17
  76: tokio::macros::scoped_tls::ScopedKey<T>::set
             at /home/joongi/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.16.1/src/macros/scoped_tls.rs:61:9
  77: tokio::runtime::thread_pool::worker::run
             at /home/joongi/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.16.1/src/runtime/thread_pool/worker.rs:373:5
  78: tokio::runtime::thread_pool::worker::Launch::launch::{{closure}}
             at /home/joongi/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.16.1/src/runtime/thread_pool/worker.rs:352:45
  79: <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll
             at /home/joongi/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.16.1/src/runtime/blocking/task.rs:42:21
  80: tokio::runtime::task::core::CoreStage<T>::poll::{{closure}}
             at /home/joongi/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.16.1/src/runtime/task/core.rs:161:17
  81: tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut
             at /home/joongi/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.16.1/src/loom/std/unsafe_cell.rs:14:9
  82: tokio::runtime::task::core::CoreStage<T>::poll
             at /home/joongi/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.16.1/src/runtime/task/core.rs:151:13
  83: tokio::runtime::task::harness::poll_future::{{closure}}
             at /home/joongi/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.16.1/src/runtime/task/harness.rs:461:19
  84: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/core/src/panic/unwind_safe.rs:271:9
  85: std::panicking::try::do_call
             at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/std/src/panicking.rs:492:40
  86: std::panicking::try
             at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/std/src/panicking.rs:456:19
  87: std::panic::catch_unwind
             at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/std/src/panic.rs:137:14
  88: tokio::runtime::task::harness::poll_future
             at /home/joongi/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.16.1/src/runtime/task/harness.rs:449:18
  89: tokio::runtime::task::harness::Harness<T,S>::poll_inner
             at /home/joongi/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.16.1/src/runtime/task/harness.rs:98:27
  90: tokio::runtime::task::harness::Harness<T,S>::poll
             at /home/joongi/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.16.1/src/runtime/task/harness.rs:53:15
  91: tokio::runtime::task::raw::poll
             at /home/joongi/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.16.1/src/runtime/task/raw.rs:113:5
  92: tokio::runtime::task::raw::RawTask::poll
             at /home/joongi/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.16.1/src/runtime/task/raw.rs:70:18
  93: tokio::runtime::task::UnownedTask<S>::run
             at /home/joongi/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.16.1/src/runtime/task/mod.rs:380:9
  94: tokio::runtime::blocking::pool::Task::run
             at /home/joongi/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.16.1/src/runtime/blocking/pool.rs:91:9
  95: tokio::runtime::blocking::pool::Inner::run
15:19:45.84 [INFO] Filesystem changed during run: retrying `Run` in 500ms...

The error message is repeated until the user interrupts with SIGINT (Ctrl+C).

Pants version
The master branch (ba9dc62) with a Linux aarch64 patch based on a comment of #12183.
Running pants from sources using export PY=$(pyenv prefix 3.10.4)/bin/python.

OS
Ubuntu 20.04.4 LTS as UTM (v3.1.5) virtual machine on macOS 12.3.1 at Apple M1 Max.

Additional info
Might be related with #14466 but not sure.
@achimnol achimnol added the bug label May 3, 2022
@achimnol
Copy link
Author

achimnol commented May 3, 2022 •
edited
Loading

Once this error happens, even reverting the dependency addition keeps the error happening afterwards. I did somehow reset this status in prior attempts but could not catch the exact reset condition yet.

@achimnol achimnol mentioned this issue May 3, 2022
Closed
@stuhood
Copy link
Member

stuhood commented May 3, 2022 •
edited
Loading

It looks like the issue is that you are running Pants with Python 3.10, which isn't supported yet: see #14111.
Until more progress is made on #14111, I would recommend developing Pants with 3.9.

The etype argument was removed from format_exception in 3.10 apparently: https://docs.python.org/3/library/traceback.html#traceback.format_exception

Changed in version 3.10: This function’s behavior and signature were modified to match print_exception().

... having said that, it looks like the etype field hasn't been used for a long time (since 3.5): I'll submit a quick patch to remove it.

After that issue is resolved, you should get a useful stacktrace out, and can re-open this issue if it's not clear.

@stuhood stuhood mentioned this issue May 3, 2022
Merged
@achimnol
Copy link
Author

achimnol commented May 4, 2022 •
edited
Loading

Yeah, I'm now trying Python 3.9 as the pants runtime.
FYI: The reason why I've changed the pants runtime Python version was due to interpreter selection error fixed by #15307 which prevented me from testing VSCode development setup integration and that my project is based on Python 3.10.

@achimnol
Copy link
Author

achimnol commented May 4, 2022

And using Python 3.9, it reveals my human mistake on a typo in one of BUILD files with a kind error message. 👍🏼 😱 🙏🏼

stuhood added a commit that referenced this issue May 5, 2022
@stuhood
Remove deprecated traceback formatting argument. (#15320)
27e2f79 
The [`etype` argument to `format_exception`](https://docs.python.org/3/library/traceback.html#traceback.format_exception) has been ignored since Python `3.5`, and is removed in Python `3.10`.

Fixes #15310.

[ci skip-build-wheels]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Remove deprecated traceback formatting argument stuhood/pants
2 participants
@stuhood @achimnol