Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Problems starting basic example #17

Open
flyaruu opened this issue Aug 17, 2020 · 17 comments
Open

Problems starting basic example #17

flyaruu opened this issue Aug 17, 2020 · 17 comments

Comments

@flyaruu
Copy link

flyaruu commented Aug 17, 2020

Hi, I'm trying to get Noria working with JDBC. I'm aware of JDBC issues, I thought I'd take a stab at seeing where it fails, so I looked at issue #15 but I fail to even get the basics working.
Hopefully I'm doing something wrong.

I'm starting a zookeeper instance:

docker run --rm -it --name some-zookeeper -p 2181:2181 zookeeper

Seems alright. In a new shell, clean noria checkout:

cargo r --release --bin noria-server -- --deployment myapp --no-reuse --shards 0

Seems ok, in another shell, in a clean noria-mysql checkout:

cargo run --release -- --deployment myapp -z 127.0.0.1:2181

Starts up fine.
Then in a final terminal:

mysql -h 127.0.0.1

Seems to work:

Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 8
Server version: 5.1.10-alpha-msql-proxy

Copyright (c) 2000, 2020, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> 

Create a table (like in #15):

CREATE TABLE Cats (id int, name VARCHAR(255), PRIMARY KEY(id));
Query OK, 0 rows affected (0.01 sec)

But the next statement fails:

mysql> INSERT INTO Cats (id, name) VALUES (2, "Burt");
ERROR 1105 (HY000): TransportError(ClientDropped

   0: backtrace::backtrace::trace
   1: backtrace::capture::Backtrace::new_unresolved
   2: failure::backtrace::internal::InternalBacktrace::new
   3: failure::error::Error::from_boxed_compat
   4: <noria::table::TableError as core::convert::From<alloc::boxed::Box<dyn std::error::Error+core::marker::Send+core::marker::Sync>>>::from
   5: <futures_util::future::either::Either<A,B> as core::future::future::Future>::poll
   6: <core::future::from_generator::GenFuture<T> as core::
mysql> 

noria-mysql trace:

thread 'tokio-runtime-worker' panicked at 'BrokenTransportRecv', /Users/frank/.cargo/git/checkouts/noria-8612be89bc6721f9/e427387/noria/src/table.rs:237:62
stack backtrace:
 151.517275884s ERROR noria_mysql::backend: failed error=Client was dropped
   0:        0x101dcc8b4 - std::backtrace_rs::backtrace::libunwind::trace::he9497f4525a08d2b
                               at /rustc/7e6d6e5f535321c2223f044caba16f97b825009c/library/std/src/../../backtrace/src/backtrace/libunwind.rs:96
   1:        0x101dcc8b4 - std::backtrace_rs::backtrace::trace_unsynchronized::h10a0a59d96f4a16b
                               at /rustc/7e6d6e5f535321c2223f044caba16f97b825009c/library/std/src/../../backtrace/src/backtrace/mod.rs:66
   2:        0x101dcc8b4 - std::sys_common::backtrace::_print_fmt::h6fd8fbf82b7783c5
                               at /rustc/7e6d6e5f535321c2223f044caba16f97b825009c/library/std/src/sys_common/backtrace.rs:79
   3:        0x101dcc8b4 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::hfb2db7374ccd4d34
                               at /rustc/7e6d6e5f535321c2223f044caba16f97b825009c/library/std/src/sys_common/backtrace.rs:58
   4:        0x101debfec - core::fmt::write::h7e381316a10f5e82
                               at /rustc/7e6d6e5f535321c2223f044caba16f97b825009c/library/core/src/fmt/mod.rs:1117
   5:        0x101dc6629 - std::io::Write::write_fmt::h53862e22ed93af5e
                               at /rustc/7e6d6e5f535321c2223f044caba16f97b825009c/library/std/src/io/mod.rs:1510
   6:        0x101dce525 - std::sys_common::backtrace::_print::h5642011e0f38c919
                               at /rustc/7e6d6e5f535321c2223f044caba16f97b825009c/library/std/src/sys_common/backtrace.rs:61
   7:        0x101dce525 - std::sys_common::backtrace::print::hbcc5d5a374cdc250
                               at /rustc/7e6d6e5f535321c2223f044caba16f97b825009c/library/std/src/sys_common/backtrace.rs:48
   8:        0x101dce525 - std::panicking::default_hook::{{closure}}::h1a87f1c0565a051b
                               at /rustc/7e6d6e5f535321c2223f044caba16f97b825009c/library/std/src/panicking.rs:200
   9:        0x101dce262 - std::panicking::default_hook::he29ab2038604c86f
                               at /rustc/7e6d6e5f535321c2223f044caba16f97b825009c/library/std/src/panicking.rs:219
  10:        0x101dceac5 - std::panicking::rust_panic_with_hook::h9dab6d214201c5da
                               at /rustc/7e6d6e5f535321c2223f044caba16f97b825009c/library/std/src/panicking.rs:569
  11:        0x101dce65b - std::panicking::begin_panic_handler::{{closure}}::hf4bcec37fb7265f5
                               at /rustc/7e6d6e5f535321c2223f044caba16f97b825009c/library/std/src/panicking.rs:476
  12:        0x101dcccf8 - std::sys_common::backtrace::__rust_end_short_backtrace::h8c49942bd9c628f8
                               at /rustc/7e6d6e5f535321c2223f044caba16f97b825009c/library/std/src/sys_common/backtrace.rs:153
  13:        0x101dce61a - rust_begin_unwind
                               at /rustc/7e6d6e5f535321c2223f044caba16f97b825009c/library/std/src/panicking.rs:475
  14:        0x101e0d84b - std::panicking::begin_panic_fmt::h769a003a4fabac0e
                               at /rustc/7e6d6e5f535321c2223f044caba16f97b825009c/library/std/src/panicking.rs:429
  15:        0x101b9a912 - <noria::table::TableEndpoint as tower_service::Service<()>>::call::{{closure}}::{{closure}}::head2a7e0d2cc3450
                               at /Users/frank/.cargo/git/checkouts/noria-8612be89bc6721f9/e427387/noria/src/table.rs:237
  16:        0x101b9a912 - tokio_tower::multiplex::client::Client<T,E,Request>::with_error_handler::{{closure}}::ha8bb50bc2271906b
                               at /Users/frank/.cargo/git/checkouts/tokio-tower-98a883543a8cb142/e58bb90/src/multiplex/client.rs:197
  17:        0x101b9a912 - <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll::he442d7d330fcd79a
                               at /rustc/7e6d6e5f535321c2223f044caba16f97b825009c/library/core/src/future/mod.rs:78
  18:        0x101b3c1c9 - tokio::runtime::task::core::Core<T,S>::poll::{{closure}}::h3d742ba66b05f6e6
                               at /Users/frank/.cargo/git/checkouts/tokio-377c595163f99a10/3137c6f/tokio/src/runtime/task/core.rs:163
  19:        0x101b3c1c9 - tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut::h38277425b57d9489
                               at /Users/frank/.cargo/git/checkouts/tokio-377c595163f99a10/3137c6f/tokio/src/loom/std/unsafe_cell.rs:14
  20:        0x101b3c1c9 - tokio::runtime::task::core::Core<T,S>::poll::h9f81880f2d783c9d
                               at /Users/frank/.cargo/git/checkouts/tokio-377c595163f99a10/3137c6f/tokio/src/runtime/task/core.rs:148
  21:        0x101b4d4bd - tokio::runtime::task::harness::Harness<T,S>::poll::{{closure}}::hc2cb169c4543b548
                               at /Users/frank/.cargo/git/checkouts/tokio-377c595163f99a10/3137c6f/tokio/src/runtime/task/harness.rs:108
  22:        0x101b4d4bd - core::ops::function::FnOnce::call_once::ha83341f8a50e8565
                               at /rustc/7e6d6e5f535321c2223f044caba16f97b825009c/library/core/src/ops/function.rs:233
  23:        0x101b4d4bd - <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h5f2483b0736864a1
                               at /rustc/7e6d6e5f535321c2223f044caba16f97b825009c/library/std/src/panic.rs:318
  24:        0x101b92bb8 - std::panicking::try::do_call::h2863e39e882eb1e0
                               at /rustc/7e6d6e5f535321c2223f044caba16f97b825009c/library/std/src/panicking.rs:373
  25:        0x101b92bb8 - std::panicking::try::hd3cd23ca9b4aac77
                               at /rustc/7e6d6e5f535321c2223f044caba16f97b825009c/library/std/src/panicking.rs:337
  26:        0x101b92bb8 - std::panic::catch_unwind::h85aaa291c5135e2c
                               at /rustc/7e6d6e5f535321c2223f044caba16f97b825009c/library/std/src/panic.rs:394
  27:        0x101b92bb8 - tokio::runtime::task::harness::Harness<T,S>::poll::h4598909b59fa3963
                               at /Users/frank/.cargo/git/checkouts/tokio-377c595163f99a10/3137c6f/tokio/src/runtime/task/harness.rs:84
  28:        0x101d2a15e - tokio::runtime::task::raw::RawTask::poll::h84b006a1e3342a09
                               at /Users/frank/.cargo/git/checkouts/tokio-377c595163f99a10/3137c6f/tokio/src/runtime/task/raw.rs:66
  29:        0x101d2a15e - tokio::runtime::task::Notified<S>::run::h46ad02afe7f7542b
                               at /Users/frank/.cargo/git/checkouts/tokio-377c595163f99a10/3137c6f/tokio/src/runtime/task/mod.rs:169
  30:        0x101d2a15e - tokio::runtime::thread_pool::worker::Context::run_task::{{closure}}::h4de65ade9c6380b1
                               at /Users/frank/.cargo/git/checkouts/tokio-377c595163f99a10/3137c6f/tokio/src/runtime/thread_pool/worker.rs:308
  31:        0x101d2a01c - tokio::coop::budget::{{closure}}::h161591f49f5d979a
                               at /Users/frank/.cargo/git/checkouts/tokio-377c595163f99a10/3137c6f/tokio/src/coop.rs:97
  32:        0x101d2a01c - std::thread::local::LocalKey<T>::try_with::ha2ee22c28756a0b2
                               at /rustc/7e6d6e5f535321c2223f044caba16f97b825009c/library/std/src/thread/local.rs:267
  33:        0x101d2a01c - std::thread::local::LocalKey<T>::with::h5ca0065eec38a20c
                               at /rustc/7e6d6e5f535321c2223f044caba16f97b825009c/library/std/src/thread/local.rs:243
  34:        0x101d2a01c - tokio::coop::budget::h238e8b9c04d253e0
                               at /Users/frank/.cargo/git/checkouts/tokio-377c595163f99a10/3137c6f/tokio/src/coop.rs:79
  35:        0x101d2a01c - tokio::runtime::thread_pool::worker::Context::run_task::ha306ca1dcf61ac54
                               at /Users/frank/.cargo/git/checkouts/tokio-377c595163f99a10/3137c6f/tokio/src/runtime/thread_pool/worker.rs:307
  36:        0x101d297ce - tokio::runtime::thread_pool::worker::Context::run::h2fffcc7ed0a4f790
                               at /Users/frank/.cargo/git/checkouts/tokio-377c595163f99a10/3137c6f/tokio/src/runtime/thread_pool/worker.rs:279
  37:        0x101d19e63 - tokio::runtime::thread_pool::worker::run::{{closure}}::h90c666296baf9cc6
                               at /Users/frank/.cargo/git/checkouts/tokio-377c595163f99a10/3137c6f/tokio/src/runtime/thread_pool/worker.rs:264
  38:        0x101d19e63 - tokio::macros::scoped_tls::ScopedKey<T>::set::h54e39d6f67b0a67d
                               at /Users/frank/.cargo/git/checkouts/tokio-377c595163f99a10/3137c6f/tokio/src/macros/scoped_tls.rs:64
  39:        0x101d292c4 - tokio::runtime::thread_pool::worker::run::h1bfb873f95891b28
                               at /Users/frank/.cargo/git/checkouts/tokio-377c595163f99a10/3137c6f/tokio/src/runtime/thread_pool/worker.rs:261
  40:        0x101d0ede3 - tokio::runtime::thread_pool::worker::Launch::launch::{{closure}}::hbbcaf2cabc73169d
                               at /Users/frank/.cargo/git/checkouts/tokio-377c595163f99a10/3137c6f/tokio/src/runtime/thread_pool/worker.rs:240
  41:        0x101d0ede3 - <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll::h88b256f57da30ac6
                               at /Users/frank/.cargo/git/checkouts/tokio-377c595163f99a10/3137c6f/tokio/src/runtime/blocking/task.rs:38
  42:        0x101d0ede3 - tokio::runtime::task::core::Core<T,S>::poll::{{closure}}::h34ee386b95d91512
                               at /Users/frank/.cargo/git/checkouts/tokio-377c595163f99a10/3137c6f/tokio/src/runtime/task/core.rs:163
  43:        0x101d0ede3 - tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut::h13a5f633ba7f2696
                               at /Users/frank/.cargo/git/checkouts/tokio-377c595163f99a10/3137c6f/tokio/src/loom/std/unsafe_cell.rs:14
  44:        0x101d2e2a7 - tokio::runtime::task::core::Core<T,S>::poll::hc9e3fae42dd9cfb9
                               at /Users/frank/.cargo/git/checkouts/tokio-377c595163f99a10/3137c6f/tokio/src/runtime/task/core.rs:148
  45:        0x101d2e2a7 - tokio::runtime::task::harness::Harness<T,S>::poll::{{closure}}::h84d381f280474d6b
                               at /Users/frank/.cargo/git/checkouts/tokio-377c595163f99a10/3137c6f/tokio/src/runtime/task/harness.rs:108
  46:        0x101d2e2a7 - core::ops::function::FnOnce::call_once::h6ac9ab05edebdd81
                               at /rustc/7e6d6e5f535321c2223f044caba16f97b825009c/library/core/src/ops/function.rs:233
  47:        0x101d2e2a7 - <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h5286e8a3ca09e6f3
                               at /rustc/7e6d6e5f535321c2223f044caba16f97b825009c/library/std/src/panic.rs:318
  48:        0x101d0d351 - std::panicking::try::do_call::h1f1664dd3b5e303b
                               at /rustc/7e6d6e5f535321c2223f044caba16f97b825009c/library/std/src/panicking.rs:373
  49:        0x101d0d351 - std::panicking::try::h929c6a8ee482c1d3
                               at /rustc/7e6d6e5f535321c2223f044caba16f97b825009c/library/std/src/panicking.rs:337
  50:        0x101d0d351 - std::panic::catch_unwind::h1a2e1c6a3c6621a3
                               at /rustc/7e6d6e5f535321c2223f044caba16f97b825009c/library/std/src/panic.rs:394
  51:        0x101d0d351 - tokio::runtime::task::harness::Harness<T,S>::poll::h36246b0d5d13a035
                               at /Users/frank/.cargo/git/checkouts/tokio-377c595163f99a10/3137c6f/tokio/src/runtime/task/harness.rs:84
  52:        0x101d0feff - tokio::runtime::task::raw::RawTask::poll::h84b006a1e3342a09
                               at /Users/frank/.cargo/git/checkouts/tokio-377c595163f99a10/3137c6f/tokio/src/runtime/task/raw.rs:66
  53:        0x101d0feff - tokio::runtime::task::Notified<S>::run::h7bbcf40f3dff391d
                               at /Users/frank/.cargo/git/checkouts/tokio-377c595163f99a10/3137c6f/tokio/src/runtime/task/mod.rs:169
  54:        0x101d0feff - tokio::runtime::blocking::pool::Inner::run::hae12476ec7e28d9e
                               at /Users/frank/.cargo/git/checkouts/tokio-377c595163f99a10/3137c6f/tokio/src/runtime/blocking/pool.rs:230
  55:        0x101d1af82 - tokio::runtime::blocking::pool::Spawner::spawn_thread::{{closure}}::{{closure}}::h98f502aeb1ad0003
                               at /Users/frank/.cargo/git/checkouts/tokio-377c595163f99a10/3137c6f/tokio/src/runtime/blocking/pool.rs:210
  56:        0x101d1af82 - tokio::runtime::context::enter::h3c16f4071e58296d
                               at /Users/frank/.cargo/git/checkouts/tokio-377c595163f99a10/3137c6f/tokio/src/runtime/context.rs:72
  57:        0x101d194fe - tokio::runtime::handle::Handle::enter::h06b6d376502de6b9
                               at /Users/frank/.cargo/git/checkouts/tokio-377c595163f99a10/3137c6f/tokio/src/runtime/handle.rs:39
  58:        0x101d194fe - tokio::runtime::blocking::pool::Spawner::spawn_thread::{{closure}}::hb8e3fa9f6c1ffde2
                               at /Users/frank/.cargo/git/checkouts/tokio-377c595163f99a10/3137c6f/tokio/src/runtime/blocking/pool.rs:209
  59:        0x101d194fe - std::sys_common::backtrace::__rust_begin_short_backtrace::hb72a8072285616e3
                               at /rustc/7e6d6e5f535321c2223f044caba16f97b825009c/library/std/src/sys_common/backtrace.rs:137
  60:        0x101d1cec1 - std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}}::hd81198c5ab0d100d
                               at /rustc/7e6d6e5f535321c2223f044caba16f97b825009c/library/std/src/thread/mod.rs:475
  61:        0x101d1cec1 - <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h545f887c76e457c1
                               at /rustc/7e6d6e5f535321c2223f044caba16f97b825009c/library/std/src/panic.rs:318
  62:        0x101d1cec1 - std::panicking::try::do_call::hfcbf1e36cf2b3658
                               at /rustc/7e6d6e5f535321c2223f044caba16f97b825009c/library/std/src/panicking.rs:373
  63:        0x101d1cec1 - std::panicking::try::hc1941a32d0ce03b0
                               at /rustc/7e6d6e5f535321c2223f044caba16f97b825009c/library/std/src/panicking.rs:337
  64:        0x101d1cec1 - std::panic::catch_unwind::haba6d86a749d0033
                               at /rustc/7e6d6e5f535321c2223f044caba16f97b825009c/library/std/src/panic.rs:394
  65:        0x101d1cec1 - std::thread::Builder::spawn_unchecked::{{closure}}::h25d389d6eab7bd3a
                               at /rustc/7e6d6e5f535321c2223f044caba16f97b825009c/library/std/src/thread/mod.rs:474
  66:        0x101d1cec1 - core::ops::function::FnOnce::call_once{{vtable.shim}}::h043a476368fcf197
                               at /rustc/7e6d6e5f535321c2223f044caba16f97b825009c/library/core/src/ops/function.rs:233
  67:        0x101dd19cd - <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once::h85dfeaba80b7a665
                               at /rustc/7e6d6e5f535321c2223f044caba16f97b825009c/library/alloc/src/boxed.rs:1025
  68:        0x101dd19cd - <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once::h05752f0c5860f139
                               at /rustc/7e6d6e5f535321c2223f044caba16f97b825009c/library/alloc/src/boxed.rs:1025
  69:        0x101dd19cd - std::sys::unix::thread::Thread::new::thread_start::h39b071b0d2fb7aea
                               at /rustc/7e6d6e5f535321c2223f044caba16f97b825009c/library/std/src/sys/unix/thread.rs:87
  70:     0x7fff717ba109 - __pthread_start

Noria terminates with a segfault:

Segmentation fault: 11

Ideas?

@jonhoo
Copy link
Collaborator

jonhoo commented Aug 17, 2020

Huh, that is very weird indeed. What platform are you on?
Can you try adding -v to where you launch noria-server? That should give you some more details about what it did just before it failed.

@jonhoo
Copy link
Collaborator

jonhoo commented Aug 17, 2020

Also, can you try the griddle branch and see if the same issue occurs there?

@flyaruu
Copy link
Author

flyaruu commented Aug 17, 2020

The noria output doesn't add much:

     Running `target/release/noria-server --deployment myapp --no-reuse --shards 0 -v -z '127.0.0.1:2181'`
Aug 17 20:48:53.223 INFO became leader at epoch 3
Aug 17 20:48:53.232 INFO found initial leader
Aug 17 20:48:53.232 INFO leader listening on external address 127.0.0.1:6033
Aug 17 20:48:53.233 INFO connected to controller, src: 127.0.0.1:52725
Aug 17 20:48:53.233 INFO listening for reads, on: 127.0.0.1:52726
Aug 17 20:48:53.233 WARN Connected to new leader
Aug 17 20:48:53.237 INFO new worker registered from 127.0.0.1:52725, which listens on 127.0.0.1:52723
Aug 17 20:49:28.533 INFO starting migration
Aug 17 20:49:28.533 INFO Schema version advanced from 0 to 1
Aug 17 20:49:28.536 INFO adding new base, node: 1
Aug 17 20:49:28.536 INFO finalizing migration, #nodes: 1
Aug 17 20:49:28.539 INFO sending domain 0.0 to worker Ok(127.0.0.1:52723)
Aug 17 20:49:28.542 INFO booted domain, nodes: 1, shard: 0, domain: 0
Aug 17 20:49:28.543 INFO informed controller that domain 0.0 is at 127.0.0.1:52735
Aug 17 20:49:28.543 INFO bringing up inter-domain connections
Aug 17 20:49:28.543 INFO initializing new materializations
Aug 17 20:49:28.544 INFO adding lookup index to view, columns: [0], node: 1
Aug 17 20:49:28.544 INFO no need to replay empty new base, node: 1
Aug 17 20:49:28.569 WARN migration completed, ms: 35
Segmentation fault: 11

I'll build the griddle branch and get back..

@jonhoo
Copy link
Collaborator

jonhoo commented Aug 17, 2020

Ah, sorry, yes, you'll also want to swap the # on these two lines to enable trace logging: https://github.com/mit-pdos/noria/blob/f54dfcef4057b490ccb43f4a31c6aaf35c923fb7/server/Cargo.toml#L42-L41

@flyaruu
Copy link
Author

flyaruu commented Aug 17, 2020

Not much more info:

Aug 17 21:29:53.331 TRCE readying empty node, local: 0, shard: 0, domain: 0
Aug 17 21:29:53.332 TRCE node ready, node: 1
Aug 17 21:29:53.332 WARN migration completed, ms: 24
Aug 17 21:29:53.332 INFO starting migration
Aug 17 21:29:53.332 DEBG 1 queries, 0 of which are named, version: 5
Aug 17 21:29:53.332 INFO Schema version advanced from 4 to 5
Aug 17 21:29:53.332 INFO finalizing migration, #nodes: 0
Aug 17 21:29:53.332 DEBG booting new domains
Aug 17 21:29:53.332 DEBG mutating existing domains
Aug 17 21:29:53.332 INFO bringing up inter-domain connections
Aug 17 21:29:53.332 INFO initializing new materializations
Aug 17 21:29:53.332 WARN migration completed, ms: 0
Aug 17 21:29:53.332 INFO starting migration
Aug 17 21:29:53.332 DEBG 1 queries, 0 of which are named, version: 6
Aug 17 21:29:53.332 INFO Schema version advanced from 5 to 6
Aug 17 21:29:53.332 INFO finalizing migration, #nodes: 0
Aug 17 21:29:53.332 DEBG booting new domains
Aug 17 21:29:53.332 DEBG mutating existing domains
Aug 17 21:29:53.332 INFO bringing up inter-domain connections
Aug 17 21:29:53.332 INFO initializing new materializations
Aug 17 21:29:53.332 WARN migration completed, ms: 0
Aug 17 21:30:06.161 TRCE creating table, for: Cats
Aug 17 21:30:18.252 DEBG accepted new connection, from: 127.0.0.1:53190, id: 0.0
Aug 17 21:30:18.252 DEBG established new connection, base: true, id: 0.0
Segmentation fault: 11

It's on a macbook, rust version:

frank@MacBook:~/git/noria (griddle)$ rustc --version
rustc 1.47.0-nightly (7e6d6e5f5 2020-08-16)

@jonhoo
Copy link
Collaborator

jonhoo commented Aug 17, 2020

That's really strange indeed. I haven't tested Noria much on macOS, as all the benchmarks and dev work I've done has been on Linux. If it works on macOS, it's by accident 😅 It certainly should not be segfaulting though. If you could figure out where it segfaults by running through gdb, that would help a lot in trying to nail this down!

@flyaruu
Copy link
Author

flyaruu commented Aug 18, 2020

Ok, first tried (without much success) to debug on Mac. Then, just to check, dusted off my linux box, and I have the exact same problem. I'll attempt debugging on the Linux box later, but for now I can say it doesn't seem to be Mac related.

@jonhoo
Copy link
Collaborator

jonhoo commented Aug 18, 2020

@ms705 Is this something you have seen before?

@flyaruu
Copy link
Author

flyaruu commented Aug 22, 2020

Ok, I got the same issue now with gdb on my linux box, looks something like this:

Aug 22 16:28:34.597 TRCE creating table, for: Cats2
Aug 22 16:28:34.605 TRCE creating table, for: Cats3
Aug 22 16:28:36.342 DEBG accepted new connection, from: 127.0.0.1:58644, id: 1.0
Aug 22 16:28:36.342 DEBG established new connection, base: true, id: 1.0
--Type <RET> for more, q to quit, c to continue without paging--

Thread 8 "worker" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff43fd700 (LWP 3927862)]
0x00005555564ce92a in noria_dataflow::payload::Packet::dst (self=0x7fffde229500) at server/dataflow/src/payload.rs:246
246	            Packet::Input { ref inner, .. } => unsafe { inner.deref() }.dst,
(gdb) where
#0  0x00005555564ce92a in noria_dataflow::payload::Packet::dst (self=0x7fffde229500) at server/dataflow/src/payload.rs:246
#1  0x00005555561a622a in noria_dataflow::group_commit::GroupCommitQueueSet::should_append (self=0x7fffdd653510, p=0x7fffde229500, nodes=0x7fffdd653238) at server/dataflow/src/group_commit.rs:24
#2  0x00005555561a4bfb in noria_dataflow::domain::Domain::on_event (self=0x7fffdd653218, executor=..., event=...) at server/dataflow/src/domain/mod.rs:3008
#3  0x0000555555bfbe71 in <noria_server::worker::replica::Replica as core::future::future::Future>::poll::{{closure}} (p=0x7fffde229500) at server/src/worker/replica.rs:640
#4  0x0000555555e5b5e9 in <noria_server::worker::replica::Replica as core::future::future::Future>::poll (self=..., cx=0x7ffff43f94d0) at server/src/worker/replica.rs:640
#5  0x000055555594ce98 in noria_server::worker::listen_df::{{closure}}::{{closure}}::{{closure}} () at /home/frank/git/noria/server/src/worker/mod.rs:341
#6  0x0000555555aaf1c9 in <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll (self=..., cx=0x7ffff43f94d0) at /rustc/792c645ca7d11a8d254df307d019c5bf01445c37/library/core/src/future/mod.rs:79
#7  0x00005555558a2609 in tokio::runtime::task::core::Core<T,S>::poll::{{closure}} (ptr=0x7fffdd652c38) at /home/frank/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.22/src/runtime/task/core.rs:173
#8  0x00005555559facd7 in tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut (self=0x7fffdd652c38, f=...) at /home/frank/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.22/src/loom/std/unsafe_cell.rs:14
#9  0x00005555558a1323 in tokio::runtime::task::core::Core<T,S>::poll (self=0x7fffdd652c30, header=0x7fffdd652c00) at /home/frank/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.22/src/runtime/task/core.rs:158
#10 0x0000555555839a3c in tokio::runtime::task::harness::Harness<T,S>::poll::{{closure}} () at /home/frank/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.22/src/runtime/task/harness.rs:107
#11 0x000055555599c270 in core::ops::function::FnOnce::call_once () at /rustc/792c645ca7d11a8d254df307d019c5bf01445c37/library/core/src/ops/function.rs:233
#12 0x000055555595f65a in <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once (self=..., _args=()) at /rustc/792c645ca7d11a8d254df307d019c5bf01445c37/library/std/src/panic.rs:318
#13 0x0000555555a35c4d in std::panicking::try::do_call (data=0x7ffff43f9850 "ș?\364\377\177\000") at /rustc/792c645ca7d11a8d254df307d019c5bf01445c37/library/std/src/panicking.rs:373
#14 0x0000555555a3a75d in __rust_try ()
#15 0x0000555555a32b75 in std::panicking::try (f=...) at /rustc/792c645ca7d11a8d254df307d019c5bf01445c37/library/std/src/panicking.rs:337
#16 0x00005555559610aa in std::panic::catch_unwind (f=...) at /rustc/792c645ca7d11a8d254df307d019c5bf01445c37/library/std/src/panic.rs:394
#17 0x0000555555829c89 in tokio::runtime::task::harness::Harness<T,S>::poll (self=...) at /home/frank/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.22/src/runtime/task/harness.rs:89
#18 0x00005555559de320 in tokio::runtime::task::raw::poll (ptr=...) at /home/frank/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.22/src/runtime/task/raw.rs:104
#19 0x000055555755894f in tokio::runtime::task::raw::RawTask::poll (self=...) at /home/frank/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.22/src/runtime/task/raw.rs:66
#20 0x0000555557531601 in tokio::runtime::task::Notified<S>::run (self=...) at /home/frank/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.22/src/runtime/task/mod.rs:169
#21 0x0000555557544042 in tokio::runtime::thread_pool::worker::Context::run_task::{{closure}} () at /home/frank/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.22/src/runtime/thread_pool/worker.rs:353
#22 0x0000555557524546 in tokio::coop::with_budget::{{closure}} (cell=0x7ffff43fd2ca) at /home/frank/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.22/src/coop.rs:127
#23 0x00005555575488b0 in std::thread::local::LocalKey<T>::try_with (self=0x55555809b128, f=...) at /rustc/792c645ca7d11a8d254df307d019c5bf01445c37/library/std/src/thread/local.rs:267
#24 0x0000555557547b7e in std::thread::local::LocalKey<T>::with (self=0x55555809b128, f=...) at /rustc/792c645ca7d11a8d254df307d019c5bf01445c37/library/std/src/thread/local.rs:243
#25 0x0000555557543ea4 in tokio::coop::with_budget (budget=..., f=...) at /home/frank/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.22/src/coop.rs:120
#26 tokio::coop::budget (f=...) at /home/frank/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.22/src/coop.rs:96
#27 tokio::runtime::thread_pool::worker::Context::run_task (self=0x7ffff43fa138, task=..., core=0x7ffff7617640) at /home/frank/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.22/src/runtime/thread_pool/worker.rs:352
#28 0x0000555557543856 in tokio::runtime::thread_pool::worker::Context::run (self=0x7ffff43fa138, core=0x7ffff7617640) at /home/frank/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.22/src/runtime/thread_pool/worker.rs:324
#29 0x0000555557543643 in tokio::runtime::thread_pool::worker::run::{{closure}} () at /home/frank/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.22/src/runtime/thread_pool/worker.rs:309
#30 0x0000555557523426 in tokio::macros::scoped_tls::ScopedKey<T>::set (self=0x55555809b550 <tokio::runtime::thread_pool::worker::CURRENT>, t=0x7ffff43fa138, f=...)
    at /home/frank/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.22/src/macros/scoped_tls.rs:63
#31 0x000055555754353f in tokio::runtime::thread_pool::worker::run (worker=Arc(strong=8, weak=0) = {...}) at /home/frank/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.22/src/runtime/thread_pool/worker.rs:306
#32 0x0000555555b438db in tokio::runtime::thread_pool::worker::block_in_place::{{closure}}::{{closure}} () at /home/frank/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.22/src/runtime/thread_pool/worker.rs:259
#33 0x00005555558d372b in <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll (self=..., _cx=0x7ffff43fa2b0)
    at /home/frank/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.22/src/runtime/blocking/task.rs:41
#34 0x00005555558a3e79 in tokio::runtime::task::core::Core<T,S>::poll::{{closure}} (ptr=0x7ffff7636130) at /home/frank/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.22/src/runtime/task/core.rs:173
#35 0x00005555559fdc77 in tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut (self=0x7ffff7636130, f=...) at /home/frank/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.22/src/loom/std/unsafe_cell.rs:14
#36 0x00005555558a1dcd in tokio::runtime::task::core::Core<T,S>::poll (self=0x7ffff7636130, header=0x7ffff7636100) at /home/frank/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.22/src/runtime/task/core.rs:158
#37 0x000055555583c79c in tokio::runtime::task::harness::Harness<T,S>::poll::{{closure}} () at /home/frank/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.22/src/runtime/task/harness.rs:107
#38 0x000055555599ca70 in core::ops::function::FnOnce::call_once () at /rustc/792c645ca7d11a8d254df307d019c5bf01445c37/library/core/src/ops/function.rs:233
#39 0x000055555595fb5a in <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once (self=..., _args=()) at /rustc/792c645ca7d11a8d254df307d019c5bf01445c37/library/std/src/panic.rs:318
#40 0x0000555555a3703d in std::panicking::try::do_call (data=0x7ffff43fa630 "\250\247?\364\377\177\000") at /rustc/792c645ca7d11a8d254df307d019c5bf01445c37/library/std/src/panicking.rs:373
#41 0x0000555555a3a75d in __rust_try ()
#42 0x0000555555a300b5 in std::panicking::try (f=...) at /rustc/792c645ca7d11a8d254df307d019c5bf01445c37/library/std/src/panicking.rs:337
#43 0x0000555555960c2a in std::panic::catch_unwind (f=...) at /rustc/792c645ca7d11a8d254df307d019c5bf01445c37/library/std/src/panic.rs:394
#44 0x0000555555831159 in tokio::runtime::task::harness::Harness<T,S>::poll (self=...) at /home/frank/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.22/src/runtime/task/harness.rs:89
#45 0x00005555559de5f0 in tokio::runtime::task::raw::poll (ptr=...) at /home/frank/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.22/src/runtime/task/raw.rs:104
#46 0x000055555755894f in tokio::runtime::task::raw::RawTask::poll (self=...) at /home/frank/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.22/src/runtime/task/raw.rs:66
#47 0x0000555557531671 in tokio::runtime::task::Notified<S>::run (self=...) at /home/frank/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.22/src/runtime/task/mod.rs:169
#48 0x000055555755a0f3 in tokio::runtime::blocking::pool::Inner::run (self=0x7ffff763a250) at /home/frank/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.22/src/runtime/blocking/pool.rs:230
#49 0x0000555557559e3e in tokio::runtime::blocking::pool::Spawner::spawn_thread::{{closure}}::{{closure}} () at /home/frank/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.22/src/runtime/blocking/pool.rs:210
#50 0x000055555750a2e3 in tokio::runtime::context::enter (new=..., f=...) at /home/frank/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.22/src/runtime/context.rs:72
#51 0x00005555574f80bd in tokio::runtime::handle::Handle::enter (self=0x7ffff43fad58, f=...) at /home/frank/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.22/src/runtime/handle.rs:76
--Type <RET> for more, q to quit, c to continue without paging--
#52 0x0000555557559ecf in tokio::runtime::blocking::pool::Spawner::spawn_thread::{{closure}} () at /home/frank/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.22/src/runtime/blocking/pool.rs:209
#53 0x0000555557507472 in std::sys_common::backtrace::__rust_begin_short_backtrace (f=...) at /rustc/792c645ca7d11a8d254df307d019c5bf01445c37/library/std/src/sys_common/backtrace.rs:137
#54 0x00005555574dda61 in std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}} () at /rustc/792c645ca7d11a8d254df307d019c5bf01445c37/library/std/src/thread/mod.rs:475
#55 0x0000555557511081 in <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once (self=..., _args=()) at /rustc/792c645ca7d11a8d254df307d019c5bf01445c37/library/std/src/panic.rs:318
#56 0x000055555753cad2 in std::panicking::try::do_call (data=0x7ffff43fafb8 "\002\000") at /rustc/792c645ca7d11a8d254df307d019c5bf01445c37/library/std/src/panicking.rs:373
#57 0x00005555575474bd in __rust_try ()
#58 0x000055555753c9b4 in std::panicking::try (f=...) at /rustc/792c645ca7d11a8d254df307d019c5bf01445c37/library/std/src/panicking.rs:337
#59 0x0000555557511143 in std::panic::catch_unwind (f=...) at /rustc/792c645ca7d11a8d254df307d019c5bf01445c37/library/std/src/panic.rs:394
#60 0x00005555574dd859 in std::thread::Builder::spawn_unchecked::{{closure}} () at /rustc/792c645ca7d11a8d254df307d019c5bf01445c37/library/std/src/thread/mod.rs:474
#61 0x000055555752855f in core::ops::function::FnOnce::call_once{{vtable-shim}} () at /rustc/792c645ca7d11a8d254df307d019c5bf01445c37/library/core/src/ops/function.rs:233

#62 0x00005555577f7f7a in <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once () at /rustc/792c645ca7d11a8d254df307d019c5bf01445c37/library/alloc/src/boxed.rs:1025
#63 <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once () at /rustc/792c645ca7d11a8d254df307d019c5bf01445c37/library/alloc/src/boxed.rs:1025
#64 std::sys::unix::thread::Thread::new::thread_start () at library/std/src/sys/unix/thread.rs:87
#65 0x00007ffff7fa6609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#66 0x00007ffff7ca2103 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
(gdb) 

Does this clarify anything?

@jonhoo
Copy link
Collaborator

jonhoo commented Aug 23, 2020

Oh, that is very weird. That should only segfault if you have called the hidden i_promise_dst_is_same_process method, which I don't think you have. My guess is that there's a version mismatch between the different ways in which the noria crate is compiled here. In particular, noria-mysql is probably linked to an older version of noria, which causes something to go awry in the serialization/deserialization. Perhaps try cargo update in the noria-mysql directory before building it?

@flyaruu
Copy link
Author

flyaruu commented Sep 1, 2020

I've tried it on another box. Nothing rust related on it. Rustup-ed, checked out all, did the same steps, with the same result.
cargo update did not seem to help. If there is a version mismatch, it's on the git master branches as well

@jonhoo
Copy link
Collaborator

jonhoo commented Sep 1, 2020

So, just to check:

$ git clone https://github.com/mit-pdos/noria.git noria
$ git clone https://github.com/mit-pdos/noria-mysql.git noria-mysql
$ cd noria && cargo update
$ cd noria-mysql && cargo update

And then you run cargo run --release both binaries?

Can you try patching this function:
https://github.com/mit-pdos/noria/blob/f54dfcef4057b490ccb43f4a31c6aaf35c923fb7/noria/src/internal/proto.rs#L50

in each match arm so that it prints whether the received thing is "local" or not? You'll probably have to modify the patch.crates-io section of noria-mysql/Cargo.toml to point at the local path you've checked out noria into.

@flyaruu
Copy link
Author

flyaruu commented Sep 1, 2020

Ok, deleted all the Noria things, re-build & ran noria (same result). Pointed the noria-mysql dependencies to the local:

-noria = { git = "https://github.com/mit-pdos/noria.git" }
-noria-server = { git = "https://github.com/mit-pdos/noria.git" }
-# noria = { path = "../noria/noria/" }
-# noria-server = { path = "../noria/server/" }
+# noria = { git = "https://github.com/mit-pdos/noria.git" }
+# noria-server = { git = "https://github.com/mit-pdos/noria.git" }
+noria = { path = "../noria/noria/" }
+noria-server = { path = "../noria/server/" }

(same result)

Patched the proto.rs:

     pub unsafe fn deref(&self) -> &T {
         match self {
-            LocalOrNotInner::Local(ref l) => l.deref(),
-            LocalOrNotInner::Not(ref t) => t,
+            LocalOrNotInner::Local(ref l) => { println!("Local!");  l.deref()},
+            LocalOrNotInner::Not(ref t) => { println!("Not!"); t},
         }
     }

Results in:

% cargo r --release --bin noria-server -- --deployment myapp --no-reuse --shards 0
   Compiling noria v0.7.0 (/Users/f.lyaruu/git/noria/noria)
   Compiling noria-common v0.7.0 (/Users/f.lyaruu/git/noria/server/common)
   Compiling noria-dataflow v0.7.0 (/Users/f.lyaruu/git/noria/server/dataflow)
   Compiling noria-mir v0.7.0 (/Users/f.lyaruu/git/noria/server/mir)
   Compiling noria-server v0.7.0 (/Users/f.lyaruu/git/noria/server)
    Finished release [optimized + debuginfo] target(s) in 2m 43s
     Running `target/release/noria-server --deployment myapp --no-reuse --shards 0`
Local!
zsh: segmentation fault  cargo r --release --bin noria-server -- --deployment myapp --no-reuse --shard
% 

And we see local!

@jonhoo
Copy link
Collaborator

jonhoo commented Sep 1, 2020

Thank you for digging! That's bizarre indeed... Local should only be used when for_local_transfer is used:
https://github.com/mit-pdos/noria/blob/f54dfcef4057b490ccb43f4a31c6aaf35c923fb7/noria/src/internal/proto.rs#L105

for_local_transfer is in turn only used for writes when dst_is_local is true:
https://github.com/mit-pdos/noria/blob/f54dfcef4057b490ccb43f4a31c6aaf35c923fb7/noria/src/table.rs#L501-L502
https://github.com/mit-pdos/noria/blob/f54dfcef4057b490ccb43f4a31c6aaf35c923fb7/noria/src/table.rs#L541-L543

dst_is_local defaults to false
https://github.com/mit-pdos/noria/blob/f54dfcef4057b490ccb43f4a31c6aaf35c923fb7/noria/src/table.rs#L383

and is only set to true if i_promise_dst_is_same_process is called:
https://github.com/mit-pdos/noria/blob/f54dfcef4057b490ccb43f4a31c6aaf35c923fb7/noria/src/table.rs#L613-L614

and that is only ever called in the vote and vote-dbtoaster-style binaries, neither of which you are running, and does not appear in noria-mysql. This is what makes me think it must be serialization error somehow --- it should not be possible for a Local to be generated (and therefore it should not be possible for the server to receive one).

Looking at the implementation, this line looks a little questionable, but I also suspect that it probably isn't hit anyway (maybe try adding a print):
https://github.com/mit-pdos/noria/blob/f54dfcef4057b490ccb43f4a31c6aaf35c923fb7/noria/src/internal/proto.rs#L34

Just for my own sanity's sake, could you try removing the tokio patch in noria-mysql/Cargo.toml? It shouldn't make a difference, but worth to double-check.

@flyaruu
Copy link
Author

flyaruu commented Sep 12, 2020

Ok, went back to this one.

I've added some debug around the deserialization code:

        println!("Before deserialize!");
        let result = usize::deserialize(deserializer).map(|p| LocalBypass(p as *mut T));
        println!("After deserialize!");

And it does get hit:

Before deserialize!
After deserialize!
Local!
zsh: segmentation fault

Removing the tokio patch does not lead to discernible effect.
It baffles me that I can reproduce the problem so consistently, yet you can not.
I've tried on two different MacBooks, a Linux pc, and a Linux VM inside a Windows box, with completely consistent results.

I had a theory that it was somehow related to my mysql client (because that was the same accross all my attempts, an up-to-date mysql 8.0.21 client, but somehow an older client 8.0.15 does not connect at all. I also tried a JS based mysql client but that also does not seem to be able to connect at all.

@alexrenda
Copy link

When cloning the latest versions of noria and noria-mysql (with noria-mysql patched to look locally), I got this same issue. I also saw this warning in the Noria-mysql build:

Patch `noria-server v0.7.0 (/home/alex/noria/server)` was not used in the crate graph

I was able to fix this with the following Cargo.toml patch:

12c12
< noria = "0.4"
---
> noria = "0.7"
30c30
< noria-server = "0.4"
---
> noria-server = "0.7"

along with a patch in noria-mysql to not refer to a function that has been removed:

 sed -i 's%noria::trace_my_next_op%// noria::trace_my_next_op%g' src/backend.rs

With this, I'm able to run those example SQL statements without anything crashing

@flyaruu
Copy link
Author

flyaruu commented Oct 25, 2020

Yes, works for me! So I guess it would make sense then to release Noria 0.7, as the master of Noria MySQL depends on it...?
Would it help to set up a ci pipeline for Noria? Issues like the one we just had are really tricky to debug but should be pretty straightforward to detect in a CI pipeline.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants