Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

bug: hummock uploader assertion failure when seal_epoch #7208

Closed
Tracked by #6640
zwang28 opened this issue Jan 5, 2023 · 2 comments · Fixed by #7263
Closed
Tracked by #6640

bug: hummock uploader assertion failure when seal_epoch #7208

zwang28 opened this issue Jan 5, 2023 · 2 comments · Fixed by #7263
Labels
found-by-longevity-test type/bug Something isn't working
Milestone
release-0.1.16

Comments

@zwang28
Copy link
Contributor

zwang28 commented Jan 5, 2023 •
edited
Loading

Describe the bug

Slack link:
https://risingwave-labs.slack.com/archives/C048NM5LNKX/p1672899254269749
Namespace: rwc-3-longevity-20230104-180851
Pod: risingwave-compute-0, risingwave-compute-1, risingwave-compute-2

compute_1.csv
compute_2.csv
compute_3.csv

1672868808671,"{""log"":""2023-01-04T21:46:48.671403148Z stderr F thread 'risingwave-main' panicked at 'some epoch 3646001036656640 older than epoch to seal 3646003135053824', src/storage/src/hummock/event_handler/uploader.rs:442:13"",""kubernetes"":{""pod_name"":""risingwave-compute-1"",""namespace_name"":""rwc-3-longevity-20230104-180851"",""pod_id"":""a3782c08-c034-459e-bd3f-4b90dca776c1"",""host"":""ip-10-0-75-129.ap-southeast-1.compute.internal"",""container_name"":""compute"",""docker_id"":""59ffb6e3f61a217d837642c2956c9cf11ed8c40110ffae987c77f5beccb96b7b"",""container_hash"":""ghcr.io/risingwavelabs/risingwave@sha256:a0b61ebfc9910a161caf40a571530a9add3168528f78cc8bcf26097748441fdf"",""container_image"":""ghcr.io/risingwavelabs/risingwave:nightly-20230103""}}"
1672868808671,"{""log"":""2023-01-04T21:46:48.671438439Z stderr F stack backtrace:"",""kubernetes"":{""pod_name"":""risingwave-compute-1"",""namespace_name"":""rwc-3-longevity-20230104-180851"",""pod_id"":""a3782c08-c034-459e-bd3f-4b90dca776c1"",""host"":""ip-10-0-75-129.ap-southeast-1.compute.internal"",""container_name"":""compute"",""docker_id"":""59ffb6e3f61a217d837642c2956c9cf11ed8c40110ffae987c77f5beccb96b7b"",""container_hash"":""ghcr.io/risingwavelabs/risingwave@sha256:a0b61ebfc9910a161caf40a571530a9add3168528f78cc8bcf26097748441fdf"",""container_image"":""ghcr.io/risingwavelabs/risingwave:nightly-20230103""}}"
1672868808671,"{""log"":""2023-01-04T21:46:48.671443309Z stderr F    0:     0x558b2509ee1a - std::backtrace_rs::backtrace::libunwind::trace::h3bfe63c4aecfa0e2"",""kubernetes"":{""pod_name"":""risingwave-compute-1"",""namespace_name"":""rwc-3-longevity-20230104-180851"",""pod_id"":""a3782c08-c034-459e-bd3f-4b90dca776c1"",""host"":""ip-10-0-75-129.ap-southeast-1.compute.internal"",""container_name"":""compute"",""docker_id"":""59ffb6e3f61a217d837642c2956c9cf11ed8c40110ffae987c77f5beccb96b7b"",""container_hash"":""ghcr.io/risingwavelabs/risingwave@sha256:a0b61ebfc9910a161caf40a571530a9add3168528f78cc8bcf26097748441fdf"",""container_image"":""ghcr.io/risingwavelabs/risingwave:nightly-20230103""}}"
1672868808671,"{""log"":""2023-01-04T21:46:48.671447229Z stderr F                                at /rustc/bdb07a8ec8e77aa10fb84fae1d4ff71c21180bb4/library/std/src/../../backtrace/src/backtrace/libunwind.rs:93:5"",""kubernetes"":{""pod_name"":""risingwave-compute-1"",""namespace_name"":""rwc-3-longevity-20230104-180851"",""pod_id"":""a3782c08-c034-459e-bd3f-4b90dca776c1"",""host"":""ip-10-0-75-129.ap-southeast-1.compute.internal"",""container_name"":""compute"",""docker_id"":""59ffb6e3f61a217d837642c2956c9cf11ed8c40110ffae987c77f5beccb96b7b"",""container_hash"":""ghcr.io/risingwavelabs/risingwave@sha256:a0b61ebfc9910a161caf40a571530a9add3168528f78cc8bcf26097748441fdf"",""container_image"":""ghcr.io/risingwavelabs/risingwave:nightly-20230103""}}"
1672868808671,"{""log"":""2023-01-04T21:46:48.671457079Z stderr F    1:     0x558b2509ee1a - std::backtrace_rs::backtrace::trace_unsynchronized::h978c02637c6cb8f9"",""kubernetes"":{""pod_name"":""risingwave-compute-1"",""namespace_name"":""rwc-3-longevity-20230104-180851"",""pod_id"":""a3782c08-c034-459e-bd3f-4b90dca776c1"",""host"":""ip-10-0-75-129.ap-southeast-1.compute.internal"",""container_name"":""compute"",""docker_id"":""59ffb6e3f61a217d837642c2956c9cf11ed8c40110ffae987c77f5beccb96b7b"",""container_hash"":""ghcr.io/risingwavelabs/risingwave@sha256:a0b61ebfc9910a161caf40a571530a9add3168528f78cc8bcf26097748441fdf"",""container_image"":""ghcr.io/risingwavelabs/risingwave:nightly-20230103""}}"
1672868808671,"{""log"":""2023-01-04T21:46:48.671467549Z stderr F                                at /rustc/bdb07a8ec8e77aa10fb84fae1d4ff71c21180bb4/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5"",""kubernetes"":{""pod_name"":""risingwave-compute-1"",""namespace_name"":""rwc-3-longevity-20230104-180851"",""pod_id"":""a3782c08-c034-459e-bd3f-4b90dca776c1"",""host"":""ip-10-0-75-129.ap-southeast-1.compute.internal"",""container_name"":""compute"",""docker_id"":""59ffb6e3f61a217d837642c2956c9cf11ed8c40110ffae987c77f5beccb96b7b"",""container_hash"":""ghcr.io/risingwavelabs/risingwave@sha256:a0b61ebfc9910a161caf40a571530a9add3168528f78cc8bcf26097748441fdf"",""container_image"":""ghcr.io/risingwavelabs/risingwave:nightly-20230103""}}"
1672868808671,"{""log"":""2023-01-04T21:46:48.671471129Z stderr F    2:     0x558b2509ee1a - std::sys_common::backtrace::_print_fmt::h86a708dcb1665377"",""kubernetes"":{""pod_name"":""risingwave-compute-1"",""namespace_name"":""rwc-3-longevity-20230104-180851"",""pod_id"":""a3782c08-c034-459e-bd3f-4b90dca776c1"",""host"":""ip-10-0-75-129.ap-southeast-1.compute.internal"",""container_name"":""compute"",""docker_id"":""59ffb6e3f61a217d837642c2956c9cf11ed8c40110ffae987c77f5beccb96b7b"",""container_hash"":""ghcr.io/risingwavelabs/risingwave@sha256:a0b61ebfc9910a161caf40a571530a9add3168528f78cc8bcf26097748441fdf"",""container_image"":""ghcr.io/risingwavelabs/risingwave:nightly-20230103""}}"
1672868808671,"{""log"":""2023-01-04T21:46:48.671474149Z stderr F                                at /rustc/bdb07a8ec8e77aa10fb84fae1d4ff71c21180bb4/library/std/src/sys_common/backtrace.rs:65:5"",""kubernetes"":{""pod_name"":""risingwave-compute-1"",""namespace_name"":""rwc-3-longevity-20230104-180851"",""pod_id"":""a3782c08-c034-459e-bd3f-4b90dca776c1"",""host"":""ip-10-0-75-129.ap-southeast-1.compute.internal"",""container_name"":""compute"",""docker_id"":""59ffb6e3f61a217d837642c2956c9cf11ed8c40110ffae987c77f5beccb96b7b"",""container_hash"":""ghcr.io/risingwavelabs/risingwave@sha256:a0b61ebfc9910a161caf40a571530a9add3168528f78cc8bcf26097748441fdf"",""container_image"":""ghcr.io/risingwavelabs/risingwave:nightly-20230103""}}"
1672868808671,"{""log"":""2023-01-04T21:46:48.671477919Z stderr F    3:     0x558b2509ee1a - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::hdbf89765373fb801"",""kubernetes"":{""pod_name"":""risingwave-compute-1"",""namespace_name"":""rwc-3-longevity-20230104-180851"",""pod_id"":""a3782c08-c034-459e-bd3f-4b90dca776c1"",""host"":""ip-10-0-75-129.ap-southeast-1.compute.internal"",""container_name"":""compute"",""docker_id"":""59ffb6e3f61a217d837642c2956c9cf11ed8c40110ffae987c77f5beccb96b7b"",""container_hash"":""ghcr.io/risingwavelabs/risingwave@sha256:a0b61ebfc9910a161caf40a571530a9add3168528f78cc8bcf26097748441fdf"",""container_image"":""ghcr.io/risingwavelabs/risingwave:nightly-20230103""}}"
1672868808671,"{""log"":""2023-01-04T21:46:48.67149904Z stderr F                                at /rustc/bdb07a8ec8e77aa10fb84fae1d4ff71c21180bb4/library/std/src/sys_common/backtrace.rs:44:22"",""kubernetes"":{""pod_name"":""risingwave-compute-1"",""namespace_name"":""rwc-3-longevity-20230104-180851"",""pod_id"":""a3782c08-c034-459e-bd3f-4b90dca776c1"",""host"":""ip-10-0-75-129.ap-southeast-1.compute.internal"",""container_name"":""compute"",""docker_id"":""59ffb6e3f61a217d837642c2956c9cf11ed8c40110ffae987c77f5beccb96b7b"",""container_hash"":""ghcr.io/risingwavelabs/risingwave@sha256:a0b61ebfc9910a161caf40a571530a9add3168528f78cc8bcf26097748441fdf"",""container_image"":""ghcr.io/risingwavelabs/risingwave:nightly-20230103""}}"
1672868808675,"{""log"":""2023-01-04T21:46:48.675161645Z stderr F    4:     0x558b250cb96e - core::fmt::write::hbf11a3bb4e6d2aba"",""kubernetes"":{""pod_name"":""risingwave-compute-1"",""namespace_name"":""rwc-3-longevity-20230104-180851"",""pod_id"":""a3782c08-c034-459e-bd3f-4b90dca776c1"",""host"":""ip-10-0-75-129.ap-southeast-1.compute.internal"",""container_name"":""compute"",""docker_id"":""59ffb6e3f61a217d837642c2956c9cf11ed8c40110ffae987c77f5beccb96b7b"",""container_hash"":""ghcr.io/risingwavelabs/risingwave@sha256:a0b61ebfc9910a161caf40a571530a9add3168528f78cc8bcf26097748441fdf"",""container_image"":""ghcr.io/risingwavelabs/risingwave:nightly-20230103""}}"
1672868808675,"{""log"":""2023-01-04T21:46:48.675185325Z stderr F                                at /rustc/bdb07a8ec8e77aa10fb84fae1d4ff71c21180bb4/library/core/src/fmt/mod.rs:1208:17"",""kubernetes"":{""pod_name"":""risingwave-compute-1"",""namespace_name"":""rwc-3-longevity-20230104-180851"",""pod_id"":""a3782c08-c034-459e-bd3f-4b90dca776c1"",""host"":""ip-10-0-75-129.ap-southeast-1.compute.internal"",""container_name"":""compute"",""docker_id"":""59ffb6e3f61a217d837642c2956c9cf11ed8c40110ffae987c77f5beccb96b7b"",""container_hash"":""ghcr.io/risingwavelabs/risingwave@sha256:a0b61ebfc9910a161caf40a571530a9add3168528f78cc8bcf26097748441fdf"",""container_image"":""ghcr.io/risingwavelabs/risingwave:nightly-20230103""}}"
1672868808675,"{""log"":""2023-01-04T21:46:48.675188825Z stderr F    5:     0x558b25096ae5 - std::io::Write::write_fmt::hd68261eacc348bbd"",""kubernetes"":{""pod_name"":""risingwave-compute-1"",""namespace_name"":""rwc-3-longevity-20230104-180851"",""pod_id"":""a3782c08-c034-459e-bd3f-4b90dca776c1"",""host"":""ip-10-0-75-129.ap-southeast-1.compute.internal"",""container_name"":""compute"",""docker_id"":""59ffb6e3f61a217d837642c2956c9cf11ed8c40110ffae987c77f5beccb96b7b"",""container_hash"":""ghcr.io/risingwavelabs/risingwave@sha256:a0b61ebfc9910a161caf40a571530a9add3168528f78cc8bcf26097748441fdf"",""container_image"":""ghcr.io/risingwavelabs/risingwave:nightly-20230103""}}"
1672868808675,"{""log"":""2023-01-04T21:46:48.675204476Z stderr F                                at /rustc/bdb07a8ec8e77aa10fb84fae1d4ff71c21180bb4/library/std/src/io/mod.rs:1682:15"",""kubernetes"":{""pod_name"":""risingwave-compute-1"",""namespace_name"":""rwc-3-longevity-20230104-180851"",""pod_id"":""a3782c08-c034-459e-bd3f-4b90dca776c1"",""host"":""ip-10-0-75-129.ap-southeast-1.compute.internal"",""container_name"":""compute"",""docker_id"":""59ffb6e3f61a217d837642c2956c9cf11ed8c40110ffae987c77f5beccb96b7b"",""container_hash"":""ghcr.io/risingwavelabs/risingwave@sha256:a0b61ebfc9910a161caf40a571530a9add3168528f78cc8bcf26097748441fdf"",""container_image"":""ghcr.io/risingwavelabs/risingwave:nightly-20230103""}}"
1672868808675,"{""log"":""2023-01-04T21:46:48.675213716Z stderr F    6:     0x558b2509ebe5 - std::sys_common::backtrace::_print::ha8e375b58cdeca4a"",""kubernetes"":{""pod_name"":""risingwave-compute-1"",""namespace_name"":""rwc-3-longevity-20230104-180851"",""pod_id"":""a3782c08-c034-459e-bd3f-4b90dca776c1"",""host"":""ip-10-0-75-129.ap-southeast-1.compute.internal"",""container_name"":""compute"",""docker_id"":""59ffb6e3f61a217d837642c2956c9cf11ed8c40110ffae987c77f5beccb96b7b"",""container_hash"":""ghcr.io/risingwavelabs/risingwave@sha256:a0b61ebfc9910a161caf40a571530a9add3168528f78cc8bcf26097748441fdf"",""container_image"":""ghcr.io/risingwavelabs/risingwave:nightly-20230103""}}"
1672868808675,"{""log"":""2023-01-04T21:46:48.675216956Z stderr F                                at /rustc/bdb07a8ec8e77aa10fb84fae1d4ff71c21180bb4/library/std/src/sys_common/backtrace.rs:47:5"",""kubernetes"":{""pod_name"":""risingwave-compute-1"",""namespace_name"":""rwc-3-longevity-20230104-180851"",""pod_id"":""a3782c08-c034-459e-bd3f-4b90dca776c1"",""host"":""ip-10-0-75-129.ap-southeast-1.compute.internal"",""container_name"":""compute"",""docker_id"":""59ffb6e3f61a217d837642c2956c9cf11ed8c40110ffae987c77f5beccb96b7b"",""container_hash"":""ghcr.io/risingwavelabs/risingwave@sha256:a0b61ebfc9910a161caf40a571530a9add3168528f78cc8bcf26097748441fdf"",""container_image"":""ghcr.io/risingwavelabs/risingwave:nightly-20230103""}}"
1672868808675,"{""log"":""2023-01-04T21:46:48.675250707Z stderr F    7:     0x558b2509ebe5 - std::sys_common::backtrace::print::he3086c07df1167ca"",""kubernetes"":{""pod_name"":""risingwave-compute-1"",""namespace_name"":""rwc-3-longevity-20230104-180851"",""pod_id"":""a3782c08-c034-459e-bd3f-4b90dca776c1"",""host"":""ip-10-0-75-129.ap-southeast-1.compute.internal"",""container_name"":""compute"",""docker_id"":""59ffb6e3f61a217d837642c2956c9cf11ed8c40110ffae987c77f5beccb96b7b"",""container_hash"":""ghcr.io/risingwavelabs/risingwave@sha256:a0b61ebfc9910a161caf40a571530a9add3168528f78cc8bcf26097748441fdf"",""container_image"":""ghcr.io/risingwavelabs/risingwave:nightly-20230103""}}"
1672868808675,"{""log"":""2023-01-04T21:46:48.675257597Z stderr F                                at /rustc/bdb07a8ec8e77aa10fb84fae1d4ff71c21180bb4/library/std/src/sys_common/backtrace.rs:34:9"",""kubernetes"":{""pod_name"":""risingwave-compute-1"",""namespace_name"":""rwc-3-longevity-20230104-180851"",""pod_id"":""a3782c08-c034-459e-bd3f-4b90dca776c1"",""host"":""ip-10-0-75-129.ap-southeast-1.compute.internal"",""container_name"":""compute"",""docker_id"":""59ffb6e3f61a217d837642c2956c9cf11ed8c40110ffae987c77f5beccb96b7b"",""container_hash"":""ghcr.io/risingwavelabs/risingwave@sha256:a0b61ebfc9910a161caf40a571530a9add3168528f78cc8bcf26097748441fdf"",""container_image"":""ghcr.io/risingwavelabs/risingwave:nightly-20230103""}}"
1672868808675,"{""log"":""2023-01-04T21:46:48.675266877Z stderr F    8:     0x558b250a036f - std::panicking::default_hook::{{closure}}::h0c12a9095c948e43"",""kubernetes"":{""pod_name"":""risingwave-compute-1"",""namespace_name"":""rwc-3-longevity-20230104-180851"",""pod_id"":""a3782c08-c034-459e-bd3f-4b90dca776c1"",""host"":""ip-10-0-75-129.ap-southeast-1.compute.internal"",""container_name"":""compute"",""docker_id"":""59ffb6e3f61a217d837642c2956c9cf11ed8c40110ffae987c77f5beccb96b7b"",""container_hash"":""ghcr.io/risingwavelabs/risingwave@sha256:a0b61ebfc9910a161caf40a571530a9add3168528f78cc8bcf26097748441fdf"",""container_image"":""ghcr.io/risingwavelabs/risingwave:nightly-20230103""}}"
1672868808675,"{""log"":""2023-01-04T21:46:48.675290918Z stderr F                                at /rustc/bdb07a8ec8e77aa10fb84fae1d4ff71c21180bb4/library/std/src/panicking.rs:267:22"",""kubernetes"":{""pod_name"":""risingwave-compute-1"",""namespace_name"":""rwc-3-longevity-20230104-180851"",""pod_id"":""a3782c08-c034-459e-bd3f-4b90dca776c1"",""host"":""ip-10-0-75-129.ap-southeast-1.compute.internal"",""container_name"":""compute"",""docker_id"":""59ffb6e3f61a217d837642c2956c9cf11ed8c40110ffae987c77f5beccb96b7b"",""container_hash"":""ghcr.io/risingwavelabs/risingwave@sha256:a0b61ebfc9910a161caf40a571530a9add3168528f78cc8bcf26097748441fdf"",""container_image"":""ghcr.io/risingwavelabs/risingwave:nightly-20230103""}}"
1672868808675,"{""log"":""2023-01-04T21:46:48.67539671Z stderr F    9:     0x558b250a00ab - std::panicking::default_hook::h1ae5478ad617eb24"",""kubernetes"":{""pod_name"":""risingwave-compute-1"",""namespace_name"":""rwc-3-longevity-20230104-180851"",""pod_id"":""a3782c08-c034-459e-bd3f-4b90dca776c1"",""host"":""ip-10-0-75-129.ap-southeast-1.compute.internal"",""container_name"":""compute"",""docker_id"":""59ffb6e3f61a217d837642c2956c9cf11ed8c40110ffae987c77f5beccb96b7b"",""container_hash"":""ghcr.io/risingwavelabs/risingwave@sha256:a0b61ebfc9910a161caf40a571530a9add3168528f78cc8bcf26097748441fdf"",""container_image"":""ghcr.io/risingwavelabs/risingwave:nightly-20230103""}}"
1672868808675,"{""log"":""2023-01-04T21:46:48.6754123Z stderr F                                at /rustc/bdb07a8ec8e77aa10fb84fae1d4ff71c21180bb4/library/std/src/panicking.rs:286:9"",""kubernetes"":{""pod_name"":""risingwave-compute-1"",""namespace_name"":""rwc-3-longevity-20230104-180851"",""pod_id"":""a3782c08-c034-459e-bd3f-4b90dca776c1"",""host"":""ip-10-0-75-129.ap-southeast-1.compute.internal"",""container_name"":""compute"",""docker_id"":""59ffb6e3f61a217d837642c2956c9cf11ed8c40110ffae987c77f5beccb96b7b"",""container_hash"":""ghcr.io/risingwavelabs/risingwave@sha256:a0b61ebfc9910a161caf40a571530a9add3168528f78cc8bcf26097748441fdf"",""container_image"":""ghcr.io/risingwavelabs/risingwave:nightly-20230103""}}"
1672868808677,"{""log"":""2023-01-04T21:46:48.677174736Z stderr F   10:     0x558b20500222 - <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call::h67ef0af2d71063c8"",""kubernetes"":{""pod_name"":""risingwave-compute-1"",""namespace_name"":""rwc-3-longevity-20230104-180851"",""pod_id"":""a3782c08-c034-459e-bd3f-4b90dca776c1"",""host"":""ip-10-0-75-129.ap-southeast-1.compute.internal"",""container_name"":""compute"",""docker_id"":""59ffb6e3f61a217d837642c2956c9cf11ed8c40110ffae987c77f5beccb96b7b"",""container_hash"":""ghcr.io/risingwavelabs/risingwave@sha256:a0b61ebfc9910a161caf40a571530a9add3168528f78cc8bcf26097748441fdf"",""container_image"":""ghcr.io/risingwavelabs/risingwave:nightly-20230103""}}"
1672868808677,"{""log"":""2023-01-04T21:46:48.677193097Z stderr F                                at /rustc/bdb07a8ec8e77aa10fb84fae1d4ff71c21180bb4/library/alloc/src/boxed.rs:2032:9"",""kubernetes"":{""pod_name"":""risingwave-compute-1"",""namespace_name"":""rwc-3-longevity-20230104-180851"",""pod_id"":""a3782c08-c034-459e-bd3f-4b90dca776c1"",""host"":""ip-10-0-75-129.ap-southeast-1.compute.internal"",""container_name"":""compute"",""docker_id"":""59ffb6e3f61a217d837642c2956c9cf11ed8c40110ffae987c77f5beccb96b7b"",""container_hash"":""ghcr.io/risingwavelabs/risingwave@sha256:a0b61ebfc9910a161caf40a571530a9add3168528f78cc8bcf26097748441fdf"",""container_image"":""ghcr.io/risingwavelabs/risingwave:nightly-20230103""}}"
1672868808677,"{""log"":""2023-01-04T21:46:48.677196417Z stderr F   11:     0x558b20500222 - risingwave_rt::set_panic_hook::{{closure}}::h96b4ca6e6434dde3"",""kubernetes"":{""pod_name"":""risingwave-compute-1"",""namespace_name"":""rwc-3-longevity-20230104-180851"",""pod_id"":""a3782c08-c034-459e-bd3f-4b90dca776c1"",""host"":""ip-10-0-75-129.ap-southeast-1.compute.internal"",""container_name"":""compute"",""docker_id"":""59ffb6e3f61a217d837642c2956c9cf11ed8c40110ffae987c77f5beccb96b7b"",""container_hash"":""ghcr.io/risingwavelabs/risingwave@sha256:a0b61ebfc9910a161caf40a571530a9add3168528f78cc8bcf26097748441fdf"",""container_image"":""ghcr.io/risingwavelabs/risingwave:nightly-20230103""}}"
1672868808677,"{""log"":""2023-01-04T21:46:48.677238258Z stderr F                                at /risingwave/src/utils/runtime/src/lib.rs:99:9"",""kubernetes"":{""pod_name"":""risingwave-compute-1"",""namespace_name"":""rwc-3-longevity-20230104-180851"",""pod_id"":""a3782c08-c034-459e-bd3f-4b90dca776c1"",""host"":""ip-10-0-75-129.ap-southeast-1.compute.internal"",""container_name"":""compute"",""docker_id"":""59ffb6e3f61a217d837642c2956c9cf11ed8c40110ffae987c77f5beccb96b7b"",""container_hash"":""ghcr.io/risingwavelabs/risingwave@sha256:a0b61ebfc9910a161caf40a571530a9add3168528f78cc8bcf26097748441fdf"",""container_image"":""ghcr.io/risingwavelabs/risingwave:nightly-20230103""}}"
1672868808677,"{""log"":""2023-01-04T21:46:48.677244298Z stderr F   12:     0x558b20500222 - std::panicking::update_hook::{{closure}}::h09d2b6d9a56b274a"",""kubernetes"":{""pod_name"":""risingwave-compute-1"",""namespace_name"":""rwc-3-longevity-20230104-180851"",""pod_id"":""a3782c08-c034-459e-bd3f-4b90dca776c1"",""host"":""ip-10-0-75-129.ap-southeast-1.compute.internal"",""container_name"":""compute"",""docker_id"":""59ffb6e3f61a217d837642c2956c9cf11ed8c40110ffae987c77f5beccb96b7b"",""container_hash"":""ghcr.io/risingwavelabs/risingwave@sha256:a0b61ebfc9910a161caf40a571530a9add3168528f78cc8bcf26097748441fdf"",""container_image"":""ghcr.io/risingwavelabs/risingwave:nightly-20230103""}}"
1672868808677,"{""log"":""2023-01-04T21:46:48.677246838Z stderr F                                at /rustc/bdb07a8ec8e77aa10fb84fae1d4ff71c21180bb4/library/std/src/panicking.rs:232:47"",""kubernetes"":{""pod_name"":""risingwave-compute-1"",""namespace_name"":""rwc-3-longevity-20230104-180851"",""pod_id"":""a3782c08-c034-459e-bd3f-4b90dca776c1"",""host"":""ip-10-0-75-129.ap-southeast-1.compute.internal"",""container_name"":""compute"",""docker_id"":""59ffb6e3f61a217d837642c2956c9cf11ed8c40110ffae987c77f5beccb96b7b"",""container_hash"":""ghcr.io/risingwavelabs/risingwave@sha256:a0b61ebfc9910a161caf40a571530a9add3168528f78cc8bcf26097748441fdf"",""container_image"":""ghcr.io/risingwavelabs/risingwave:nightly-20230103""}}"
1672868808677,"{""log"":""2023-01-04T21:46:48.677290099Z stderr F   13:     0x558b250a0afd - <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call::h237cca5e72caa7e7"",""kubernetes"":{""pod_name"":""risingwave-compute-1"",""namespace_name"":""rwc-3-longevity-20230104-180851"",""pod_id"":""a3782c08-c034-459e-bd3f-4b90dca776c1"",""host"":""ip-10-0-75-129.ap-southeast-1.compute.internal"",""container_name"":""compute"",""docker_id"":""59ffb6e3f61a217d837642c2956c9cf11ed8c40110ffae987c77f5beccb96b7b"",""container_hash"":""ghcr.io/risingwavelabs/risingwave@sha256:a0b61ebfc9910a161caf40a571530a9add3168528f78cc8bcf26097748441fdf"",""container_image"":""ghcr.io/risingwavelabs/risingwave:nightly-20230103""}}"
1672868808677,"{""log"":""2023-01-04T21:46:48.677299309Z stderr F                                at /rustc/bdb07a8ec8e77aa10fb84fae1d4ff71c21180bb4/library/alloc/src/boxed.rs:2032:9"",""kubernetes"":{""pod_name"":""risingwave-compute-1"",""namespace_name"":""rwc-3-longevity-20230104-180851"",""pod_id"":""a3782c08-c034-459e-bd3f-4b90dca776c1"",""host"":""ip-10-0-75-129.ap-southeast-1.compute.internal"",""container_name"":""compute"",""docker_id"":""59ffb6e3f61a217d837642c2956c9cf11ed8c40110ffae987c77f5beccb96b7b"",""container_hash"":""ghcr.io/risingwavelabs/risingwave@sha256:a0b61ebfc9910a161caf40a571530a9add3168528f78cc8bcf26097748441fdf"",""container_image"":""ghcr.io/risingwavelabs/risingwave:nightly-20230103""}}"
1672868808677,"{""log"":""2023-01-04T21:46:48.677302849Z stderr F   14:     0x558b250a0afd - std::panicking::rust_panic_with_hook::h63a24f957ccb5d93"",""kubernetes"":{""pod_name"":""risingwave-compute-1"",""namespace_name"":""rwc-3-longevity-20230104-180851"",""pod_id"":""a3782c08-c034-459e-bd3f-4b90dca776c1"",""host"":""ip-10-0-75-129.ap-southeast-1.compute.internal"",""container_name"":""compute"",""docker_id"":""59ffb6e3f61a217d837642c2956c9cf11ed8c40110ffae987c77f5beccb96b7b"",""container_hash"":""ghcr.io/risingwavelabs/risingwave@sha256:a0b61ebfc9910a161caf40a571530a9add3168528f78cc8bcf26097748441fdf"",""container_image"":""ghcr.io/risingwavelabs/risingwave:nightly-20230103""}}"
1672868808677,"{""log"":""2023-01-04T21:46:48.677305349Z stderr F                                at /rustc/bdb07a8ec8e77aa10fb84fae1d4ff71c21180bb4/library/std/src/panicking.rs:692:13"",""kubernetes"":{""pod_name"":""risingwave-compute-1"",""namespace_name"":""rwc-3-longevity-20230104-180851"",""pod_id"":""a3782c08-c034-459e-bd3f-4b90dca776c1"",""host"":""ip-10-0-75-129.ap-southeast-1.compute.internal"",""container_name"":""compute"",""docker_id"":""59ffb6e3f61a217d837642c2956c9cf11ed8c40110ffae987c77f5beccb96b7b"",""container_hash"":""ghcr.io/risingwavelabs/risingwave@sha256:a0b61ebfc9910a161caf40a571530a9add3168528f78cc8bcf26097748441fdf"",""container_image"":""ghcr.io/risingwavelabs/risingwave:nightly-20230103""}}"
1672868808677,"{""log"":""2023-01-04T21:46:48.67734246Z stderr F   15:     0x558b250a0879 - std::panicking::begin_panic_handler::{{closure}}::h1499194b322f3b28"",""kubernetes"":{""pod_name"":""risingwave-compute-1"",""namespace_name"":""rwc-3-longevity-20230104-180851"",""pod_id"":""a3782c08-c034-459e-bd3f-4b90dca776c1"",""host"":""ip-10-0-75-129.ap-southeast-1.compute.internal"",""container_name"":""compute"",""docker_id"":""59ffb6e3f61a217d837642c2956c9cf11ed8c40110ffae987c77f5beccb96b7b"",""container_hash"":""ghcr.io/risingwavelabs/risingwave@sha256:a0b61ebfc9910a161caf40a571530a9add3168528f78cc8bcf26097748441fdf"",""container_image"":""ghcr.io/risingwavelabs/risingwave:nightly-20230103""}}"
1672868808677,"{""log"":""2023-01-04T21:46:48.67735292Z stderr F                                at /rustc/bdb07a8ec8e77aa10fb84fae1d4ff71c21180bb4/library/std/src/panicking.rs:579:13"",""kubernetes"":{""pod_name"":""risingwave-compute-1"",""namespace_name"":""rwc-3-longevity-20230104-180851"",""pod_id"":""a3782c08-c034-459e-bd3f-4b90dca776c1"",""host"":""ip-10-0-75-129.ap-southeast-1.compute.internal"",""container_name"":""compute"",""docker_id"":""59ffb6e3f61a217d837642c2956c9cf11ed8c40110ffae987c77f5beccb96b7b"",""container_hash"":""ghcr.io/risingwavelabs/risingwave@sha256:a0b61ebfc9910a161caf40a571530a9add3168528f78cc8bcf26097748441fdf"",""container_image"":""ghcr.io/risingwavelabs/risingwave:nightly-20230103""}}"
1672868808677,"{""log"":""2023-01-04T21:46:48.67736322Z stderr F   16:     0x558b2509f2cc - std::sys_common::backtrace::__rust_end_short_backtrace::h5569dab7b5c29ca8"",""kubernetes"":{""pod_name"":""risingwave-compute-1"",""namespace_name"":""rwc-3-longevity-20230104-180851"",""pod_id"":""a3782c08-c034-459e-bd3f-4b90dca776c1"",""host"":""ip-10-0-75-129.ap-southeast-1.compute.internal"",""container_name"":""compute"",""docker_id"":""59ffb6e3f61a217d837642c2956c9cf11ed8c40110ffae987c77f5beccb96b7b"",""container_hash"":""ghcr.io/risingwavelabs/risingwave@sha256:a0b61ebfc9910a161caf40a571530a9add3168528f78cc8bcf26097748441fdf"",""container_image"":""ghcr.io/risingwavelabs/risingwave:nightly-20230103""}}"
1672868808677,"{""log"":""2023-01-04T21:46:48.6773682Z stderr F                                at /rustc/bdb07a8ec8e77aa10fb84fae1d4ff71c21180bb4/library/std/src/sys_common/backtrace.rs:137:18"",""kubernetes"":{""pod_name"":""risingwave-compute-1"",""namespace_name"":""rwc-3-longevity-20230104-180851"",""pod_id"":""a3782c08-c034-459e-bd3f-4b90dca776c1"",""host"":""ip-10-0-75-129.ap-southeast-1.compute.internal"",""container_name"":""compute"",""docker_id"":""59ffb6e3f61a217d837642c2956c9cf11ed8c40110ffae987c77f5beccb96b7b"",""container_hash"":""ghcr.io/risingwavelabs/risingwave@sha256:a0b61ebfc9910a161caf40a571530a9add3168528f78cc8bcf26097748441fdf"",""container_image"":""ghcr.io/risingwavelabs/risingwave:nightly-20230103""}}"
1672868808677,"{""log"":""2023-01-04T21:46:48.67737075Z stderr F   17:     0x558b250a0582 - rust_begin_unwind"",""kubernetes"":{""pod_name"":""risingwave-compute-1"",""namespace_name"":""rwc-3-longevity-20230104-180851"",""pod_id"":""a3782c08-c034-459e-bd3f-4b90dca776c1"",""host"":""ip-10-0-75-129.ap-southeast-1.compute.internal"",""container_name"":""compute"",""docker_id"":""59ffb6e3f61a217d837642c2956c9cf11ed8c40110ffae987c77f5beccb96b7b"",""container_hash"":""ghcr.io/risingwavelabs/risingwave@sha256:a0b61ebfc9910a161caf40a571530a9add3168528f78cc8bcf26097748441fdf"",""container_image"":""ghcr.io/risingwavelabs/risingwave:nightly-20230103""}}"
1672868808677,"{""log"":""2023-01-04T21:46:48.67737281Z stderr F                                at /rustc/bdb07a8ec8e77aa10fb84fae1d4ff71c21180bb4/library/std/src/panicking.rs:575:5"",""kubernetes"":{""pod_name"":""risingwave-compute-1"",""namespace_name"":""rwc-3-longevity-20230104-180851"",""pod_id"":""a3782c08-c034-459e-bd3f-4b90dca776c1"",""host"":""ip-10-0-75-129.ap-southeast-1.compute.internal"",""container_name"":""compute"",""docker_id"":""59ffb6e3f61a217d837642c2956c9cf11ed8c40110ffae987c77f5beccb96b7b"",""container_hash"":""ghcr.io/risingwavelabs/risingwave@sha256:a0b61ebfc9910a161caf40a571530a9add3168528f78cc8bcf26097748441fdf"",""container_image"":""ghcr.io/risingwavelabs/risingwave:nightly-20230103""}}"
1672868808677,"{""log"":""2023-01-04T21:46:48.677381621Z stderr F   18:     0x558b250c8b83 - core::panicking::panic_fmt::h81b99a9e904184d4"",""kubernetes"":{""pod_name"":""risingwave-compute-1"",""namespace_name"":""rwc-3-longevity-20230104-180851"",""pod_id"":""a3782c08-c034-459e-bd3f-4b90dca776c1"",""host"":""ip-10-0-75-129.ap-southeast-1.compute.internal"",""container_name"":""compute"",""docker_id"":""59ffb6e3f61a217d837642c2956c9cf11ed8c40110ffae987c77f5beccb96b7b"",""container_hash"":""ghcr.io/risingwavelabs/risingwave@sha256:a0b61ebfc9910a161caf40a571530a9add3168528f78cc8bcf26097748441fdf"",""container_image"":""ghcr.io/risingwavelabs/risingwave:nightly-20230103""}}"
1672868808677,"{""log"":""2023-01-04T21:46:48.677396391Z stderr F                                at /rustc/bdb07a8ec8e77aa10fb84fae1d4ff71c21180bb4/library/core/src/panicking.rs:64:14"",""kubernetes"":{""pod_name"":""risingwave-compute-1"",""namespace_name"":""rwc-3-longevity-20230104-180851"",""pod_id"":""a3782c08-c034-459e-bd3f-4b90dca776c1"",""host"":""ip-10-0-75-129.ap-southeast-1.compute.internal"",""container_name"":""compute"",""docker_id"":""59ffb6e3f61a217d837642c2956c9cf11ed8c40110ffae987c77f5beccb96b7b"",""container_hash"":""ghcr.io/risingwavelabs/risingwave@sha256:a0b61ebfc9910a161caf40a571530a9add3168528f78cc8bcf26097748441fdf"",""container_image"":""ghcr.io/risingwavelabs/risingwave:nightly-20230103""}}"
1672868808681,"{""log"":""2023-01-04T21:46:48.681884023Z stderr F   19:     0x558b228b728c - risingwave_storage::hummock::event_handler::uploader::HummockUploader::seal_epoch::ha2397e68288b16ba"",""kubernetes"":{""pod_name"":""risingwave-compute-1"",""namespace_name"":""rwc-3-longevity-20230104-180851"",""pod_id"":""a3782c08-c034-459e-bd3f-4b90dca776c1"",""host"":""ip-10-0-75-129.ap-southeast-1.compute.internal"",""container_name"":""compute"",""docker_id"":""59ffb6e3f61a217d837642c2956c9cf11ed8c40110ffae987c77f5beccb96b7b"",""container_hash"":""ghcr.io/risingwavelabs/risingwave@sha256:a0b61ebfc9910a161caf40a571530a9add3168528f78cc8bcf26097748441fdf"",""container_image"":""ghcr.io/risingwavelabs/risingwave:nightly-20230103""}}"
1672868808682,"{""log"":""2023-01-04T21:46:48.682045776Z stderr F   20:     0x558b21a56af2 - risingwave_storage::hummock::event_handler::hummock_event_handler::HummockEventHandler::start_hummock_event_handler_worker::{{closure}}::hcf0a2df6116dfe59"",""kubernetes"":{""pod_name"":""risingwave-compute-1"",""namespace_name"":""rwc-3-longevity-20230104-180851"",""pod_id"":""a3782c08-c034-459e-bd3f-4b90dca776c1"",""host"":""ip-10-0-75-129.ap-southeast-1.compute.internal"",""container_name"":""compute"",""docker_id"":""59ffb6e3f61a217d837642c2956c9cf11ed8c40110ffae987c77f5beccb96b7b"",""container_hash"":""ghcr.io/risingwavelabs/risingwave@sha256:a0b61ebfc9910a161caf40a571530a9add3168528f78cc8bcf26097748441fdf"",""container_image"":""ghcr.io/risingwavelabs/risingwave:nightly-20230103""}}"
1672868808682,"{""log"":""2023-01-04T21:46:48.682059307Z stderr F                                at /risingwave/src/storage/src/hummock/event_handler/hummock_event_handler.rs:462:29"",""kubernetes"":{""pod_name"":""risingwave-compute-1"",""namespace_name"":""rwc-3-longevity-20230104-180851"",""pod_id"":""a3782c08-c034-459e-bd3f-4b90dca776c1"",""host"":""ip-10-0-75-129.ap-southeast-1.compute.internal"",""container_name"":""compute"",""docker_id"":""59ffb6e3f61a217d837642c2956c9cf11ed8c40110ffae987c77f5beccb96b7b"",""container_hash"":""ghcr.io/risingwavelabs/risingwave@sha256:a0b61ebfc9910a161caf40a571530a9add3168528f78cc8bcf26097748441fdf"",""container_image"":""ghcr.io/risingwavelabs/risingwave:nightly-20230103""}}"

To Reproduce

No response

Expected behavior

No response

Additional context

There has been several cluster recovery during the test.
@zwang28 zwang28 added the type/bug Something isn't working label Jan 5, 2023
@github-actions github-actions bot added this to the release-0.1.16 milestone Jan 5, 2023
@zwang28 zwang28 changed the title hummock uploader: some epoch 3646003068272640 older than epoch to seal 3646003135053824 hummock uploader: assertion failure when seal_epoch Jan 5, 2023
@zwang28 zwang28 mentioned this issue Jan 5, 2023
Open
65 tasks
@zwang28
Copy link
Contributor Author

zwang28 commented Jan 5, 2023

Is shared buffer not cleaned up when recovery?

@zwang28 zwang28 changed the title hummock uploader: assertion failure when seal_epoch bug: hummock uploader assertion failure when seal_epoch Jan 5, 2023
@zwang28
Copy link
Contributor Author

zwang28 commented Jan 9, 2023 •
edited
Loading

We abort actor's task but don't await it to terminate, I suspect it can lead to unexpected write from actors that is expected to have been dropped but actually still alive, after clear_storage_buffer.

Verifying.

@zwang28 zwang28 mentioned this issue Jan 9, 2023
Merged
1 task
@mergify mergify bot closed this as completed in #7263 Jan 10, 2023
mergify bot pushed a commit that referenced this issue Jan 10, 2023
@zwang28 @wangrunji0408
fix: join actor when dropping (#7263)
5d25f9c 
Ensure actor task is completely dropped after calling `drop_all_actors`, see #7208 (comment).
Also remove unused method `wait_all`.

Approved-By: yezizp2012

Co-Authored-By: zwang28 <84491488@qq.com>
Co-Authored-By: Runji Wang <wangrunji0408@163.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
found-by-longevity-test type/bug Something isn't working
Projects
None yet
Milestone
release-0.1.16
Development

Successfully merging a pull request may close this issue.

fix: join actor when dropping risingwavelabs/risingwave
1 participant
@zwang28