Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CI Failure (exceptional future ignored) in ConsumerOffsetsConsistencyTest.test_flipping_leadership #13278

Closed
rockwotj opened this issue Sep 6, 2023 · 33 comments
Closed

CI Failure (exceptional future ignored) in ConsumerOffsetsConsistencyTest.test_flipping_leadership #13278

rockwotj opened this issue Sep 6, 2023 · 33 comments
Labels
area/replication ci-failure kind/bug Something isn't working sev/low Bugs which are non-functional paper cuts, e.g. typos, issues in log messages

Comments

@rockwotj
Copy link
Contributor

rockwotj commented Sep 6, 2023 •
edited by jira bot
Loading

https://buildkite.com/redpanda/redpanda/builds/36451#018a6738-582b-4a44-96dd-699b359d5375

Module: rptest.tests.consumer_offsets_consistency_test
Class:  ConsumerOffsetsConsistencyTest
Method: test_flipping_leadership

test_id:    rptest.tests.consumer_offsets_consistency_test.ConsumerOffsetsConsistencyTest.test_flipping_leadership
status:     FAIL
run time:   8 minutes 37.887 seconds


    <BadLogLines nodes=docker-rp-15(1) example="WARN  2023-09-05 21:54:57,241 [shard 0:main] seastar - Exceptional future ignored: seastar::no_sharded_instance_exception (sharded instance does not exist: seastar::abort_source), backtrace: 0xedf06 /opt/redpanda_installs/ci/lib/libseastar.so+0x58a53f2 /opt/redpanda_installs/ci/lib/libseastar.so+0x58a4fd5 /opt/redpanda_installs/ci/lib/libseastar.so+0x58a960e /opt/redpanda_installs/ci/lib/libseastar.so+0x58abfef /opt/redpanda_installs/ci/lib/libseastar.so+0x4237e65 0x19adf2 0x19ac84 /opt/redpanda_installs/ci/lib/libv_v_application.so+0x5a45246 /opt/redpanda_installs/ci/lib/libseastar.so+0x423d5bd /opt/redpanda_installs/ci/lib/libseastar.so+0x423db87 /opt/redpanda_installs/ci/lib/libseastar.so+0x45fb744 /opt/redpanda_installs/ci/lib/libseastar.so+0x46089c1 /opt/redpanda_installs/ci/lib/libseastar.so+0x460ec2b /opt/redpanda_installs/ci/lib/libseastar.so+0x460c605 /opt/redpanda_installs/ci/lib/libseastar.so+0x4078ccc /opt/redpanda_installs/ci/lib/libseastar.so+0x40766c6 /opt/redpanda_installs/ci/lib/libv_v_application.so+0x70b6d22 0x17fe74 /opt/redpanda_installs/ci/lib/libc.so.6+0x2d58f /opt/redpanda_installs/ci/lib/libc.so.6+0x2d648 0xa8e64">
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 135, in run
    data = self.run_test()
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 227, in run_test
    return self.test_context.function(self.test)
  File "/root/tests/rptest/services/cluster.py", line 142, in wrapped
    redpanda.raise_on_bad_logs(
  File "/root/tests/rptest/services/redpanda.py", line 1237, in raise_on_bad_logs
    raise BadLogLines(bad_lines)
rptest.services.utils.BadLogLines: <BadLogLines nodes=docker-rp-15(1) example="WARN  2023-09-05 21:54:57,241 [shard 0:main] seastar - Exceptional future ignored: seastar::no_sharded_instance_exception (sharded instance does not exist: seastar::abort_source), backtrace: 0xedf06 /opt/redpanda_installs/ci/lib/libseastar.so+0x58a53f2 /opt/redpanda_installs/ci/lib/libseastar.so+0x58a4fd5 /opt/redpanda_installs/ci/lib/libseastar.so+0x58a960e /opt/redpanda_installs/ci/lib/libseastar.so+0x58abfef /opt/redpanda_installs/ci/lib/libseastar.so+0x4237e65 0x19adf2 0x19ac84 /opt/redpanda_installs/ci/lib/libv_v_application.so+0x5a45246 /opt/redpanda_installs/ci/lib/libseastar.so+0x423d5bd /opt/redpanda_installs/ci/lib/libseastar.so+0x423db87 /opt/redpanda_installs/ci/lib/libseastar.so+0x45fb744 /opt/redpanda_installs/ci/lib/libseastar.so+0x46089c1 /opt/redpanda_installs/ci/lib/libseastar.so+0x460ec2b /opt/redpanda_installs/ci/lib/libseastar.so+0x460c605 /opt/redpanda_installs/ci/lib/libseastar.so+0x4078ccc /opt/redpanda_installs/ci/lib/libseastar.so+0x40766c6 /opt/redpanda_installs/ci/lib/libv_v_application.so+0x70b6d22 0x17fe74 /opt/redpanda_installs/ci/lib/libc.so.6+0x2d58f /opt/redpanda_installs/ci/lib/libc.so.6+0x2d648 0xa8e64">

JIRA Link: CORE-1432
@rockwotj rockwotj added kind/bug Something isn't working ci-failure sev/medium Bugs that do not meet criteria for high or critical, but are more severe than low. labels Sep 6, 2023
@rockwotj rockwotj mentioned this issue Sep 6, 2023
Merged
7 tasks
@rockwotj
Copy link
Contributor Author

rockwotj commented Sep 6, 2023

This is what I could decode, not exactly helpful... ☹️

0xedf06 
void seastar::backtrace<seastar::current_backtrace_tasklocal()::$_0>(seastar::current_backtrace_tasklocal()::$_0&&) at /v/build/v_deps_build/seastar-prefix/src/seastar/include/seastar/util/backtrace.hh:64
seastar::current_backtrace_tasklocal() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/util/backtrace.cc:98
seastar::current_tasktrace() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/util/backtrace.cc:149
seastar::current_backtrace() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/util/backtrace.cc:182
seastar::report_failed_future(std::exception_ptr const&) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/future.cc:216
0x19adf2 
0x19ac84 
[/opt/redpanda_installs/ci/lib/libv_v_application.so+0x5a45246 ](seastar::promise<void>::set_exception(std::exception_ptr&&) at /vectorized/include/seastar/core/future.hh:978)
seastar::parallel_for_each_state::wait_for_one() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/future-util.cc:79
seastar::parallel_for_each_state::run_and_dispose() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/future-util.cc:91
seastar::reactor::run_tasks(seastar::reactor::task_queue&) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2750
seastar::reactor::run_some_tasks() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:3213
seastar::reactor::do_run() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:3397
seastar::reactor::run() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:3265
seastar::app_template::run_deprecated(int, char**, std::__1::function<void ()>&&) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/app-template.cc:276
seastar::app_template::run(int, char**, std::__1::function<seastar::future<int> ()>&&) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/app-template.cc:167
[/opt/redpanda_installs/ci/lib/libv_v_application.so+0x70b6d22 ](application::run(int, char**) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-085e8b9097d3be57f-1/redpanda/redpanda/src/v/redpanda/application.cc:348)
0x17fe74 
/opt/redpanda_installs/ci/lib/libc.so.6+0x2d58f 
/opt/redpanda_installs/ci/lib/libc.so.6+0x2d648 
0xa8e64

@rockwotj rockwotj added sev/low Bugs which are non-functional paper cuts, e.g. typos, issues in log messages and removed sev/medium Bugs that do not meet criteria for high or critical, but are more severe than low. labels Sep 6, 2023
@rockwotj
Copy link
Contributor Author

rockwotj commented Sep 6, 2023

This happens during shutdown, downgrading to sev/low

@rystsov
Copy link
Contributor

rystsov commented Sep 12, 2023

https://buildkite.com/redpanda/redpanda/builds/36199

@bharathv bharathv mentioned this issue Sep 25, 2023
Merged
7 tasks
@abhijat
Copy link
Contributor

abhijat commented Sep 28, 2023

https://buildkite.com/redpanda/redpanda/builds/37716

@rockwotj
Copy link
Contributor Author

https://buildkite.com/redpanda/redpanda/builds/38771#018b21b8-c392-4154-8a8a-e28f1e53cc9e

This was referenced Oct 12, 2023
Merged
Closed
@vbotbuildovich
Copy link
Collaborator

*https://buildkite.com/redpanda/redpanda/builds/40282

@andijcr andijcr mentioned this issue Nov 21, 2023
Open
7 tasks
@vbotbuildovich
Copy link
Collaborator

*https://buildkite.com/redpanda/redpanda/builds/41798

@vbotbuildovich
Copy link
Collaborator

*https://buildkite.com/redpanda/redpanda/builds/42006

@dotnwat
Copy link
Member

dotnwat commented Dec 15, 2023

Assigning to replication as Michal is the only author of the test that is failing.

Note that it appears that the failure mode described in the cover letter isn't the same failure mode as some of these other reported failures.

@vbotbuildovich
Copy link
Collaborator

*https://buildkite.com/redpanda/redpanda/builds/43292

@vbotbuildovich
Copy link
Collaborator

*https://buildkite.com/redpanda/redpanda/builds/43423

@vbotbuildovich
Copy link
Collaborator

*https://buildkite.com/redpanda/redpanda/builds/43796

@vbotbuildovich
Copy link
Collaborator

*https://buildkite.com/redpanda/redpanda/builds/44998

@vbotbuildovich
Copy link
Collaborator

*https://buildkite.com/redpanda/redpanda/builds/45109

@vbotbuildovich
Copy link
Collaborator

*https://buildkite.com/redpanda/redpanda/builds/45181

@vbotbuildovich
Copy link
Collaborator

*https://buildkite.com/redpanda/redpanda/builds/45505

@vbotbuildovich
Copy link
Collaborator

*https://buildkite.com/redpanda/vtools/builds/12069

@vbotbuildovich
Copy link
Collaborator

*https://buildkite.com/redpanda/vtools/builds/12141#018e12c9-e355-4a78-a3b1-c03e5779f19d

@piyushredpanda piyushredpanda mentioned this issue Mar 7, 2024
Merged
@vbotbuildovich
Copy link
Collaborator

*https://buildkite.com/redpanda/redpanda/builds/46542

@vbotbuildovich
Copy link
Collaborator

*https://buildkite.com/redpanda/redpanda/builds/46944

@vbotbuildovich
Copy link
Collaborator

*https://buildkite.com/redpanda/redpanda/builds/47094

@vbotbuildovich
Copy link
Collaborator

*https://buildkite.com/redpanda/vtools/builds/12611

@vbotbuildovich
Copy link
Collaborator

*https://buildkite.com/redpanda/redpanda/builds/47526
*https://buildkite.com/redpanda/vtools/builds/12792

@vbotbuildovich
Copy link
Collaborator

*https://buildkite.com/redpanda/vtools/builds/13280

@vbotbuildovich
Copy link
Collaborator

*https://buildkite.com/redpanda/vtools/builds/13369

@vbotbuildovich
Copy link
Collaborator

*https://buildkite.com/redpanda/redpanda/builds/48718

@vbotbuildovich
Copy link
Collaborator

*https://buildkite.com/redpanda/vtools/builds/13739

@vbotbuildovich
Copy link
Collaborator

*https://buildkite.com/redpanda/redpanda/builds/49554

@vbotbuildovich
Copy link
Collaborator

*https://buildkite.com/redpanda/vtools/builds/14509

@vbotbuildovich
Copy link
Collaborator

*https://buildkite.com/redpanda/vtools/builds/13280
*https://buildkite.com/redpanda/vtools/builds/13369
*https://buildkite.com/redpanda/redpanda/builds/48718
*https://buildkite.com/redpanda/vtools/builds/13739
*https://buildkite.com/redpanda/redpanda/builds/49554
*https://buildkite.com/redpanda/vtools/builds/14509

2 similar comments
@vbotbuildovich
Copy link
Collaborator

*https://buildkite.com/redpanda/vtools/builds/13280
*https://buildkite.com/redpanda/vtools/builds/13369
*https://buildkite.com/redpanda/redpanda/builds/48718
*https://buildkite.com/redpanda/vtools/builds/13739
*https://buildkite.com/redpanda/redpanda/builds/49554
*https://buildkite.com/redpanda/vtools/builds/14509

@vbotbuildovich
Copy link
Collaborator

*https://buildkite.com/redpanda/vtools/builds/13280
*https://buildkite.com/redpanda/vtools/builds/13369
*https://buildkite.com/redpanda/redpanda/builds/48718
*https://buildkite.com/redpanda/vtools/builds/13739
*https://buildkite.com/redpanda/redpanda/builds/49554
*https://buildkite.com/redpanda/vtools/builds/14509

@piyushredpanda
Copy link
Contributor

Closing older-bot-filed CI issues as we transition to a more reliable system.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/replication ci-failure kind/bug Something isn't working sev/low Bugs which are non-functional paper cuts, e.g. typos, issues in log messages
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@dotnwat @abhijat @rystsov @rockwotj @vbotbuildovich @piyushredpanda