Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CI Failure (leader count on shard (2, 0) (3) is < 4) in AutomaticLeadershipBalancingTest.test_automatic_rebalance #17150

Closed
vbotbuildovich opened this issue Mar 16, 2024 · 36 comments · Fixed by #18497
Closed

CI Failure (leader count on shard (2, 0) (3) is < 4) in AutomaticLeadershipBalancingTest.test_automatic_rebalance #17150

vbotbuildovich opened this issue Mar 16, 2024 · 36 comments · Fixed by #18497
Assignees
@ztlpn
Labels
area/replication auto-triaged used to know which issues have been opened from a CI job ci-failure ci-rca/redpanda CI Root Cause Analysis - Redpanda Issue

Comments

@vbotbuildovich
Copy link
Collaborator

vbotbuildovich commented Mar 16, 2024 •
edited by jira bot
Loading

https://buildkite.com/redpanda/vtools/builds/12307

Module: rptest.tests.leadership_transfer_test
Class: AutomaticLeadershipBalancingTest
Method: test_automatic_rebalance

test_id:    AutomaticLeadershipBalancingTest.test_automatic_rebalance
status:     FAIL
run time:   91.740 seconds

AssertionError('leader count on shard (2, 0) (3) is < 4')
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 184, in _do_run
    data = self.run_test()
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 269, in run_test
    return self.test_context.function(self.test)
  File "/home/ubuntu/redpanda/tests/rptest/services/cluster.py", line 104, in wrapped
    r = f(self, *args, **kwargs)
  File "/home/ubuntu/redpanda/tests/rptest/tests/leadership_transfer_test.py", line 324, in test_automatic_rebalance
    assert count >= expected_min, \
AssertionError: leader count on shard (2, 0) (3) is < 4

JIRA Link: CORE-1883
@vbotbuildovich vbotbuildovich added auto-triaged used to know which issues have been opened from a CI job ci-failure labels Mar 16, 2024
@vbotbuildovich
Copy link
Collaborator Author

*https://buildkite.com/redpanda/vtools/builds/12321

@michael-redpanda michael-redpanda changed the title CI Failure (key symptom) in AutomaticLeadershipBalancingTest.test_automatic_rebalance CI Failure (leader count on shard (2, 0) (3) is < 4) in AutomaticLeadershipBalancingTest.test_automatic_rebalance Mar 19, 2024
@vbotbuildovich
Copy link
Collaborator Author

*https://buildkite.com/redpanda/vtools/builds/12578
*https://buildkite.com/redpanda/redpanda/builds/47080

@vbotbuildovich
Copy link
Collaborator Author

*https://buildkite.com/redpanda/vtools/builds/12592

@vbotbuildovich
Copy link
Collaborator Author

*https://buildkite.com/redpanda/redpanda/builds/47264

1 similar comment
@vbotbuildovich
Copy link
Collaborator Author

*https://buildkite.com/redpanda/redpanda/builds/47264

@vbotbuildovich
Copy link
Collaborator Author

*https://buildkite.com/redpanda/redpanda/builds/47372
*https://buildkite.com/redpanda/vtools/builds/12679

@vbotbuildovich
Copy link
Collaborator Author

*https://buildkite.com/redpanda/redpanda/builds/47416
*https://buildkite.com/redpanda/redpanda/builds/47612

@vbotbuildovich
Copy link
Collaborator Author

*https://buildkite.com/redpanda/redpanda/builds/47756

@vbotbuildovich
Copy link
Collaborator Author

*https://buildkite.com/redpanda/vtools/builds/12865
*https://buildkite.com/redpanda/vtools/builds/12866

@vbotbuildovich
Copy link
Collaborator Author

*https://buildkite.com/redpanda/redpanda/builds/47786

@vbotbuildovich
Copy link
Collaborator Author

*https://buildkite.com/redpanda/redpanda/builds/47838
*https://buildkite.com/redpanda/redpanda/builds/47843
*https://buildkite.com/redpanda/vtools/builds/12890

@vbotbuildovich
Copy link
Collaborator Author

*https://buildkite.com/redpanda/vtools/builds/12994
*https://buildkite.com/redpanda/redpanda/builds/48064
*https://buildkite.com/redpanda/redpanda/builds/48086
*https://buildkite.com/redpanda/redpanda/builds/48099

@ztlpn ztlpn added the ci-rca/redpanda CI Root Cause Analysis - Redpanda Issue label Apr 23, 2024
@ztlpn
Copy link
Contributor

ztlpn commented Apr 23, 2024

The test is failing because a newly restarted node is sending an incomplete health report (some partitions haven't started yet). I guess we can give newly restarted nodes a grace period and mute them for a bit before transferring leadership there (a good idea for other reasons as well).

@vbotbuildovich
Copy link
Collaborator Author

*https://buildkite.com/redpanda/vtools/builds/13100

@ztlpn ztlpn mentioned this issue Apr 24, 2024
Merged
@vbotbuildovich
Copy link
Collaborator Author

*https://buildkite.com/redpanda/vtools/builds/13147

@vbotbuildovich
Copy link
Collaborator Author

*https://buildkite.com/redpanda/vtools/builds/13189

@vbotbuildovich
Copy link
Collaborator Author

*https://buildkite.com/redpanda/vtools/builds/13219
*https://buildkite.com/redpanda/vtools/builds/13228

@vbotbuildovich
Copy link
Collaborator Author

*https://buildkite.com/redpanda/redpanda/builds/48366

@vbotbuildovich
Copy link
Collaborator Author

*https://buildkite.com/redpanda/redpanda/builds/48606

@vbotbuildovich
Copy link
Collaborator Author

*https://buildkite.com/redpanda/redpanda/builds/48666
*https://buildkite.com/redpanda/redpanda/builds/48667

@vbotbuildovich
Copy link
Collaborator Author

*https://buildkite.com/redpanda/redpanda/builds/48717

@vbotbuildovich
Copy link
Collaborator Author

*https://buildkite.com/redpanda/vtools/builds/13497

@piyushredpanda piyushredpanda mentioned this issue May 10, 2024
Merged
7 tasks
@vbotbuildovich
Copy link
Collaborator Author

*https://buildkite.com/redpanda/vtools/builds/13656

@vbotbuildovich
Copy link
Collaborator Author

*https://buildkite.com/redpanda/redpanda/builds/48992
*https://buildkite.com/redpanda/redpanda/builds/48990
*https://buildkite.com/redpanda/redpanda/builds/49002

@vbotbuildovich
Copy link
Collaborator Author

*https://buildkite.com/redpanda/redpanda/builds/49075

ztlpn added a commit to ztlpn/redpanda that referenced this issue May 15, 2024
@ztlpn
tests: relax check in AutomaticLeadershipBalancingTest
ca483a4 
Fixes redpanda-data#17150
@ztlpn ztlpn mentioned this issue May 15, 2024
Merged
7 tasks
@vbotbuildovich
Copy link
Collaborator Author

*https://buildkite.com/redpanda/redpanda/builds/49143
*https://buildkite.com/redpanda/vtools/builds/13742

ztlpn added a commit to ztlpn/redpanda that referenced this issue May 16, 2024
@ztlpn
tests: relax check in AutomaticLeadershipBalancingTest
e68016b 
Fixes redpanda-data#17150
@vbotbuildovich
Copy link
Collaborator Author

*https://buildkite.com/redpanda/redpanda/builds/49202
*https://buildkite.com/redpanda/vtools/builds/13769
*https://buildkite.com/redpanda/vtools/builds/13773
*https://buildkite.com/redpanda/vtools/builds/13892

@vbotbuildovich
Copy link
Collaborator Author

*https://buildkite.com/redpanda/redpanda/builds/49468

@vbotbuildovich
Copy link
Collaborator Author

*https://buildkite.com/redpanda/redpanda/builds/49567

@vbotbuildovich
Copy link
Collaborator Author

*https://buildkite.com/redpanda/vtools/builds/14105

@vbotbuildovich
Copy link
Collaborator Author

*https://buildkite.com/redpanda/redpanda/builds/49631

@vbotbuildovich
Copy link
Collaborator Author

*https://buildkite.com/redpanda/vtools/builds/14181

@vbotbuildovich
Copy link
Collaborator Author

*https://buildkite.com/redpanda/redpanda/builds/49701
*https://buildkite.com/redpanda/redpanda/builds/49721
*https://buildkite.com/redpanda/vtools/builds/14209
*https://buildkite.com/redpanda/redpanda/builds/49742

@graphcareful graphcareful mentioned this issue Jun 4, 2024
Merged
@vbotbuildovich
Copy link
Collaborator Author

*https://buildkite.com/redpanda/redpanda/builds/49779
*https://buildkite.com/redpanda/redpanda/builds/49863

@vbotbuildovich
Copy link
Collaborator Author

*https://buildkite.com/redpanda/redpanda/builds/49979
*https://buildkite.com/redpanda/redpanda/builds/49990

@vbotbuildovich
Copy link
Collaborator Author

*https://buildkite.com/redpanda/redpanda/builds/50144

@vbotbuildovich vbotbuildovich mentioned this issue Jun 12, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ztlpn ztlpn

Labels
area/replication auto-triaged used to know which issues have been opened from a CI job ci-failure ci-rca/redpanda CI Root Cause Analysis - Redpanda Issue
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Mute just restarted nodes in leader_balancer ztlpn/redpanda
3 participants
@ztlpn @vbotbuildovich @michael-redpanda