CI Failure (Timeout waiting for partitions to move) in NodesDecommissioningTest.test_decommissioning_finishes_after_manual_cancellation

[michael-redpanda]

https://buildkite.com/redpanda/vtools/builds/8050#0188a6e9-85d8-48ed-81ef-8307498c7772

Module: rptest.tests.nodes_decommissioning_test
Class:  NodesDecommissioningTest
Method: test_decommissioning_finishes_after_manual_cancellation
Arguments:
{
  "delete_topic": false
}

test_id:    rptest.tests.nodes_decommissioning_test.NodesDecommissioningTest.test_decommissioning_finishes_after_manual_cancellation.delete_topic=False
status:     FAIL
run time:   3 minutes 51.443 seconds


    TimeoutError('')
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 135, in run
    data = self.run_test()
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 227, in run_test
    return self.test_context.function(self.test)
  File "/usr/local/lib/python3.10/dist-packages/ducktape/mark/_mark.py", line 481, in wrapper
    return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
  File "/home/ubuntu/redpanda/tests/rptest/services/cluster.py", line 79, in wrapped
    r = f(self, *args, **kwargs)
  File "/home/ubuntu/redpanda/tests/rptest/tests/nodes_decommissioning_test.py", line 593, in test_decommissioning_finishes_after_manual_cancellation
    wait_until(lambda: self._partitions_moving(node=survivor_node),
  File "/usr/local/lib/python3.10/dist-packages/ducktape/utils/util.py", line 57, in wait_until
    raise TimeoutError(err_msg() if callable(err_msg) else err_msg) from last_exception
ducktape.errors.TimeoutError

[michael-redpanda]

https://buildkite.com/redpanda/vtools/builds/8047#0188a456-d37f-4920-9f18-6235fd222b27 (delete_topic = true)

[ztlpn]

https://buildkite.com/redpanda/redpanda/builds/31189#0188b56e-3d9c-40ed-b801-ac50ffdbf8cd
https://buildkite.com/redpanda/vtools/builds/8065#0188b139-3f45-49fb-9528-0ce49fefce98

[ztlpn]

I'll chase this one

[ztlpn]

This is a test error: the test first decommissions, then sets the recovery rate:

        self.logger.info(f"decommissioning node: {node_id}", )
        self._decommission(node_id)

        self._set_recovery_rate(100)

decommission manages to finish before the recovery rate is lowered :)

[michael-redpanda]

https://buildkite.com/redpanda/redpanda/builds/31350#0188bf15-8931-464d-ad09-9898472ceaf4
https://buildkite.com/redpanda/redpanda/builds/31258#0188b8b4-5aeb-427b-a96f-2e85a7854e23
https://buildkite.com/redpanda/redpanda/builds/31243#0188b830-9434-496b-aaf3-fbca98319ddd
https://buildkite.com/redpanda/vtools/builds/8088#0188b660-206c-4d5b-9054-263a2a463905