fix(scan_operations): add retry policy to cql query

[aleksbykov]

The node where scan operations was started could be
used by disruptive nemesis. If node was restarted/stopped
while scan query had been running, the scan operation would
be terminated and error event and message will mark
test as failed.

Add to cql session ExponentialBackoffRetryPolicy
which allow to retry the query, if node was down
and once it back, query will be succesfully finished

Fixes: #9284

Testing

• Test 1 - Passed
• Test 2 - Failed - failed due to c-s error: java.io.IOException: Operation x10 on key(s) [32374e4d3230504b3030]: Data returned was not validated. Errors related to Scan operations are not reproduced
• Test 3 - Failed - job failed due to issue scylla server hangs on shutdown - on mapreduce (eventually getting killed after 15 minutes with no progress) scylladb#21568, Errors related to Scan operations were not reproduced

PR pre-checks (self review)

• I added the relevant backport labels
• I didn't leave commented-out/debugging code

Reminders

• Add New configuration option and document them (in sdcm/sct_config.py)
• Add unit tests to cover my changes (under unit-test/ folder)
• Update the Readme/doc folder relevant to this change (if needed)

[soyacz]

is there a difference between self._request_default_timeout and self._request_default_timeout? maybe it could be reused?

[fruch]

this is a replacement for @temichus trials in #9370 ?

[fruch]

this is a bit weird it comes next to the code executing the query, and no the code creating the session.

I would recommend consolidating the session creating code into something like:

@property
def cql_connection(self, **kwargs):
    with self.fullscan_params.db_cluster.cql_connection_patient(
                    node=self.db_node,
                    user=self.fullscan_params.user,
                    password=self.fullscan_params.user_password, **kwargs) as session:
        session.cluster.default_retry_policy = ExponentialBackoffRetryPolicy(**self._exp_backoff_retry_policy_params)
        session.default_timeout = self._request_default_timeout
        yield session

there way too many repetitions of applying this retry, and it should be across the board for all of the sessions.

[temichus]

this is a replacement for @temichus trials in #9370 ?

yes

[fruch]

you need to pop out the node from kwargs, if you want to specify it later
or use setdefault

the unittest demonstrate it's currently broken

[fruch]

@aleksbykov anything blocking this one ?

[yarongilor]

There are various errors of scan operation repeating in Azure-3h for a while:

2024-12-25 03:32:29.893: (FullScanAggregateEvent Severity.ERROR) period_type=end event_id=d9d1ca8c-f6a1-4a31-8cfc-d0a5071f0bc0 during_nemesis=NodetoolDecommission duration=1m26s node=longevity-10gb-3h-master-db-node-9aaba4c5-eastus-4 select_from=keyspace1.standard1 message=FullScanAggregatesOperation operation failed: Fullscan failed - 'mapreduce_service_requests_dispatched_to_other_nodes' was not triggered

https://argus.scylladb.com/tests/scylla-cluster-tests/9aaba4c5-7332-4337-8abc-2ca5e180934c

2025-01-14 07:36:48.326: (FullScanAggregateEvent Severity.ERROR) period_type=end event_id=7c049364-179d-44b5-8dae-af8b787768c2 during_nemesis=RollingConfigChangeInternodeCompression duration=25s node=longevity-10gb-3h-master-db-node-df48f23c-eastus-4 select_from=keyspace1.standard1 message=FullScanAggregatesOperation operation failed, ReadTimeout error: ReadTimeout('Error from server: code=1200 [Coordinator node timed out waiting for replica nodes\' responses] message="Operation failed for keyspace1.standard1 - received 0 responses and 1 failures from 1 CL=ONE." info={\'consistency\': \'ONE\', \'required_responses\': 1, \'received_responses\': 0}')

https://argus.scylladb.com/tests/scylla-cluster-tests/df48f23c-a07f-4bc6-bd47-e5e2e40d764e
@fruch , @aleksbykov , please advise if these errors are addressed by this PR (or by #9370 ?).
cc: @pehala