Investigate Cassandra tests flakiness

[piodul]

Recently we started to get random failures in test runs with Cassandra. Some examples of failed runs:

• https://github.com/scylladb/scylla-rust-driver/actions/runs/3846334491/jobs/6551514332
• https://github.com/scylladb/scylla-rust-driver/actions/runs/3842229469/jobs/6543369559

The above runs are for PRs, but the failures don't seem related in any way to the changes they introduce. I can see timeouts, queries not returning all expected data, Unknown CF errors...

This should be investigated, but my initial suspicion is that we are stressing the Cassandra cluster too much with parallel tests (would explain timeouts) and may use consistency that is too low (would explain missing data in reads). This might also be related in some way to the fact that we switched to a 3-node Cassandra cluster.

[Gor027]

The #629 PR changed Cassandra version from 4.1(latest) to 4.0.7 in CI to avoid the flakiness problem, however, the root cause of the flakiness is still unknown. I have tried to look at the Cassandra logs after a failed CI and found an issue that is probably the root cause of the flakiness but it needs to be somehow approved.

[piodul]

I opened a new issue about failing tests with Cassandra 4.1: #633

found an issue that is probably the root cause of the flakiness but it needs to be somehow approved.

Could you explain in more detail what you mean? The link you provided points to this issue. Do you know the real root cause of the flakiness? If so, then let's continue the discussion on the new issue.

[Gor027]

Sorry, that was a copy-paste issue, I have updated the link to the Cassandra issue. Unfortunately, I cannot confirm that the patch resolving the mentioned issue is the root cause of the problem but the issue with the shared logs may help in further investigations.