K8SSAND-1167 ⁃ Check for schema agreement before applying schema changes

[jsanda]

What is missing?
This is a follow up to #21. With the changes being introduced in #262, there several places/scenarios where we can apply schema changes in Cassandra. Specifically we change the replication of several keyspaces. We should be checking for schema agreement in the C* cluster before we apply each change.

Why do we need it?
Applying schema changes when the version is not consistent across the cluster can introduce instability. Its a good practice in general to check for schema version agreement before applying changes.

This should be done as a follow up to #262.

Environment

• K8ssandra Operator version:

0538e957

┆Issue is synchronized with this Jira Bug by Unito
┆Reviewer: Miles Garnsey
┆fixVersions: k8ssandra-operator-v1.0.0
┆friendlyId: K8SSAND-1167
┆priority: High

[jsanda]

This is being handled by #262.

[jsanda]

I started looking into this. This really needs to be done in defaultManagementApiFacade. Schema version checks should be done in each of the following methods:

• CreateKeyspaceIfNotExists
• AlterKeyspace
• CreateTable
• EnsureKeyspaceReplication

If we do not have schema agreement, then those methods should not make any changes. We want to requeue the reconcile request at that point.

I will introduce a custom error type to indicate schema disagreement. The caller can check the error type to determine if a requeue is necessary.