Starting up a new cluster using the k8ssandra operator using helm chart 1.10.2 reaper does not work

[jbiskur]

When I start a simple 3 node cluster using k8ssandra I get an error when the operator tries to add the cluster to reaper.

WARN   [2023-11-11 02:39:37,839] [clustername-nio-worker-0] c.d.d.c.RequestHandler - Unexpected error while querying production-cluster-datacenter1-service/10.242.1.254:9042 - [com.datastax.driver.core.exceptions.InvalidQueryException: Keyspace 'null' does not exist]. Find next host to query.
INFO   [2023-11-11 02:39:39,136] [dw-31 - PUT /cluster/production-cluster?seedHost=production-cluster-seed-service] i.c.r.ClusterResource - PUT addOrUpdateCluster called with: cluster_name = production-cluster, seedHost = production-cluster-seed-service 
INFO   [2023-11-11 02:39:39,140] [dw-31 - PUT /cluster/production-cluster?seedHost=production-cluster-seed-service] i.c.r.ClusterResource - POST/PUT on cluster resource production-cluster called with seedHost production-cluster-seed-service belonging to different cluster testcluster
INFO   [2023-11-11 02:39:50,691] [dw-38 - PUT /cluster/production-cluster?seedHost=production-cluster-seed-service] i.c.r.ClusterResource - PUT addOrUpdateCluster called with: cluster_name = production-cluster, seedHost = production-cluster-seed-service 
INFO   [2023-11-11 02:39:50,694] [dw-38 - PUT /cluster/production-cluster?seedHost=production-cluster-seed-service] i.c.r.ClusterResource - POST/PUT on cluster resource production-cluster called with seedHost production-cluster-seed-service belonging to different cluster testcluster
INFO   [2023-11-11 02:40:03,322] [dw-37 - PUT /cluster/production-cluster?seedHost=production-cluster-seed-service] i.c.r.ClusterResource - PUT addOrUpdateCluster called with: cluster_name = production-cluster, seedHost = production-cluster-seed-service 
INFO   [2023-11-11 02:40:03,326] [dw-37 - PUT /cluster/production-cluster?seedHost=production-cluster-seed-service] i.c.r.ClusterResource - POST/PUT on cluster resource production-cluster called with seedHost production-cluster-seed-service belonging to different cluster testcluster
INFO   [2023-11-11 02:40:26,187] [dw-35 - PUT /cluster/production-cluster?seedHost=production-cluster-seed-service] i.c.r.ClusterResource - PUT addOrUpdateCluster called with: cluster_name = production-cluster, seedHost = production-cluster-seed-service 
INFO   [2023-11-11 02:40:26,192] [dw-35 - PUT /cluster/production-cluster?seedHost=production-cluster-seed-service] i.c.r.ClusterResource - POST/PUT on cluster resource production-cluster called with seedHost production-cluster-seed-service belonging to different cluster testcluster
INFO   [2023-11-11 02:41:12,894] [dw-35 - PUT /cluster/production-cluster?seedHost=production-cluster-seed-service] i.c.r.ClusterResource - PUT addOrUpdateCluster called with: cluster_name = production-cluster, seedHost = production-cluster-seed-service 
INFO   [2023-11-11 02:41:12,899] [dw-35 - PUT /cluster/production-cluster?seedHost=production-cluster-seed-service] i.c.r.ClusterResource - POST/PUT on cluster resource production-cluster called with seedHost production-cluster-seed-service belonging to different cluster testcluster

I also don't see any cluster in the reaper UI.

Did you expect to see something different?

How to reproduce it (as minimally and precisely as possible):

Environment

• K8ssandra Operator version:
  1.10.2

• Kubernetes version information:

Client Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.1", GitCommit:"4c9411232e10168d7b050c49a1b59f6df9d7ea4b", GitTreeState:"clean", BuildDate:"2023-04-14T13:14:41Z", GoVersion:"go1.20.3", Compiler:"gc", Platform:"darwin/arm64"}
Kustomize Version: v5.0.1
Server Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.6", GitCommit:"7ffcdf755d47c73903854cc5955afcdcd8c95225", GitTreeState:"clean", BuildDate:"2023-10-09T14:43:34Z", GoVersion:"go1.19.10", Compiler:"gc", Platform:"linux/amd64"}

• Kubernetes cluster kind:

AKS

• Manifests:

apiVersion: k8ssandra.io/v1alpha1
kind: K8ssandraCluster
.....
spec:
  auth: true
  cassandra:
    datacenters:
      - config:
          jvmOptions:
            gc: G1GC
            heapSize: 2G
        metadata:
          name: datacenter1
        perNodeConfigInitContainerImage: mikefarah/yq:4
        size: 7
        stopped: false
    jmxInitContainerImage:
      name: busybox
      registry: docker.io
      repository: library
      tag: latest
    perNodeConfigInitContainerImage: mikefarah/yq:4
    resources:
      limits:
        memory: 8Gi
      requests:
        memory: 4Gi
    serverType: cassandra
    serverVersion: 4.1.0
    storageConfig:
      cassandraDataVolumeClaimSpec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 300Gi
        storageClassName: managed-csi
    superuserSecretRef:
      name: production-cluster-superuser
  reaper:
    ServiceAccountName: default
    autoScheduling:
      enabled: true
      initialDelayPeriod: PT15S
      percentUnrepairedThreshold: 10
      periodBetweenPolls: PT10M
      repairType: AUTO
      scheduleSpreadPeriod: PT6H
      timeBeforeFirstSchedule: PT5M
    cassandraUserSecretRef:
      name: production-reaper-user
    containerImage:
      name: cassandra-reaper
      registry: docker.io
      repository: thelastpickle
      tag: 3.2.1
    deploymentMode: PER_DC
    heapSize: 256Mi
    initContainerImage:
      name: cassandra-reaper
      registry: docker.io
      repository: thelastpickle
      tag: 3.2.1
    keyspace: reaper_db
    secretsProvider: internal
    uiUserSecretRef:
      name: production-reaper-ui-user
  secretsProvider: internal

• K8ssandra Operator Logs:

2023-11-11T02:49:16.272Z	INFO	Reaper Deployment ready	{"controller": "reaper", "controllerGroup": "reaper.k8ssandra.io", "controllerKind": "Reaper", "Reaper": {"name":"production-cluster-datacenter1-reaper","namespace":"datastores"}, "namespace": "datastores", "name": "production-cluster-datacenter1-reaper", "reconcileID": "3540404d-a59b-4c71-bb8c-cf9dd5e6713a", "Reaper": "datastores/production-cluster-datacenter1-reaper", "Deployment": "datastores/production-cluster-datacenter1-reaper"}
2023-11-11T02:49:16.272Z	INFO	Reconciling Reaper Service	{"controller": "reaper", "controllerGroup": "reaper.k8ssandra.io", "controllerKind": "Reaper", "Reaper": {"name":"production-cluster-datacenter1-reaper","namespace":"datastores"}, "namespace": "datastores", "name": "production-cluster-datacenter1-reaper", "reconcileID": "3540404d-a59b-4c71-bb8c-cf9dd5e6713a", "Reaper": "datastores/production-cluster-datacenter1-reaper", "Service": "datastores/production-cluster-datacenter1-reaper-service"}
2023-11-11T02:49:16.272Z	INFO	Reaper Service is ready	{"controller": "reaper", "controllerGroup": "reaper.k8ssandra.io", "controllerKind": "Reaper", "Reaper": {"name":"production-cluster-datacenter1-reaper","namespace":"datastores"}, "namespace": "datastores", "name": "production-cluster-datacenter1-reaper", "reconcileID": "3540404d-a59b-4c71-bb8c-cf9dd5e6713a", "Reaper": "datastores/production-cluster-datacenter1-reaper", "Service": "datastores/production-cluster-datacenter1-reaper-service"}
2023-11-11T02:49:16.285Z	INFO	registering cluster with reaper	{"controller": "reaper", "controllerGroup": "reaper.k8ssandra.io", "controllerKind": "Reaper", "Reaper": {"name":"production-cluster-datacenter1-reaper","namespace":"datastores"}, "namespace": "datastores", "name": "production-cluster-datacenter1-reaper", "reconcileID": "3540404d-a59b-4c71-bb8c-cf9dd5e6713a", "Reaper": "datastores/production-cluster-datacenter1-reaper"}
2023-11-11T02:49:16.296Z	ERROR	failed to register cluster with reaper	{"controller": "reaper", "controllerGroup": "reaper.k8ssandra.io", "controllerKind": "Reaper", "Reaper": {"name":"production-cluster-datacenter1-reaper","namespace":"datastores"}, "namespace": "datastores", "name": "production-cluster-datacenter1-reaper", "reconcileID": "3540404d-a59b-4c71-bb8c-cf9dd5e6713a", "Reaper": "datastores/production-cluster-datacenter1-reaper", "error": "failed to create cluster production-cluster: POST/PUT on cluster resource production-cluster called with seedHost production-cluster-seed-service belonging to different cluster testcluster (HTTP status 400)"}
github.com/k8ssandra/k8ssandra-operator/controllers/reaper.(*ReaperReconciler).configureReaper
	/workspace/controllers/reaper/reaper_controller.go:322
github.com/k8ssandra/k8ssandra-operator/controllers/reaper.(*ReaperReconciler).reconcile
	/workspace/controllers/reaper/reaper_controller.go:111
github.com/k8ssandra/k8ssandra-operator/controllers/reaper.(*ReaperReconciler).Reconcile
	/workspace/controllers/reaper/reaper_controller.go:78
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem

I have looked at the resources in kubernetes related to the operator and k8ssandra and Cassandra as well as the cass-operator, I cant find the "testcluster" anywhere. The cassandra.yaml has the cluster name set to "Test Cluster" for some reason, even though I have also tried to override the name using the clusterName key in the CRD.

[adejanovski]

Looking at your manifest and the reported cluster names (testcluster vs production-cluster) I think this could be related to this issue.
We'll soon release a new patch version with a fix for the cluster name setting when using Cassandra 4.1+

[adejanovski]

We've released k8ssandra-operator v1.10.3 with a new version of the config builder which fixes that issue.