K8SSAND-962 ⁃ Investigate making the root file system in cassandra container read-only

[jsanda]

Why do we need it?
The root file system should be read-only for improved security. This is considered a best practice for security.

#196 was created specifically for DSE. The goal for this ticket is to address both OSS Cassandra and DSE. If it turns out that there are substantially different changes required for Cassandra vs DSE, then they should be addressed in separate PRs.

Here is an example manifest that configures the cassandra container with a read-only root file system:

apiVersion: cassandra.datastax.com/v1beta1
kind: CassandraDatacenter
metadata:
  name: dc1
spec:
  clusterName: test
  serverType: cassandra
  serverVersion: "4.0.0"
  systemLoggerImage:
  serverImage:
  size: 1
  storageConfig:
    cassandraDataVolumeClaimSpec:
      storageClassName: standard
      accessModes:
        - ReadWriteOnce
      resources:
        requests:
          storage: 5Gi
  podTemplateSpec:
    spec:
      containers:
        - name: "cassandra"
          securityContext:
            readOnlyRootFilesystem: true
            runAsNonRoot: true
        config:
    jvm-server-options:
      initial_heap_size: "800M"
      max_heap_size: "800M"

The cassandra goes into a crash loop with this in the log:

Starting Management API
cp: cannot create regular file '/opt/cassandra/conf/cassandra-env.sh': Read-only file system
cp: cannot create regular file '/opt/cassandra/conf/cassandra-rackdc.properties': Read-only file system
cp: cannot create regular file '/opt/cassandra/conf/cassandra.yaml': Read-only file system
cp: cannot create regular file '/opt/cassandra/conf/jvm11-server.options': Read-only file system
cp: cannot create regular file '/opt/cassandra/conf/jvm8-server.options': Read-only file system
cp: cannot create regular file '/opt/cassandra/conf/jvm.options': Read-only file system
cp: cannot create regular file '/opt/cassandra/conf/jvm-server.options': Read-only file system
cp: cannot create regular file '/opt/cassandra/conf/logback.xml': Read-only file system

Environment

• Cass Operator version:
  v1.8.0

Anything else we need to know?:
We have already implemented a solution (or at least partial) in the k8ssandra Helm chart. Look at the cassdc.yaml template here. Here are the highlights:

• Add a cassandra-config volume to be used to /etc/cassandra
• The base-config-init init container copies out of box configs to cassandra-config
• The server-config-init init container (i.e., config builder) runs and writes files to the server-config volume
• The cassandra container mounts server-config at /config
• The cassandra container mounts cassandra-config at /etc/cassandra
• The entry point script of the cassandra container copies files from /config to /etc/cassandra

Again this what is already done in k8ssandra. We basically need to backport it.

Fixes

Preview Give feedback

1. K8SSAND-954 ⁃ Unable to create CassandraDatacenter if Setup containers.securityContext.readOnlyRootFilesystem: true #196
   bug done +2

┆Issue is synchronized with this Jira Task by Unito
┆epic: Security Phase II
┆friendlyId: K8SSAND-962
┆priority: Medium

[jsanda]

There are additional changes needed. The management-api writes to /opt. Logs are written to /var/log/cassandra.

This makes a number of places where we need to introduce volume mounts. This begs the question of whether we want to continue using init containers to copy things around as we are currently doing or make changes in the images themselves.

I did some prototyping for this earlier in the week. I pushed it here https://github.com/jsanda/cass-operator-1/tree/non-root-fs-cassandra.

I do not think that the current solution with copying stuff with init containers is a scalable and maintainable solution. I think we would we better off in the long run to update the images with the necessary changes. Note that investigation is still required to determined what all is necessary.

The management-api uses base images of Cassandra from https://github.com/docker-library/cassandra. We might need a different base image that is designed with a read-only root file system in mind.

Ultimately I think we have to configure the containers with read-only root file systems to discover all the changes that are necessary. I will revisit my prototype to get a better sense of what is involved and then start trying to break the work down into smaller tickes.

[jsanda]

I thought some more about config files/directories, /etc/cassandra in particular. Here is what I propose we do:

Store all config files in a ConfigMap. It becomes part of the operator installation. We need a ConfigMap for each Cassandra version that the operator supports. The ConfigMap can be installed via Kustomize or Helm along with all other operator resources. To be clear, I am not suggesting we generate the ConfigMap in the operator Go code.

We add a ConfigMap volume to the StatefulSet. The ConfigMap volume should be ready only. It will have the appropriate base configuration. Each pod needs its own configuration though. We add an EmptyDir volume for /etc/cassandra.

An init container with a volume mount for the ConfigMap will copy the base configuration onto the /etc/cassandra volume. This init container should run before any other init containers.

Currently the management-api's entry point script copies files generated by config-builder over to /etc/cassandra (see here). I don't think that will be necessary any more. The config builder init container can write directly to the /etc/cassandra volume mount.

In k8ssandra I think we could get rid of the base-config-init init container as well.

[emerkle826]

An init container with a volume mount for the ConfigMap will copy the base configuration onto the /etc/cassandra volume. This init container should run before any other init containers.

Currently the management-api's entry point script copies files generated by config-builder over to /etc/cassandra (see here). I don't think that will be necessary any more. The config builder init container can write directly to the /etc/cassandra volume mount.

This sounds good to me. My only question is if this is sufficient for the read-only root filesystem, or if this is just a part of the solution as it seems it only addresses write to /etc/cassandra? We still have to deal with writes to /opt/cassandra/logs//var/log/cassandra and potentially /opt/cassandra/data//var/lib/cassandra.

[jsanda]

My idea is only focused on addressing /etc/cassandra in particular. The approach is applicable though for configs is general. For some situations it might be overkill.

[jsanda]

I am doing some more work on this and wanted to point out that there is already an EmptyDir volume for /var/log/cassandra since the log directory needs to be accessible by the server-system-logger container.

There is already a PersistenVolume for /var/lib/cassandra. I think that leaves us with the following directories to worry about:

• /tmp
• /etc/cassandra
• /opt/mcac/config

[adejanovski]

@emerkle826 @burmanm, I think this is what we'll be discussing next week around moving more logic into the Dockerfiles and remove it from the entrypoint, right?

[burmanm]

This would be one of the tickets (involving cass-operator changes for new empty mounts).