Hardened Security Context for Elasticsearch

[barkbay]

If Elasticsearch configuration files are copied before symbolic links are created, it is possible to run all the Elasticsearch containers with the following restricted securityContext

    securityContext:
      allowPrivilegeEscalation: false
      capabilities:
        drop:
        - ALL
      privileged: false
      readOnlyRootFilesystem: true

Fixes #6126

See my comment here if you want to review this PR: #6703 (comment)

[barkbay]

buildkite test this -f p=gke,s=8.7.0,s=7.17.8

[thbkrkr]

buildkite test this -f p=gke,s=8.7.0,s=7.17.8

This will only run the e2e tests with s=7.17.8. When you want to test one variable with different values (here s the stack version), you need to declare this variable and its values in the --mixed|-m flag.

• buildkite test this --fixed p=gke --mixed s=8.7.0,s=7.17.8
• buildkite test this -f p=gke -m s=8.7.0,s=7.17.8 (with the short flags)

[barkbay]

@thbkrkr Thanks! I think I spotted a problem with this PR, I will use your advice next time 🙇

[barkbay]

• TestVolumeEmptyDir failed because this "feature" is itself broken: emptyDir volume is not mounted #6186. I added a function to detect, on a best effort basis, whether or not data are in a volume.
• TestKibanaStandalone also failed, but I'm not sure to understand why 🤷

[thbkrkr]

TestKibanaStandalone looks very similar to #6693 with all sub-tests failing as we don't have the end of the test execution in the log because the log stream was interrupted.

log

{
    "Time": "2023-04-19T17:16:36.671150963Z",
    "Action": "output",
    "Package": "github.com/elastic/cloud-on-k8s/v2/test/e2e/kb",
    "Test": "TestKibanaStandalone/ES_HTTP_certificate_authority_should_be_set_and_deployed",
    "Output": "=== RUN   TestKibanaStandalone/ES_HTTP_certificate_authority_should_be_set_and_deployed\n"
}
{
    "log.level": "info",
    "@timestamp": "2023-04-19T17:30:05.540Z",
    "log.logger": "e2e-pdzcj",
    "message": "Log stream ended",
    "service.version": "0.0.0-SNAPSHOT+00000000",
    "service.type": "eck",
    "ecs.version": "1.4.0",
    "pod_name": "eck-e2e-pdzcj-cl86s"
}
{
    "log.level": "info",
    "@timestamp": "2023-04-19T17:30:05.540Z",
    "log.logger": "e2e-pdzcj",
    "message": "Streaming pod logs",
    "service.version": "0.0.0-SNAPSHOT+00000000",
    "service.type": "eck",
    "ecs.version": "1.4.0",
    "pod_name": "eck-e2e-pdzcj-cl86s"
}

[barkbay]

buildkite test this --fixed p=gke --mixed s=8.7.0,s=7.17.8

[barkbay]

buildkite test this -f p=gke -m s=8.7.0,s=7.17.8

[barkbay]

Failures to investigate from the last e2e tests session:

I think this is because I added a check to validate that all the containers in the Elasticsearch Pod are running with the expected securityContext. That's probably not the case for Beat sidecars when stack monitoring is enabled.

[barkbay]

buildkite test this -f p=gke -m s=8.7.0,s=7.17.8

[barkbay]

buildkite test this -f p=gke -m s=8.7.0,s=7.17.8

[barkbay]

buildkite test this -f p=gke -m s=8.0.1,s=8.8.0-SNAPSHOT

[barkbay]

• Tests passed for 8.7.0, 7.17.8 on GKE 🎉
• I started an e2e test session on 8.8.0-SNAPSHOT for which RunAsNonRoot is expected to be true thanks to Use UID for dockerfile to allow runAsNonRoot to be used. elasticsearch#95390
• I performed preliminary tests on OpenShift and things seem to work as expected.

Some "noteworthy" things if you want to review this PR:

• All stack monitoring sidecar containers (both Metricbeat and Filebeat) now have their /usr/share/metricbeat/data directory mounted in an emptyDir to allow readOnlyRootFilesystem to be true.
• However, only Elasticsearch sidecar containers have their securityContext updated. I haven't had time to do extensive testing on other components, I think that if we want to target 2.8.0 it is safer to limit this change to Elasticsearch.
• ReadOnlyRootFilesystem is set to true iif it is obvious from the operator point of view that data are written in a volume: https://github.com/elastic/cloud-on-k8s/pull/6703/files#diff-0e1da5b5ab6c4b7fa20fcc3151370567119207c7a353e1e9bf9bb2bdb9a53d8eR106-R109 (see also my comment about TestVolumeEmptyDir here: Hardened Security Context for Elasticsearch #6703 (comment))
• Containers SecurityContext is set automatically iff none is initially set. In other words, it's not merged with a SecurityContext that would come from the podTemplate: https://github.com/elastic/cloud-on-k8s/pull/6703/files#diff-e0b41f58404fb3608247e1e16a04b3b27a1abcafd46e1507d5b86b8c05aa30ffR349-R359

[barkbay]

buildkite test this -f p=ocp,s=8.7.0

[barkbay]

Results from the last e2e test sessions:

• OCP4 🟢
• 7.17.8/8.0.1/8.7.0 on GKE 🟢
• 8.8.0-SNAPSHOT 🟡 - 140/144 tests passed, TestKibanaStandalone failed. As for this failure, I don't think it is related to the changes in this PR.

[thbkrkr]

LGTM, this is great!

Looks also like we're writing code specifically to handle #6186 but why not fix it first? We would no longer need to detect if the data directory is mounted in a volume as this is normally impossible, no? So we could call WithContainersSecurityContext after stackmon.WithMonitoring, to reuse the same code to set the SecurityContext for beats sidecars.

[barkbay]

Looks also like we're writing code specifically to handle #6186 but why not fix it first?

At first I thought this code was specifically handling #6186. But I've been wondering if, in general, we should not set readOnlyRootFilesystem to true only if we are pretty confident in the fact that the data are written in a volume, not in the container. I'm thinking about users that will upgrade to 2.8.0: if we set readOnlyRootFilesystem to true too "aggressively" we may end up with issues during upgrades if Elasticsearch is not configured properly. That's why I decided to set readOnlyRootFilesystem to true only if the default data volume is mounted as expected. If a user has a more involved storage configuration, with different volumes, it would have to test this flag manually first.

I have to admit this is still brittle, as there are other ways to mis-configure Elasticsearch with the default volume, ending up with data being written into the container. As you suggest, we could also always enable readOnlyRootFilesystem and let the user fix the configuration if the Pods crash, I'm trying to avoid that situation by keeping this code.

[thbkrkr]

But I've been wondering if, in general, we should not set readOnlyRootFilesystem to true only if we are pretty confident in the fact that the data are written in a volume, not in the container. I'm thinking about users that will upgrade to 2.8.0: if we set readOnlyRootFilesystem to true too "aggressively" we may end up with issues during upgrades if Elasticsearch is not configured properly.

👍 👍

[pebrc]

Code LGTM I have not done many tests yet.

[pebrc]

Can we use a shared variable or function here given that we use the exact same context twice? Also potentially we could refer to the new securitycontext package here, to have all contexts in one place?

[barkbay]

👍 done in 18818fc

[pebrc]

LGTM I managed to run a few tests on GKE and OCP and things seem to work fine!

[barkbay]

I'll open an issue to update https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-security-context.html before the next release as it is definitely outdated.

Edit: I think we can actually work on this as part of #5913