From b7022817b4ceefb74ca3c79cbf979560657a022d Mon Sep 17 00:00:00 2001 From: Anshuman Tripathi Date: Fri, 9 Feb 2024 23:57:19 -0800 Subject: [PATCH] Security hardening guide for scheduler configuration Signed-off-by: Anshuman Tripathi [WIP] Security hardening guide for scheduler configurations Signed-off-by: Anshuman Tripathi Updates after passing through hemmingway.app Signed-off-by: Anshuman Tripathi Update scheduling configurations Signed-off-by: Anshuman Tripathi Apply suggestions from code review Co-authored-by: Tim Bannister Co-authored-by: Daniel Register Updates based on PR feedback Signed-off-by: Anshuman Tripathi Update bind-address definition Signed-off-by: Anshuman Tripathi Update phrasing of permit-address-sharing Signed-off-by: Anshuman Tripathi Add -- to args Signed-off-by: Anshuman Tripathi Sentence case in table title Signed-off-by: Anshuman Tripathi Reword and correct grammer based on feedback Signed-off-by: Anshuman Tripathi Remove verbatim argument description Signed-off-by: Anshuman Tripathi More updates Signed-off-by: Anshuman Tripathi Update custom scheduler heading and description Signed-off-by: Anshuman Tripathi Remove dashes on args Signed-off-by: Anshuman Tripathi Apply suggestions from code review Co-authored-by: Tim Bannister Signed-off-by: Anshuman Tripathi Update table title Signed-off-by: Anshuman Tripathi Update based on feedback Signed-off-by: Anshuman Tripathi node selector Signed-off-by: Anshuman Tripathi Feedback Signed-off-by: Anshuman Tripathi --- .../security/hardening-guide/scheduler.md | 92 +++++++++++++++++++ 1 file changed, 92 insertions(+) create mode 100644 content/en/docs/concepts/security/hardening-guide/scheduler.md diff --git a/content/en/docs/concepts/security/hardening-guide/scheduler.md b/content/en/docs/concepts/security/hardening-guide/scheduler.md new file mode 100644 index 0000000000000..1387726e2c05c --- /dev/null +++ b/content/en/docs/concepts/security/hardening-guide/scheduler.md @@ -0,0 +1,92 @@ +--- +title: "Hardening Guide - Scheduler Configuration" +description: > + Information about how to make the Kubernetes scheduler more secure. +content_type: concept +weight: 90 +--- + + +The Kubernetes {{< glossary_tooltip text="scheduler" term_id="kube-scheduler" >}} is +one of the critical components of the +{{< glossary_tooltip text="control plane" term_id="control-plane" >}}. + +This document covers how to improve the security posture of the Scheduler. + +A misconfigured scheduler can have security implications. Such a scheduler can target specific nodes and evict the workloads or applications that are sharing the node and its resources. This can aid an attacker with a [Yo-Yo attack](https://arxiv.org/abs/2105.00542): an attack on a vulnerable autoscaler. + +## kube-scheduler configuration + +### Scheduler authentication & authorization command line options +{{}} +| Command line argument | Security hardening advice | +| ---------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- | +| `authentication-kubeconfig` | Make sure to provide a proper kubeconfig so that the server calls are secure. This kubeconfig file should also be maintained securely. | +| `authentication-tolerate-lookup-failure` | Set to `false` to make sure invalid authentication configurations do not lead to requests passing off as anonymous | +| `authentication-skip-lookup` | This should be set to `false` to make sure all missing authentication configuration falls back to the authentication kubeconfig. | +| `authorization-always-allow-paths` | These paths should respond with data that is appropriate for anonymous authorization. Defaults to `/healthz,/readyz,/livez`. | +{{
}} + +### Scheduler networking command line options +{{}} +| Command line argument | Security hardening advice | +| ------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `bind-address` | In most cases, the kube-scheduler does not need to be externally accessible. Setting the bind address to `localhost` is a secure practice. | +| `permit-address-sharing` | Set this to `false` to disable connection sharing through `SO_REUSEADDR`. `SO_REUSEADDR` can lead to reuse of terminated connections that are in `TIME_WAIT` state. | +| `permit-port-sharing` | Default `false`. Use the default unless you are confident you understand the security implications. | +{{
}} + +### Scheduler TLS command line options +{{}} +| Command line argument | Security hardening advice | +| ------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `requestheader-client-ca-file` | Generally do not depend on authorization being already done for incoming requests. Always provide the root certificate bundle. This allows authorization to happen on each incoming request through `requestheader-allowed-names`. | +| `tls-cipher-suites` | Always provide a list of preferred cipher suites. This ensures encryption never happens with insecure cipher suites. | +{{
}} + +## Scheduling configurations for custom schedulers +This section covers security hardening when using custom schedulers based on the Kubernetes +scheduling code. +The cluster administrator needs be careful with the plugins that use the `queueSort`, `prefilter`, `filter`, or permit [extension points](https://github.com/docs/reference/scheduling/config/#extension-points). +Scheduling happens in a series of stages that are exposed through the extension points. +As a cluster administrator, you can enable plugins that define their own extension points. +Doing so can affect the defined scheduling behaviors of the kube-scheduler in your cluster. + +When using a custom scheduler, plugin extension points such as `queueSort`, `prefilter`, `filter` and `permit` should be used with care. + +Exactly one plugin that uses the `queueSort` extension point can be enabled at a time. Any plugins that use `queueSort` should be scrutinized. + +Plugins that implement the `prefilter` or `filter` extension point can potentially mark all nodes as unschedulable. This can bring scheduling of new pods to a halt. + +Plugins that implement the `permit` extension point can prevent or delay the binding of a Pod. Such plugins should be thoroughly reviewed by the cluster administrator. + +When using a plugin that is not one of the [default plugins](/docs/reference/scheduling/config/#scheduling-plugins), consider disabling the `queueSort`, `filter` and `permit` extension points as follows: + +```yaml +apiVersion: kubescheduler.config.k8s.io/v1 +kind: KubeSchedulerConfiguration +profiles: + - schedulerName: default-scheduler + - schedulerName: my-custom-scheduler + plugins: + queueSort: + disabled: + # Globally only one queueSort can be specified. + - name: my-custom-scheduler + # - name: default-scheduler + permit: + disabled: + - name: '*' +``` +This creates two scheduler profiles: `default-scheduler` and ` my-custom-scheduler`. +Whenever the `.spec` of a Pod does not have a value for `.spec.schedulerName`, +the kube-scheduler runs for that Pod, using its main configuration, and default plugins. +If you define a Pod with `.spec.schedulerName` set to `my-custom-scheduler`, the +kube-scheduler also runs but with a custom configuration; in that custom configuration, +the `queueSort`, `filter` and `permit` extension points are disabled. +If you use this kube-scheduler configuration, and don't run any custom scheduler, and +you then define a Pod with `.spec.schedulerName` set to `nonexistent-scheduler` (or +any other scheduler name that doesn't exist in your cluster), no events would be generated for a pod. + +## Disallow labeling nodes +A cluster administrator should ensure that cluster users cannot label the nodes. A malicious actor can use `nodeSelector` to schedule workloads on nodes where those workloads should not be present.