Skip to content

Commit

Permalink
Merge pull request #18 from avast/avast/feature/on-demand-cluster-scan
Browse files Browse the repository at this point in the history
Introduce subcommands for main binary, add scanner functionality
  • Loading branch information
Jirka Korejtko authored Jun 24, 2019
2 parents dee12a8 + 36936a3 commit 3e96db1
Show file tree
Hide file tree
Showing 7 changed files with 387 additions and 227 deletions.
2 changes: 1 addition & 1 deletion Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -23,4 +23,4 @@ WORKDIR /app

COPY --from=builder /go/src/github.com/avast/k8s-admission-webhook/k8s-admission-webhook .

ENTRYPOINT ["./k8s-admission-webhook"]
ENTRYPOINT ["./k8s-admission-webhook", "webhook"]
4 changes: 4 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -53,3 +53,7 @@ copy-image-for-test: build-image-for-test
docker rmi $(TEST_IMAGE_NAME)

deploy-webhook-for-test: copy-image-for-test apply-webhook

build-binary:
# You might need to call "glide install -v" to download all libs
go build -ldflags="-w -s"
35 changes: 32 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,8 @@

A general-purpose Kubernetes [admission webhook](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/) to aid with enforcing best practices within your cluster.

Apart from webhook validation, this tool can also do [one-time outside of cluster scan](#on-demand-outside-of-cluster-scan) for objects that are violating validation rules.

Can be set up to validate that:
* containers have their resource limits specified (`memory`, `cpu`)
* containers have their resource requests specified (`memory`, `cpu`)
Expand Down Expand Up @@ -43,7 +45,7 @@ This implementation currently acts as a **validating admission webhook** to vali
## Configuration options
The webhook application can be configured using the flags outlined below.

Note that every option can also be specified via an environment variable: environment variables should upper-case, using `_` instead of `-` as seen in the flag name. E.g.: `--rule-resource-limit-cpu-required` can be alternatively set via an environment variable `RULE_RESOURCE_LIMIT_CPU_REQUIRED=1`.
Note that every option can also be specified via an environment variable. Environment variables should be in uppercase, using `_` instead of `-` as seen in the flag name. E.g.: `--rule-resource-limit-cpu-required` can be alternatively set via an environment variable `RULE_RESOURCE_LIMIT_CPU_REQUIRED=1`.

```
--listen-port int32 Port to listen on. (default 443)
Expand Down Expand Up @@ -119,6 +121,33 @@ Certain places of the example configuration YAML contain references to variables
* `${WEBHOOK_CA_BUNDLE}` (in `ValidatingWebhookConfiguration`) is the cluster's CA bundle.
- This can be retrieved by running `$(kubectl get configmap -n kube-system extension-apiserver-authentication -o=jsonpath='{.data.client-ca-file}' | base64 | tr -d '\n')`. You'll need to have `kubectl` on your `PATH`, with its current context pointing at the target cluster.

## On-demand outside of cluster scan
This tool provides also function to scan cluster (based on **current context** from `~/.kube/config`) for objects that are violating given rules.

Usage:
* Create binary via `make build-binary`
* Run `./k8s-admission-webhook scanner [options]`

Configuration options for cluster scanner:
```
--namespace Whether specific namespace should be scanned. If omitted, all namespaces are scanned.
--rule-resource-limit-cpu-must-be-nonzero Whether 'cpu' limit in resource specifications must be a nonzero value.
--rule-resource-limit-cpu-required Whether 'cpu' limit in resource specifications is required.
--rule-resource-limit-memory-must-be-nonzero Whether 'memory' limit in resource specifications must be a nonzero value.
--rule-resource-limit-memory-required Whether 'memory' limit in resource specifications is required.
--rule-resource-request-cpu-must-be-nonzero Whether 'cpu' request in resource specifications must be a nonzero value.
--rule-resource-request-cpu-required Whether 'cpu' request in resource specifications is required.
--rule-resource-request-memory-must-be-nonzero Whether 'memory' request in resource specifications must be a nonzero value.
--rule-resource-request-memory-required Whether 'memory' request in resource specifications is required.
--rule-security-readonly-rootfs-required Whether 'readOnlyRootFilesystem' in security context specifications is required.
--rule-security-readonly-rootfs-required-whitelist-enabled Whether rule 'readOnlyRootFilesystem' in security context can be ignored by container whitelisting.
--rule-resource-violation-message Additional message to be included whenever any of the resource-related rules are violated.
--rule-ingress-collision Whether ingress tls and host collision should be checked
--rule-ingress-violation-message Additional message to be included whenever any of the ingress-related rules are violated.
--annotations-prefix What prefix should be used for admission validation annotations.
```
Note that every option can also be specified via an environment variable. Environment variables should be in uppercase, using `_` instead of `-` as seen in the flag name. E.g.: `--rule-resource-limit-cpu-required` can be alternatively set via an environment variable `RULE_RESOURCE_LIMIT_CPU_REQUIRED=1`.

## Development
The webhook is written in Go and uses [Glide](https://glide.sh/) for dependency management.

Expand All @@ -133,10 +162,10 @@ These are some `make` targets intended to be used during local development. Note
#### Running the end-to-end tests
On the CI server, end-to-end tests are run via:
```
make ci-e2e-test KUBERNETES_VERSION=1.9
make ci-e2e-test KUBERNETES_VERSION=1.13
```

Other than `1.9`, tests can currently run also against `1.10` and `1.11` (see (.travis.yml)[travis.yml]) for more details.
Tests can currently run against versions `1.9` to `1.13` (see [.travis.yml](.travis.yml) for more details).

While `ci-e2e-test` spins up the whole cluster on every run, a more convenient workflow to run end-to-end tests during
local development is available via:
Expand Down
78 changes: 16 additions & 62 deletions config.go
Original file line number Diff line number Diff line change
@@ -1,12 +1,7 @@
package main

import (
"strings"

"github.com/pkg/errors"
log "github.com/sirupsen/logrus"
"github.com/spf13/cobra"
"github.com/spf13/viper"
)

type config struct {
Expand All @@ -28,82 +23,41 @@ type config struct {
RuleIngressCollision bool `mapstructure:"rule-ingress-collision"`
RuleIngressViolationMessage string `mapstructure:"rule-ingress-violation-message"`
AnnotationsPrefix string `mapstructure:"annotations-prefix"`
Namespace string `mapstructure:"namespace"`
}

var rootCmd = &cobra.Command{
Use: "k8s-admission-webhook",
Long: "Kubernetes Admission Webhook",
Run: func(cmd *cobra.Command, args []string) {},
}

func initialize() (*config, error) {
rootCmd.Flags().String("tls-cert-file", "",
"Path to the certificate file. Required, unless --no-tls is set.")
rootCmd.Flags().Bool("no-tls", false,
"Do not use TLS.")
rootCmd.Flags().String("tls-private-key-file", "",
"Path to the certificate key file. Required, unless --no-tls is set.")
rootCmd.Flags().Int32("listen-port", 443,
"Port to listen on.")

func initCommonFlags(cmd *cobra.Command) {
//pod
rootCmd.Flags().String("rule-resource-violation-message", "",
cmd.Flags().String("rule-resource-violation-message", "",
"Additional message to be included whenever any of the resource-related rules are violated.")
rootCmd.Flags().Bool("rule-resource-limit-cpu-required", false,
cmd.Flags().Bool("rule-resource-limit-cpu-required", false,
"Whether 'cpu' limit in resource specifications is required.")
rootCmd.Flags().Bool("rule-resource-limit-cpu-must-be-nonzero", false,
cmd.Flags().Bool("rule-resource-limit-cpu-must-be-nonzero", false,
"Whether 'cpu' limit in resource specifications must be a nonzero value.")
rootCmd.Flags().Bool("rule-resource-limit-memory-required", false,
cmd.Flags().Bool("rule-resource-limit-memory-required", false,
"Whether 'memory' limit in resource specifications is required.")
rootCmd.Flags().Bool("rule-resource-limit-memory-must-be-nonzero", false,
cmd.Flags().Bool("rule-resource-limit-memory-must-be-nonzero", false,
"Whether 'memory' limit in resource specifications must be a nonzero value.")
rootCmd.Flags().Bool("rule-resource-request-cpu-required", false,
cmd.Flags().Bool("rule-resource-request-cpu-required", false,
"Whether 'cpu' request in resource specifications is required.")
rootCmd.Flags().Bool("rule-resource-request-cpu-must-be-nonzero", false,
cmd.Flags().Bool("rule-resource-request-cpu-must-be-nonzero", false,
"Whether 'cpu' request in resource specifications must be a nonzero value.")
rootCmd.Flags().Bool("rule-resource-request-memory-required", false,
cmd.Flags().Bool("rule-resource-request-memory-required", false,
"Whether 'memory' request in resource specifications is required.")
rootCmd.Flags().Bool("rule-resource-request-memory-must-be-nonzero", false,
cmd.Flags().Bool("rule-resource-request-memory-must-be-nonzero", false,
"Whether 'memory' request in resource specifications must be a nonzero value.")
rootCmd.Flags().Bool("rule-security-readonly-rootfs-required", false,
cmd.Flags().Bool("rule-security-readonly-rootfs-required", false,
"Whether 'readOnlyRootFilesystem' in security context specifications is required.")
rootCmd.Flags().Bool("rule-security-readonly-rootfs-required-whitelist-enabled", false,
cmd.Flags().Bool("rule-security-readonly-rootfs-required-whitelist-enabled", false,
"Whether rule 'readOnlyRootFilesystem' in security context can be ignored by container whitelisting.")

//ingress
rootCmd.Flags().String("rule-ingress-violation-message", "",
cmd.Flags().String("rule-ingress-violation-message", "",
"Additional message to be included whenever any of the ingress-related rules are violated.")
rootCmd.Flags().Bool("rule-ingress-collision", false,
cmd.Flags().Bool("rule-ingress-collision", false,
"Whether ingress tls and host collision should be checked")

//customizations
rootCmd.Flags().String("annotations-prefix", "admission.validation.avast.com",
cmd.Flags().String("annotations-prefix", "admission.validation.avast.com",
"What prefix should be used for admission validation annotations.")

if err := viper.BindPFlags(rootCmd.Flags()); err != nil {
return errorWithUsage(err)
}

viper.AutomaticEnv()
viper.SetEnvKeyReplacer(strings.NewReplacer("-", "_"))

if err := rootCmd.Execute(); err != nil {
return errorWithUsage(err)
}

c := &config{}
if err := viper.Unmarshal(c); err != nil {
return errorWithUsage(err)
}

if !c.NoTLS && (c.TLSPrivateKeyFile == "" || c.TLSCertFile == "") {
return errorWithUsage(errors.New("Both --tls-cert-file and --tls-private-key-file are required (unless TLS is disabled by setting --no-tls)"))
}

return c, nil
}

func errorWithUsage(err error) (*config, error) {
log.Error(rootCmd.UsageString())
return nil, err
}
172 changes: 11 additions & 161 deletions main.go
Original file line number Diff line number Diff line change
@@ -1,181 +1,31 @@
package main

import (
"fmt"
"net/http"
"os"

log "github.com/sirupsen/logrus"
"k8s.io/api/admission/v1beta1"
appsv1 "k8s.io/api/apps/v1"
batchv1 "k8s.io/api/batch/v1"
batchv1beta1 "k8s.io/api/batch/v1beta1"
corev1 "k8s.io/api/core/v1"
extv1beta1 "k8s.io/api/extensions/v1beta1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/client-go/kubernetes"
"github.com/spf13/cobra"
)

func validate(ar v1beta1.AdmissionReview, config *config, clientSet *kubernetes.Clientset) *v1beta1.AdmissionResponse {
validation := &objectValidation{ar.Request.Kind.Kind, nil, &validationViolationSet{}}
deserializer := codecs.UniversalDeserializer()

raw := ar.Request.Object.Raw
var configMessage string
switch ar.Request.Kind.Kind {
case "Pod":
configMessage = config.RuleResourceViolationMessage
pod := corev1.Pod{}
if _, _, err := deserializer.Decode(raw, nil, &pod); err != nil {
log.Error(err)
return toAdmissionResponse(err)
}

log.Debugf("Admitting Pod: %+v", pod)
validation.ObjMeta = &pod.ObjectMeta
validatePodSpec(validation, &pod.ObjectMeta, &pod.Spec, config)

case "ReplicaSet":
configMessage = config.RuleResourceViolationMessage
replicaSet := appsv1.ReplicaSet{}
if _, _, err := deserializer.Decode(raw, nil, &replicaSet); err != nil {
log.Error(err)
return toAdmissionResponse(err)
}

log.Debugf("Admitting ReplicaSet: %+v", replicaSet)
validation.ObjMeta = &replicaSet.ObjectMeta
validatePodSpec(validation, &replicaSet.Spec.Template.ObjectMeta, &replicaSet.Spec.Template.Spec, config)

case "Deployment":
configMessage = config.RuleResourceViolationMessage
deployment := appsv1.Deployment{}
if _, _, err := deserializer.Decode(raw, nil, &deployment); err != nil {
log.Error(err)
return toAdmissionResponse(err)
}

log.Debugf("Admitting deployment: %+v", deployment)
validation.ObjMeta = &deployment.ObjectMeta
validatePodSpec(validation, &deployment.Spec.Template.ObjectMeta, &deployment.Spec.Template.Spec, config)

case "DaemonSet":
configMessage = config.RuleResourceViolationMessage
daemonSet := appsv1.DaemonSet{}
if _, _, err := deserializer.Decode(raw, nil, &daemonSet); err != nil {
log.Error(err)
return toAdmissionResponse(err)
}

log.Debugf("Admitting DaemonSet: %+v", daemonSet)
validation.ObjMeta = &daemonSet.ObjectMeta
validatePodSpec(validation, &daemonSet.Spec.Template.ObjectMeta, &daemonSet.Spec.Template.Spec, config)

case "Job":
configMessage = config.RuleResourceViolationMessage
job := batchv1.Job{}
if _, _, err := deserializer.Decode(raw, nil, &job); err != nil {
log.Error(err)
return toAdmissionResponse(err)
}

log.Debugf("Admitting Job: %+v", job)
validation.ObjMeta = &job.ObjectMeta
validatePodSpec(validation, &job.Spec.Template.ObjectMeta, &job.Spec.Template.Spec, config)

case "CronJob":
configMessage = config.RuleResourceViolationMessage
cronJob := batchv1beta1.CronJob{}
if _, _, err := deserializer.Decode(raw, nil, &cronJob); err != nil {
log.Error(err)
return toAdmissionResponse(err)
}

log.Debugf("Admitting CronJob: %+v", cronJob)
validation.ObjMeta = &cronJob.ObjectMeta
validatePodSpec(validation, &cronJob.Spec.JobTemplate.Spec.Template.ObjectMeta, &cronJob.Spec.JobTemplate.Spec.Template.Spec, config)

case "Ingress":
configMessage = config.RuleIngressViolationMessage
ingress := extv1beta1.Ingress{}
if _, _, err := deserializer.Decode(raw, nil, &ingress); err != nil {
log.Error(err)
return toAdmissionResponse(err)
}

log.Debugf("Admitting Ingress: %+v", ingress)
validation.ObjMeta = &ingress.ObjectMeta
err := ValidateIngress(validation, &ingress, config, clientSet)
if err != nil {
return toAdmissionResponse(err)
}

case "StatefulSet":
configMessage = config.RuleResourceViolationMessage
statefulSet := appsv1.StatefulSet{}
if _, _, err := deserializer.Decode(raw, nil, &statefulSet); err != nil {
log.Error(err)
return toAdmissionResponse(err)
}

log.Debugf("Admitting stateful set: %+v", statefulSet)
validation.ObjMeta = &statefulSet.ObjectMeta
validatePodSpec(validation, &statefulSet.Spec.Template.ObjectMeta, &statefulSet.Spec.Template.Spec, config)

default:
log.Warnf("Admitted an unexpected resource: %v", ar.Request.Kind)
}

reviewResponse := v1beta1.AdmissionResponse{}

message := validation.message(configMessage)
if len(message) > 0 {
reviewResponse.Allowed = false
reviewResponse.Result = &metav1.Status{Message: message}
} else {
reviewResponse.Allowed = true
}

return &reviewResponse
var rootCmd = &cobra.Command {
Use: "k8s-admission-webhook",
Long: "Kubernetes Admission Webhook",
}

func main() {
initLogger()

//parse command line arguments
config, configErr := initialize()
if configErr != nil {
log.Fatal(configErr)
}
log.Debugf("Configuration is: %+v", config)

//initialize kube client
kubeClientSet, kubeClientSetErr := KubeClientSet(true)
if kubeClientSetErr != nil {
log.Fatal(kubeClientSetErr)
if err := rootCmd.Execute(); err != nil {
errorWithUsage(err)
}

http.HandleFunc("/validate", admitFunc(validate).serve(config, kubeClientSet))

addr := fmt.Sprintf(":%v", config.ListenPort)
var httpErr error
if config.NoTLS {
log.Infof("Starting webserver at %v (no TLS)", addr)
httpErr = http.ListenAndServe(addr, nil)
} else {
log.Infof("Starting webserver at %v (TLS)", addr)
httpErr = http.ListenAndServeTLS(addr, config.TLSCertFile, config.TLSPrivateKeyFile, nil)
}

if httpErr != nil {
log.Fatal(httpErr)
} else {
log.Info("Finished")
}

}

func initLogger() {
log.SetOutput(os.Stdout)
log.SetLevel(log.DebugLevel)
}

func errorWithUsage(err error) {
log.Error(rootCmd.UsageString())
log.Fatal(err)
}
Loading

0 comments on commit 3e96db1

Please sign in to comment.