Add Beat config library #3406
Add Beat config library #3406
Conversation
| securityContext: | ||
| runAsUser: 0 | ||
| # If using Red Hat OpenShift uncomment this: | ||
| #privileged: true |
There was a problem hiding this comment.
The privileged field does not exist in the PodSecurityContext. It is only available in the container's SecurityContext.
Also I'm wondering if it is required for all versions of Openshift since it's seems to work out of the box on Openshift 4 according to our e2e tests ? I think it is required on Openshift 3.11 to bypass SELinux on the hostPath volume.
There was a problem hiding this comment.
I've fixed the placement, I'll try to test at some point to see if only 3.11 is affected. FYI, the comment was taken from Beats examples, but you are right, it works in our E2E tests.
| customize := func(builder beat.Builder) beat.Builder { | ||
| return builder. | ||
| WithRoles(beat.PSPClusterRoleName, beat.AutodiscoverClusterRoleName). | ||
| WithESValidations( |
There was a problem hiding this comment.
I think it's a bit odd that we define test validations as part of the builder, that builds the test subjects, but I guess this is a general problem in our e2e setup at the moment.
Co-authored-by: Peter Brachwitz <peter.brachwitz@gmail.com>
Co-authored-by: Peter Brachwitz <peter.brachwitz@gmail.com>
|
I went and added missing RBAC, but then I realized that those kubernetes processors are not adding anything to the events, so I removed them. In case of auditbeat and journalbeat I'm not sure if kubernetes metadata makes sense at all as they interact with hosts. For packetbeat, I think it would make sense to annotate events with src/dst containers, but it doesn't seem to happen and I'm not sure if this is a Beat limitation or something is wrong with the config. I think that I'd leave that for a future improvement. |
k8s nodes still have labels and annotations, so I could see how it might happen, but going by your message it does not 🤷♀️ maybe worth opening a docs issue just to clarify what resources/beats actually add metadata |
The three filebeat examples share a beat name so overlap each other, I was a bit confused at first when the k8s metadata was not being added. Maybe this is fine, but we probably want to update the initial PR if so. Otherwise we may want to change the name of the beats |
|
Don't think this is a blocker but: I'm at least having trouble with the auditbeat recipe, the only events in the dashboard are
Even after I deleted the dset and let ECK recreate it. So hopped on and saw
And of course @jordansissel is forever in our hearts and already ran into the same issue: I wonder if we want to just link to that as a known COS issue. I can also see the argument for including the fix in a recipe though since we have a lot of GKE customers. |
| @@ -0,0 +1,39 @@ | |||
| # Beats Configuration Library | |||
|
|
|||
| This directory contains yaml manifests with example configurations for Beats. These manifests are self-contained and work out-of-the-box on any non-secured Kubernetes cluster. All of them contain three-node Elasticsearch cluster and single Kibana instance. All Beat configurations set up Kibana dashboards if they are available for a given Beat and all required RBAC resources. | |||
There was a problem hiding this comment.
Should we mention how the Elasticsearch spec is not suitable to a production environment?
There was a problem hiding this comment.
This is already there in https://github.com/elastic/cloud-on-k8s/blob/master/config/recipes/README.asciidoc, should we just repeat it here?
There was a problem hiding this comment.
I'm not sure people will notice the root README, it's worth repeating here imo.
config/recipes/beats/README.md
Outdated
| @@ -0,0 +1,39 @@ | |||
| # Beats Configuration Library | |||
There was a problem hiding this comment.
I'm wondering how those examples will be referenced from the documentation on the website. I think there's value in listing all the examples explicitly in the docs (ie. copying this markdown file content, basically).
There was a problem hiding this comment.
I tried to include the samples in the documentation a long time ago but the docs build only checks out the docs directory so that wasn't successful. We'll either have to move the samples inside the docs directory or figure out the magic incantation required by the docs build to checkout more directories.
There was a problem hiding this comment.
I planned to copy it to the docs almost 1:1. The only thing I'd add in the docs is kubectl apply command for each example.
I forgot to mention I tested on AKS where I did not run into the auditbeat issue @anyasabo found. |
But is "self-contained" implying they won't overlap? I didn't mean to imply the latter. I'd be reluctant to rename, because we'd also need to rename RBAC pieces and ES/Kibana for consistency. Right now they are fairly easy to ready, but if we start encoding more things into the names they might get more complicated. |
|
@ChrsMark FYI |
I think that combined with the suggestion to just:
to review is what threw me off. I'm good with leaving the names as is, just a note for other reviewers 👍 |
There was a problem hiding this comment.
I'm good with this as is but would suggest a couple changes:
- adding a note in the readme about packetbeat+gke w/ COS being broken, or including another recipe that works for GKE
- adding comments justifying the host network usage in beats where it's not necessary (which I think is everything but packetbeat)
charith-elastic
added
:ci
* Add Beat config library * Fix commented security context placement * Apply suggestions from code review Co-authored-by: Peter Brachwitz <peter.brachwitz@gmail.com> * Add beats config library readme * Apply suggestions from code review Co-authored-by: Peter Brachwitz <peter.brachwitz@gmail.com> * Remove add_kubernetes_metadata processor * PR fixes * Add note about namespace expectations for heartbeat example * Add hostNetwork comments, add fix for GKE Co-authored-by: Peter Brachwitz <peter.brachwitz@gmail.com>
* Add Beat config library * Fix commented security context placement * Apply suggestions from code review Co-authored-by: Peter Brachwitz <peter.brachwitz@gmail.com> * Add beats config library readme * Apply suggestions from code review Co-authored-by: Peter Brachwitz <peter.brachwitz@gmail.com> * Remove add_kubernetes_metadata processor * PR fixes * Add note about namespace expectations for heartbeat example * Add hostNetwork comments, add fix for GKE Co-authored-by: Peter Brachwitz <peter.brachwitz@gmail.com>
… (#3459) * Add Beat config library (#3406) * Add Beat config library * Fix commented security context placement * Apply suggestions from code review Co-authored-by: Peter Brachwitz <peter.brachwitz@gmail.com> * Add beats config library readme * Apply suggestions from code review Co-authored-by: Peter Brachwitz <peter.brachwitz@gmail.com> * Remove add_kubernetes_metadata processor * PR fixes * Add note about namespace expectations for heartbeat example * Add hostNetwork comments, add fix for GKE Co-authored-by: Peter Brachwitz <peter.brachwitz@gmail.com> * Fix recipes E2E tests (#3462) * Add permissions for e2e runner necessary for recipes testing * Add a note about examples purpose * Sync Beat examples descriptions between readme and docs Co-authored-by: Peter Brachwitz <peter.brachwitz@gmail.com>
This PR introduces a configuration library for Beat CRD. Yaml files in
config/recipes/beatsare self-contained and should be fully functional in non-secured cluster (PSP/SCC is out of scope).Each recipe is tested as an E2E test with custom (per recipe) setup and checks. Docs linking to those recipes will come as a separate PR. Related: #3319.