Skip to content

Commit

Permalink
Allow adding extra fields and lint yaml
Browse files Browse the repository at this point in the history
  • Loading branch information
mosheavni committed Mar 24, 2021
1 parent 32d507d commit bab7bbb
Show file tree
Hide file tree
Showing 4 changed files with 163 additions and 139 deletions.
151 changes: 85 additions & 66 deletions filebeat/README.md
Original file line number Diff line number Diff line change
@@ -1,34 +1,45 @@
# Logzio-k8s-logs

Helm is a tool for managing packages of pre-configured Kubernetes resources using Charts.
Logzio-k8s-logs allows you to ship logs from your Kubernetes cluster to Logz.io.
You can either deploy this Daemonset with the standrad configuration, or with autodiscover configuration. For further information about Filebeat's autodiscover please see [Autodiscover documentation](https://www.elastic.co/guide/en/beats/filebeat/current/configuration-autodiscover.html).
Helm is a tool for managing packages of pre-configured Kubernetes resources
using Charts. Logzio-k8s-logs allows you to ship logs from your Kubernetes
cluster to Logz.io. You can either deploy this Daemonset with the standrad
configuration, or with autodiscover configuration. For further information about
Filebeat's autodiscover please see [Autodiscover documentation](https://www.elastic.co/guide/en/beats/filebeat/current/configuration-autodiscover.html).

## Prerequisites

### Prerequisites:
* [Helm CLI](https://helm.sh/docs/intro/install/) installed
* Allow outgoing traffic to destination port 5015

## Deployment

You have two options for deployment:

* [Standard configuration](#standard-config)
* [Autodiscover configuration](#autodiscover-config)

**Note:** Helm 2 will reach [EOL on November 2020](https://helm.sh/blog/2019-10-22-helm-2150-released/#:~:text=6%20months%20after%20Helm%203's,Helm%202%20will%20formally%20end). This document follows the command syntax recommended for Helm 3, but the Chart will work with both Helm 2 and Helm 3.
<div id="standard-config">
**Note:** Helm 2 will reach [EOL on November
2020](https://helm.sh/blog/2019-10-22-helm-2150-released/#:~:text=6%20months%20after%20Helm%203's,Helm%202%20will%20formally%20end).
This document follows the command syntax recommended for Helm 3, but the Chart
will work with both Helm 2 and Helm 3.

### Standard configuration deployment:
### Add logzio-k8s-logs repo to your helm repo list

#### 1. Add logzio-k8s-logs repo to your helm repo list
```shell
helm repo add logzio-helm https://logzio.github.io/logzio-helm/filebeat
```

```shell
helm repo add logzio-helm https://logzio.github.io/logzio-helm/filebeat
```
<div id="standard-config">

#### 2. Deploy
### Standard configuration deployment

Replace `<<SHIPPING-TOKEN>>` with the [token](https://app.logz.io/#/dashboard/settings/general) of the account you want to ship to.
Replace `<<SHIPPING-TOKEN>>` with the
[token](https://app.logz.io/#/dashboard/settings/general) of the account you
want to ship to.
Replace `<<LISTENER-REGION>>` with your region’s code (for example, `eu`). For more information on finding your account’s region, see [Account region](https://docs.logz.io/user-guide/accounts/account-region.html).
Replace `<<LISTENER-REGION>>` with your region’s code (for example, `eu`). For
more information on finding your account’s region, see [Account
region](https://docs.logz.io/user-guide/accounts/account-region.html).
Replace `<<CLUSTER-NAME>>` with your cluster's name.
Expand All @@ -40,32 +51,32 @@ helm install --namespace=kube-system \
logzio-k8s-logs logzio-helm/logzio-k8s-logs
```
#### 3. Check Logz.io for your logs
Give your logs some time to get from your system to ours, and then open [Logz.io](https://app.logz.io/).

</div>
<div id="autodiscover-config">
### Autodiscover configuration deployment:

Autodiscover allows you to adapt settings as changes happen. By defining configuration templates, the autodiscover subsystem can monitor services as they start running.
### Autodiscover configuration deployment
#### 1. Add logzio-k8s-logs repo to your helm repo list

```shell
helm repo add logzio-helm https://logzio.github.io/logzio-helm/filebeat
```
Autodiscover allows you to adapt settings as changes happen. By defining
configuration templates, the autodiscover subsystem can monitor services as they
start running.
#### 3. Deploy
In the following commands, make the following changes:
* Replace `<<SHIPPING-TOKEN>>` with the [token](https://app.logz.io/#/dashboard/settings/general) of the account you want to ship to.
* Replace `<<LISTENER-REGION>>` with your region’s code (for example, `eu`). For more information on finding your account’s region, see [Account region](https://docs.logz.io/user-guide/accounts/account-region.html).
* Replace `<<SHIPPING-TOKEN>>` with the
[token](https://app.logz.io/#/dashboard/settings/general) of the account you
want to ship to.
* Replace `<<LISTENER-REGION>>` with your region’s code (for example, `eu`). For
more information on finding your account’s region, see [Account
region](https://docs.logz.io/user-guide/accounts/account-region.html).
* Replace `<<CLUSTER-NAME>>` with your cluster's name.
This Daemonset's default autodiscover configuration is [hints based](https://www.elastic.co/guide/en/beats/filebeat/current/configuration-autodiscover-hints.html). If you wish to deploy it use:
This Daemonset's default autodiscover configuration is [hints
based](https://www.elastic.co/guide/en/beats/filebeat/current/configuration-autodiscover-hints.html).
If you wish to deploy it use:
```shell
helm install --namespace=kube-system \
--set configType='autodiscover' \
Expand All @@ -74,46 +85,55 @@ helm install --namespace=kube-system \
--set secrets.clusterName='<<CLUSTER-NAME>>' \
logzio-k8s-logs logzio-helm/logzio-k8s-logs
```
If you have a custom configuration, deploy with:
```shell
helm install --namespace=kube-system \
--set configType='auto-custom' \
--set configType='custom' \
--set secrets.logzioShippingToken='<<SHIPPING-TOKEN>>' \
--set secrets.logzioRegion='<<LISTENER-REGION>>' \
--set secrets.clusterName='<<CLUSTER-NAME>>' \
--set-file filebeatConfig.autoCustomConfig=/path/to/your/config.yaml \
logzio-k8s-logs logzio-helm/logzio-k8s-logs
```
*Note:* If you're using a custom config, please make sure that you're using a `.yaml` file in the following structure:
```
filebeat.yml: |-
filebeat.autodiscover:
#....
# your autodiscover config
# ...
processors:
- add_cloud_metadata: ~
fields:
logzio_codec: ${LOGZIO_CODEC}
token: ${LOGZIO_LOGS_SHIPPING_TOKEN}
cluster: ${CLUSTER_NAME}
type: ${LOGZIO_TYPE}
fields_under_root: ${FIELDS_UNDER_ROOT}
ignore_older: ${IGNORE_OLDER}
output:
logstash:
hosts: ["${LOGZIO_LOGS_LISTENER_HOST}:5015"]
ssl:
certificate_authorities: ['/etc/pki/tls/certs/SectigoRSADomainValidationSecureServerCA.crt']
*Note:* If you're using a custom config, please make sure that you're using a
`.yaml` file in the following structure:
```shell
filebeatConfig:
customConfig: |-
filebeat.autodiscover:
#....
# your autodiscover config
# ...
processors:
- add_cloud_metadata: ~
fields:
logzio_codec: ${LOGZIO_CODEC}
token: ${LOGZIO_LOGS_SHIPPING_TOKEN}
cluster: ${CLUSTER_NAME}
type: ${LOGZIO_TYPE}
fields_under_root: ${FIELDS_UNDER_ROOT}
ignore_older: ${IGNORE_OLDER}
output:
logstash:
hosts: ["${LOGZIO_LOGS_LISTENER_HOST}:5015"]
ssl:
certificate_authorities: ['/etc/pki/tls/certs/SectigoRSADomainValidationSecureServerCA.crt']
```
#### 4. Check Logz.io for your logs
### Check Logz.io for your logs
Give your logs some time to get from your system to ours, and then open [Logz.io](https://app.logz.io/).
Also, you can verify you don't have any [drop filters](https://app.logz.io/#/dashboard/tools/drop-filters)
that may drop your newly sent logs.
</div>
### Configuration
## Configuration
| Parameter | Description | Default |
|---|---|---|
Expand All @@ -132,9 +152,8 @@ Give your logs some time to get from your system to ours, and then open [Logz.io
| `clusterRoleRules` | Configurable [cluster role rules](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#role-and-clusterrole) that Filebeat uses to access Kubernetes resources. | See [values.yaml](https://github.com/logzio/logzio-helm/blob/master/filebeat/values.yaml) |
| `logzioCert` | Logzio public SSL certificate. | See [values.yaml](https://github.com/logzio/logzio-helm/blob/master/filebeat/values.yaml) |
| `configType` | Specifies which configuration to use for Filebeat. Set to `autodiscover` to use autodiscover. | `standard` |
| `filebeatConfig.standardConfig` | Standard Filebeat configuration, using `filebeat.input`. | See [values.yaml](https://github.com/logzio/logzio-helm/blob/master/filebeat/values.yaml) |
| `filebeatConfig.autodiscoverConfig` | Autodiscover Filebeat configuration, using `filebeat.autodiscover`. | See [values.yaml](https://github.com/logzio/logzio-helm/blob/master/filebeat/values.yaml) |
| `filebeatConfig.autoCustomConfig` | Autodiscover Filebeat custom configuration, using `filebeat.autodiscover`. Should be used if you want to use your custimized autodiscover config | {} |
| `filebeatConfig.customConfig` | Filebeat custom configuration, using `filebeat.autodiscover`. Should be used if you want to use your custimized autodiscover config | {} |
| `filebeatConfig.extraFields` | Filebeat processor add_fields to send extra fields to every log line | {} |
| `serviceAccount.create` | Specifies whether a service account should be created. | `true` |
| `serviceAccount.name` | Name of the service account. | `filebeat` |
| `terminationGracePeriod` | Termination period (in seconds) to wait before killing Filebeat pod process on pod shutdown. | `30` |
Expand All @@ -153,27 +172,27 @@ Give your logs some time to get from your system to ours, and then open [Logz.io
| `secrets.logzioRegion`| Secret with your [logzio region](https://docs.logz.io/user-guide/accounts/account-region.html). Defaults to US East. | `" "` |
| `secrets.clusterName`| Secret with your cluster name. | `""` |

If you wish to change the default values, specify each parameter using the `--set key=value` argument to `helm install`. For example,
If you wish to change the default values, specify each parameter using the
`--set key=value` argument to `helm install`. For example,
```shell
helm install --namespace=kube-system logzio-k8s-logs logzio-helm/logzio-k8s-logs \
--set imageTag=7.7.0 \
--set terminationGracePeriodSeconds=30
```
### Uninstalling the Chart
## Uninstalling the Chart
The command removes all the k8s components associated with the chart and deletes the release.
To uninstall the `logzio-k8s-logs` deployment:
The command removes all the k8s components associated with the chart and deletes
the release. To uninstall the `logzio-k8s-logs` deployment:
```shell
helm uninstall --namespace=kube-system logzio-k8s-logs
```

## Change log
- **0.0.2**:
- Added option to set tolerations for daemonset (Thanks [jlewis42lines](https://github.com/jlewis42lines)!).
- **0.0.1**:
- Initial release.
* **0.0.2**:
* Added option to set tolerations for daemonset (Thanks [jlewis42lines](https://github.com/jlewis42lines)!).
* **0.0.1**:
* Initial release.
14 changes: 13 additions & 1 deletion filebeat/templates/_helpers.tpl
Original file line number Diff line number Diff line change
Expand Up @@ -37,4 +37,16 @@ Convert logzio region code to listener host
{{- else if or ( eq $.Values.secrets.logzioRegion "au" ) ( eq $.Values.secrets.logzioRegion "ca" ) ( eq $.Values.secrets.logzioRegion "eu" ) ( eq $.Values.secrets.logzioRegion "nl" ) ( eq $.Values.secrets.logzioRegion "uk" ) ( eq $.Values.secrets.logzioRegion "wa" ) }}
{{- printf "listener-%s.logz.io" .Values.secrets.logzioRegion -}}
{{- end -}}
{{- end -}}
{{- end -}}

{{/*
Generate list of additional fields
*/}}
{{- define "logzio.extraFields" }}
{{- if .Values.filebeatConfig.extraFields }}
- add_fields:
target: ''
fields:
{{- toYaml .Values.filebeatConfig.extraFields | nindent 6 }}
{{- end -}}
{{- end -}}
78 changes: 61 additions & 17 deletions filebeat/templates/configmap.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -24,20 +24,64 @@ metadata:
labels:
k8s-app: filebeat
data:
{{- if eq .Values.configType "standard" }}
{{- range $path, $config := .Values.filebeatConfig.standardConfig }}
{{ $path }}: |-
{{ $config | indent 4 -}}
{{- end -}}
{{- else if eq .Values.configType "autodiscover" }}
{{- range $path, $config := .Values.filebeatConfig.autodiscoverConfig }}
{{ $path }}: |-
{{ $config | indent 4 -}}
{{- end -}}
{{- else if eq .Values.configType "auto-custom" }}
{{- range $path, $config := fromYaml .Values.filebeatConfig.autoCustomConfig }}
{{ $path }}: |-
{{ $config | indent 4 -}}
{{- end -}}
{{- end -}}
{{- end -}}
{{- if eq .Values.configType "standard" }}
filebeat.yml: |-
filebeat.inputs:
- type: container
paths:
- /var/log/containers/*.log
processors:
- add_kubernetes_metadata:
host: ${NODE_NAME}
matchers:
- logs_path:
logs_path: "/var/log/containers/"
processors:
- add_cloud_metadata: ~
{{- include "logzio.extraFields" . | nindent 6 }}
fields:
logzio_codec: ${LOGZIO_CODEC}
token: ${LOGZIO_LOGS_SHIPPING_TOKEN}
cluster: ${CLUSTER_NAME}
type: ${LOGZIO_TYPE}
fields_under_root: ${FIELDS_UNDER_ROOT}
ignore_older: ${IGNORE_OLDER}
output:
logstash:
hosts: ["${LOGZIO_LOGS_LISTENER_HOST}:5015"]
ssl:
certificate_authorities: ['/etc/pki/tls/certs/SectigoRSADomainValidationSecureServerCA.crt']
{{- else if eq .Values.configType "autodiscover" }}
filebeat.yml: |-
filebeat.autodiscover:
providers:
- type: kubernetes
node: ${NODE_NAME}
hints.enabled: true
hints.default_config:
type: container
paths:
- /var/log/containers/*-${data.kubernetes.container.id}.log
include_annotations: '*'
processors:
- add_cloud_metadata: ~
{{- include "logzio.extraFields" . | nindent 6 }}
fields:
logzio_codec: ${LOGZIO_CODEC}
token: ${LOGZIO_LOGS_SHIPPING_TOKEN}
cluster: ${CLUSTER_NAME}
type: ${LOGZIO_TYPE}
fields_under_root: ${FIELDS_UNDER_ROOT}
ignore_older: ${IGNORE_OLDER}
output:
logstash:
hosts: ["${LOGZIO_LOGS_LISTENER_HOST}:5015"]
ssl:
certificate_authorities: ['/etc/pki/tls/certs/SectigoRSADomainValidationSecureServerCA.crt']
{{- else if eq .Values.configType "custom" }}
filebeat.yml:
{{- toYaml .Values.filebeatConfig.customConfig | nindent 4 }}
{{- end }}
{{- end }}
Loading

0 comments on commit bab7bbb

Please sign in to comment.