Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fluentbit에서 다수의 elastic cluster를 지원하도록 수정함 #132

Merged
merged 4 commits into from
Aug 2, 2022
Merged

fluentbit에서 다수의 elastic cluster를 지원하도록 수정함 #132

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
3b7b81d
remove operator
intelliguy Jul 22, 2022
3f6a350
fluentbit: support multi-elasticsearch
intelliguy Jul 23, 2022
3b5fce7
fluentbit: add value override example
Jul 26, 2022
5eb5a6b
fluentbit: update cluster name in example
Aug 2, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion fluentbit-operator/Chart.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,4 +13,5 @@ maintainers:
name: fluentbit-operator
sources:
- https://github.com/intelliguy/fluentbit-operator
version: 1.2.2
version: 1.3.0
# Support Multi-ElasticSeach clusters (1.3)
210 changes: 210 additions & 0 deletions fluentbit-operator/examples/multi-es.valueoverride.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,210 @@
fluentbit:
alerts:
makeAlertRule: true
clusterName: cluster.local
daemonset:
spec:
pod:
tolerations:
- key: node-role.kubernetes.io/master
operator: Exists
- key: node-role.kubernetes.io/node
operator: Exists
enabled: true
exclude:
- key: $kubernetes['container_name']
value: kibana|elasticsearch|fluent-bit
nodeSelector:
taco-lma: enabled
outputs:
es:
- name: taco-es
host: eck-elasticsearch-es-http.lma.svc.cluster.local
port: 9200
createuser: true
username: taco-fluentbit
password: tacoword
elasticPasswordSecret: eck-elasticsearch-es-elastic-user
template:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

template 하위에 templates 항목이 있는데 굉장히 어색합니다. template을 index (혹은 index_template? 이 것도 동어반복되는 느낌이...) 로 변경하는 것 어떻게 생각하시나요?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

template는 템플릿이라는 큰 카테고리이고
그에 포함된 내역으로 실제 템플릿들(templates)과
템플릿으로 부터 만들어지는 인덱스를 관리하는 내역(ilms)을 포함하고 있습니다.
뭔가 정확한 용어로 정리하는 것은 좋아보이지만 제시하신 단어는 맞지 않네요.
그리고 본 pr과는 무관한 내용이므로 추후 refactoring시 고려하도록 하겠습니다.

enabled: true
ilms:
- name: hot-delete-14days
json:
policy:
phases:
delete:
actions:
delete: {}
min_age: 14d
hot:
actions:
rollover:
max_age: 1d
max_docs: 5000000000
max_size: 30gb
set_priority:
priority: 100
- name: hot-delete-7days
json:
policy:
phases:
delete:
actions:
delete: {}
min_age: 7d
hot:
actions:
rollover:
max_age: 1d
max_docs: 5000000000
max_size: 30gb
set_priority:
priority: 100
- name: hot-delete-3hour
json:
policy:
phases:
delete:
actions:
delete: {}
min_age: 3h
hot:
actions:
rollover:
max_age: 1h
max_docs: 5000000000
max_size: 30gb
set_priority:
priority: 100
templates:
- name: platform
json:
index_patterns: platform*
settings:
index.lifecycle.name: hot-delete-14days
index.lifecycle.rollover_alias: platform
number_of_replicas: 1
number_of_shards: 3
refresh_interval: 30s
- name: syslog
json:
index_patterns: syslog*
settings:
index.lifecycle.name: hot-delete-14days
index.lifecycle.rollover_alias: syslog
number_of_replicas: 1
number_of_shards: 2
refresh_interval: 30s
- name: external-es
host: eck-elasticsearch-es-http.byoh.svc.cluster.local
port: 9200
createuser: true
username: taco-fluentbit
password: tacoword
elasticPasswordSecret: eck-elasticsearch-es-elastic-user
template:
enabled: true
ilms:
- name: hot-delete-14days
json:
policy:
phases:
delete:
actions:
delete: {}
min_age: 14d
hot:
actions:
rollover:
max_age: 1d
max_docs: 5000000000
max_size: 30gb
set_priority:
priority: 100
- name: hot-delete-7days
json:
policy:
phases:
delete:
actions:
delete: {}
min_age: 7d
hot:
actions:
rollover:
max_age: 1d
max_docs: 5000000000
max_size: 30gb
set_priority:
priority: 100
templates:
- name: coreapp
json:
index_patterns: coreapp*
settings:
index.lifecycle.name: hot-delete-14days
index.lifecycle.rollover_alias: coreapp
number_of_replicas: 1
number_of_shards: 3
refresh_interval: 30s
- name: svcapp
json:
index_patterns: svcapp*
settings:
index.lifecycle.name: hot-delete-14days
index.lifecycle.rollover_alias: svcapp
number_of_replicas: 1
number_of_shards: 2
refresh_interval: 30s
http:
enabled: true
kafka:
broker: YOUR_BORKER_INFO
enabled: false
topic: YOUR_TACO_LOG_INFO
targetLogs:
- name: dockerlog
do_not_store_as_default: true
bufferChunkSize: 2M
bufferMaxSize: 5M
memBufLimit: 20MB
multi_index:
- index: platform
es_name: taco-es
key: $kubernetes['namespace_name']
value: kube-system|lma|fed|argo|openstack|istio-system|istio-services|trident|registry
- index: coreapp
es_name: external-es
key: $kubernetes['namespace_name']
value: byoh|byoh-system
- index: svcapp
es_name: external-es
key: $kubernetes['container_name']
value: csi-rbdplugin
# value: cap[a-z,\-,0-9]+|ceph[a-z,\-,0-9]+
parser: docker
path: /var/log/containers/*.log
tag: kube.*
type: fluent
- name: syslog
do_not_store_as_default: false
es_name: taco-es
index: syslog
parser: syslog-rfc5424
path: /var/log/syslog
tag: syslog.*
type: syslog
fluentbitOperator:
cleanupCustomResource: false
createCustomResource: false
enabled: false
fullnameOverride: fbcr
global:
base_cluster_url: cluster.local
nodeSelector:
taco-lma: enabled
image:
hyperkube:
tag: v1.17.6
logExporter:
enabled: false
Loading