Skip to content

Commit 404459c

Browse files
labrenbelabrenbe
authored
feat: Use Opensearch operator and OpenSearch Dashboards image in logging demo (#315)
* use opensearch-operator and opensearch-dashboards image in demo * fix vector aggregator * update docs * address feedback on PR * move tls pod override to role level * address feedback in PR * fix default mode
1 parent c5d38c4 commit 404459c

File tree

8 files changed

+210
-145
lines changed

8 files changed

+210
-145
lines changed

‎docs/modules/demos/pages/logging.adoc‎

Lines changed: 10 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -26,16 +26,6 @@ To run this demo, your system needs at least:
2626

2727
If you use MacOS or Windows and use Docker to run Kubernetes, set the RAM to at least 4 GB in _Preferences > Resources_.
2828

29-
==== Linux
30-
31-
OpenSearch uses a mmapfs directory by default to store its indices. The default operating system limits on mmap counts
32-
are likely too low - usually 65530, which may result in out-of-memory exceptions. So, the Linux setting
33-
`vm.max_map_count` on the host machine where the containers are running must be set to at least 262144.
34-
35-
This is automatically set by default in this demo (via the `setSysctlMaxMapCount` Stack parameter).
36-
37-
OpenSearch has more information about this setting in their https://opensearch.org/docs/2.12/install-and-configure/install-opensearch/index/#important-settings[documentation].
38-
3929
== Overview
4030

4131
This demo will
@@ -63,15 +53,16 @@ To list the installed Stackable services run the following command:
6353
[source,console]
6454
----
6555
$ stackablectl stacklet list
66-
┌───────────────────────┬───────────────────────┬───────────┬─────────────────────────────────────────────────┬─────────────────────────────────┐
67-
│ PRODUCT ┆ NAME ┆ NAMESPACE ┆ ENDPOINTS ┆ CONDITIONS │
68-
╞═══════════════════════╪═══════════════════════╪═══════════╪═════════════════════════════════════════════════╪═════════════════════════════════╡
69-
│ zookeeper ┆ simple-zk ┆ default ┆ server-zk ┆ Available, Reconciling, Running │
70-
│ ┆ ┆ ┆ simple-zk-server.default.svc.cluster.local:2282 ┆ │
71-
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
72-
│ opensearch-dashboards ┆ opensearch-dashboards ┆ default ┆ http http://172.18.0.2:31734 ┆ │
73-
│ ┆ ┆ ┆ metrics 172.18.0.2:32120 ┆ │
74-
└───────────────────────┴───────────────────────┴───────────┴─────────────────────────────────────────────────┴─────────────────────────────────┘
56+
┌───────────────────────┬───────────────────────┬───────────┬────────────────────────────────────────────────────────────────────────────────────┬─────────────────────────────────┐
57+
│ PRODUCT ┆ NAME ┆ NAMESPACE ┆ ENDPOINTS ┆ CONDITIONS │
58+
╞═══════════════════════╪═══════════════════════╪═══════════╪════════════════════════════════════════════════════════════════════════════════════╪═════════════════════════════════╡
59+
│ opensearch ┆ opensearch ┆ default ┆ nodes-default-http http://opensearch-nodes-default.default.svc.cluster.local:9200 ┆ Available, Reconciling, Running │
60+
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
61+
│ zookeeper ┆ simple-zk ┆ default ┆ server-zk simple-zk-server.default.svc.cluster.local:2282 ┆ Available, Reconciling, Running │
62+
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
63+
│ opensearch-dashboards ┆ opensearch-dashboards ┆ default ┆ http http://172.18.0.2:30595 ┆ │
64+
│ ┆ ┆ ┆ metrics 172.18.0.2:31767 ┆ │
65+
└───────────────────────┴───────────────────────┴───────────┴────────────────────────────────────────────────────────────────────────────────────┴─────────────────────────────────┘
7566
----
7667

7768
include::partial$instance-hint.adoc[]

‎stacks/_templates/opensearch-dashboards.yaml‎

Lines changed: 10 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -4,8 +4,12 @@ name: opensearch-dashboards
44
repo:
55
name: opensearch-dashboards
66
url: https://opensearch-project.github.io/helm-charts
7-
version: 2.30.0 # 2.19.2
7+
version: {{ opensearchVersion }}
88
options:
9+
opensearchHosts: https://opensearch:9200
10+
image:
11+
repository: oci.stackable.tech/sdp/opensearch-dashboards
12+
tag: "{{ opensearchVersion }}-stackable{{ stackableReleaseVersion }}"
913
labels:
1014
stackable.tech/vendor: Stackable
1115
service:
@@ -19,9 +23,8 @@ options:
1923
stackable.tech/vendor: Stackable
2024
opensearchAccount:
2125
secret: opensearch-dashboard-user
22-
extraEnvs:
23-
- name: OPEN_SEARCH_ADMIN_PASSWORD
24-
valueFrom:
25-
secretKeyRef:
26-
name: opensearch-user
27-
key: password
26+
serviceAccount:
27+
create: false
28+
# Use the ServiceAccount of OpenSearch because its permissions are already configured to work on
29+
# OpenShift.
30+
name: opensearch-serviceaccount

‎stacks/_templates/opensearch.yaml‎

Lines changed: 0 additions & 110 deletions
This file was deleted.

‎stacks/_templates/vector-aggregator.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -23,7 +23,7 @@ options:
2323
inputs:
2424
- vector
2525
endpoints:
26-
- https://opensearch-cluster-master.default.svc.cluster.local:9200
26+
- https://opensearch.default.svc.cluster.local:9200
2727
mode: bulk
2828
# The auto-detection of the API version does not work in Vector
2929
# 0.41.1 for OpenSearch, so the version must be set explicitly

‎stacks/argo-cd-git-ops/applicationsets/stackable-operators.yaml‎

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -20,8 +20,7 @@ spec:
2020
- operator: kafka
2121
- operator: nifi
2222
- operator: opa
23-
# TODO: enable Opensearch operator
24-
# - opensearch
23+
- operator: opensearch
2524
- operator: spark-k8s
2625
- operator: superset
2726
- operator: trino

‎stacks/logging/opensearch.yaml‎

Lines changed: 179 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,179 @@
1+
apiVersion: opensearch.stackable.tech/v1alpha1
2+
kind: OpenSearchCluster
3+
metadata:
4+
name: opensearch
5+
spec:
6+
image:
7+
productVersion: {{ opensearchVersion }}
8+
pullPolicy: IfNotPresent
9+
clusterConfig:
10+
vectorAggregatorConfigMapName: vector-aggregator-discovery
11+
nodes:
12+
config:
13+
logging:
14+
enableVectorAgent: true
15+
roleGroups:
16+
default:
17+
config:
18+
listenerClass: cluster-internal
19+
replicas: 1
20+
configOverrides:
21+
opensearch.yml:
22+
# Disable memory mapping in this stack; If memory mapping were activated, the kernel setting
23+
# vm.max_map_count would have to be increased to 262144 on the node.
24+
node.store.allow_mmap: "false"
25+
# Disable the disk allocation decider in this stack; Otherwise depending on the disk
26+
# usage of the node and if the relative watermark set in
27+
# `cluster.routing.allocation.disk.watermark.high` is reached the security index can't
28+
# be created even if enough disk space would be available.
29+
cluster.routing.allocation.disk.threshold_enabled: "false"
30+
plugins.security.allow_default_init_securityindex: "true"
31+
plugins.security.ssl.transport.enabled: "true"
32+
plugins.security.ssl.transport.pemcert_filepath: /stackable/opensearch/config/tls/tls.crt
33+
plugins.security.ssl.transport.pemkey_filepath: /stackable/opensearch/config/tls/tls.key
34+
plugins.security.ssl.transport.pemtrustedcas_filepath: /stackable/opensearch/config/tls/ca.crt
35+
plugins.security.ssl.http.enabled: "true"
36+
plugins.security.ssl.http.pemcert_filepath: /stackable/opensearch/config/tls/tls.crt
37+
plugins.security.ssl.http.pemkey_filepath: /stackable/opensearch/config/tls/tls.key
38+
plugins.security.ssl.http.pemtrustedcas_filepath: /stackable/opensearch/config/tls/ca.crt
39+
podOverrides:
40+
spec:
41+
containers:
42+
- name: opensearch
43+
volumeMounts:
44+
- name: security-config
45+
mountPath: /stackable/opensearch/config/opensearch-security
46+
readOnly: true
47+
- name: tls
48+
mountPath: /stackable/opensearch/config/tls
49+
readOnly: true
50+
volumes:
51+
- name: security-config
52+
secret:
53+
secretName: opensearch-security-config
54+
defaultMode: 0o660
55+
- name: tls
56+
ephemeral:
57+
volumeClaimTemplate:
58+
metadata:
59+
annotations:
60+
secrets.stackable.tech/class: tls
61+
secrets.stackable.tech/scope: node,pod,service=opensearch,service=opensearch-nodes-default-headless
62+
spec:
63+
storageClassName: secrets.stackable.tech
64+
accessModes:
65+
- ReadWriteOnce
66+
resources:
67+
requests:
68+
storage: "1"
69+
---
70+
apiVersion: v1
71+
kind: Secret
72+
metadata:
73+
name: opensearch-security-config
74+
stringData:
75+
action_groups.yml: |
76+
---
77+
_meta:
78+
type: actiongroups
79+
config_version: 2
80+
allowlist.yml: |
81+
---
82+
_meta:
83+
type: allowlist
84+
config_version: 2
85+
86+
config:
87+
enabled: false
88+
audit.yml: |
89+
---
90+
_meta:
91+
type: audit
92+
config_version: 2
93+
94+
config:
95+
enabled: false
96+
config.yml: |
97+
---
98+
_meta:
99+
type: config
100+
config_version: 2
101+
102+
config:
103+
dynamic:
104+
authc:
105+
basic_internal_auth_domain:
106+
description: Authenticate via HTTP Basic against internal users database
107+
http_enabled: true
108+
transport_enabled: true
109+
order: 1
110+
http_authenticator:
111+
type: basic
112+
challenge: true
113+
authentication_backend:
114+
type: intern
115+
authz: {}
116+
internal_users.yml: |
117+
---
118+
_meta:
119+
type: internalusers
120+
config_version: 2
121+
122+
admin:
123+
hash: {{ bcrypt(password=openSearchAdminPassword) }}
124+
reserved: true
125+
backend_roles:
126+
- admin
127+
description: OpenSearch admin user
128+
129+
kibanaserver:
130+
hash: {{ bcrypt(password=openSearchDashboardPassword) }}
131+
reserved: true
132+
description: OpenSearch Dashboards user
133+
nodes_dn.yml: |
134+
---
135+
_meta:
136+
type: nodesdn
137+
config_version: 2
138+
roles.yml: |
139+
---
140+
_meta:
141+
type: roles
142+
config_version: 2
143+
roles_mapping.yml: |
144+
---
145+
_meta:
146+
type: rolesmapping
147+
config_version: 2
148+
149+
all_access:
150+
reserved: false
151+
backend_roles:
152+
- admin
153+
154+
kibana_server:
155+
reserved: true
156+
users:
157+
- kibanaserver
158+
tenants.yml: |
159+
---
160+
_meta:
161+
type: tenants
162+
config_version: 2
163+
---
164+
apiVersion: v1
165+
kind: Secret
166+
metadata:
167+
name: opensearch-user
168+
stringData:
169+
username: admin
170+
password: {{ openSearchAdminPassword }}
171+
---
172+
apiVersion: v1
173+
kind: Secret
174+
metadata:
175+
name: opensearch-dashboard-user
176+
stringData:
177+
username: kibanaserver
178+
password: {{ openSearchDashboardPassword }}
179+
cookie: {{ random_password() }}

‎stacks/logging/setup-opensearch-dashboards.yaml‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,6 @@ spec:
1717
key: password
1818
command:
1919
- bash
20-
- -x
2120
- -euo
2221
- pipefail
2322
- -c

0 commit comments

Comments
 (0)