[AMORO-1335]Amoro AMS Support On K8S (#1909)

* [AMORO-1335][helm-chart] Amoro k8s helm chart doc * [AMORO-1335][helm-chart] Amoro k8s helm chart yaml
apache · Sep 12, 2023 · b1ba50b · b1ba50b
1 parent 80239e1
commit b1ba50b
Show file tree

Hide file tree

Showing 12 changed files with 706 additions and 0 deletions.
diff --git a/charts/amoro/.helmignore b/charts/amoro/.helmignore
@@ -0,0 +1,23 @@
+# Patterns to ignore when building packages.
+# This supports shell glob matching, relative path matching, and
+# negation (prefixed with !). Only one pattern per line.
+.DS_Store
+# Common VCS dirs
+.git/
+.gitignore
+.bzr/
+.bzrignore
+.hg/
+.hgignore
+.svn/
+# Common backup files
+*.swp
+*.bak
+*.tmp
+*.orig
+*~
+# Various IDEs
+.project
+.idea/
+*.tmproj
+.vscode/
diff --git a/charts/amoro/Chart.yaml b/charts/amoro/Chart.yaml
@@ -0,0 +1,27 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+apiVersion: v2
+name: amoro
+description: A Helm chart for Amoro
+type: application
+version: 0.1.0
+appVersion: "0.5.0"
+home: https://amoro.netease.com/
+icon: https://amoro.netease.com/img/amoro-logo-icon.png
+sources:
+  - https://github.com/NetEase/amoro
diff --git a/charts/amoro/README.md b/charts/amoro/README.md
@@ -0,0 +1,59 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements.  See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership.  The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License.  You may obtain a copy of the License at
+
+   http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied.  See the License for the
+ specific language governing permissions and limitations
+ under the License.
+ -->
+
+# Helm Chart for Amoro
+
+[Amoro](https://amoro.netease.com) is a Lakehouse management system built on open data lake formats.
+Working with compute engines including Flink, Spark, and Trino, Amoro brings pluggable and self-managed features for Lakehouse to provide out-of-the-box data warehouse experience, 
+and helps data platforms or products easily build infra-decoupled, stream-and-batch-fused and lake-native architecture.
+
+
+## Introduction
+
+This chart will bootstrap an [Amoro](https://amoro.netease.com) deployment on a [Kubernetes](http://kubernetes.io)
+cluster using the [Helm](https://helm.sh) package manager.
+
+## Requirements
+
+- Kubernetes cluster
+- Helm 3.0+
+
+## Template rendering
+
+When you want to test the template rendering, but not actually install anything. [Debugging templates](https://helm.sh/docs/chart_template_guide/debugging/) provide a quick way of viewing the generated content without YAML parse errors blocking.
+
+There are two ways to render templates. It will return the rendered template to you so you can see the output.
+
+- Local rendering chart templates
+```shell
+helm template --debug ../amoro
+```
+- Server side rendering chart templates
+```shell
+helm install --dry-run --debug --generate-name ../amoro
+```
+<!-- ## Features -->
+
+## Documentation
+
+Configuration guide documentation for Amoro lives [on the website](https://amoro.netease.com/docs/latest). (Not just for Helm Chart)
+
+## Contributing
+
+Want to help build Amoro? Check out our [contributing documentation](https://amoro.netease.com/join-community).
diff --git a/charts/amoro/templates/NOTES.txt b/charts/amoro/templates/NOTES.txt
@@ -0,0 +1,31 @@
+{{/*
+  Licensed to the Apache Software Foundation (ASF) under one or more
+  contributor license agreements.  See the NOTICE file distributed with
+  this work for additional information regarding copyright ownership.
+  The ASF licenses this file to You under the Apache License, Version 2.0
+  (the "License"); you may not use this file except in compliance with
+  the License.  You may obtain a copy of the License at
+
+     http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+*/}}
+
+The Amoro Helm chart has been installed!
+
+If you want to go ams web ui,you can use this cmd forward to local:
+
+ $ kubectl --namespace {{ .Release.Namespace }} port-forward services/{{ .Release.Name }}-rest {{ .Values.server.rest.port}}:{{ .Values.server.rest.port}}
+
+Additional Resources Docs
+
+================================================================================================================
+
+        * Documentation: https://amoro.netease.com/docs/latest/
+        * Version build Info : https://github.com/NetEase/amoro/releases/tag/v{{ .Chart.AppVersion }}
+
+================================================================================================================
diff --git a/charts/amoro/templates/_helpers.tpl b/charts/amoro/templates/_helpers.tpl
@@ -0,0 +1,33 @@
+{{/*
+Expand the name of the chart.
+*/}}
+{{- define "amoro.name" -}}
+{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" }}
+{{- end }}
+
+{{/*
+Create chart name and version as used by the chart label.
+*/}}
+{{- define "amoro.chart" -}}
+{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" }}
+{{- end }}
+
+{{/*
+Common labels
+*/}}
+{{- define "amoro.labels" -}}
+helm.sh/chart: {{ include "amoro.chart" . }}
+{{ include "amoro.selectorLabels" . }}
+{{- if .Chart.AppVersion }}
+app.kubernetes.io/version: {{ .Chart.AppVersion | quote }}
+{{- end }}
+app.kubernetes.io/managed-by: {{ .Release.Service }}
+{{- end }}
+
+{{/*
+Selector labels
+*/}}
+{{- define "amoro.selectorLabels" -}}
+app.kubernetes.io/name: {{ include "amoro.name" . }}
+app.kubernetes.io/instance: {{ .Release.Name }}
+{{- end }}
diff --git a/charts/amoro/templates/amoro-configmap.yaml b/charts/amoro/templates/amoro-configmap.yaml
@@ -0,0 +1,132 @@
+{{/*
+  Licensed to the Apache Software Foundation (ASF) under one or more
+  contributor license agreements.  See the NOTICE file distributed with
+  this work for additional information regarding copyright ownership.
+  The ASF licenses this file to You under the Apache License, Version 2.0
+  (the "License"); you may not use this file except in compliance with
+  the License.  You may obtain a copy of the License at
+
+     http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+*/}}
+
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: {{ .Release.Name }}
+  labels:
+    {{- include "amoro.labels" . | nindent 4 }}
+data:
+  ## Helm chart provided Amoro configurations
+  config.yaml: |
+    ams:
+      admin-username: admin
+      admin-password: admin
+      server-bind-host: "0.0.0.0"
+      server-expose-host: "127.0.0.1"
+
+      thrift-server:
+        max-message-size: 104857600 # 100MB
+        selector-thread-count: 2
+        selector-queue-size: 4
+        table-service:
+          bind-port: {{ .Values.server.table.port }}
+          worker-thread-count: 20
+        optimizing-service:
+          bind-port: {{ .Values.server.optimizing.port }}
+
+      http-server:
+        bind-port: {{ .Values.server.rest.port }}
+
+      refresh-external-catalogs:
+        interval: 180000 # 3min
+
+      refresh-tables:
+        thread-count: 10
+        interval: 60000 # 1min
+
+      self-optimizing:
+        commit-thread-count: 10
+
+      optimizer:
+        heart-beat-timeout: 60000 # 1min
+        task-ack-timeout: 30000 # 30s
+
+      blocker:
+        timeout: 60000 # 1min
+
+      # optional features
+      expire-snapshots:
+        enabled: true
+        thread-count: 10
+
+      clean-orphan-files:
+        enabled: true
+        thread-count: 10
+
+      sync-hive-tables:
+        enabled: true
+        thread-count: 10
+
+      database:
+        type: {{ .Values.amoroConf.database.type }}
+        url: {{ .Values.amoroConf.database.url }}
+        jdbc-driver-class: {{ .Values.amoroConf.database.driver | quote }}
+        {{- if eq .Values.amoroConf.database.type "mysql" }}
+        username: {{ .Values.amoroConf.database.username | quote }}
+        password: {{ .Values.amoroConf.database.password | quote }}
+        {{- end }}
+
+      #  MySQL database configuration.
+      #  database:
+      #    type: mysql
+      #    jdbc-driver-class: com.mysql.cj.jdbc.Driver
+      #    url: jdbc:mysql://127.0.0.1:3306/db?useUnicode=true&characterEncoding=UTF8&autoReconnect=true&useAffectedRows=true&useSSL=false
+      #    username: root
+      #    password: root
+
+      terminal:
+        backend: local
+        local.spark.sql.iceberg.handle-timestamp-without-timezone: false
+
+      #  Kyuubi terminal backend configuration.
+      #  terminal:
+      #    backend: kyuubi
+      #    kyuubi.jdbc.url: jdbc:hive2://127.0.0.1:10009/
+
+    #  High availability configuration.
+    #  ha:
+    #    enabled: true
+    #    cluster-name: default
+    #    zookeeper-address: 127.0.0.1:2181,127.0.0.1:2182,127.0.0.1:2183
+
+    containers:
+      - name: localContainer
+        container-impl: com.netease.arctic.optimizer.LocalOptimizerContainer
+        properties:
+          export.JAVA_HOME: "/opt/java"   # JDK environment
+
+            #containers:
+            #  - name: flinkContainer
+            #    container-impl: com.netease.arctic.optimizer.FlinkOptimizerContainer
+            #    properties:
+            #      flink-home: "/opt/flink/"                                     # Flink install home
+            #      export.JVM_ARGS: "-Djava.security.krb5.conf=/opt/krb5.conf"   # Flink launch jvm args, like kerberos config when ues kerberos
+            #      export.HADOOP_CONF_DIR: "/etc/hadoop/conf/"                   # Hadoop config dir
+            #      export.HADOOP_USER_NAME: "hadoop"                             # Hadoop user submit on yarn
+          #      export.FLINK_CONF_DIR: "/etc/hadoop/conf/"                    # Flink config dir
+
+      ## User provided Amoro configurations
+    {{- with .Values.amoroConf.amoroDefaults }}
+      {{- tpl . $ | nindent 4 }}
+    {{- end }}
+  {{- with .Values.amoroConf.log4j2 }}
+  log4j2.xml: |
+    {{- tpl . $ | nindent 4 }}
+---
+  {{- end }}