From 85f61cb1e925d77ac01b37b84fb2a7204b6bb095 Mon Sep 17 00:00:00 2001 From: Seb Kim <75765303+sb-sebkim@users.noreply.github.com> Date: Mon, 21 Nov 2022 11:36:31 +0900 Subject: [PATCH] [DATA-6011] Upgrade 0.9.2 (#3) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * upgrading to 8.40 (#139) * Option to add podAnnotations to jobs. (#137) * feat(helm): add pod annotations to jobs * feat(helm): add pod annotation to cron jobs. * docs(helm): Add podAnnotation documentation and default values. * feat(helm) chart version bump. * update to v0.8.41 (#143) * fix(mysql): update mysql helm chart version (#146) * fix(metrics): fix Prometheus env variable in the MCE consumer deployment (#148) * feat(datahub-frontend): Option to configure OIDC auth easily (#142) * fix(kafka): update kafka chart version (#147) * feat(datahub-frontend): Allow lifecycle hooks (#141) * (datahub-frontend): Allow lifecycle hooks This PR adds a lifecycle node to the datahub-frontend deployment template. It can be used to execute lifecycle hooks, e.g. postStart actions, like replacing the default user.props file. * Bump chart version to 0.2.86 Co-authored-by: Pedro Silva * Update to v0.8.43 (#151) * feat(upgrade): allow setting batch args (#155) * feat(datahub-upgrade): Restore Indices Resources (#152) * (datahub-upgrade): Restore Indices Resources This PR adds the ability for end-users to specify resource requests and limits for the datahub upgrade job 'restore indices'. This is needed for situations where the restore indices job will OOM due to a large backfill of data. I've seen this happening with our deployment of datahub and am unable to tune the resources for the restore indices job because the current requests and limits are hard-coded. * fix(gms): remove unused JMX sidecar (#156) * fix(gms): remove unused JMX sidecar in favour of the already included JVM agent * Update Chart.yaml * Update Chart.yaml * Bumping to v0.8.44 (#160) * feat(ci): close stale issues/prs (#164) * feat: add extra labels to ingress configurations (#163) * feat(prerequisites): add postgresql to prerequistes Helm chart as an optional dependency (#121) * feat(charts): add an option to configure revisionHistoryLimit to subcharts (#159) * feat: datahub-frontend add oidc authentication client secret reference (#165) * fix(monitoring) Fixes monitoring default values + gms chart version (#161) * chore: bump datahub-actions version (#168) * Feat(actions): Support mounting a customer defined k8s secrets as files needed in ingestion recipes (#169) * Mount a customer defined k8s secret for ssl secrets needed in ingestion recipes * Bump chart versions * Address comment * Update readme * Address Pedro's comment * feat(monitoring): enable monitoring on frontend (#171) * Add kafka-exporter to expose prometheus metrics for kafka cluster (#170) * Add kafka-exporter to expose prometheus metrics for kafka cluster * Kafka-exporter: Updated values.yaml to support tls/sasl authentication while connecting kafka cluster * Kafka-exporter: Updated values.yaml to remove default values and correct intendation * Add kafka-exporter to expose prometheus metrics for kafka cluster * Kafka-exporter: Updated values.yaml to support tls/sasl authentication while connecting kafka cluster * Kafka-exporter: Updated values.yaml to remove default values and correct intendation * Updated chart version * Bumping actions (#173) * feat(elasticsearch-setup): Add support for insecure curl connections (#166) * fix versions, clean jmx exporter (#175) * bumping helm charts for 0.8.45 (#178) * bumping to 0.8.45 * Fixing * feat(monitoring): Add optional ServiceMonitors to all JVM based services (#181) * feat(frontend): add service monitoring * feat(gms): add service monitoring * gms: servicemonitor * feat(mae): enable service monitoring * update READMEs * fix service ports * bump version * Add missing default value * Update charts/datahub/Chart.yaml * delete jmx config and rename port to http * feat(secrets): Allow users to specify their own secret values at first boot (#180) * fix(auth-secret): Make template use correct retrieval method `datahub-auth-secret` yaml was incorrectly using the `index` template function which is for list variables. For dictionaries as is the case with the `$secret.data` object, the correct method is: https://helm.sh/docs/chart_template_guide/function_list/#get Tested locally, that this works against a running cluster. * Updates encryption secret to use get helm function * feat(secrets): Allow users to specify their own secret values at first boot * fix(datahub-frontend): Remove unused env var (#176) * fix(datahub-frontend): Remove unused env var * Bump chart version * Change default versions 0.9.0 (#184) * feat(release): automatically bumbp versions for all subcharts (#185) * feat(release): automatically bumbp versions for all subcharts * Remove kafka export as subchart Kafka exporter is useful for monitoring datahub, but it's not part of it. It should be deploy sepratly, as we do in production * Add script to bump versions together * Add tagging to the workflow * Don't run release on CI commits * fix(CI): remove duplicate id (#189) * fix(ci): us sh instead of bash; increase tag by patch (#190) * DataHub Release v0.9.1 (#191) * fix(v0.9.1): Fix v0.9.1 release versions (#192) * Release v0.9.1 Helm * Bump chart versions * Fixing final 210 * Bumping to release Helm charts for v0.9.2 * stg actions image upgrade * service annotations scp applied * filters deprecated in favor of orFilters * filters deprecated in favor of orFilters version applied in yaml * release(v0.9.2) Release Helm Charts for DataHub v0.9.2 (#194) * Bumping to release Helm charts for v0.9.2 * bump version by running: ./.github/scripts/bump-chart-versions.sh 0.2.112 Co-authored-by: szalai1 * fix(gha): revert auto bump release changes (#197) Co-authored-by: Gabe Lyons Co-authored-by: Lukáš Novotný <42157644+novotl@users.noreply.github.com> Co-authored-by: Aseem Bansal Co-authored-by: RyanHolstien Co-authored-by: Justin Marozas Co-authored-by: Tomáš Kubín Co-authored-by: Toby Irmer Co-authored-by: Pedro Silva Co-authored-by: Pedro Silva Co-authored-by: Hunter Elbourn Co-authored-by: Peter Szalai Co-authored-by: John Joyce Co-authored-by: Upendra Rao Vedullapalli Co-authored-by: Felix Lüdin <13187726+Masterchen09@users.noreply.github.com> Co-authored-by: Bumsoo Kim Co-authored-by: Tony Ouyang <43738225+TonyOuyangGit@users.noreply.github.com> Co-authored-by: Harshal Sheth Co-authored-by: Jinlin Yang <86577891+jinlintt@users.noreply.github.com> Co-authored-by: Navin Sharma <103643430+NavinSharma13@users.noreply.github.com> Co-authored-by: Bogdan Antoniu --- .github/scripts/bump-chart-versions.sh | 19 +++++ .github/workflows/close-stale-issues.yml | 25 ++++++ .github/workflows/lint-test.yaml | 2 +- .github/workflows/release.yaml | 3 +- charts/datahub/Chart.yaml | 19 ++--- charts/datahub/README.md | 35 +++++--- .../datahub/quickstart-values-with-neo4j.yaml | 18 ++--- .../acryl-datahub-actions/Chart.yaml | 7 +- .../subcharts/acryl-datahub-actions/README.md | 4 +- .../templates/deployment.yaml | 11 +++ .../subcharts/datahub-frontend/Chart.yaml | 6 +- .../subcharts/datahub-frontend/README.md | 12 +++ .../templates/config-jmx-exporter.yaml | 17 ---- .../templates/deployment.yaml | 59 ++++++++++++-- .../datahub-frontend/templates/ingress.yaml | 3 + .../datahub-frontend/templates/service.yaml | 6 ++ .../templates/servicemonitor.yaml | 25 ++++++ .../subcharts/datahub-frontend/values.yaml | 34 +++++++- .../datahub/subcharts/datahub-gms/Chart.yaml | 7 +- .../datahub/subcharts/datahub-gms/README.md | 6 +- .../templates/config-jmx-exporter.yaml | 25 ------ .../datahub-gms/templates/deployment.yaml | 10 +-- .../datahub-gms/templates/ingress.yaml | 3 + .../datahub-gms/templates/service.yaml | 6 ++ .../datahub-gms/templates/servicemonitor.yaml | 25 ++++++ .../datahub/subcharts/datahub-gms/values.yaml | 64 +++------------ .../datahub-ingestion-cron/Chart.yaml | 7 +- .../datahub-ingestion-cron/README.md | 1 + .../templates/cron.yaml | 7 +- .../datahub-ingestion-cron/values.yaml | 4 + .../subcharts/datahub-jmxexporter/Chart.yaml | 21 ----- .../templates/_container.tpl | 32 -------- .../subcharts/datahub-mae-consumer/Chart.yaml | 7 +- .../subcharts/datahub-mae-consumer/README.md | 4 +- .../templates/config-jmx-exporter.yaml | 25 ------ .../templates/deployment.yaml | 10 +-- .../templates/service.yaml | 6 ++ .../templates/servicemonitor.yaml | 25 ++++++ .../datahub-mae-consumer/values.yaml | 8 +- .../subcharts/datahub-mce-consumer/Chart.yaml | 7 +- .../subcharts/datahub-mce-consumer/README.md | 4 +- .../templates/config-jmx-exporter.yaml | 25 ------ .../templates/deployment.yaml | 19 ++--- .../templates/service.yaml | 26 ++++++ .../templates/servicemonitor.yaml | 25 ++++++ .../datahub-mce-consumer/values.yaml | 21 ++++- .../templates/datahub-auth-secrets.yml | 20 +++-- .../templates/datahub-encryption-secrets.yml | 14 +++- .../templates/datahub-upgrade/_upgrade.tpl | 4 + .../datahub-cleanup-job-template.yml | 10 ++- .../datahub-restore-indices-job-template.yml | 21 +++-- .../datahub-upgrade/datahub-upgrade-job.yml | 14 +++- .../templates/elasticsearch-setup-job.yml | 14 +++- charts/datahub/templates/kafka-setup-job.yml | 10 ++- charts/datahub/templates/mysql-setup-job.yml | 10 ++- .../templates/postgresql-setup-job.yml | 10 ++- charts/datahub/values.yaml | 80 ++++++++++++++++--- charts/datahub/values_stg.yaml | 2 +- charts/prerequisites/Chart.yaml | 10 ++- charts/prerequisites/values.yaml | 6 ++ .../custom-datahub-actions/Dockerfile | 4 +- .../gql_get_sb_regions.py | 22 +++-- 62 files changed, 627 insertions(+), 359 deletions(-) create mode 100755 .github/scripts/bump-chart-versions.sh create mode 100644 .github/workflows/close-stale-issues.yml delete mode 100644 charts/datahub/subcharts/datahub-frontend/templates/config-jmx-exporter.yaml create mode 100644 charts/datahub/subcharts/datahub-frontend/templates/servicemonitor.yaml delete mode 100644 charts/datahub/subcharts/datahub-gms/templates/config-jmx-exporter.yaml create mode 100644 charts/datahub/subcharts/datahub-gms/templates/servicemonitor.yaml delete mode 100644 charts/datahub/subcharts/datahub-jmxexporter/Chart.yaml delete mode 100644 charts/datahub/subcharts/datahub-jmxexporter/templates/_container.tpl delete mode 100644 charts/datahub/subcharts/datahub-mae-consumer/templates/config-jmx-exporter.yaml create mode 100644 charts/datahub/subcharts/datahub-mae-consumer/templates/servicemonitor.yaml delete mode 100644 charts/datahub/subcharts/datahub-mce-consumer/templates/config-jmx-exporter.yaml create mode 100644 charts/datahub/subcharts/datahub-mce-consumer/templates/service.yaml create mode 100644 charts/datahub/subcharts/datahub-mce-consumer/templates/servicemonitor.yaml diff --git a/.github/scripts/bump-chart-versions.sh b/.github/scripts/bump-chart-versions.sh new file mode 100755 index 000000000..358fcdaff --- /dev/null +++ b/.github/scripts/bump-chart-versions.sh @@ -0,0 +1,19 @@ +#! /bin/sh + +export TAG=$1 + +function updateVersion() { + TAG=$1 yq -i e '.version |= env(TAG) ' $2; + echo "Version is updated to $1 in $2" +} + +# updating subcharts +for file in charts/datahub/*/*/Chart.yaml; do + updateVersion $TAG $file +done + +# updating datahub chart +updateVersion $TAG charts/datahub/Chart.yaml + + + yq -i e ".dependencies[].version |= env(TAG)" charts/datahub/Chart.yaml \ No newline at end of file diff --git a/.github/workflows/close-stale-issues.yml b/.github/workflows/close-stale-issues.yml new file mode 100644 index 000000000..0a5c3c3a8 --- /dev/null +++ b/.github/workflows/close-stale-issues.yml @@ -0,0 +1,25 @@ +name: Close inactive issues +on: + schedule: + - cron: "30 1 * * *" + +jobs: + close-issues: + runs-on: ubuntu-latest + permissions: + issues: write + pull-requests: write + steps: + - uses: actions/stale@v5 + with: + days-before-issue-stale: 30 + days-before-issue-close: 30 + stale-issue-label: "stale" + stale-issue-message: "This issue is stale because it has been open for 30 days with no activity. If you believe this is still an issue on the latest DataHub release please leave a comment with the version that you tested it with. If this is a question/discussion please head to https://slack.datahubproject.io. For feature requests please use https://feature-requests.datahubproject.io" + close-issue-message: "This issue was closed because it has been inactive for 30 days since being marked as stale." + days-before-pr-stale: 30 + days-before-pr-close: 30 + stale-pr-label: "stale" + stale-pr-message: "This PR is stale. We will close it in 30 days if there is no comment or activity. If you want feedback but not able to get it on github please head to #contribute channel in slack at https://slack.datahubproject.io." + close-pr-message: "Closing stale PR." + repo-token: ${{ secrets.GITHUB_TOKEN }} diff --git a/.github/workflows/lint-test.yaml b/.github/workflows/lint-test.yaml index b2e1e17da..c4721be95 100644 --- a/.github/workflows/lint-test.yaml +++ b/.github/workflows/lint-test.yaml @@ -44,4 +44,4 @@ jobs: fi - name: Run chart-testing (lint) - run: ct lint + run: ct lint diff --git a/.github/workflows/release.yaml b/.github/workflows/release.yaml index 602d43722..8f1be8623 100644 --- a/.github/workflows/release.yaml +++ b/.github/workflows/release.yaml @@ -1,3 +1,5 @@ + + name: Release Charts on: @@ -23,7 +25,6 @@ jobs: run: | git config user.name "$GITHUB_ACTOR" git config user.email "$GITHUB_ACTOR@users.noreply.github.com" - - name: Install Helm uses: azure/setup-helm@v1 with: diff --git a/charts/datahub/Chart.yaml b/charts/datahub/Chart.yaml index 9e25dba15..cf6339fac 100644 --- a/charts/datahub/Chart.yaml +++ b/charts/datahub/Chart.yaml @@ -4,36 +4,33 @@ description: A Helm chart for LinkedIn DataHub type: application # This is the chart version. This version number should be incremented each time you make changes # to the chart and its templates, including the app version. -version: 0.2.84 +version: 0.2.112 # This is the version number of the application being deployed. This version number should be # incremented each time you make changes to the application. -appVersion: 0.8.39 +appVersion: 0.9.1 dependencies: - name: datahub-gms - version: 0.2.5 + version: 0.2.112 repository: file://./subcharts/datahub-gms condition: datahub-gms.enabled - name: datahub-frontend - version: 0.2.2 + version: 0.2.112 repository: file://./subcharts/datahub-frontend condition: datahub-frontend.enabled - name: datahub-mae-consumer - version: 0.2.5 + version: 0.2.112 repository: file://./subcharts/datahub-mae-consumer condition: global.datahub_standalone_consumers_enabled - name: datahub-mce-consumer - version: 0.2.5 + version: 0.2.112 repository: file://./subcharts/datahub-mce-consumer condition: global.datahub_standalone_consumers_enabled - name: datahub-ingestion-cron - version: 0.2.3 + version: 0.2.112 repository: file://./subcharts/datahub-ingestion-cron condition: datahub-ingestion-cron.enabled - - name: datahub-jmxexporter - version: 0.2.1 - repository: file://./subcharts/datahub-jmxexporter - name: acryl-datahub-actions - version: 0.0.2 + version: 0.2.112 repository: file://./subcharts/acryl-datahub-actions condition: acryl-datahub-actions.enabled maintainers: diff --git a/charts/datahub/README.md b/charts/datahub/README.md index e18efe538..5e077742f 100644 --- a/charts/datahub/README.md +++ b/charts/datahub/README.md @@ -22,37 +22,43 @@ helm install datahub datahub/datahub --values <> |-----|------|---------|-------------| | datahub-frontend.enabled | bool | `true` | Enable Datahub Front-end | | datahub-frontend.image.repository | string | `"linkedin/datahub-frontend-react"` | Image repository for datahub-frontend | -| datahub-frontend.image.tag | string | `"v0.8.39"` | Image tag for datahub-frontend | +| datahub-frontend.image.tag | string | `"v0.9.2"` | Image tag for datahub-frontend | | datahub-gms.enabled | bool | `true` | Enable GMS | | datahub-gms.image.repository | string | `"linkedin/datahub-gms"` | Image repository for datahub-gms | -| datahub-gms.image.tag | string | `"v0.8.39"` | Image tag for datahub-gms | +| datahub-gms.image.tag | string | `"v0.9.2"` | Image tag for datahub-gms | | datahub-mae-consumer.image.repository | string | `"linkedin/datahub-mae-consumer"` | Image repository for datahub-mae-consumer | -| datahub-mae-consumer.image.tag | string | `"v0.8.39"` | Image tag for datahub-mae-consumer | +| datahub-mae-consumer.image.tag | string | `"v0.9.2"` | Image tag for datahub-mae-consumer | | datahub-mce-consumer.image.repository | string | `"linkedin/datahub-mce-consumer"` | Image repository for datahub-mce-consumer | -| datahub-mce-consumer.image.tag | string | `"v0.8.39"` | Image tag for datahub-mce-consumer | +| datahub-mce-consumer.image.tag | string | `"v0.9.2"` | Image tag for datahub-mce-consumer | | datahub-ingestion-cron.enabled | bool | `false` | Enable cronjob for periodic ingestion | | datahubUpgrade.podSecurityContext | object | `{}` | Pod security context for datahubUpgrade jobs | | datahubUpgrade.securityContext | object | `{}` | Container security context for datahubUpgrade jobs | +| datahubUpgrade.podAnnotations | object | `{}` | Pod annotations for datahubUpgrade jobs | +| datahubUpgrade.restoreIndices.resources | object | '{}' | Kube Resource definitions for the datahub upgrade job 'restore indices' | | elasticsearchSetupJob.enabled | bool | `true` | Enable setup job for elasicsearch | | elasticsearchSetupJob.image.repository | string | `"linkedin/datahub-elasticsearch-setup"` | Image repository for elasticsearchSetupJob | -| elasticsearchSetupJob.image.tag | string | `"v0.8.39"` | Image repository for elasticsearchSetupJob | +| elasticsearchSetupJob.image.tag | string | `"v0.9.2"` | Image repository for elasticsearchSetupJob | | elasticsearchSetupJob.podSecurityContext | object | `{"fsGroup": 1000}` | Pod security context for elasticsearchSetupJob | | elasticsearchSetupJob.securityContext | object | `{"runAsUser": 1000}` | Container security context for elasticsearchSetupJob | +| elasticsearchSetupJob.podAnnotations | object | `{}` | Pod annotations for elasticsearchSetupJob | | kafkaSetupJob.enabled | bool | `true` | Enable setup job for kafka | | kafkaSetupJob.image.repository | string | `"linkedin/datahub-kafka-setup"` | Image repository for kafkaSetupJob | -| kafkaSetupJob.image.tag | string | `"v0.8.39"` | Image repository for kafkaSetupJob | +| kafkaSetupJob.image.tag | string | `"v0.9.2"` | Image repository for kafkaSetupJob | | kafkaSetupJob.podSecurityContext | object | `{"fsGroup": 1000}` | Pod security context for kafkaSetupJob | | kafkaSetupJob.securityContext | object | `{"runAsUser": 1000}` | Container security context for kafkaSetupJob | +| kafkaSetupJob.podAnnotations | object | `{}` | Pod annotations for kafkaSetupJob | | mysqlSetupJob.enabled | bool | `false` | Enable setup job for mysql | | mysqlSetupJob.image.repository | string | `"acryldata/datahub-mysql-setup"` | Image repository for mysqlSetupJob | -| mysqlSetupJob.image.tag | string | `"v0.8.39"` | Image repository for mysqlSetupJob | +| mysqlSetupJob.image.tag | string | `"v0.9.2"` | Image repository for mysqlSetupJob | | mysqlSetupJob.podSecurityContext | object | `{"fsGroup": 1000}` | Pod security context for mysqlSetupJob | | mysqlSetupJob.securityContext | object | `{"runAsUser": 1000}` | Container security context for mysqlSetupJob | +| mysqlSetupJob.podAnnotations | object | `{}` | Pod annotations for mysqlSetupJob | | postgresqlSetupJob.enabled | bool | `false` | Enable setup job for postgresql | | postgresqlSetupJob.image.repository | string | `"acryldata/datahub-postgres-setup"` | Image repository for postgresqlSetupJob | -| postgresqlSetupJob.image.tag | string | `"v0.8.39"` | Image repository for postgresqlSetupJob | +| postgresqlSetupJob.image.tag | string | `"v0.9.2"` | Image repository for postgresqlSetupJob | | postgresqlSetupJob.podSecurityContext | object | `{"fsGroup": 1000}` | Pod security context for mysqlSetupJob | | postgresqlSetupJob.securityContext | object | `{"runAsUser": 1000}` | Container security context for mysqlSetupJob | +| postgresqlSetupJob.podAnnotations | object | `{}` | Pod annotations for mysqlSetupJob | | global.datahub_standalone_consumers_enabled | boolean | true | Enable standalone consumers for kafka | | global.datahub_analytics_enabled | boolean | true | Enable datahub usage analytics | | global.datahub.appVersion | string | `"1.0"` | App version for annotation | @@ -81,6 +87,7 @@ helm install datahub datahub/datahub --values <> | Key | Type | Default | Description | |-----|------|---------|-------------| +| acryl-datahub-actions.ingestionSecretFiles.name | string | `""` | Name of the k8s secret that holds any secret files (e.g., SSL certificates and private keys) that are used in your ingestion recipes. The keys in the secret will be mounted as individual files under `/etc/datahub/ingestion-secret-files` | | global.credentialsAndCertsSecrets.name | string | `""` | Name of the secret that holds SSL certificates (keystores, truststores) | | global.credentialsAndCertsSecrets.path | string | `"/mnt/certs"` | Path to mount the SSL certificates | | global.credentialsAndCertsSecrets.secureEnv | map | `{}` | Map of SSL config name and the corresponding value in the secret | @@ -100,11 +107,17 @@ helm install datahub datahub/datahub --values <> | global.datahub.metadata_service_authentication.tokenService.signingKey.secretKey | string | `token_service_signing_key` | The key of a secret containing the internal system secret that is used to sign JWT auth tokens issued by DataHub GMS. If a secret reference is not provided, a random one will be generated for you in a Kubernetes secret value named `token_service_signing_key` within a secret named `datahub-auth-secrets`. | | global.datahub.metadata_service_authentication.tokenService.salt.secretRef | string | `datahub-auth-secrets` | The reference to a secret containing the internal system secret that is used to salt JWT auth tokens signatures issued by DataHub GMS that is part of the metadata graph. If a secret reference is not provided, a random one will be generated for you in a Kubernetes secret called `datahub-auth-secrets`. | | global.datahub.metadata_service_authentication.tokenService.salt.secretKey | string | `token_service_salt` | The key of a secret containing the internal system secret that is used to salt JWT auth tokens signatures issued by DataHub GMS that is part of the metadata graph. If a secret reference is not provided, a random one will be generated for you in a Kubernetes secret value named `token_service_salt` within a secret named `datahub-auth-secrets`. | -| global.datahub.metadata_service_authentication.provisionSecrets | bool | `true` | Whether auth secrets (token signing key & system client secret) should be provisioned on the first deployment for you. Set this to false if you are overriding global.datahub.metadata_service_authentication.tokenService.signingKey.secretRef or global.datahub.metadata_service_authentication systemClientSecret.secretRef. | +| global.datahub.metadata_service_authentication.provisionSecrets.enabled | bool | `true` | Whether auth secrets (system client secret, token signing key & token service salt) should be provisioned on the first deployment for you. Set this to false if you are overriding `global.datahub.metadata_service_authentication.tokenService.signingKey.secretRef` or `global.datahub.metadata_service_authentication systemClientSecret.secretRef`. | +| global.datahub.metadata_service_authentication.provisionSecrets.autoGenerate | bool | `true` | Whether auth secrets (token signing key, system client secret & token service salt) should be provisioned on the first deployment for you **with a random seed** on the first deployment for you. Set this to false and use `global.datahub.metadata_service_authentication.provisionSecrets.secretValues.*` if you would like to specify the secret values directly. | +| global.datahub.encryptionKey.provisionSecrets.secretValues.secret | string | `` | The system client secret key value to be used if specified directly. | +| global.datahub.encryptionKey.provisionSecrets.secretValues.signingkey | string | `` | The system signing key value to be used if specified directly. | +| global.datahub.encryptionKey.provisionSecrets.secretValues.salt | string | `` | The token service salt value to be used if specified directly. | | global.datahub.managed_ingestion.enabled | bool | `true` | Whether or not UI-based ingestion experience is enabled. | | global.datahub.encryptionKey.secretRef | string | `datahub-encryption-secrets` | The reference to a secret containing an alpha-numeric encryption key, which is used to encrypt Secrets on DataHub. If a secret reference is not provided, a random one will be generated for you in a Kubernetes secret named `datahub-encryption-secrets`. | | global.datahub.encryptionKey.secretKey | string | `encryption_key_secret` | The key of a secret containing an alpha-numeric encryption key, which is used to encrypt Secrets on DataHub. If a secret reference is not provided, a random one will be generated for you in a Kubernetes secret value named `encryption_key_secret` within a secret named `datahub-encryption-secrets`. | -| global.datahub.managed_ingestion.defaultCliVersion | string | `0.8.39.4` | This is the version of the DataHub CLI to use for UI ingestion, by default. | -| global.datahub.encryptionKey.provisionSecret | bool | `true` | Whether an encryption key secret should be provisioned on the first deployment for you. Set this to false if you are overriding global.datahub.encryptionKey.secretRef. | +| global.datahub.managed_ingestion.defaultCliVersion | string | `0.9.1` | This is the version of the DataHub CLI to use for UI ingestion, by default. | +| global.datahub.encryptionKey.provisionSecret.enabled | bool | `true` | Whether an encryption key secret should be provisioned on the first deployment for you. Set this to false if you are overriding global.datahub.encryptionKey.secretRef. | +| global.datahub.encryptionKey.provisionSecret.autoGenerate | bool | `true` | Whether an encryption key secret should be provisioned for you **with a random seed** on the first deployment for you. Set this to false and use `global.datahub.encryptionKey.provisionSecret.secretValues.encryptionKey` if you would like to specify the secret values directly. | +| global.datahub.encryptionKey.provisionSecret.secretValues.encryptionKey | string | `` | The encryption key value to be used if specified directly. | | global.datahub.enable_retention | bool | `false` | Whether or not to enable retention on local DB | | global.sql.datasource.hostForpostgresqlClient | string | `""` | SQL database host (without port) when using postgresqlSetupJob | diff --git a/charts/datahub/quickstart-values-with-neo4j.yaml b/charts/datahub/quickstart-values-with-neo4j.yaml index acd7c2343..4f441bca7 100644 --- a/charts/datahub/quickstart-values-with-neo4j.yaml +++ b/charts/datahub/quickstart-values-with-neo4j.yaml @@ -4,13 +4,13 @@ datahub-gms: enabled: true image: repository: linkedin/datahub-gms - tag: "v0.8.39" + tag: "v0.9.2" datahub-frontend: enabled: true image: repository: linkedin/datahub-frontend-react - tag: "v0.8.39" + tag: "v0.9.2" # Set up ingress to expose react front-end ingress: enabled: false @@ -19,7 +19,7 @@ acryl-datahub-actions: enabled: true image: repository: acryldata/datahub-actions - tag: "v0.0.4" + tag: "v0.0.7" resources: limits: cpu: 500m @@ -32,31 +32,31 @@ elasticsearchSetupJob: enabled: true image: repository: linkedin/datahub-elasticsearch-setup - tag: "v0.8.39" + tag: "v0.9.2" kafkaSetupJob: enabled: true image: repository: linkedin/datahub-kafka-setup - tag: "v0.8.39" + tag: "v0.9.2" mysqlSetupJob: enabled: true image: repository: acryldata/datahub-mysql-setup - tag: "v0.8.39" + tag: "v0.9.2" datahubUpgrade: enabled: true image: repository: acryldata/datahub-upgrade - tag: "v0.8.39" + tag: "v0.9.2" datahub-ingestion-cron: enabled: false image: repository: acryldata/datahub-ingestion - tag: "v0.8.39" + tag: "v0.9.2" global: graph_service_impl: neo4j @@ -102,4 +102,4 @@ global: managed_ingestion: enabled: true - defaultCliVersion: "0.8.39" + defaultCliVersion: "0.9.1" diff --git a/charts/datahub/subcharts/acryl-datahub-actions/Chart.yaml b/charts/datahub/subcharts/acryl-datahub-actions/Chart.yaml index dca17b9d5..58d1a4a04 100644 --- a/charts/datahub/subcharts/acryl-datahub-actions/Chart.yaml +++ b/charts/datahub/subcharts/acryl-datahub-actions/Chart.yaml @@ -1,7 +1,6 @@ apiVersion: v2 name: acryl-datahub-actions description: A Helm chart for Kubernetes - # A chart can be either an 'application' or a 'library' chart. # # Application charts are a collection of templates that can be packaged into versioned archives @@ -11,11 +10,9 @@ description: A Helm chart for Kubernetes # a dependency of application charts to inject those utilities and functions into the rendering # pipeline. Library charts do not define any templates and therefore cannot be deployed. type: application - # This is the chart version. This version number should be incremented each time you make changes # to the chart and its templates, including the app version. -version: 0.0.2 - +version: 0.2.112 # This is the version number of the application being deployed. This version number should be # incremented each time you make changes to the application. -appVersion: 0.0.2 +appVersion: 0.0.7 diff --git a/charts/datahub/subcharts/acryl-datahub-actions/README.md b/charts/datahub/subcharts/acryl-datahub-actions/README.md index eb301cd4f..6b98f0376 100644 --- a/charts/datahub/subcharts/acryl-datahub-actions/README.md +++ b/charts/datahub/subcharts/acryl-datahub-actions/README.md @@ -2,7 +2,7 @@ acryl-datahub-actions ================ A Helm chart for acryl-datahub-actions -Current chart version is `0.0.1` +Current chart version is `0.0.3` ## Chart Values @@ -17,7 +17,7 @@ Current chart version is `0.0.1` | global.datahub.gms.port | string | `"8080"` | | | image.pullPolicy | string | `"IfNotPresent"` | | | image.repository | string | `"acryldata/datahub-actions"` | | -| image.tag | string | `"v0.0.4"` | | +| image.tag | string | `"v0.0.6"` | | | imagePullSecrets | list | `[]` | | | nameOverride | string | `""` | | | nodeSelector | object | `{}` | | diff --git a/charts/datahub/subcharts/acryl-datahub-actions/templates/deployment.yaml b/charts/datahub/subcharts/acryl-datahub-actions/templates/deployment.yaml index 7dfca7d4e..fee5f1855 100644 --- a/charts/datahub/subcharts/acryl-datahub-actions/templates/deployment.yaml +++ b/charts/datahub/subcharts/acryl-datahub-actions/templates/deployment.yaml @@ -35,6 +35,12 @@ spec: defaultMode: 0444 secretName: {{ .name }} {{- end }} + {{- with .Values.ingestionSecretFiles }} + - name: ingestion-secret-files + secret: + defaultMode: 0444 + secretName: {{ .name }} + {{- end }} {{- if .Values.extraVolumes }} {{ toYaml .Values.extraVolumes | nindent 8 }} {{- end }} @@ -103,6 +109,11 @@ spec: {{- with .Values.extraVolumeMounts }} {{- toYaml . | nindent 12 }} {{- end }} + {{- with .Values.ingestionSecretFiles }} + - name: ingestion-secret-files + readOnly: true + mountPath: "/etc/datahub/ingestion-secret-files" + {{- end }} resources: {{- toYaml .Values.resources | nindent 12 }} diff --git a/charts/datahub/subcharts/datahub-frontend/Chart.yaml b/charts/datahub/subcharts/datahub-frontend/Chart.yaml index f6e3c8b60..732d554b3 100644 --- a/charts/datahub/subcharts/datahub-frontend/Chart.yaml +++ b/charts/datahub/subcharts/datahub-frontend/Chart.yaml @@ -1,7 +1,6 @@ apiVersion: v2 name: datahub-frontend description: A Helm chart for Kubernetes - # A chart can be either an 'application' or a 'library' chart. # # Application charts are a collection of templates that can be packaged into versioned archives @@ -11,11 +10,10 @@ description: A Helm chart for Kubernetes # a dependency of application charts to inject those utilities and functions into the rendering # pipeline. Library charts do not define any templates and therefore cannot be deployed. type: application - # This is the chart version. This version number should be incremented each time you make changes # to the chart and its templates, including the app version. -version: 0.2.2 +version: 0.2.112 # This is the version number of the application being deployed. This version number should be # incremented each time you make changes to the application. -appVersion: 0.3.2 +appVersion: v0.9.2 diff --git a/charts/datahub/subcharts/datahub-frontend/README.md b/charts/datahub/subcharts/datahub-frontend/README.md index 60c7bc227..ee8e6e289 100644 --- a/charts/datahub/subcharts/datahub-frontend/README.md +++ b/charts/datahub/subcharts/datahub-frontend/README.md @@ -24,6 +24,7 @@ Current chart version is `0.2.0` | imagePullSecrets | list | `[]` | | | ingress.annotations | object | `{}` | | | ingress.enabled | bool | `false` | | +| ingress.extraLabels | object | `{}` | provides extra labels for ingress configuration | | ingress.hosts[0].host | string | `"chart-example.local"` | | | ingress.hosts[0].paths | list | `[]` | | | ingress.hosts[0].redirectPaths | list | `[]` | | @@ -33,12 +34,22 @@ Current chart version is `0.2.0` | livenessProbe.failureThreshold | int | `4` | | | nameOverride | string | `""` | | | nodeSelector | object | `{}` | | +| oidcAuthentication.enabled | boolean | `false` | Enable [OIDC authentication](https://datahubproject.io/docs/authentication/guides/sso/configure-oidc-react) | +| oidcAuthentication.provider | string | `""` | One of the supported OIDC providers: [google](https://datahubproject.io/docs/authentication/guides/sso/configure-oidc-react-google), [okta](https://datahubproject.io/docs/authentication/guides/sso/configure-oidc-react-okta), or [azure](https://datahubproject.io/docs/authentication/guides/sso/configure-oidc-react-azure) | +| oidcAuthentication.clientId | string | `""` | A unique identifier for your application with the identity provider | +| oidcAuthentication.clientSecret | string | `""` | A shared secret to use for exchange between you and your identity provider | +| oidcAuthentication.clientSecretRef.secretRef | string | `"nil"` | Optional, this is the reference to the shared secret to use for exchange between you and your identity provider | +| oidcAuthentication.clientSecretRef.secretKey | string | `"nil"` | Optional, this is the key of the shared secret to use for exchange between you and your identity provider | +| oidcAuthentication.oktaDomain | string | `""` | Okta domain, e.g. `dev-12345.okta.com`; needed only if `provider` is set to `okta` | +| oidcAuthentication.azureTenantId | string | `""` | Azure directory (tenant) ID; neede only if `provider` is set to `azure` | | podAnnotations | object | `{}` | | | podSecurityContext | object | `{}` | | | readinessProbe.initialDelaySeconds | int | `60` | | | readinessProbe.periodSeconds | int | `30` | | | readinessProbe.failureThreshold | int | `4` | | | replicaCount | int | `1` | | +| revisionHistoryLimit | int | `10` | | +| lifecycle | object | `{}` | | | resources | object | `{}` | | | securityContext | object | `{}` | | | service.port | int | `9001` | | @@ -47,6 +58,7 @@ Current chart version is `0.2.0` | serviceAccount.annotations | object | `{}` | | | serviceAccount.create | bool | `true` | | | serviceAccount.name | string | `nil` | | +| serviceMonitoring.create | bool | `false` | If set true and `global.datahub.monitoring.enablePrometheus` is set `true` it will create a ServiceMonitor resource | | tolerations | list | `[]` | | | global.elasticsearch.host | string | `"elasticsearch"` | | | global.elasticsearch.port | string | `"9200"` | | diff --git a/charts/datahub/subcharts/datahub-frontend/templates/config-jmx-exporter.yaml b/charts/datahub/subcharts/datahub-frontend/templates/config-jmx-exporter.yaml deleted file mode 100644 index f4304f6a7..000000000 --- a/charts/datahub/subcharts/datahub-frontend/templates/config-jmx-exporter.yaml +++ /dev/null @@ -1,17 +0,0 @@ -{{- if .Values.exporters.jmx.enabled }} -apiVersion: v1 -kind: ConfigMap -metadata: - name: {{ include "datahub-frontend.fullname" . }}-config-jmx-exporter - labels: - {{- include "datahub-frontend.labels" . | nindent 4 }} -data: - config.yml: |- - hostPort: localhost:{{ .Values.env.JMXPORT }} - lowercaseOutputName: {{ .Values.exporters.jmx.config.lowercaseOutputName }} - lowercaseOutputLabelNames: {{ .Values.exporters.jmx.config.lowercaseOutputLabelNames }} - rules: -{{ .Values.exporters.jmx.config.rules | toYaml | indent 6 }} - ssl: false - startDelaySeconds: {{ .Values.exporters.jmx.config.startDelaySeconds }} -{{- end }} diff --git a/charts/datahub/subcharts/datahub-frontend/templates/deployment.yaml b/charts/datahub/subcharts/datahub-frontend/templates/deployment.yaml index ef310559c..e6b9cd16a 100644 --- a/charts/datahub/subcharts/datahub-frontend/templates/deployment.yaml +++ b/charts/datahub/subcharts/datahub-frontend/templates/deployment.yaml @@ -6,6 +6,7 @@ metadata: {{- include "datahub-frontend.labels" . | nindent 4 }} spec: replicas: {{ .Values.replicaCount }} + revisionHistoryLimit: {{ .Values.revisionHistoryLimit }} selector: matchLabels: {{- include "datahub-frontend.selectorLabels" . | nindent 6 }} @@ -35,11 +36,6 @@ spec: defaultMode: 0444 secretName: {{ .name }} {{- end }} - {{- if .Values.exporters.jmx.enabled }} - - name: config-jmx-exporter - configMap: - name: {{ include "datahub-gms.fullname" . }}-config-jmx-exporter - {{- end }} {{- with .Values.extraVolumes }} {{- toYaml . | nindent 8 }} {{- end }} @@ -53,10 +49,16 @@ spec: {{- toYaml .Values.securityContext | nindent 12 }} image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}" imagePullPolicy: {{ .Values.image.pullPolicy }} + lifecycle: {{ .Values.lifecycle }} ports: - name: http containerPort: 9002 protocol: TCP + {{- if or .Values.global.datahub.monitoring.enablePrometheus .Values.global.datahub.monitoring.enableJMXPort }} + - name: jmx + containerPort: 4318 + protocol: TCP + {{- end }} livenessProbe: httpGet: path: /admin @@ -72,6 +74,10 @@ spec: periodSeconds: {{ .Values.readinessProbe.periodSeconds }} failureThreshold: {{ .Values.readinessProbe.failureThreshold }} env: + {{- if .Values.global.datahub.monitoring.enablePrometheus }} + - name: ENABLE_PROMETHEUS + value: "true" + {{- end }} - name: DATAHUB_GMS_HOST value: {{ printf "%s-%s" .Release.Name "datahub-gms" }} - name: DATAHUB_GMS_PORT @@ -151,6 +157,47 @@ spec: name: {{ .Values.global.datahub.metadata_service_authentication.systemClientSecret.secretRef }} key: {{ .Values.global.datahub.metadata_service_authentication.systemClientSecret.secretKey }} {{- end }} + {{- with .Values.oidcAuthentication }} + {{- if .enabled }} + - name: AUTH_OIDC_ENABLED + value: "true" + - name: AUTH_OIDC_CLIENT_ID + value: {{ .clientId }} + - name: AUTH_OIDC_CLIENT_SECRET + {{- if .clientSecretRef }} + valueFrom: + secretKeyRef: + name: {{ .clientSecretRef.secretRef }} + key: {{ .clientSecretRef.secretKey }} + {{- else }} + value: {{ .clientSecret }} + {{- end }} + - name: AUTH_OIDC_BASE_URL + value: https://{{ (first $.Values.ingress.hosts).host }} + {{- if eq .provider "google" }} + - name: AUTH_OIDC_DISCOVERY_URI + value: https://accounts.google.com/.well-known/openid-configuration + - name: AUTH_OIDC_SCOPE + value: "openid profile email" + - name: AUTH_OIDC_USER_NAME_CLAIM + value: email + - name: AUTH_OIDC_USER_NAME_CLAIM_REGEX + value: ([^@]+) + {{- else if eq .provider "okta" }} + - name: AUTH_OIDC_DISCOVERY_URI + value: https://{{ .oktaDomain }}/.well-known/openid-configuration + - name: AUTH_OIDC_SCOPE + value: "openid profile email groups" + {{- else if eq .provider "azure" }} + - name: AUTH_OIDC_DISCOVERY_URI + value: https://login.microsoftonline.com/{{ .azureTenantId }}/v2.0/.well-known/openid-configuration + - name: AUTH_OIDC_SCOPE + value: "openid profile email" + {{- else }} + {{- fail (printf "unsupported .oidcAuthentication.provider value '%s'" .provider) }} + {{- end }} + {{- end }} + {{- end }} {{- with .Values.extraEnvs }} {{- toYaml . | nindent 12 }} {{- end }} @@ -165,8 +212,6 @@ spec: resources: {{- toYaml .Values.resources | nindent 12 }} - {{- include "datahub-jmxexporter.container" . }} - {{- with .Values.nodeSelector }} nodeSelector: {{- toYaml . | nindent 8 }} diff --git a/charts/datahub/subcharts/datahub-frontend/templates/ingress.yaml b/charts/datahub/subcharts/datahub-frontend/templates/ingress.yaml index 43c678035..845e45c88 100644 --- a/charts/datahub/subcharts/datahub-frontend/templates/ingress.yaml +++ b/charts/datahub/subcharts/datahub-frontend/templates/ingress.yaml @@ -8,6 +8,9 @@ metadata: name: {{ $fullName }} labels: {{- include "datahub-frontend.labels" . | nindent 4 }} + {{- range $key, $val := .Values.ingress.extraLabels }} + {{ $key }}: {{ $val }} + {{- end }} {{- with .Values.ingress.annotations }} annotations: {{- toYaml . | nindent 4 }} diff --git a/charts/datahub/subcharts/datahub-frontend/templates/service.yaml b/charts/datahub/subcharts/datahub-frontend/templates/service.yaml index fc21cc15f..98b21248b 100644 --- a/charts/datahub/subcharts/datahub-frontend/templates/service.yaml +++ b/charts/datahub/subcharts/datahub-frontend/templates/service.yaml @@ -20,6 +20,12 @@ spec: nodePort: {{ . }} {{- end }} {{- end }} + {{- if .Values.global.datahub.monitoring.enablePrometheus }} + - name: jmx + port: 4318 + targetPort: jmx + protocol: TCP + {{- end }} selector: {{- include "datahub-frontend.selectorLabels" . | nindent 4 }} {{- with .Values.service.spec.loadBalancerSourceRanges }} diff --git a/charts/datahub/subcharts/datahub-frontend/templates/servicemonitor.yaml b/charts/datahub/subcharts/datahub-frontend/templates/servicemonitor.yaml new file mode 100644 index 000000000..5df3ce9be --- /dev/null +++ b/charts/datahub/subcharts/datahub-frontend/templates/servicemonitor.yaml @@ -0,0 +1,25 @@ +{{- if and .Values.serviceMonitor.create .Values.global.datahub.monitoring.enablePrometheus -}} +apiVersion: monitoring.coreos.com/v1 +kind: ServiceMonitor +metadata: + name: {{ printf "%s-%s" .Release.Name "datahub-frontend" }} + labels: + {{- include "datahub-frontend.labels" . | nindent 4 }} + {{- with .Values.serviceMonitor.annotations }} + annotations: + {{- toYaml . | nindent 4 }} + {{- end }} +spec: + endpoints: + - port: jmx + relabelings: + - separator: / + sourceLabels: + - namespace + - pod + targetLabel: instance + selector: + matchLabels: + app.kubernetes.io/managed-by: Helm + app.kubernetes.io/name: datahub-frontend +{{- end -}} \ No newline at end of file diff --git a/charts/datahub/subcharts/datahub-frontend/values.yaml b/charts/datahub/subcharts/datahub-frontend/values.yaml index 205777258..227bfd41a 100644 --- a/charts/datahub/subcharts/datahub-frontend/values.yaml +++ b/charts/datahub/subcharts/datahub-frontend/values.yaml @@ -4,6 +4,8 @@ replicaCount: 1 +revisionHistoryLimit: 10 + image: repository: linkedin/datahub-frontend-react tag: "head" @@ -42,14 +44,18 @@ service: targetPort: http protocol: TCP name: http - # Annotations to add to the service, this will help in adding - # Internal load balancer or various other annotation support in AWS + # Annotations to add to the service, this will help in adding + # Internal load balancer or various other annotation support in AWS annotations: {} # service.beta.kubernetes.io/aws-load-balancer-internal: "true" +serviceMonitor: + create: false + ingress: # className: "" enabled: false + extraLabels: {} annotations: {} # kubernetes.io/ingress.class: nginx # kubernetes.io/tls-acme: "true" @@ -62,6 +68,22 @@ ingress: # hosts: # - chart-example.local +# OIDC auth based on https://datahubproject.io/docs/authentication/guides/sso/configure-oidc-react +oidcAuthentication: + enabled: false + # provider: google/okta/azure <- choose only one + # clientId: your-client-id + # clientSecret: your-client-secret + # only needed if you would like to store the client secret in secret + # clientSecretRef: + # secretRef: + # secretKey: + # only needed if provider is `okta` + # oktaDomain: your-okta-domain.com + + # only neede if provider is `azure` + # azureTenantId: your-azure-tenant-id + # Extra environment variables # This will be appended to the current 'env:' key. You can use any of the kubernetes env # syntax here @@ -80,6 +102,12 @@ extraVolumeMounts: [] extraInitContainers: [] +lifecycle: + # To add a new user to datahub in JAAS config without mounting the user.props file + # postStart: + # exec: + # command: ["/bin/sh", "-c", "echo "test_user:password" >> datahub-frontend/conf/user.props"] + resources: {} # We usually recommend not to specify default resources and to leave this as a conscious # choice for the user. This also increases chances charts run on environments with little @@ -174,7 +202,7 @@ global: gms: port: "8080" appVersion: "1.0" - metadata_service_authentication: + metadata_service_authentication: enabled: false systemClientId: "__datahub_system" # systemClientSecret: diff --git a/charts/datahub/subcharts/datahub-gms/Chart.yaml b/charts/datahub/subcharts/datahub-gms/Chart.yaml index 2fa34d73a..4e0a41eb1 100644 --- a/charts/datahub/subcharts/datahub-gms/Chart.yaml +++ b/charts/datahub/subcharts/datahub-gms/Chart.yaml @@ -1,7 +1,6 @@ apiVersion: v2 name: datahub-gms description: A Helm chart for LinkedIn DataHub's datahub-gms component - # A chart can be either an 'application' or a 'library' chart. # # Application charts are a collection of templates that can be packaged into versioned archives @@ -11,11 +10,9 @@ description: A Helm chart for LinkedIn DataHub's datahub-gms component # a dependency of application charts to inject those utilities and functions into the rendering # pipeline. Library charts do not define any templates and therefore cannot be deployed. type: application - # This is the chart version. This version number should be incremented each time you make changes # to the chart and its templates, including the app version. -version: 0.2.5 - +version: 0.2.112 # This is the version number of the application being deployed. This version number should be # incremented each time you make changes to the application. -appVersion: 0.3.2 +appVersion: v0.9.2 diff --git a/charts/datahub/subcharts/datahub-gms/README.md b/charts/datahub/subcharts/datahub-gms/README.md index 485ec4226..d71df4af0 100644 --- a/charts/datahub/subcharts/datahub-gms/README.md +++ b/charts/datahub/subcharts/datahub-gms/README.md @@ -9,7 +9,6 @@ Current chart version is `0.2.0` | Key | Type | Default | Description | |-----|------|---------|-------------| | affinity | object | `{}` | | -| exporters.jmx.enabled | boolean | false | | | extraEnvs | Extra [environment variables][] which will be appended to the `env:` definition for the container | `[]` | | extraVolumes | Templatable string of additional `volumes` to be passed to the `tpl` function | "" | | extraVolumeMounts | Templatable string of additional `volumeMounts` to be passed to the `tpl` function | "" | @@ -44,6 +43,7 @@ Current chart version is `0.2.0` | imagePullSecrets | list | `[]` | | | ingress.annotations | object | `{}` | | | ingress.enabled | bool | `false` | | +| ingress.extraLabels | object | `{}` | provides extra labels for ingress configuration | | ingress.hosts[0].host | string | `"chart-example.local"` | | | ingress.hosts[0].paths | list | `[]` | | | ingress.tls | list | `[]` | | @@ -58,6 +58,7 @@ Current chart version is `0.2.0` | readinessProbe.periodSeconds | int | `30` | | | readinessProbe.failureThreshold | int | `8` | | | replicaCount | int | `1` | | +| revisionHistoryLimit | int | `10` | | | resources | object | `{}` | | | securityContext | object | `{}` | | | service.port | int | `8080` | | @@ -65,6 +66,7 @@ Current chart version is `0.2.0` | serviceAccount.annotations | object | `{}` | | | serviceAccount.create | bool | `true` | | | serviceAccount.name | string | `nil` | | +| serviceMonitoring.create | bool | `false` | If set true and `global.datahub.monitoring.enablePrometheus` is set `true` it will create a ServiceMonitor resource | | tolerations | list | `[]` | | | datahub.metadata_service_authentication.enabled | bool | `false` | Whether Metadata Service Authentication is enabled. | | global.datahub.metadata_service_authentication.systemClientId | string | `"__datahub_system"` | The internal system id that is used to communicate with DataHub GMS. Required if metadata_service_authentication is 'true'. | @@ -77,5 +79,5 @@ Current chart version is `0.2.0` | global.datahub.managed_ingestion.enabled | bool | `true` | Whether or not UI-based ingestion experience is enabled. | | global.datahub.encryptionKey.secretRef | string | `nil` | The reference to a secret containing an alpha-numeric encryption key, which is used to encrypt Secrets on DataHub. Required if managed_ingestion_enabled is 'true'. | | global.datahub.encryptionKey.secretKey | string | `nil` | The key of a secret containing an alpha-numeric encryption key, which is used to encrypt Secrets on DataHub. Required if managed_ingestion_enabled is 'true'. | -| global.datahub.managed_ingestion.defaultCliVersion | string | `0.8.39.4` | This is the version of the DataHub CLI to use for UI ingestion, by default. You do not need to explicitly provide this. By default the underlying datahub-gms container will provide a latest version compatible with the server. | +| global.datahub.managed_ingestion.defaultCliVersion | string | `0.9.1` | This is the version of the DataHub CLI to use for UI ingestion, by default. You do not need to explicitly provide this. By default the underlying datahub-gms container will provide a latest version compatible with the server. | | global.datahub.enable_retention | bool | `false` | Whether or not to enable retention on local DB | diff --git a/charts/datahub/subcharts/datahub-gms/templates/config-jmx-exporter.yaml b/charts/datahub/subcharts/datahub-gms/templates/config-jmx-exporter.yaml deleted file mode 100644 index 9071aa1e0..000000000 --- a/charts/datahub/subcharts/datahub-gms/templates/config-jmx-exporter.yaml +++ /dev/null @@ -1,25 +0,0 @@ -{{- if .Values.exporters.jmx.enabled }} -apiVersion: v1 -kind: ConfigMap -metadata: - name: {{ include "datahub-gms.fullname" . }}-config-jmx-exporter - labels: - {{- include "datahub-gms.labels" . | nindent 4 }} -data: - config.yml: |- - hostPort: localhost:{{ .Values.env.JMXPORT }} - lowercaseOutputName: {{ .Values.exporters.jmx.config.lowercaseOutputName }} - lowercaseOutputLabelNames: {{ .Values.exporters.jmx.config.lowercaseOutputLabelNames }} - {{- with .Values.exporters.jmx.config.whitelistObjectNames }} - whitelistObjectNames: - {{- toYaml . | nindent 6 }} - {{- end }} - {{- with .Values.exporters.jmx.config.blacklistObjectNames }} - blacklistObjectNames: - {{- toYaml . | nindent 6 }} - {{- end }} - rules: -{{ .Values.exporters.jmx.config.rules | toYaml | indent 6 }} - ssl: false - startDelaySeconds: {{ .Values.exporters.jmx.config.startDelaySeconds }} -{{- end }} diff --git a/charts/datahub/subcharts/datahub-gms/templates/deployment.yaml b/charts/datahub/subcharts/datahub-gms/templates/deployment.yaml index a4328b4b1..cb316cae6 100644 --- a/charts/datahub/subcharts/datahub-gms/templates/deployment.yaml +++ b/charts/datahub/subcharts/datahub-gms/templates/deployment.yaml @@ -6,6 +6,7 @@ metadata: {{- include "datahub-gms.labels" . | nindent 4 }} spec: replicas: {{ .Values.replicaCount }} + revisionHistoryLimit: {{ .Values.revisionHistoryLimit }} selector: matchLabels: {{- include "datahub-gms.selectorLabels" . | nindent 6 }} @@ -39,11 +40,6 @@ spec: defaultMode: 0444 secretName: {{ .name }} {{- end }} - {{- if .Values.exporters.jmx.enabled }} - - name: config-jmx-exporter - configMap: - name: {{ include "datahub-gms.fullname" . }}-config-jmx-exporter - {{- end }} {{- with .Values.extraVolumes }} {{- toYaml . | nindent 8 }} {{- end }} @@ -132,6 +128,8 @@ spec: value: "{{ .Values.global.elasticsearch.host }}" - name: ELASTICSEARCH_PORT value: "{{ .Values.global.elasticsearch.port }}" + - name: SKIP_ELASTICSEARCH_CHECK + value: "{{ .Values.global.elasticsearch.skipcheck }}" {{- with .Values.global.elasticsearch.useSSL }} - name: ELASTICSEARCH_USE_SSL value: {{ . | quote }} @@ -236,8 +234,6 @@ spec: resources: {{- toYaml .Values.resources | nindent 12 }} - {{- include "datahub-jmxexporter.container" . }} - {{- with .Values.nodeSelector }} nodeSelector: {{- toYaml . | nindent 8 }} diff --git a/charts/datahub/subcharts/datahub-gms/templates/ingress.yaml b/charts/datahub/subcharts/datahub-gms/templates/ingress.yaml index 0197f80d5..4f82e05a1 100644 --- a/charts/datahub/subcharts/datahub-gms/templates/ingress.yaml +++ b/charts/datahub/subcharts/datahub-gms/templates/ingress.yaml @@ -8,6 +8,9 @@ metadata: name: {{ $fullName }} labels: {{- include "datahub-gms.labels" . | nindent 4 }} + {{- range $key, $val := .Values.ingress.extraLabels }} + {{ $key }}: {{ $val }} + {{- end }} {{- with .Values.ingress.annotations }} annotations: {{- toYaml . | nindent 4 }} diff --git a/charts/datahub/subcharts/datahub-gms/templates/service.yaml b/charts/datahub/subcharts/datahub-gms/templates/service.yaml index 5c8ce026e..cc417b271 100644 --- a/charts/datahub/subcharts/datahub-gms/templates/service.yaml +++ b/charts/datahub/subcharts/datahub-gms/templates/service.yaml @@ -20,5 +20,11 @@ spec: nodePort: {{ . }} {{- end }} {{- end }} + {{- if .Values.global.datahub.monitoring.enablePrometheus }} + - name: jmx + port: 4318 + targetPort: jmx + protocol: TCP + {{- end }} selector: {{- include "datahub-gms.selectorLabels" . | nindent 4 }} diff --git a/charts/datahub/subcharts/datahub-gms/templates/servicemonitor.yaml b/charts/datahub/subcharts/datahub-gms/templates/servicemonitor.yaml new file mode 100644 index 000000000..bf1bc4996 --- /dev/null +++ b/charts/datahub/subcharts/datahub-gms/templates/servicemonitor.yaml @@ -0,0 +1,25 @@ +{{- if and .Values.serviceMonitor.create .Values.global.datahub.monitoring.enablePrometheus -}} +apiVersion: monitoring.coreos.com/v1 +kind: ServiceMonitor +metadata: + name: {{ printf "%s-%s" .Release.Name "datahub-gms" }} + labels: + {{- include "datahub-gms.labels" . | nindent 4 }} + {{- with .Values.serviceMonitor.annotations }} + annotations: + {{- toYaml . | nindent 4 }} + {{- end }} +spec: + endpoints: + - port: jmx + relabelings: + - separator: / + sourceLabels: + - namespace + - pod + targetLabel: instance + selector: + matchLabels: + app.kubernetes.io/managed-by: Helm + app.kubernetes.io/name: datahub-gms +{{- end -}} \ No newline at end of file diff --git a/charts/datahub/subcharts/datahub-gms/values.yaml b/charts/datahub/subcharts/datahub-gms/values.yaml index 3f3685f9c..5bee12e51 100644 --- a/charts/datahub/subcharts/datahub-gms/values.yaml +++ b/charts/datahub/subcharts/datahub-gms/values.yaml @@ -4,6 +4,8 @@ replicaCount: 1 +revisionHistoryLimit: 10 + image: repository: linkedin/datahub-gms pullPolicy: IfNotPresent @@ -22,6 +24,10 @@ serviceAccount: # If not set and create is true, a name is generated using the fullname template name: +serviceMonitor: + create: false + + podAnnotations: {} # co.elastic.logs/enabled: "true" @@ -42,14 +48,15 @@ service: targetPort: http protocol: TCP name: http - # Annotations to add to the service, this will help in adding - # Internal load balancer or various other annotation support in AWS + # Annotations to add to the service, this will help in adding + # Internal load balancer or various other annotation support in AWS annotations: {} # service.beta.kubernetes.io/aws-load-balancer-internal: "true" ingress: # className: "" enabled: false + extraLabels: {} annotations: {} # kubernetes.io/ingress.class: nginx # kubernetes.io/tls-acme: "true" @@ -98,52 +105,6 @@ tolerations: [] affinity: {} -env: - JMXPORT: 1099 - -exporters: - jmx: - enabled: false -# image: -# repository: bitnami/jmx-exporter -# tag: 0.15.0 -# pullPolicy: IfNotPresent -# config: -# lowercaseOutputName: true -# lowercaseOutputLabelNames: true -# rules: -# - pattern: ".*" -# startDelaySeconds: 30 -# env: {} -# resources: {} -# path: /metrics -# ports: -# jmxxp: -# containerPort: 5556 -# protocol: TCP -# livenessProbe: -# httpGet: -# path: /metrics -# port: jmxxp -# initialDelaySeconds: 30 -# periodSeconds: 15 -# timeoutSeconds: 60 -# failureThreshold: 8 -# successThreshold: 1 -# readinessProbe: -# httpGet: -# path: /metrics -# port: jmxxp -# initialDelaySeconds: 30 -# periodSeconds: 15 -# timeoutSeconds: 60 -# failureThreshold: 8 -# successThreshold: 1 -# serviceMonitor: -# interval: 30s -# scrapeTimeout: 30s -# scheme: http - livenessProbe: initialDelaySeconds: 60 periodSeconds: 30 @@ -163,6 +124,7 @@ global: elasticsearch: host: "elasticsearch" port: "9200" + skipcheck: "false" kafka: bootstrap: @@ -198,12 +160,12 @@ global: managed_ingestion: enabled: true - # defaultCliVersion: "X.X.X" --> Optional: Controls the acryl-datahub package version downloaded from PyPI. + # defaultCliVersion: "X.X.X" --> Optional: Controls the acryl-datahub package version downloaded from PyPI. - metadata_service_authentication: + metadata_service_authentication: enabled: false # tokenService: - # signingKey: + # signingKey: # secretRef: # secretKey: # salt: diff --git a/charts/datahub/subcharts/datahub-ingestion-cron/Chart.yaml b/charts/datahub/subcharts/datahub-ingestion-cron/Chart.yaml index de2562900..75d0b3fa4 100644 --- a/charts/datahub/subcharts/datahub-ingestion-cron/Chart.yaml +++ b/charts/datahub/subcharts/datahub-ingestion-cron/Chart.yaml @@ -1,7 +1,6 @@ apiVersion: v2 name: datahub-ingestion-cron description: A Helm chart for Kubernetes - # A chart can be either an 'application' or a 'library' chart. # # Application charts are a collection of templates that can be packaged into versioned archives @@ -11,11 +10,9 @@ description: A Helm chart for Kubernetes # a dependency of application charts to inject those utilities and functions into the rendering # pipeline. Library charts do not define any templates and therefore cannot be deployed. type: application - # This is the chart version. This version number should be incremented each time you make changes # to the chart and its templates, including the app version. -version: 0.2.3 - +version: 0.2.112 # This is the version number of the application being deployed. This version number should be # incremented each time you make changes to the application. -appVersion: 0.3.2 +appVersion: v0.9.2 diff --git a/charts/datahub/subcharts/datahub-ingestion-cron/README.md b/charts/datahub/subcharts/datahub-ingestion-cron/README.md index 3995b123a..435948462 100644 --- a/charts/datahub/subcharts/datahub-ingestion-cron/README.md +++ b/charts/datahub/subcharts/datahub-ingestion-cron/README.md @@ -26,3 +26,4 @@ A Helm chart for datahub's metadata-ingestion framework with kerberos authentica | crons.extraVolumeMounts | array | `[]` | Additional volume mounts to add to the pods | | crons.extraInitContainers | object | `{}` | Init containers to add to the cronjob container | | crons.serviceAccountName | string | | Service account name used for the cronjob container | +| crons.podAnnotations | object | `{}` | Annotations to add to the pods | diff --git a/charts/datahub/subcharts/datahub-ingestion-cron/templates/cron.yaml b/charts/datahub/subcharts/datahub-ingestion-cron/templates/cron.yaml index 4c318d5ee..b5f08a944 100644 --- a/charts/datahub/subcharts/datahub-ingestion-cron/templates/cron.yaml +++ b/charts/datahub/subcharts/datahub-ingestion-cron/templates/cron.yaml @@ -11,7 +11,12 @@ spec: schedule: {{ default "0 0 * * *" .schedule | quote}} jobTemplate: spec: - template: + template: + {{- with $val.podAnnotations }} + metadata: + annotations: + {{- toYaml . | nindent 12 }} + {{- end }} spec: {{- with $.Values.imagePullSecrets }} imagePullSecrets: diff --git a/charts/datahub/subcharts/datahub-ingestion-cron/values.yaml b/charts/datahub/subcharts/datahub-ingestion-cron/values.yaml index 1aee39875..ad7b31b39 100644 --- a/charts/datahub/subcharts/datahub-ingestion-cron/values.yaml +++ b/charts/datahub/subcharts/datahub-ingestion-cron/values.yaml @@ -54,3 +54,7 @@ crons: {} ## If you want to specify your own service account, set its name like so. ## #serviceAccountName: "my-cron-service" + + ## Add your own pod annotations. + ## + #podAnnotations: {} diff --git a/charts/datahub/subcharts/datahub-jmxexporter/Chart.yaml b/charts/datahub/subcharts/datahub-jmxexporter/Chart.yaml deleted file mode 100644 index cc96e4b07..000000000 --- a/charts/datahub/subcharts/datahub-jmxexporter/Chart.yaml +++ /dev/null @@ -1,21 +0,0 @@ -apiVersion: v2 -name: datahub-jmxexporter -description: A Helm chart for Kubernetes - -# A chart can be either an 'application' or a 'library' chart. -# -# Application charts are a collection of templates that can be packaged into versioned archives -# to be deployed. -# -# Library charts provide useful utilities or functions for the chart developer. They're included as -# a dependency of application charts to inject those utilities and functions into the rendering -# pipeline. Library charts do not define any templates and therefore cannot be deployed. -type: library - -# This is the chart version. This version number should be incremented each time you make changes -# to the chart and its templates, including the app version. -version: 0.2.1 - -# This is the version number of the application being deployed. This version number should be -# incremented each time you make changes to the application. -appVersion: 0.3.1 diff --git a/charts/datahub/subcharts/datahub-jmxexporter/templates/_container.tpl b/charts/datahub/subcharts/datahub-jmxexporter/templates/_container.tpl deleted file mode 100644 index 47ee83419..000000000 --- a/charts/datahub/subcharts/datahub-jmxexporter/templates/_container.tpl +++ /dev/null @@ -1,32 +0,0 @@ -{{- define "datahub-jmxexporter.container" -}} -{{- if .Values.exporters.jmx.enabled }} - - name: jmx-exporter - image: "{{ .Values.exporters.jmx.image.repository }}:{{ .Values.exporters.jmx.image.tag }}" - imagePullPolicy: {{ .Values.exporters.jmx.image.pullPolicy }} - args: ["{{ .Values.exporters.jmx.ports.jmxxp.containerPort}}", "/opt/jmx_exporter/config.yml"] - ports: - {{- range $key, $port := .Values.exporters.jmx.ports }} - - name: {{ $key }} -{{ toYaml $port | indent 14 }} - {{- end }} - livenessProbe: -{{ toYaml .Values.exporters.jmx.livenessProbe | indent 12 }} - readinessProbe: -{{ toYaml .Values.exporters.jmx.readinessProbe | indent 12 }} - env: - - name: SERVICE_PORT - value: {{ .Values.exporters.jmx.ports.jmxxp.containerPort | quote }} - {{- with .Values.exporters.jmx.env }} - {{- range $key, $value := . }} - - name: {{ $key | upper | replace "." "_" }} - value: {{ $value | quote }} - {{- end }} - {{- end }} - resources: -{{ toYaml .Values.exporters.jmx.resources | indent 12 }} - volumeMounts: - - name: config-jmx-exporter - mountPath: /opt/jmx_exporter/config.yml - subPath: config.yml -{{- end }} -{{- end }} diff --git a/charts/datahub/subcharts/datahub-mae-consumer/Chart.yaml b/charts/datahub/subcharts/datahub-mae-consumer/Chart.yaml index da1d0c9ff..14cb13dc6 100644 --- a/charts/datahub/subcharts/datahub-mae-consumer/Chart.yaml +++ b/charts/datahub/subcharts/datahub-mae-consumer/Chart.yaml @@ -1,7 +1,6 @@ apiVersion: v2 name: datahub-mae-consumer description: A Helm chart for Kubernetes - # A chart can be either an 'application' or a 'library' chart. # # Application charts are a collection of templates that can be packaged into versioned archives @@ -11,11 +10,9 @@ description: A Helm chart for Kubernetes # a dependency of application charts to inject those utilities and functions into the rendering # pipeline. Library charts do not define any templates and therefore cannot be deployed. type: application - # This is the chart version. This version number should be incremented each time you make changes # to the chart and its templates, including the app version. -version: 0.2.5 - +version: 0.2.112 # This is the version number of the application being deployed. This version number should be # incremented each time you make changes to the application. -appVersion: 0.3.2 +appVersion: v0.9.2 diff --git a/charts/datahub/subcharts/datahub-mae-consumer/README.md b/charts/datahub/subcharts/datahub-mae-consumer/README.md index 950eb69b2..3ddb698fe 100644 --- a/charts/datahub/subcharts/datahub-mae-consumer/README.md +++ b/charts/datahub/subcharts/datahub-mae-consumer/README.md @@ -52,6 +52,7 @@ Current chart version is `0.2.0` | readinessProbe.periodSeconds | int | `30` | | | readinessProbe.failureThreshold | int | `8` | | | replicaCount | int | `1` | | +| revisionHistoryLimit | int | `10` | | | resources | object | `{}` | | | securityContext | object | `{}` | | | service.port | int | `80` | | @@ -59,8 +60,9 @@ Current chart version is `0.2.0` | serviceAccount.annotations | object | `{}` | | | serviceAccount.create | bool | `true` | | | serviceAccount.name | string | `nil` | | +| serviceMonitoring.create | bool | `false` | If set true and `global.datahub.monitoring.enablePrometheus` is set `true` it will create a ServiceMonitor resource | | tolerations | list | `[]` | | | datahub.metadata_service_authentication.enabled | bool | `false` | Whether Metadata Service Authentication is enabled. | | global.datahub.metadata_service_authentication.systemClientId | string | `"__datahub_system"` | The internal system id that is used to communicate with DataHub GMS. Required if metadata_service_authentication is 'true'. | | global.datahub.metadata_service_authentication.systemClientSecret.secretRef | string | `nil` | The reference to a secret containing the internal system secret that is used to communicate with DataHub GMS. Required if metadata_service_authentication is 'true'. | -| global.datahub.metadata_service_authentication.systemClientSecret.secretKey | string | `nil` | The key of a secret containing the internal system secret that is used to communicate with DataHub GMS. Required if metadata_service_authentication is 'true'. | \ No newline at end of file +| global.datahub.metadata_service_authentication.systemClientSecret.secretKey | string | `nil` | The key of a secret containing the internal system secret that is used to communicate with DataHub GMS. Required if metadata_service_authentication is 'true'. | diff --git a/charts/datahub/subcharts/datahub-mae-consumer/templates/config-jmx-exporter.yaml b/charts/datahub/subcharts/datahub-mae-consumer/templates/config-jmx-exporter.yaml deleted file mode 100644 index f3b6f0bab..000000000 --- a/charts/datahub/subcharts/datahub-mae-consumer/templates/config-jmx-exporter.yaml +++ /dev/null @@ -1,25 +0,0 @@ -{{- if .Values.exporters.jmx.enabled }} -apiVersion: v1 -kind: ConfigMap -metadata: - name: {{ include "datahub-mae-consumer.fullname" . }}-config-jmx-exporter - labels: - {{- include "datahub-mae-consumer.labels" . | nindent 4 }} -data: - config.yml: |- - hostPort: localhost:{{ .Values.env.JMXPORT }} - lowercaseOutputName: {{ .Values.exporters.jmx.config.lowercaseOutputName }} - lowercaseOutputLabelNames: {{ .Values.exporters.jmx.config.lowercaseOutputLabelNames }} - {{- with .Values.exporters.jmx.config.whitelistObjectNames }} - whitelistObjectNames: - {{- toYaml . | nindent 6 }} - {{- end }} - {{- with .Values.exporters.jmx.config.blacklistObjectNames }} - blacklistObjectNames: - {{- toYaml . | nindent 6 }} - {{- end }} - rules: -{{ .Values.exporters.jmx.config.rules | toYaml | indent 6 }} - ssl: false - startDelaySeconds: {{ .Values.exporters.jmx.config.startDelaySeconds }} -{{- end }} diff --git a/charts/datahub/subcharts/datahub-mae-consumer/templates/deployment.yaml b/charts/datahub/subcharts/datahub-mae-consumer/templates/deployment.yaml index 2031564e6..6a74316e1 100644 --- a/charts/datahub/subcharts/datahub-mae-consumer/templates/deployment.yaml +++ b/charts/datahub/subcharts/datahub-mae-consumer/templates/deployment.yaml @@ -6,6 +6,7 @@ metadata: {{- include "datahub-mae-consumer.labels" . | nindent 4 }} spec: replicas: {{ .Values.replicaCount }} + revisionHistoryLimit: {{ .Values.revisionHistoryLimit }} selector: matchLabels: {{- include "datahub-mae-consumer.selectorLabels" . | nindent 6 }} @@ -39,11 +40,6 @@ spec: defaultMode: 0444 secretName: {{ .name }} {{- end }} - {{- if .Values.exporters.jmx.enabled }} - - name: config-jmx-exporter - configMap: - name: {{ include "datahub-mae-consumer.fullname" . }}-config-jmx-exporter - {{- end }} {{- with .Values.extraVolumes }} {{- toYaml . | nindent 8 }} {{- end }} @@ -117,6 +113,8 @@ spec: value: "{{ .Values.global.elasticsearch.host }}" - name: ELASTICSEARCH_PORT value: "{{ .Values.global.elasticsearch.port }}" + - name: SKIP_ELASTICSEARCH_CHECK + value: "{{ .Values.global.elasticsearch.skipcheck }}" {{- with .Values.global.elasticsearch.useSSL }} - name: ELASTICSEARCH_USE_SSL value: {{ . | quote }} @@ -202,8 +200,6 @@ spec: resources: {{- toYaml .Values.resources | nindent 12 }} - {{- include "datahub-jmxexporter.container" . }} - {{- with .Values.nodeSelector }} nodeSelector: {{- toYaml . | nindent 8 }} diff --git a/charts/datahub/subcharts/datahub-mae-consumer/templates/service.yaml b/charts/datahub/subcharts/datahub-mae-consumer/templates/service.yaml index bd9ca7c8c..9bba91be5 100644 --- a/charts/datahub/subcharts/datahub-mae-consumer/templates/service.yaml +++ b/charts/datahub/subcharts/datahub-mae-consumer/templates/service.yaml @@ -16,5 +16,11 @@ spec: nodePort: {{ . }} {{- end }} {{- end }} + {{- if .Values.global.datahub.monitoring.enablePrometheus }} + - name: jmx + port: 4318 + targetPort: jmx + protocol: TCP + {{- end }} selector: {{- include "datahub-mae-consumer.selectorLabels" . | nindent 4 }} diff --git a/charts/datahub/subcharts/datahub-mae-consumer/templates/servicemonitor.yaml b/charts/datahub/subcharts/datahub-mae-consumer/templates/servicemonitor.yaml new file mode 100644 index 000000000..450fe182b --- /dev/null +++ b/charts/datahub/subcharts/datahub-mae-consumer/templates/servicemonitor.yaml @@ -0,0 +1,25 @@ +{{- if and .Values.serviceMonitor.create .Values.global.datahub.monitoring.enablePrometheus -}} +apiVersion: monitoring.coreos.com/v1 +kind: ServiceMonitor +metadata: + name: {{ printf "%s-%s" .Release.Name "datahub-mae-consumer" }} + labels: + {{- include "datahub-mae-consumer.labels" . | nindent 4 }} + {{- with .Values.serviceMonitor.annotations }} + annotations: + {{- toYaml . | nindent 4 }} + {{- end }} +spec: + endpoints: + - port: jmx + relabelings: + - separator: / + sourceLabels: + - namespace + - pod + targetLabel: instance + selector: + matchLabels: + app.kubernetes.io/managed-by: Helm + app.kubernetes.io/name: datahub-mae-consumer +{{- end -}} \ No newline at end of file diff --git a/charts/datahub/subcharts/datahub-mae-consumer/values.yaml b/charts/datahub/subcharts/datahub-mae-consumer/values.yaml index fbfb400f1..6a797f705 100644 --- a/charts/datahub/subcharts/datahub-mae-consumer/values.yaml +++ b/charts/datahub/subcharts/datahub-mae-consumer/values.yaml @@ -4,6 +4,8 @@ replicaCount: 1 +revisionHistoryLimit: 10 + image: repository: linkedin/datahub-mae-consumer pullPolicy: IfNotPresent @@ -22,6 +24,9 @@ serviceAccount: # If not set and create is true, a name is generated using the fullname template name: +serviceMonitor: + create: false + podAnnotations: {} # co.elastic.logs/enabled: "true" @@ -154,6 +159,7 @@ global: elasticsearch: host: "elasticsearch" port: "9200" + skipcheck: "false" kafka: bootstrap: @@ -176,7 +182,7 @@ global: mae_consumer: port: "9091" - metadata_service_authentication: + metadata_service_authentication: enabled: false systemClientId: "__datahub_system" # systemClientSecret: diff --git a/charts/datahub/subcharts/datahub-mce-consumer/Chart.yaml b/charts/datahub/subcharts/datahub-mce-consumer/Chart.yaml index ad5f62c9f..b7bc68541 100644 --- a/charts/datahub/subcharts/datahub-mce-consumer/Chart.yaml +++ b/charts/datahub/subcharts/datahub-mce-consumer/Chart.yaml @@ -1,7 +1,6 @@ apiVersion: v2 name: datahub-mce-consumer description: A Helm chart for Kubernetes - # A chart can be either an 'application' or a 'library' chart. # # Application charts are a collection of templates that can be packaged into versioned archives @@ -11,11 +10,9 @@ description: A Helm chart for Kubernetes # a dependency of application charts to inject those utilities and functions into the rendering # pipeline. Library charts do not define any templates and therefore cannot be deployed. type: application - # This is the chart version. This version number should be incremented each time you make changes # to the chart and its templates, including the app version. -version: 0.2.5 - +version: 0.2.112 # This is the version number of the application being deployed. This version number should be # incremented each time you make changes to the application. -appVersion: 0.3.2 +appVersion: v0.9.2 diff --git a/charts/datahub/subcharts/datahub-mce-consumer/README.md b/charts/datahub/subcharts/datahub-mce-consumer/README.md index fa92d101e..6163d4b6b 100644 --- a/charts/datahub/subcharts/datahub-mce-consumer/README.md +++ b/charts/datahub/subcharts/datahub-mce-consumer/README.md @@ -42,6 +42,7 @@ Current chart version is `0.2.0` | readinessProbe.periodSeconds | int | `30` | | | readinessProbe.failureThreshold | int | `4` | | | replicaCount | int | `1` | | +| revisionHistoryLimit | int | `10` | | | resources | object | `{}` | | | securityContext | object | `{}` | | | service.port | int | `80` | | @@ -49,8 +50,9 @@ Current chart version is `0.2.0` | serviceAccount.annotations | object | `{}` | | | serviceAccount.create | bool | `true` | | | serviceAccount.name | string | `nil` | | +| serviceMonitoring.create | bool | `false` | If set true and `global.datahub.monitoring.enablePrometheus` is set `true` it will create a ServiceMonitor resource | | tolerations | list | `[]` | | | datahub.metadata_service_authentication.enabled | bool | `false` | Whether Metadata Service Authentication is enabled. | | global.datahub.metadata_service_authentication.systemClientId | string | `"__datahub_system"` | The internal system id that is used to communicate with DataHub GMS. Required if metadata_service_authentication is 'true'. | | global.datahub.metadata_service_authentication.systemClientSecret.secretRef | string | `nil` | The reference to a secret containing the internal system secret that is used to communicate with DataHub GMS. Required if metadata_service_authentication is 'true'. | -| global.datahub.metadata_service_authentication.systemClientSecret.secretKey | string | `nil` | The key of a secret containing the internal system secret that is used to communicate with DataHub GMS. Required if metadata_service_authentication is 'true'. | \ No newline at end of file +| global.datahub.metadata_service_authentication.systemClientSecret.secretKey | string | `nil` | The key of a secret containing the internal system secret that is used to communicate with DataHub GMS. Required if metadata_service_authentication is 'true'. | diff --git a/charts/datahub/subcharts/datahub-mce-consumer/templates/config-jmx-exporter.yaml b/charts/datahub/subcharts/datahub-mce-consumer/templates/config-jmx-exporter.yaml deleted file mode 100644 index 9663d9bcb..000000000 --- a/charts/datahub/subcharts/datahub-mce-consumer/templates/config-jmx-exporter.yaml +++ /dev/null @@ -1,25 +0,0 @@ -{{- if .Values.exporters.jmx.enabled }} -apiVersion: v1 -kind: ConfigMap -metadata: - name: {{ include "datahub-mce-consumer.fullname" . }}-config-jmx-exporter - labels: - {{- include "datahub-mce-consumer.labels" . | nindent 4 }} -data: - config.yml: |- - hostPort: localhost:{{ .Values.env.JMXPORT }} - lowercaseOutputName: {{ .Values.exporters.jmx.config.lowercaseOutputName }} - lowercaseOutputLabelNames: {{ .Values.exporters.jmx.config.lowercaseOutputLabelNames }} - {{- with .Values.exporters.jmx.config.whitelistObjectNames }} - whitelistObjectNames: - {{- toYaml . | nindent 6 }} - {{- end }} - {{- with .Values.exporters.jmx.config.blacklistObjectNames }} - blacklistObjectNames: - {{- toYaml . | nindent 6 }} - {{- end }} - rules: -{{ .Values.exporters.jmx.config.rules | toYaml | indent 6 }} - ssl: false - startDelaySeconds: {{ .Values.exporters.jmx.config.startDelaySeconds }} -{{- end }} diff --git a/charts/datahub/subcharts/datahub-mce-consumer/templates/deployment.yaml b/charts/datahub/subcharts/datahub-mce-consumer/templates/deployment.yaml index a6c86e4e8..ab2d99d2e 100644 --- a/charts/datahub/subcharts/datahub-mce-consumer/templates/deployment.yaml +++ b/charts/datahub/subcharts/datahub-mce-consumer/templates/deployment.yaml @@ -6,6 +6,7 @@ metadata: {{- include "datahub-mce-consumer.labels" . | nindent 4 }} spec: replicas: {{ .Values.replicaCount }} + revisionHistoryLimit: {{ .Values.revisionHistoryLimit }} selector: matchLabels: {{- include "datahub-mce-consumer.selectorLabels" . | nindent 6 }} @@ -39,11 +40,6 @@ spec: defaultMode: 0444 secretName: {{ .Values.global.credentialsAndCertsSecrets.name }} {{- end }} - {{- if .Values.exporters.jmx.enabled }} - - name: config-jmx-exporter - configMap: - name: {{ include "datahub-mce-consumer.fullname" . }}-config-jmx-exporter - {{- end }} {{- with .Values.extraVolumes }} {{- toYaml . | nindent 8 }} {{- end }} @@ -58,6 +54,9 @@ spec: image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}" imagePullPolicy: {{ .Values.image.pullPolicy }} ports: + - name: http + containerPort: 9090 + protocol: TCP {{- if or .Values.global.datahub.monitoring.enablePrometheus .Values.global.datahub.monitoring.enableJMXPort }} - name: jmx containerPort: 4318 @@ -66,18 +65,22 @@ spec: livenessProbe: httpGet: path: /actuator/health - port: 9090 + port: http initialDelaySeconds: {{ .Values.livenessProbe.initialDelaySeconds }} periodSeconds: {{ .Values.livenessProbe.periodSeconds }} failureThreshold: {{ .Values.livenessProbe.failureThreshold }} readinessProbe: httpGet: path: /actuator/health - port: 9090 + port: http initialDelaySeconds: {{ .Values.readinessProbe.initialDelaySeconds }} periodSeconds: {{ .Values.readinessProbe.periodSeconds }} failureThreshold: {{ .Values.readinessProbe.failureThreshold }} env: + {{- if .Values.global.datahub.monitoring.enablePrometheus }} + - name: ENABLE_PROMETHEUS + value: "true" + {{- end }} - name: MCE_CONSUMER_ENABLED value: "true" - name: KAFKA_BOOTSTRAP_SERVER @@ -142,8 +145,6 @@ spec: resources: {{- toYaml .Values.resources | nindent 12 }} - {{- include "datahub-jmxexporter.container" . }} - {{- with .Values.nodeSelector }} nodeSelector: {{- toYaml . | nindent 8 }} diff --git a/charts/datahub/subcharts/datahub-mce-consumer/templates/service.yaml b/charts/datahub/subcharts/datahub-mce-consumer/templates/service.yaml new file mode 100644 index 000000000..67cd48d98 --- /dev/null +++ b/charts/datahub/subcharts/datahub-mce-consumer/templates/service.yaml @@ -0,0 +1,26 @@ +apiVersion: v1 +kind: Service +metadata: + name: {{ printf "%s-%s" .Release.Name "datahub-mce-consumer" }} + labels: + {{- include "datahub-mce-consumer.labels" . | nindent 4 }} +spec: + type: {{ .Values.service.type }} + ports: + - port: {{ .Values.service.port }} + targetPort: {{ .Values.service.targetPort }} + protocol: {{ .Values.service.protocol }} + name: {{ .Values.service.name }} + {{- if eq .Values.service.type "NodePort" }} + {{- with .Values.service.nodePort }} + nodePort: {{ . }} + {{- end }} + {{- end }} + {{- if .Values.global.datahub.monitoring.enablePrometheus }} + - name: jmx + port: 4318 + targetPort: jmx + protocol: TCP + {{- end }} + selector: + {{- include "datahub-mce-consumer.selectorLabels" . | nindent 4 }} diff --git a/charts/datahub/subcharts/datahub-mce-consumer/templates/servicemonitor.yaml b/charts/datahub/subcharts/datahub-mce-consumer/templates/servicemonitor.yaml new file mode 100644 index 000000000..0e0a54ad3 --- /dev/null +++ b/charts/datahub/subcharts/datahub-mce-consumer/templates/servicemonitor.yaml @@ -0,0 +1,25 @@ +{{- if and .Values.serviceMonitor.create .Values.global.datahub.monitoring.enablePrometheus -}} +apiVersion: monitoring.coreos.com/v1 +kind: ServiceMonitor +metadata: + name: {{ printf "%s-%s" .Release.Name "datahub-mce-consumer" }} + labels: + {{- include "datahub-mce-consumer.labels" . | nindent 4 }} + {{- with .Values.serviceMonitor.annotations }} + annotations: + {{- toYaml . | nindent 4 }} + {{- end }} +spec: + endpoints: + - port: jmx + relabelings: + - separator: / + sourceLabels: + - namespace + - pod + targetLabel: instance + selector: + matchLabels: + app.kubernetes.io/managed-by: Helm + app.kubernetes.io/name: datahub-mce-consumer +{{- end -}} \ No newline at end of file diff --git a/charts/datahub/subcharts/datahub-mce-consumer/values.yaml b/charts/datahub/subcharts/datahub-mce-consumer/values.yaml index 2c3397edf..d33c3938e 100644 --- a/charts/datahub/subcharts/datahub-mce-consumer/values.yaml +++ b/charts/datahub/subcharts/datahub-mce-consumer/values.yaml @@ -4,6 +4,8 @@ replicaCount: 1 +revisionHistoryLimit: 10 + image: repository: linkedin/datahub-mce-consumer pullPolicy: IfNotPresent @@ -22,6 +24,9 @@ serviceAccount: # If not set and create is true, a name is generated using the fullname template name: +serviceAccount: + create: false + podAnnotations: {} # co.elastic.logs/enabled: "true" @@ -37,8 +42,14 @@ securityContext: {} # runAsUser: 1000 service: - type: ClusterIP - port: 80 + type: ClusterIP # NodePort + port: "9090" + targetPort: http + protocol: TCP + name: http + +serviceMonitor: + create: false ingress: enabled: false @@ -152,9 +163,11 @@ global: url: "http://schema-registry:8081" datahub: + monitoring: + enablePrometheus: false gms: port: "8080" - metadata_service_authentication: + metadata_service_authentication: enabled: false systemClientId: "__datahub_system" # systemClientSecret: @@ -167,4 +180,4 @@ global: - "broker" - "mysql" - "elasticsearch" - - "neo4j" \ No newline at end of file + - "neo4j" diff --git a/charts/datahub/templates/datahub-auth-secrets.yml b/charts/datahub/templates/datahub-auth-secrets.yml index 6b3b33ae3..fcc2fe1ac 100644 --- a/charts/datahub/templates/datahub-auth-secrets.yml +++ b/charts/datahub/templates/datahub-auth-secrets.yml @@ -1,13 +1,23 @@ {{- $secret := lookup "v1" "Secret" .Release.Namespace "datahub-auth-secrets" -}} {{- $data := $secret.data | default dict -}} -{{- if .Values.global.datahub.metadata_service_authentication.provisionSecrets -}} +{{- with .Values.global.datahub.metadata_service_authentication.provisionSecrets }} + +{{- if .enabled }} apiVersion: v1 kind: Secret metadata: name: "datahub-auth-secrets" type: Opaque data: - system_client_secret: {{ index $data "system_client_secret" | default (randAlphaNum 32 | b64enc | quote) }} - token_service_signing_key: {{ index $data "token_service_signing_key" | default (randAlphaNum 32 | b64enc | quote) }} - token_service_salt: {{ index $data "token_service_salt" | default (randAlphaNum 32 | b64enc | quote) }} -{{- end -}} \ No newline at end of file + {{- if .autoGenerate }} + system_client_secret: {{ get $data "system_client_secret" | default (randAlphaNum 32 | b64enc | quote) }} + token_service_signing_key: {{ get $data "token_service_signing_key"| default (randAlphaNum 32 | b64enc | quote) }} + token_service_salt: {{ get $data "token_service_salt" | default (randAlphaNum 32 | b64enc | quote) }} + {{- else }} + system_client_secret: {{ .secretValues.secret | b64enc | quote }} + token_service_signing_key: {{ .secretValues.signingKey | b64enc | quote }} + token_service_salt: {{ .secretValues.salt | b64enc | quote }} + {{- end }} + +{{- end }} +{{- end -}} diff --git a/charts/datahub/templates/datahub-encryption-secrets.yml b/charts/datahub/templates/datahub-encryption-secrets.yml index ba2b3a0b5..1dc656cac 100644 --- a/charts/datahub/templates/datahub-encryption-secrets.yml +++ b/charts/datahub/templates/datahub-encryption-secrets.yml @@ -1,11 +1,19 @@ {{- $secret := lookup "v1" "Secret" .Release.Namespace "datahub-encryption-secrets" -}} {{- $data := $secret.data | default dict -}} -{{- if .Values.global.datahub.encryptionKey.provisionSecret -}} +{{- with .Values.global.datahub.encryptionKey.provisionSecret }} + +{{- if .enabled }} apiVersion: v1 kind: Secret metadata: name: "datahub-encryption-secrets" type: Opaque data: - encryption_key_secret: {{ index $data "encryption_key_secret" | default (randAlphaNum 20 | b64enc | quote) }} -{{- end -}} \ No newline at end of file + {{- if .autoGenerate }} + encryption_key_secret: {{ get $data "encryption_key_secret" | default (randAlphaNum 20 | b64enc | quote) }} + {{- else }} + encryption_key_secret: {{ .secretValues.encryptionKey | b64enc | quote }} + {{- end }} + +{{- end }} +{{- end -}} diff --git a/charts/datahub/templates/datahub-upgrade/_upgrade.tpl b/charts/datahub/templates/datahub-upgrade/_upgrade.tpl index 44c0011d7..19015b018 100644 --- a/charts/datahub/templates/datahub-upgrade/_upgrade.tpl +++ b/charts/datahub/templates/datahub-upgrade/_upgrade.tpl @@ -34,6 +34,10 @@ Return the env variables for upgrade jobs value: {{ .Values.global.elasticsearch.host | quote }} - name: ELASTICSEARCH_PORT value: {{ .Values.global.elasticsearch.port | quote }} +- name: SKIP_ELASTICSEARCH_CHECK + value: {{ .Values.global.elasticsearch.skipcheck | quote }} +- name: ELASTICSEARCH_INSECURE + value: {{ .Values.global.elasticsearch.insecure | quote }} {{- with .Values.global.elasticsearch.useSSL }} - name: ELASTICSEARCH_USE_SSL value: {{ . | quote }} diff --git a/charts/datahub/templates/datahub-upgrade/datahub-cleanup-job-template.yml b/charts/datahub/templates/datahub-upgrade/datahub-cleanup-job-template.yml index ca306fb5c..532e8c147 100644 --- a/charts/datahub/templates/datahub-upgrade/datahub-cleanup-job-template.yml +++ b/charts/datahub/templates/datahub-upgrade/datahub-cleanup-job-template.yml @@ -18,13 +18,19 @@ spec: jobTemplate: spec: template: - {{- if .Values.global.podLabels }} + {{- if or .Values.global.podLabels .Values.datahubUpgrade.podAnnotations }} metadata: + {{- with .Values.datahubUpgrade.podAnnotations }} + annotations: + {{- toYaml . | nindent 12 }} + {{- end }} + {{- with .Values.global.podLabels }} labels: - {{- range $key, $value := .Values.global.podLabels }} + {{- range $key, $value := . }} {{ $key }}: {{ $value | quote }} {{- end }} {{- end }} + {{- end }} spec: {{- with .Values.global.hostAliases }} hostAliases: diff --git a/charts/datahub/templates/datahub-upgrade/datahub-restore-indices-job-template.yml b/charts/datahub/templates/datahub-upgrade/datahub-restore-indices-job-template.yml index cf9a7150e..1357a07aa 100644 --- a/charts/datahub/templates/datahub-upgrade/datahub-restore-indices-job-template.yml +++ b/charts/datahub/templates/datahub-upgrade/datahub-restore-indices-job-template.yml @@ -18,13 +18,19 @@ spec: jobTemplate: spec: template: - {{- if .Values.global.podLabels }} + {{- if or .Values.global.podLabels .Values.datahubUpgrade.podAnnotations }} metadata: + {{- with .Values.datahubUpgrade.podAnnotations }} + annotations: + {{- toYaml . | nindent 12 }} + {{- end }} + {{- with .Values.global.podLabels }} labels: - {{- range $key, $value := .Values.global.podLabels }} + {{- range $key, $value := . }} {{ $key }}: {{ $value | quote }} {{- end }} {{- end }} + {{- end }} spec: {{- with .Values.global.hostAliases }} hostAliases: @@ -61,6 +67,10 @@ spec: args: - "-u" - "RestoreIndices" + - "-a" + - "batchSize={{ .Values.datahubUpgrade.batchSize }}" + - "-a" + - "batchDelayMs={{ .Values.datahubUpgrade.batchDelayMs }}" env: {{- include "datahub.upgrade.env" . | nindent 16}} {{- with .Values.datahubUpgrade.extraEnvs }} @@ -77,12 +87,7 @@ spec: {{- toYaml . | nindent 16 }} {{- end }} resources: - limits: - cpu: 500m - memory: 512Mi - requests: - cpu: 300m - memory: 256Mi + {{- toYaml .Values.datahubUpgrade.restoreIndices.resources | nindent 16}} {{- with .Values.datahubUpgrade.nodeSelector }} nodeSelector: {{- toYaml . | nindent 12 }} diff --git a/charts/datahub/templates/datahub-upgrade/datahub-upgrade-job.yml b/charts/datahub/templates/datahub-upgrade/datahub-upgrade-job.yml index 248bd0ad9..f22cd43a4 100644 --- a/charts/datahub/templates/datahub-upgrade/datahub-upgrade-job.yml +++ b/charts/datahub/templates/datahub-upgrade/datahub-upgrade-job.yml @@ -16,13 +16,19 @@ metadata: "helm.sh/hook-delete-policy": before-hook-creation spec: template: - {{- if .Values.global.podLabels }} + {{- if or .Values.global.podLabels .Values.datahubUpgrade.podAnnotations}} metadata: + {{- with .Values.datahubUpgrade.podAnnotations }} + annotations: + {{- toYaml . | nindent 8 }} + {{- end }} + {{- with .Values.global.podLabels }} labels: - {{- range $key, $value := .Values.global.podLabels }} + {{- range $key, $value := . }} {{ $key }}: {{ $value | quote }} {{- end }} {{- end }} + {{- end }} spec: {{- with .Values.global.hostAliases }} hostAliases: @@ -60,9 +66,9 @@ spec: - "-u" - "NoCodeDataMigration" - "-a" - - "batchSize=1000" + - "batchSize={{ .Values.datahubUpgrade.batchSize }}" - "-a" - - "batchDelayMs=100" + - "batchDelayMs={{ .Values.datahubUpgrade.batchDelayMs }}" - "-a" - "dbType={{ .Values.datahubUpgrade.noCodeDataMigration.sqlDbType }}" env: diff --git a/charts/datahub/templates/elasticsearch-setup-job.yml b/charts/datahub/templates/elasticsearch-setup-job.yml index 61ccabd11..06ecf0975 100644 --- a/charts/datahub/templates/elasticsearch-setup-job.yml +++ b/charts/datahub/templates/elasticsearch-setup-job.yml @@ -16,13 +16,19 @@ metadata: "helm.sh/hook-delete-policy": before-hook-creation spec: template: - {{- if .Values.global.podLabels }} + {{- if or .Values.global.podLabels .Values.elasticsearchSetupJob.podAnnotations }} metadata: + {{- with .Values.elasticsearchSetupJob.podAnnotations }} + annotations: + {{- toYaml . | nindent 8 }} + {{- end }} + {{- with .Values.global.podLabels }} labels: - {{- range $key, $value := .Values.global.podLabels }} + {{- range $key, $value := . }} {{ $key }}: {{ $value | quote }} {{- end }} {{- end }} + {{- end }} spec: {{- with .Values.global.hostAliases }} hostAliases: @@ -51,6 +57,10 @@ spec: value: {{ .Values.global.elasticsearch.host | quote }} - name: ELASTICSEARCH_PORT value: {{ .Values.global.elasticsearch.port | quote }} + - name: SKIP_ELASTICSEARCH_CHECK + value: {{ .Values.global.elasticsearch.skipcheck | quote }} + - name: ELASTICSEARCH_INSECURE + value: {{ .Values.global.elasticsearch.insecure | quote }} {{- with .Values.global.elasticsearch.useSSL }} - name: ELASTICSEARCH_USE_SSL value: {{ . | quote }} diff --git a/charts/datahub/templates/kafka-setup-job.yml b/charts/datahub/templates/kafka-setup-job.yml index 99115f04b..7d907e5e9 100644 --- a/charts/datahub/templates/kafka-setup-job.yml +++ b/charts/datahub/templates/kafka-setup-job.yml @@ -16,13 +16,19 @@ metadata: "helm.sh/hook-delete-policy": before-hook-creation spec: template: - {{- if .Values.global.podLabels }} + {{- if or .Values.global.podLabels .Values.kafkaSetupJob.podAnnotations }} metadata: + {{- with .Values.kafkaSetupJob.podAnnotations }} + annotations: + {{- toYaml . | nindent 8 }} + {{- end }} + {{- with .Values.global.podLabels }} labels: - {{- range $key, $value := .Values.global.podLabels }} + {{- range $key, $value := . }} {{ $key }}: {{ $value | quote }} {{- end }} {{- end }} + {{- end }} spec: {{- with .Values.global.hostAliases }} hostAliases: diff --git a/charts/datahub/templates/mysql-setup-job.yml b/charts/datahub/templates/mysql-setup-job.yml index 51b1dba3a..7f4097351 100644 --- a/charts/datahub/templates/mysql-setup-job.yml +++ b/charts/datahub/templates/mysql-setup-job.yml @@ -16,13 +16,19 @@ metadata: "helm.sh/hook-delete-policy": before-hook-creation spec: template: - {{- if .Values.global.podLabels }} + {{- if or .Values.global.podLabels .Values.mysqlSetupJob.podAnnotations }} metadata: + {{- with .Values.mysqlSetupJob.podAnnotations }} + annotations: + {{- toYaml . | nindent 8 }} + {{- end }} + {{- with .Values.global.podLabels }} labels: - {{- range $key, $value := .Values.global.podLabels }} + {{- range $key, $value := . }} {{ $key }}: {{ $value | quote }} {{- end }} {{- end }} + {{- end }} spec: {{- with .Values.global.hostAliases }} hostAliases: diff --git a/charts/datahub/templates/postgresql-setup-job.yml b/charts/datahub/templates/postgresql-setup-job.yml index bac141368..107a905f3 100644 --- a/charts/datahub/templates/postgresql-setup-job.yml +++ b/charts/datahub/templates/postgresql-setup-job.yml @@ -16,13 +16,19 @@ metadata: "helm.sh/hook-delete-policy": before-hook-creation spec: template: - {{- if .Values.global.podLabels }} + {{- if or .Values.global.podLabels .Values.postgresqlSetupJob.podAnnotations }} metadata: + {{- with .Values.postgresqlSetupJob.podAnnotations }} + annotations: + {{- toYaml . | nindent 8 }} + {{- end }} + {{- with .Values.global.podLabels }} labels: - {{- range $key, $value := .Values.global.podLabels }} + {{- range $key, $value := . }} {{ $key }}: {{ $value | quote }} {{- end }} {{- end }} + {{- end }} spec: {{- with .Values.global.hostAliases }} hostAliases: diff --git a/charts/datahub/values.yaml b/charts/datahub/values.yaml index c71056092..0e2073694 100644 --- a/charts/datahub/values.yaml +++ b/charts/datahub/values.yaml @@ -8,7 +8,7 @@ datahub-gms: enabled: true image: repository: linkedin/datahub-gms - tag: "v0.8.39" + tag: "v0.9.2" affinity: nodeAffinity: @@ -24,11 +24,15 @@ datahub-gms: cpu: 1000m memory: 8Gi + service: + annotations: + service.beta.kubernetes.io/aws-load-balancer-additional-resource-tags: owner=dataplatform,purpose=datahub,sbregion=awsregional,sendbird_region=awsregional + datahub-frontend: enabled: true image: repository: linkedin/datahub-frontend-react - tag: "v0.8.39" + tag: "v0.9.2" # Set up ingress to expose react front-end ingress: enabled: false @@ -62,6 +66,7 @@ datahub-frontend: targetPort: 9002 annotations: service.beta.kubernetes.io/aws-load-balancer-ssl-cert: 'arn:aws:acm:ap-northeast-2:012481551608:certificate/ef2e4fdd-4b9b-4dca-975e-b4eef047ee79' + service.beta.kubernetes.io/aws-load-balancer-additional-resource-tags: owner=dataplatform,purpose=datahub,sbregion=awsregional,sendbird_region=awsregional spec: loadBalancerSourceRanges: - "54.248.189.144/32" @@ -91,7 +96,7 @@ acryl-datahub-actions: enabled: true image: repository: 314716043882.dkr.ecr.ap-northeast-2.amazonaws.com/dataplatform/rivendell - tag: "custom-datahub-actions-v0.0.6-3" + tag: "custom-datahub-actions-v0.0.7-2" affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: @@ -116,70 +121,86 @@ acryl-datahub-actions: datahub-mae-consumer: image: repository: linkedin/datahub-mae-consumer - tag: "v0.8.39" + tag: "v0.9.2" datahub-mce-consumer: image: repository: linkedin/datahub-mce-consumer - tag: "v0.8.39" + tag: "v0.9.2" datahub-ingestion-cron: enabled: false image: repository: acryldata/datahub-ingestion - tag: "v0.8.39" + tag: "v0.9.2" elasticsearchSetupJob: enabled: true image: repository: linkedin/datahub-elasticsearch-setup - tag: "v0.8.39" + tag: "v0.9.2" podSecurityContext: fsGroup: 1000 securityContext: runAsUser: 1000 + podAnnotations: {} kafkaSetupJob: enabled: true image: repository: linkedin/datahub-kafka-setup - tag: "v0.8.39" + tag: "v0.9.2" podSecurityContext: fsGroup: 1000 securityContext: runAsUser: 1000 + podAnnotations: {} mysqlSetupJob: enabled: true image: repository: acryldata/datahub-mysql-setup - tag: "v0.8.39" + tag: "v0.9.2" podSecurityContext: fsGroup: 1000 securityContext: runAsUser: 1000 + podAnnotations: {} postgresqlSetupJob: enabled: false image: repository: acryldata/datahub-postgres-setup - tag: "v0.8.39" + tag: "v0.9.2" podSecurityContext: fsGroup: 1000 securityContext: runAsUser: 1000 + podAnnotations: {} datahubUpgrade: enabled: true image: repository: acryldata/datahub-upgrade - tag: "v0.8.39" + tag: "v0.9.2" + batchSize: 1000 + batchDelayMs: 100 noCodeDataMigration: sqlDbType: "MYSQL" + # sqlDbType: "POSTGRES" podSecurityContext: {} # fsGroup: 1000 securityContext: {} # runAsUser: 1000 + podAnnotations: {} + restoreIndices: + resources: + limits: + cpu: 500m + memory: 512Mi + requests: + cpu: 300m + memory: 256Mi global: graph_service_impl: neo4j @@ -189,6 +210,8 @@ global: elasticsearch: host: "elasticsearch-master" port: "9200" + skipcheck: "false" + insecure: "false" kafka: bootstrap: @@ -225,13 +248,29 @@ global: secretRef: mysql-secrets secretKey: mysql-root-password + ## Use below for usage of PostgreSQL instead of MySQL + # host: "prerequisites-postgresql:5432" + # hostForpostgresqlClient: "prerequisites-postgresql" + # port: "5432" + # url: "jdbc:postgresql://prerequisites-postgresql:5432/datahub" + # driver: "org.postgresql.Driver" + # username: "postgres" + # password: + # secretRef: postgresql-secrets + # secretKey: postgres-password + datahub: gms: port: "8080" nodePort: "30001" + + monitoring: + enablePrometheus: true + mae_consumer: port: "9091" nodePort: "30002" + appVersion: "1.0" encryptionKey: @@ -239,10 +278,16 @@ global: secretKey: "encryption_key_secret" # Set to false if you'd like to provide your own secret. provisionSecret: false +# provisionSecret: +# enabled: true +# autoGenerate: true +# # Only specify if autoGenerate set to false +# # secretValues: +# # encryptionKey: managed_ingestion: enabled: true - defaultCliVersion: "0.8.39" + defaultCliVersion: "0.9.1" metadata_service_authentication: enabled: true @@ -259,16 +304,25 @@ global: secretKey: "token_service_salt" # Set to false if you'd like to provide your own auth secrets provisionSecrets: false +# provisionSecrets: +# enabled: true +# autoGenerate: true +# # Only specify if autoGenerate set to false +# # secretValues: +# # secret: +# # signingKey: +# # salt: # hostAliases: # - ip: "192.168.0.104" # hostnames: # - "broker" # - "mysql" +# - "postgresql" # - "elasticsearch" # - "neo4j" -## Add below to enable SSL for kafka +## Add below to enable SSL for kafka # credentialsAndCertsSecrets: # name: datahub-certs # path: /mnt/datahub/certs diff --git a/charts/datahub/values_stg.yaml b/charts/datahub/values_stg.yaml index a6e4fef94..d95278ab3 100644 --- a/charts/datahub/values_stg.yaml +++ b/charts/datahub/values_stg.yaml @@ -57,7 +57,7 @@ acryl-datahub-actions: - *default-node-group image: repository: 314716043882.dkr.ecr.ap-northeast-2.amazonaws.com/dataplatform/rivendell - tag: "custom-datahub-actions-v0.0.6-stg-5" + tag: "custom-datahub-actions-v0.0.7-stg-2" extraEnvs: - name: PYTHONPATH value: /etc/datahub diff --git a/charts/prerequisites/Chart.yaml b/charts/prerequisites/Chart.yaml index 0ad2e7096..116c32235 100644 --- a/charts/prerequisites/Chart.yaml +++ b/charts/prerequisites/Chart.yaml @@ -4,7 +4,7 @@ description: A Helm chart for packages that Datahub depends on type: application # This is the chart version. This version number should be incremented each time you make changes # to the chart and its templates, including the app version. -version: 0.0.6 +version: 0.0.10 dependencies: - name: elasticsearch version: 7.16.2 @@ -21,9 +21,13 @@ dependencies: repository: https://equinor.github.io/helm-charts/charts/ condition: neo4j-community.enabled - name: mysql - version: 8.5.4 + version: 9.1.2 repository: https://charts.bitnami.com/bitnami condition: mysql.enabled + - name: postgresql + version: 11.2.2 + repository: https://charts.bitnami.com/bitnami + condition: postgresql.enabled # This chart deploys an enterprise version of kafka that requires commercial license # Note, Schema registry and kafka rest proxy do not require the commercial license - name: cp-helm-charts @@ -32,7 +36,7 @@ dependencies: condition: cp-helm-charts.enabled # This chart deploys a community version of kafka - name: kafka - version: 12.17.4 + version: 17.1.0 repository: https://charts.bitnami.com/bitnami condition: kafka.enabled maintainers: diff --git a/charts/prerequisites/values.yaml b/charts/prerequisites/values.yaml index 2db6c9419..5d316d621 100644 --- a/charts/prerequisites/values.yaml +++ b/charts/prerequisites/values.yaml @@ -101,6 +101,12 @@ mysql: persistence: size: 300Gi +postgresql: + enabled: false + auth: + # For better security, add postgresql-secrets k8s secret with postgres-password, replication-password and password + existingSecret: postgresql-secrets + cp-helm-charts: # Schema registry is under the community license cp-schema-registry: diff --git a/custom_images/custom-datahub-actions/Dockerfile b/custom_images/custom-datahub-actions/Dockerfile index bd9810348..7d2361117 100644 --- a/custom_images/custom-datahub-actions/Dockerfile +++ b/custom_images/custom-datahub-actions/Dockerfile @@ -1,9 +1,9 @@ -FROM acryldata/datahub-actions:v0.0.6 +FROM acryldata/datahub-actions:v0.0.7 USER root RUN pip install 'gql[all]==3.4.0' RUN apt install -y logrotate cron vim ADD logrotate.conf /etc/logrotate.d/datahub-actions -RUN sed -i '106s/ "max.poll.interval.ms": "10000"/ "max.poll.interval.ms": "300000"/g' /usr/local/lib/python3.9/site-packages/datahub_actions/plugin/source/kafka/kafka_event_source.py +RUN sed -i '132s/ "max.poll.interval.ms": "10000"/ "max.poll.interval.ms": "300000"/g' /usr/local/lib/python3.10/site-packages/datahub_actions/plugin/source/kafka/kafka_event_source.py ADD start_datahub_actions.sh /start_datahub_actions.sh RUN chmod 755 /start_datahub_actions.sh ADD *.py /etc/datahub diff --git a/custom_images/custom-datahub-actions/gql_get_sb_regions.py b/custom_images/custom-datahub-actions/gql_get_sb_regions.py index 211a7fb31..124aaaf0a 100644 --- a/custom_images/custom-datahub-actions/gql_get_sb_regions.py +++ b/custom_images/custom-datahub-actions/gql_get_sb_regions.py @@ -23,14 +23,22 @@ def get_sb_regions(gms_endpoint: str, token: str, gql_query: str, query: "{gql_query}", start: 0, count: 100, - filters: [ + orFilters: [ {{ - field: "platform", - value: "urn:li:dataPlatform:{platform}" - }}, - {{ - field: "origin", - value: "{ENV.upper()}" + and: [ + {{ + field: "platform", + values: [ + "urn:li:dataPlatform:{platform}" + ] + }}, + {{ + field: "origin", + values: [ + "{ENV.upper()}" + ] + }} + ] }} ] }}