This repository has been archived by the owner on Jan 11, 2024. It is now read-only.
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[DATA-7040] DataHub upgrade to 0.10.4 (#9)
* upgrading to 8.40 (acryldata#139) * Option to add podAnnotations to jobs. (acryldata#137) * feat(helm): add pod annotations to jobs * feat(helm): add pod annotation to cron jobs. * docs(helm): Add podAnnotation documentation and default values. * feat(helm) chart version bump. * update to v0.8.41 (acryldata#143) * fix(mysql): update mysql helm chart version (acryldata#146) * fix(metrics): fix Prometheus env variable in the MCE consumer deployment (acryldata#148) * feat(datahub-frontend): Option to configure OIDC auth easily (acryldata#142) * fix(kafka): update kafka chart version (acryldata#147) * feat(datahub-frontend): Allow lifecycle hooks (acryldata#141) * <feat>(datahub-frontend): Allow lifecycle hooks This PR adds a lifecycle node to the datahub-frontend deployment template. It can be used to execute lifecycle hooks, e.g. postStart actions, like replacing the default user.props file. * Bump chart version to 0.2.86 Co-authored-by: Pedro Silva <pedro.cls93@gmail.com> * Update to v0.8.43 (acryldata#151) * feat(upgrade): allow setting batch args (acryldata#155) * feat(datahub-upgrade): Restore Indices Resources (acryldata#152) * <feat>(datahub-upgrade): Restore Indices Resources This PR adds the ability for end-users to specify resource requests and limits for the datahub upgrade job 'restore indices'. This is needed for situations where the restore indices job will OOM due to a large backfill of data. I've seen this happening with our deployment of datahub and am unable to tune the resources for the restore indices job because the current requests and limits are hard-coded. * fix(gms): remove unused JMX sidecar (acryldata#156) * fix(gms): remove unused JMX sidecar in favour of the already included JVM agent * Update Chart.yaml * Update Chart.yaml * Bumping to v0.8.44 (acryldata#160) * feat(ci): close stale issues/prs (acryldata#164) * feat: add extra labels to ingress configurations (acryldata#163) * feat(prerequisites): add postgresql to prerequistes Helm chart as an optional dependency (acryldata#121) * feat(charts): add an option to configure revisionHistoryLimit to subcharts (acryldata#159) * feat: datahub-frontend add oidc authentication client secret reference (acryldata#165) * fix(monitoring) Fixes monitoring default values + gms chart version (acryldata#161) * chore: bump datahub-actions version (acryldata#168) * Feat(actions): Support mounting a customer defined k8s secrets as files needed in ingestion recipes (acryldata#169) * Mount a customer defined k8s secret for ssl secrets needed in ingestion recipes * Bump chart versions * Address comment * Update readme * Address Pedro's comment * feat(monitoring): enable monitoring on frontend (acryldata#171) * Add kafka-exporter to expose prometheus metrics for kafka cluster (acryldata#170) * Add kafka-exporter to expose prometheus metrics for kafka cluster * Kafka-exporter: Updated values.yaml to support tls/sasl authentication while connecting kafka cluster * Kafka-exporter: Updated values.yaml to remove default values and correct intendation * Add kafka-exporter to expose prometheus metrics for kafka cluster * Kafka-exporter: Updated values.yaml to support tls/sasl authentication while connecting kafka cluster * Kafka-exporter: Updated values.yaml to remove default values and correct intendation * Updated chart version * Bumping actions (acryldata#173) * feat(elasticsearch-setup): Add support for insecure curl connections (acryldata#166) * fix versions, clean jmx exporter (acryldata#175) * bumping helm charts for 0.8.45 (acryldata#178) * bumping to 0.8.45 * Fixing * feat(monitoring): Add optional ServiceMonitors to all JVM based services (acryldata#181) * feat(frontend): add service monitoring * feat(gms): add service monitoring * gms: servicemonitor * feat(mae): enable service monitoring * update READMEs * fix service ports * bump version * Add missing default value * Update charts/datahub/Chart.yaml * delete jmx config and rename port to http * feat(secrets): Allow users to specify their own secret values at first boot (acryldata#180) * fix(auth-secret): Make template use correct retrieval method `datahub-auth-secret` yaml was incorrectly using the `index` template function which is for list variables. For dictionaries as is the case with the `$secret.data` object, the correct method is: https://helm.sh/docs/chart_template_guide/function_list/#get Tested locally, that this works against a running cluster. * Updates encryption secret to use get helm function * feat(secrets): Allow users to specify their own secret values at first boot * fix(datahub-frontend): Remove unused env var (acryldata#176) * fix(datahub-frontend): Remove unused env var * Bump chart version * Change default versions 0.9.0 (acryldata#184) * feat(release): automatically bumbp versions for all subcharts (acryldata#185) * feat(release): automatically bumbp versions for all subcharts * Remove kafka export as subchart Kafka exporter is useful for monitoring datahub, but it's not part of it. It should be deploy sepratly, as we do in production * Add script to bump versions together * Add tagging to the workflow * Don't run release on CI commits * fix(CI): remove duplicate id (acryldata#189) * fix(ci): us sh instead of bash; increase tag by patch (acryldata#190) * DataHub Release v0.9.1 (acryldata#191) * fix(v0.9.1): Fix v0.9.1 release versions (acryldata#192) * Release v0.9.1 Helm * Bump chart versions * Fixing final 210 * release(v0.9.2) Release Helm Charts for DataHub v0.9.2 (acryldata#194) * Bumping to release Helm charts for v0.9.2 * bump version by running: ./.github/scripts/bump-chart-versions.sh 0.2.112 Co-authored-by: szalai1 <szalaipeti.vagyok@gmail.com> * fix(gha): revert auto bump release changes (acryldata#197) * fix(datahub-upgrade): Add system auth credentials to datahub upgrade jobs (acryldata#199) * fix(datahub-upgrade): Add system auth credentials to datahub upgrade jobs * bump chart.yaml version * feat(sql): extend sql username/password configuration (acryldata#198) * Ebean credential configuration flexbility Allows the EBEAN_DATASOURCE_USERNAME and EBEAN_DATASOURCE_PASSWORD to be specified either in global or in the gms config. Also, allows a string value to be specified as the password to enable vault secret webhooks. * Extended postgres creds config * Extended mysql setup job creds config * bump chart version * bump chart version * show example in values.yaml * Added explanation to README * fix typo * tidy values.yaml examples * consistent quote Use the same quoting solution for sql password and usernames across the file * remove space error * Bump gms sub-chart * Update Chart.yaml * add nil condition to gms deploy * ad nil conditional to mysql job * add nil conditional to postgres job * fix(frontend): increase recommended defaults (acryldata#202) * docs: add secret key information (acryldata#195) Added information about the key name in the secret, or it will fail like this: Error: couldn't find key neo4j-password in Secret default/neo4j-secrets * fix(doc): correct doc for serviceMonitor.create (acryldata#204) * Bumping charts to v0.9.3 (acryldata#207) * Update Chart.yaml (acryldata#208) * doc Add global.elasticsearch.useSSL to values.yaml (acryldata#209) * fix: only add DATAHUB_SYSTEM_CLIENT envs when metadata service auth enabled (acryldata#205) Co-authored-by: Pedro Silva <pedro@acryl.io> * fix(frontend): Correct lifecycle templating for frontend (acryldata#201) * Fix lifecycle templating for frontend Fixes `throws ValidationError(Deployment.spec.template.spec.containers[0].lifecycle): invalid type for io.k8s.api.core.v1.Lifecycle: got "string", expected "map"` * bump up Chart.yaml version * set default value for lifecycle * Bump versions Co-authored-by: Pedro Silva <pedro@acryl.io> * fix(ingestion-cron): Make hostAliases work in datahub-ingestion-cron (acryldata#206) * fix: make hostAliases work in datahub-ingestion-cron * Bump ingestion cron subchart version Fixes acryldata#200 Co-authored-by: Pedro Silva <pedro@acryl.io> * bugfix: datahub-upgrade use datasource password value if set (acryldata#210) * feat(global-version): add option to globally define versions (acryldata#212) * Update versions & bitnami index (acryldata#219) * Update prerequisite versions & bitnami repo * fix: move default tag to global.datahub.version (acryldata#214) image.tag in datahub-gms and datahub-fronted shadowed the global.datahub.version from the top datahub chart, so the resulting image was head. * fix: move default tag to global in subcharts (acryldata#220) * fix(jobs): Bump kafka setup resources + make all jobs resources configurable (acryldata#224) * Remove release for master * fix(jobs): Bump kafka setup resources + make all jobs resources configurable * Add documentation * Bump chart.yaml * Lock upgrade job to v0.9.4 Co-authored-by: Dexter Lee <dexter@acryl.io> * fix: add extra space before comment to make yamllint pass (acryldata#226) * fix(release): Revert workflow change & change upgrade job to 0.9.3 (acryldata#227) * feat(mce-consumer): removing dependency on gms for mce consumer (acryldata#216) * feat(mce-consumer): removing dependency on gms for mce consumer envs * feat(elasticsearch): adding elasticsearch index job (acryldata#228) * Feat/elasticsearch optimization ext (acryldata#229) * feat(elasticsearch): allow document count mismatch override * Bumping the versions as part of v0.9.5 release (acryldata#230) * Fix 232 (acryldata#233) * fix(cron-ingestion): referecing $ values * bump versions * feat(elasticsearch): build-indices job (acryldata#231) * feat(elasticsearch): adding elasticsearch index job * feat(elasticsearch): allow document count mismatch override * Adjusting hook order build-indices job * config(gms): enable elasticsearch reindex by default (acryldata#235) * fix(jobs): Make job pull policy consistent with subchart (acryldata#237) * fix(jobs): Make job pull policy consistent with subchart * Explicitly set schema registry type * Update charts/datahub/values.yaml Co-authored-by: Peter Szalai <szalaipeti.vagyok@gmail.com> Co-authored-by: Peter Szalai <szalaipeti.vagyok@gmail.com> * fix(mce-consumer): Correct env vars & configure resources for all components (acryldata#238) * fix(mce-consumer): Correct env vars & configure resources for all components * Update quickstart values to use global datahub version * Update elasticsearch memory configs * fix(version) Update remaining values for 0.9.6 upgrade (acryldata#239) * v0.9.6.1 release (acryldata#241) * Release bump fix (acryldata#243) * fix(release version): fix missing global release bump to 0.9.6.1 * Update values.yaml (acryldata#244) * Update values.yaml * Update Chart.yaml * fix(restore-indices): restore indices should use global datastore values (acryldata#247) * expose oidc scope (acryldata#249) * Update Chart.yaml (acryldata#251) * fix(kafka-setup): Make kafka setup use explicit topic names (acryldata#245) * fix(kafka-setup): Make kafka setup use explicit topic names * Set kafka env vars for all deployments * Update gms & actions pod env vars * Add documentation for new properties * Version bump (acryldata#250) * bump versions * Search improvements updates (acryldata#254) * Set system-update job to enabled by global flag * Datahub upgrade topic (acryldata#255) * v0.9.6.1 release * fix(release version): fix missing global release bump to 0.9.6.1 * bump versions * Update for search improvements and generic system update job * enable default graphl db to elasticsearch * lint * Set system-update job to enabled by global flag * Adding support for non-default datahub upgrade history topic * Datahub upgrade topic (acryldata#256) * v0.9.6.1 release * fix(release version): fix missing global release bump to 0.9.6.1 * bump versions * Update for search improvements and generic system update job * enable default graphl db to elasticsearch * lint * Set system-update job to enabled by global flag * Adding support for non-default datahub upgrade history topic * feat(gms): Add hazelcast distributed caching when replica count > 1.0 (acryldata#240) Co-authored-by: david-leifker <114954101+david-leifker@users.noreply.github.com> * feat(neo4j-password): be able to deploy without neo4j secret (acryldata#248) * be able to deploy without neo4j secret --------- Co-authored-by: david-leifker <114954101+david-leifker@users.noreply.github.com> * feat(gas): remove release name from service's name (acryldata#252) * service name without release name --------- Co-authored-by: david-leifker <114954101+david-leifker@users.noreply.github.com> * Bump versions v0.10.0 (acryldata#257) * Release v0.10.0 * feat: allow extra labels on deployments (acryldata#258) * feat: allow extra labels on deployments * feat: add examples for extraLabels * feat(elasticsearch): allow plain text password (acryldata#264) * feat(elasticsearch): allow plain text password * chart version bumps * fix strange line move * fix(mce): remove duplicated serviceAccount section (acryldata#260) * fix(mce): remove duplicated serviceAccount section Fixes: acryldata#259 * Update Chart.yaml * Update Chart.yaml * Add input validation for managed ingestion (acryldata#268) * Fix(managed ingestion): Add checks for MCE processor replica count with managed ingestion * Bump chart.yaml version * Add flag to disable helm chart validations * fix: remove duplicated EBEAN env vars on datahub-upgrade jobs (acryldata#269) Fixes: [acryldata#261](acryldata#261) Co-authored-by: Justas Cernas <JustasCe@users.noreply.github.com> * Update README.md (acryldata#271) * fix: add missing service account in acryl-datahub-actions (acryldata#272) * Emit mcl config (acryldata#274) * v0.9.6.1 release * fix(release version): fix missing global release bump to 0.9.6.1 * bump versions * config(mcl): add mcl emission flag * lint * Emit mcl config (acryldata#275) * fix(mcl): fix mcl emit for default true * feat(flags): adding various configuration options (acryldata#277) * Enabling new environment configuration * Diff mode flag (acryldata#278) * Fix options for true defaults for lineage/search cache * fix: remove duplicated EBEAN env vars on datahub-upgrade jobs (acryldata#269) (acryldata#270) Signed-off-by: Tomáš Novák <tomas.novak@bcas.cz> Co-authored-by: david-leifker <114954101+david-leifker@users.noreply.github.com> * Release Charts for DataHub v0.10.1 (acryldata#288) * fix(secrets): Make secret creation optional Makes datahub-auth-secrets creation dependant on whether metadata service authentication has been enabled or not. * Bump chart version & default datahub version * fix: secretkey for systemClientSecret in values.yaml file (acryldata#280) * fix: systemClientSecret key in values.yaml Updated systemClientSecret with proper secretKey from the datahub-auth-secrets file * fix: Updated Chart version --------- Co-authored-by: Upendra Vedullapalli <upendra.rao.vedullapalli@entur.org> * feat(PFP-102): Make Elasticsearch use 1 master node by default (acryldata#283) Our dependencies chart is setup to use 3 ES nodes in 3 AZ-equivalents, but is broadly intended for use as a test harness which often runs in single-node events. So let's fix this and have it be single-node by default * feat: change k8s secret mount default mode to accept value override (acryldata#289) * update deployment yaml file * bump chart version * add missing chart version update * address comments to add descriptions in README and values.yaml * address comment to remove default value of defaultMode * Update helm charts for datahub 0.10.2 release (acryldata#298) Co-authored-by: Indy Prentice <indy@Indys-MacBook-Pro.local> * feat: enable sidecar configuration (acryldata#276) * feat: enable sidecar configuration * feat: values for PSQL db name, elastic prefix (acryldata#313) * Values for PSQL database name, elastic prefix --------- Co-authored-by: david-leifker <114954101+david-leifker@users.noreply.github.com> * feat(search,schema-registry): updates for v0.10.3 release (acryldata#311) * Update kafka chart to 22.1.3 for kafka 3.4.0 (acryldata#316) * feat: add parameters to cleanupJob resources settings (acryldata#317) * feat: Add ability to specify extraPodLabels per deployment (acryldata#310) * feat: Add ability to specify extraPodLabels per deployment * Update Chart.yaml --------- Co-authored-by: jorrick <jorrick.sleijster@adyen.com> Co-authored-by: david-leifker <114954101+david-leifker@users.noreply.github.com> * fix: use common labels for (Cron)Jobs (acryldata#303) * fix(datahub): use common labels for (Cron)Jobs * chore: update version --------- Co-authored-by: Matthijs van der Loos <matthijs.vanderloos@utopiamusic.com> Co-authored-by: david-leifker <114954101+david-leifker@users.noreply.github.com> * fix: add missing global values to subchart values (acryldata#302) * fix(datahub): add missing global values to subchart values * chore: update versions --------- Co-authored-by: Matthijs van der Loos <matthijs.vanderloos@utopiamusic.com> Co-authored-by: david-leifker <114954101+david-leifker@users.noreply.github.com> * feat: allow pulling ebean username from secrets alongside password (acryldata#291) * chore(secrets): use configurable refs instead of fixed names (acryldata#323) * chore(secrets): use configurable refs instead of fixed names * Update Chart.yaml --------- Co-authored-by: david-leifker <114954101+david-leifker@users.noreply.github.com> * Update Default version to v0.10.4 (acryldata#330) * fix: Fixed indentation in datahub-cleanup-job-template.yml (acryldata#328) * feat(healthcheck): use new healthcheck endpoint for GMS (acryldata#331) Co-authored-by: Indy Prentice <indy@Indys-MacBook-Pro.local> * chore(version): version bump & indent (acryldata#324) * feat: Default User Credentials (acryldata#321) Co-authored-by: david-leifker <114954101+david-leifker@users.noreply.github.com> * feat(cloud-sql-proxy): add support for running gcloud sql proxy as prerequisite (acryldata#332) * mce-consumer subcharts typo fix * datahubUpgradeBuildIndices enabled false * kibana 7.17.3 upgrade * finalize * elastic replicas 3 * lint fix * docker build multi platform supported * upload script fix --------- Signed-off-by: Tomáš Novák <tomas.novak@bcas.cz> Co-authored-by: Gabe Lyons <itsgabelyons@gmail.com> Co-authored-by: Lukáš Novotný <42157644+novotl@users.noreply.github.com> Co-authored-by: Aseem Bansal <asmbansal2@gmail.com> Co-authored-by: RyanHolstien <RyanHolstien@users.noreply.github.com> Co-authored-by: Justin Marozas <justinas.marozas@gmail.com> Co-authored-by: Tomáš Kubín <tomas.kubin@blindspot.ai> Co-authored-by: Toby Irmer <toby@irmer.com> Co-authored-by: Pedro Silva <pedro.cls93@gmail.com> Co-authored-by: Pedro Silva <pedro@acryl.io> Co-authored-by: Hunter Elbourn <hunterelbourn@gmail.com> Co-authored-by: Peter Szalai <szalaipeti.vagyok@gmail.com> Co-authored-by: John Joyce <john@acryl.io> Co-authored-by: Upendra Rao Vedullapalli <upendrao@gmail.com> Co-authored-by: Felix Lüdin <13187726+Masterchen09@users.noreply.github.com> Co-authored-by: Bumsoo Kim <bskim45@gmail.com> Co-authored-by: Tony Ouyang <43738225+TonyOuyangGit@users.noreply.github.com> Co-authored-by: Harshal Sheth <hsheth2@gmail.com> Co-authored-by: Jinlin Yang <86577891+jinlintt@users.noreply.github.com> Co-authored-by: Navin Sharma <103643430+NavinSharma13@users.noreply.github.com> Co-authored-by: Bogdan Antoniu <bogdan.antoniu@gmail.com> Co-authored-by: Callum Richard Edwards <34392823+CallumREdwards@users.noreply.github.com> Co-authored-by: Álvaro González <alvaro.gonzalez@auctane.com> Co-authored-by: David Haglund <548769+daha@users.noreply.github.com> Co-authored-by: Andrea Scarpino <andrea@scarpino.dev> Co-authored-by: Nurlan Farajov <nur.rico@gmail.com> Co-authored-by: david-leifker <114954101+david-leifker@users.noreply.github.com> Co-authored-by: Dexter Lee <dexter@acryl.io> Co-authored-by: Adam Price <adam.price@afreshtechnologies.com> Co-authored-by: Chris Collins <chriscollins3456@gmail.com> Co-authored-by: Mike <90835468+justmike1@users.noreply.github.com> Co-authored-by: Matthijs van der Loos <33864538+matthijsvanderloos@users.noreply.github.com> Co-authored-by: Matt Matravers <mattmatravers@hotmail.com> Co-authored-by: Maiara Reinaldo <72740386+maiarareinaldo@users.noreply.github.com> Co-authored-by: Justas Cernas <JustasCe@users.noreply.github.com> Co-authored-by: Tomáš Novák <tom.nov96@gmail.com> Co-authored-by: Upendra Vedullapalli <upendra.rao.vedullapalli@entur.org> Co-authored-by: Kevin Meyer <meyerkev@users.noreply.github.com> Co-authored-by: Indy Prentice <iprentic@users.noreply.github.com> Co-authored-by: Indy Prentice <indy@Indys-MacBook-Pro.local> Co-authored-by: KonstantinVishnivetskii <120646195+KonstantinVishnivetskii@users.noreply.github.com> Co-authored-by: Jorrick Sleijster <jorricks3@gmail.com> Co-authored-by: jorrick <jorrick.sleijster@adyen.com> Co-authored-by: Matthijs van der Loos <matthijs.vanderloos@utopiamusic.com> Co-authored-by: Max Pospischil <maxpospischil@gmail.com> Co-authored-by: Sergio Gómez Villamor <sgomezvillamor@gmail.com> Co-authored-by: TusharM <tushar.madhukar@gmail.com> Co-authored-by: seokyun.ha <127274415+Seokyun-Ha@users.noreply.github.com>
- Loading branch information
48 people
authored
Jul 6, 2023
1 parent
1cc931b
commit 3421dfd