Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(elasticsearch): allow plain text password #264

Merged
merged 3 commits into from
Feb 13, 2023

Conversation

mattmatravers
Copy link
Contributor

@mattmatravers mattmatravers commented Feb 12, 2023

Fixes #262. Allows users to put passwords in the following format:

  elasticsearch:
    auth:
      username: abc123
      password:
        value: es-plaintextpassword

Checklist

  • The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
  • Links to related issues (if applicable)
  • Tests for the changes have been added/updated (if applicable)
  • Docs related to the changes have been added/updated (if applicable)

Copy link
Contributor

@szalai1 szalai1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change looks good and makes sense.

Not necessary requests:

  • add a commented out value to the main values.yaml with some comment, so it can be found by other
  • currently if both secret and plaintext password is given the plain text is used. default to secret one. (this just an opinion).

thanks for the contribution.

@mattmatravers
Copy link
Contributor Author

This change looks good and makes sense.

Not necessary requests:

  • add a commented out value to the main values.yaml with some comment, so it can be found by other
  • currently if both secret and plaintext password is given the plain text is used. default to secret one. (this just an opinion).

thanks for the contribution.

Thanks Peter!

I agree on both parts but I tried to follow the existing style. For other auth parameters the "value" takes precedence over the secrets. The values.yaml file doesn't currently contain a reference to the secrets either, so I left it as is. If you'd like me to add it, I can.

@szalai1 szalai1 merged commit 0361995 into acryldata:master Feb 13, 2023
sb-sebkim added a commit to sendbird/datahub-helm that referenced this pull request Jul 6, 2023
* upgrading to 8.40 (acryldata#139)

* Option to add podAnnotations to jobs. (acryldata#137)

* feat(helm): add pod annotations to jobs

* feat(helm): add pod annotation to cron jobs.

* docs(helm): Add podAnnotation documentation and default values.

* feat(helm) chart version bump.

* update to v0.8.41 (acryldata#143)

* fix(mysql): update mysql helm chart version (acryldata#146)

* fix(metrics): fix Prometheus env variable in the MCE consumer deployment (acryldata#148)

* feat(datahub-frontend): Option to configure OIDC auth easily (acryldata#142)

* fix(kafka): update kafka chart version (acryldata#147)

* feat(datahub-frontend): Allow lifecycle hooks (acryldata#141)

* <feat>(datahub-frontend): Allow lifecycle hooks

This PR adds a lifecycle node to the datahub-frontend deployment template.
It can be used to execute lifecycle hooks, e.g. postStart actions, like
replacing the default user.props file.

* Bump chart version to 0.2.86

Co-authored-by: Pedro Silva <pedro.cls93@gmail.com>

* Update to v0.8.43 (acryldata#151)

* feat(upgrade): allow setting batch args (acryldata#155)

* feat(datahub-upgrade): Restore Indices Resources (acryldata#152)

* <feat>(datahub-upgrade): Restore Indices Resources

This PR adds the ability for end-users to specify resource requests and limits
for the datahub upgrade job 'restore indices'. This is needed for situations
where the restore indices job will OOM due to a large backfill of data. I've
seen this happening with our deployment of datahub and am unable to tune
the resources for the restore indices job because the current requests
and limits are hard-coded.

* fix(gms): remove unused JMX sidecar (acryldata#156)

* fix(gms): remove unused JMX sidecar
in favour of the already included JVM agent

* Update Chart.yaml

* Update Chart.yaml

* Bumping to v0.8.44 (acryldata#160)

* feat(ci): close stale issues/prs (acryldata#164)

* feat: add extra labels to ingress configurations (acryldata#163)

* feat(prerequisites): add postgresql to prerequistes Helm chart as an optional dependency (acryldata#121)

* feat(charts): add an option to configure revisionHistoryLimit to subcharts (acryldata#159)

* feat: datahub-frontend add oidc authentication client secret reference (acryldata#165)

* fix(monitoring) Fixes monitoring default values + gms chart version (acryldata#161)

* chore: bump datahub-actions version (acryldata#168)

* Feat(actions): Support mounting a customer defined k8s secrets as files needed in ingestion recipes (acryldata#169)

* Mount a customer defined k8s secret for ssl secrets needed in ingestion recipes

* Bump chart versions

* Address comment

* Update readme

* Address Pedro's comment

* feat(monitoring): enable monitoring on frontend (acryldata#171)

* Add kafka-exporter to expose prometheus metrics for kafka cluster (acryldata#170)

* Add kafka-exporter to expose prometheus metrics for kafka cluster

* Kafka-exporter: Updated values.yaml to support tls/sasl authentication while connecting kafka cluster

* Kafka-exporter: Updated values.yaml to remove default values and correct intendation

* Add kafka-exporter to expose prometheus metrics for kafka cluster

* Kafka-exporter: Updated values.yaml to support tls/sasl authentication while connecting kafka cluster

* Kafka-exporter: Updated values.yaml to remove default values and correct intendation

* Updated chart version

* Bumping actions (acryldata#173)

* feat(elasticsearch-setup): Add support for insecure curl connections (acryldata#166)

* fix versions, clean jmx exporter (acryldata#175)

* bumping helm charts for 0.8.45 (acryldata#178)

* bumping to 0.8.45

* Fixing

* feat(monitoring): Add optional ServiceMonitors to all JVM based services (acryldata#181)

* feat(frontend): add service monitoring

* feat(gms): add service monitoring

* gms: servicemonitor

* feat(mae): enable service monitoring

* update READMEs

* fix service ports

* bump version

* Add missing default value

* Update charts/datahub/Chart.yaml

* delete jmx config and rename port to http

* feat(secrets): Allow users to specify their own secret values at first boot (acryldata#180)

* fix(auth-secret): Make template use correct retrieval method

`datahub-auth-secret` yaml was incorrectly using the `index` template function which is for list variables. For dictionaries as is the case with the `$secret.data` object, the correct method is: https://helm.sh/docs/chart_template_guide/function_list/#get

Tested locally, that this works against a running cluster.

* Updates encryption secret to use get helm function

* feat(secrets): Allow users to specify their own secret values at first boot

* fix(datahub-frontend): Remove unused env var (acryldata#176)

* fix(datahub-frontend): Remove unused env var

* Bump chart version

* Change default versions 0.9.0 (acryldata#184)

* feat(release): automatically bumbp versions for all subcharts (acryldata#185)

* feat(release): automatically bumbp versions for all subcharts

* Remove kafka export as subchart
Kafka exporter is useful for monitoring datahub, but it's not part of it. It should be deploy sepratly, as we do in production

* Add script to bump versions together

* Add tagging to the workflow

* Don't run release on CI commits

* fix(CI): remove duplicate id (acryldata#189)

* fix(ci): us sh instead of bash; increase tag by patch (acryldata#190)

* DataHub Release v0.9.1  (acryldata#191)

* fix(v0.9.1): Fix v0.9.1 release versions (acryldata#192)

* Release v0.9.1 Helm

* Bump chart versions

* Fixing final 210

* release(v0.9.2) Release Helm Charts for DataHub v0.9.2 (acryldata#194)

* Bumping to release Helm charts for v0.9.2

* bump version by running:
 ./.github/scripts/bump-chart-versions.sh 0.2.112

Co-authored-by: szalai1 <szalaipeti.vagyok@gmail.com>

* fix(gha): revert auto bump release changes (acryldata#197)

* fix(datahub-upgrade): Add system auth credentials to datahub upgrade jobs (acryldata#199)

* fix(datahub-upgrade): Add system auth credentials to datahub upgrade jobs

* bump chart.yaml version

* feat(sql): extend sql username/password configuration (acryldata#198)

* Ebean credential configuration flexbility

Allows the EBEAN_DATASOURCE_USERNAME and EBEAN_DATASOURCE_PASSWORD to be specified either in global or in the gms config.
Also, allows a string value to be specified as the password to enable vault secret webhooks.

* Extended postgres creds config

* Extended mysql setup job creds config

* bump chart version

* bump chart version

* show example in values.yaml

* Added explanation to README

* fix typo

* tidy values.yaml examples

* consistent quote

Use the same quoting solution for sql password and usernames across the file

* remove space error

* Bump gms sub-chart

* Update Chart.yaml

* add nil condition to gms deploy

* ad nil conditional to mysql job

* add nil conditional to postgres job

* fix(frontend): increase recommended defaults (acryldata#202)

* docs: add secret key information (acryldata#195)

Added information about the key name in the secret, or it will fail like this:
Error: couldn't find key neo4j-password in Secret default/neo4j-secrets

* fix(doc): correct doc for serviceMonitor.create (acryldata#204)

* Bumping charts to v0.9.3 (acryldata#207)

* Update Chart.yaml (acryldata#208)

* doc Add global.elasticsearch.useSSL to values.yaml (acryldata#209)

* fix: only add DATAHUB_SYSTEM_CLIENT envs when metadata service auth enabled (acryldata#205)

Co-authored-by: Pedro Silva <pedro@acryl.io>

* fix(frontend): Correct lifecycle templating for frontend (acryldata#201)

* Fix lifecycle templating for frontend
Fixes `throws ValidationError(Deployment.spec.template.spec.containers[0].lifecycle): invalid type for io.k8s.api.core.v1.Lifecycle: got "string", expected "map"`

* bump up Chart.yaml version

* set default value for lifecycle

* Bump versions

Co-authored-by: Pedro Silva <pedro@acryl.io>

* fix(ingestion-cron): Make hostAliases work in datahub-ingestion-cron (acryldata#206)

* fix: make hostAliases work in datahub-ingestion-cron
* Bump ingestion cron subchart version

Fixes acryldata#200

Co-authored-by: Pedro Silva <pedro@acryl.io>

* bugfix: datahub-upgrade use datasource password value if set (acryldata#210)

* feat(global-version): add option to globally define versions (acryldata#212)

* Update versions & bitnami index (acryldata#219)

* Update prerequisite versions & bitnami repo

* fix: move default tag to global.datahub.version (acryldata#214)

image.tag in datahub-gms and datahub-fronted shadowed the
global.datahub.version from the top datahub chart, so the resulting
image was head.

* fix: move default tag to global in subcharts (acryldata#220)

* fix(jobs): Bump kafka setup resources + make all jobs resources configurable (acryldata#224)

* Remove release for master

* fix(jobs): Bump kafka setup resources + make all jobs resources configurable

* Add documentation

* Bump chart.yaml

* Lock upgrade job to v0.9.4

Co-authored-by: Dexter Lee <dexter@acryl.io>

* fix: add extra space before comment to make yamllint pass (acryldata#226)

* fix(release): Revert workflow change & change upgrade job to 0.9.3 (acryldata#227)

* feat(mce-consumer): removing dependency on gms for mce consumer (acryldata#216)

* feat(mce-consumer): removing dependency on gms for mce consumer envs

* feat(elasticsearch): adding elasticsearch index job (acryldata#228)

* Feat/elasticsearch optimization ext (acryldata#229)

* feat(elasticsearch): allow document count mismatch override

* Bumping the versions as part of v0.9.5 release (acryldata#230)

* Fix 232 (acryldata#233)

* fix(cron-ingestion): referecing $ values

* bump versions

* feat(elasticsearch):  build-indices job (acryldata#231)

* feat(elasticsearch): adding elasticsearch index job
* feat(elasticsearch): allow document count mismatch override
* Adjusting hook order build-indices job

* config(gms): enable elasticsearch reindex by default (acryldata#235)

* fix(jobs): Make job pull policy consistent with subchart (acryldata#237)

* fix(jobs): Make job pull policy consistent with subchart

* Explicitly set schema registry type

* Update charts/datahub/values.yaml

Co-authored-by: Peter Szalai  <szalaipeti.vagyok@gmail.com>

Co-authored-by: Peter Szalai  <szalaipeti.vagyok@gmail.com>

* fix(mce-consumer): Correct env vars & configure resources for all components (acryldata#238)

* fix(mce-consumer): Correct env vars & configure resources for all components

* Update quickstart values to use global datahub version

* Update elasticsearch memory configs

* fix(version) Update remaining values for 0.9.6 upgrade  (acryldata#239)

* v0.9.6.1 release (acryldata#241)

* Release bump fix (acryldata#243)

* fix(release version): fix missing global release bump to 0.9.6.1

* Update values.yaml (acryldata#244)

* Update values.yaml

* Update Chart.yaml

* fix(restore-indices): restore indices should use global datastore values (acryldata#247)

* expose oidc scope (acryldata#249)

* Update Chart.yaml (acryldata#251)

* fix(kafka-setup): Make kafka setup use explicit topic names (acryldata#245)

* fix(kafka-setup): Make kafka setup use explicit topic names

* Set kafka env vars for all deployments

* Update gms & actions pod env vars

* Add documentation for new properties

* Version bump (acryldata#250)

* bump versions

* Search improvements updates (acryldata#254)

* Set system-update job to enabled by global flag

* Datahub upgrade topic (acryldata#255)

* v0.9.6.1 release

* fix(release version): fix missing global release bump to 0.9.6.1

* bump versions

* Update for search improvements and generic system update job

* enable default graphl db to elasticsearch

* lint

* Set system-update job to enabled by global flag

* Adding support for non-default datahub upgrade history topic

* Datahub upgrade topic (acryldata#256)

* v0.9.6.1 release

* fix(release version): fix missing global release bump to 0.9.6.1

* bump versions

* Update for search improvements and generic system update job

* enable default graphl db to elasticsearch

* lint

* Set system-update job to enabled by global flag

* Adding support for non-default datahub upgrade history topic

* feat(gms): Add hazelcast distributed caching when replica count > 1.0 (acryldata#240)

Co-authored-by: david-leifker <114954101+david-leifker@users.noreply.github.com>

* feat(neo4j-password): be able to deploy without neo4j secret (acryldata#248)

* be able to deploy without neo4j secret

---------

Co-authored-by: david-leifker <114954101+david-leifker@users.noreply.github.com>

* feat(gas): remove release name from service's name (acryldata#252)

* service name without release name

---------

Co-authored-by: david-leifker <114954101+david-leifker@users.noreply.github.com>

* Bump versions v0.10.0 (acryldata#257)

* Release v0.10.0

* feat: allow extra labels on deployments (acryldata#258)

* feat: allow extra labels on deployments

* feat: add examples for extraLabels

* feat(elasticsearch): allow plain text password (acryldata#264)

* feat(elasticsearch): allow plain text password

* chart version bumps

* fix strange line move

* fix(mce): remove duplicated serviceAccount section (acryldata#260)

* fix(mce): remove duplicated serviceAccount section

Fixes: acryldata#259

* Update Chart.yaml

* Update Chart.yaml

* Add input validation for managed ingestion (acryldata#268)

* Fix(managed ingestion): Add checks for MCE processor replica count with managed ingestion

* Bump chart.yaml version

* Add flag to disable helm chart validations

* fix: remove duplicated EBEAN env vars on datahub-upgrade jobs (acryldata#269)

Fixes: [acryldata#261](acryldata#261)

Co-authored-by: Justas Cernas <JustasCe@users.noreply.github.com>

* Update README.md (acryldata#271)

* fix: add missing service account in acryl-datahub-actions (acryldata#272)

* Emit mcl config (acryldata#274)

* v0.9.6.1 release

* fix(release version): fix missing global release bump to 0.9.6.1

* bump versions

* config(mcl): add mcl emission flag

* lint

* Emit mcl config (acryldata#275)

* fix(mcl): fix mcl emit for default true

* feat(flags): adding various configuration options (acryldata#277)

* Enabling new environment configuration

* Diff mode flag (acryldata#278)

* Fix options for true defaults for lineage/search cache

* fix: remove duplicated EBEAN env vars on datahub-upgrade jobs (acryldata#269) (acryldata#270)

Signed-off-by: Tomáš Novák <tomas.novak@bcas.cz>
Co-authored-by: david-leifker <114954101+david-leifker@users.noreply.github.com>

* Release Charts for DataHub v0.10.1 (acryldata#288)

* fix(secrets): Make secret creation optional

Makes datahub-auth-secrets creation dependant on whether metadata service authentication has been enabled or not.

* Bump chart version & default datahub version

* fix: secretkey for systemClientSecret in values.yaml file (acryldata#280)

* fix: systemClientSecret key in values.yaml

Updated systemClientSecret with proper secretKey from the datahub-auth-secrets file

* fix: Updated Chart version

---------

Co-authored-by: Upendra Vedullapalli <upendra.rao.vedullapalli@entur.org>

* feat(PFP-102): Make Elasticsearch use 1 master node by default (acryldata#283)

Our dependencies chart is setup to use 3 ES nodes in 3 AZ-equivalents, but
is broadly intended for use as a test harness which often runs in single-node events.

So let's fix this and have it be single-node by default

* feat: change k8s secret mount default mode to accept value override (acryldata#289)

* update deployment yaml file

* bump chart version

* add missing chart version update

* address comments to add descriptions in README and values.yaml

* address comment to remove default value of defaultMode

* Update helm charts for datahub 0.10.2 release (acryldata#298)

Co-authored-by: Indy Prentice <indy@Indys-MacBook-Pro.local>

* feat: enable sidecar configuration (acryldata#276)

* feat: enable sidecar configuration

* feat: values for PSQL db name, elastic prefix (acryldata#313)

* Values for PSQL database name, elastic prefix
---------

Co-authored-by: david-leifker <114954101+david-leifker@users.noreply.github.com>

* feat(search,schema-registry): updates for v0.10.3 release (acryldata#311)

* Update kafka chart to 22.1.3 for kafka 3.4.0 (acryldata#316)

* feat: add parameters to cleanupJob resources settings (acryldata#317)

* feat: Add ability to specify extraPodLabels per deployment (acryldata#310)

* feat: Add ability to specify extraPodLabels per deployment

* Update Chart.yaml

---------

Co-authored-by: jorrick <jorrick.sleijster@adyen.com>
Co-authored-by: david-leifker <114954101+david-leifker@users.noreply.github.com>

* fix: use common labels for (Cron)Jobs (acryldata#303)

* fix(datahub): use common labels for (Cron)Jobs

* chore: update version

---------

Co-authored-by: Matthijs van der Loos <matthijs.vanderloos@utopiamusic.com>
Co-authored-by: david-leifker <114954101+david-leifker@users.noreply.github.com>

* fix: add missing global values to subchart values (acryldata#302)

* fix(datahub): add missing global values to subchart values

* chore: update versions

---------

Co-authored-by: Matthijs van der Loos <matthijs.vanderloos@utopiamusic.com>
Co-authored-by: david-leifker <114954101+david-leifker@users.noreply.github.com>

* feat: allow pulling ebean username from secrets alongside password (acryldata#291)

* chore(secrets): use configurable refs instead of fixed names (acryldata#323)

* chore(secrets): use configurable refs instead of fixed names

* Update Chart.yaml

---------

Co-authored-by: david-leifker <114954101+david-leifker@users.noreply.github.com>

* Update Default version to v0.10.4 (acryldata#330)

* fix: Fixed indentation in  datahub-cleanup-job-template.yml (acryldata#328)

* feat(healthcheck): use new healthcheck endpoint for GMS (acryldata#331)

Co-authored-by: Indy Prentice <indy@Indys-MacBook-Pro.local>

* chore(version): version bump & indent (acryldata#324)

* feat: Default User Credentials (acryldata#321)

Co-authored-by: david-leifker <114954101+david-leifker@users.noreply.github.com>

* feat(cloud-sql-proxy): add support for running gcloud sql proxy as prerequisite (acryldata#332)

* mce-consumer subcharts typo fix

* datahubUpgradeBuildIndices enabled false

* kibana 7.17.3 upgrade

* finalize

* elastic replicas 3

* lint fix

* docker build multi platform supported

* upload script fix

---------

Signed-off-by: Tomáš Novák <tomas.novak@bcas.cz>
Co-authored-by: Gabe Lyons <itsgabelyons@gmail.com>
Co-authored-by: Lukáš Novotný <42157644+novotl@users.noreply.github.com>
Co-authored-by: Aseem Bansal <asmbansal2@gmail.com>
Co-authored-by: RyanHolstien <RyanHolstien@users.noreply.github.com>
Co-authored-by: Justin Marozas <justinas.marozas@gmail.com>
Co-authored-by: Tomáš Kubín <tomas.kubin@blindspot.ai>
Co-authored-by: Toby Irmer <toby@irmer.com>
Co-authored-by: Pedro Silva <pedro.cls93@gmail.com>
Co-authored-by: Pedro Silva <pedro@acryl.io>
Co-authored-by: Hunter Elbourn <hunterelbourn@gmail.com>
Co-authored-by: Peter Szalai <szalaipeti.vagyok@gmail.com>
Co-authored-by: John Joyce <john@acryl.io>
Co-authored-by: Upendra Rao Vedullapalli <upendrao@gmail.com>
Co-authored-by: Felix Lüdin <13187726+Masterchen09@users.noreply.github.com>
Co-authored-by: Bumsoo Kim <bskim45@gmail.com>
Co-authored-by: Tony Ouyang <43738225+TonyOuyangGit@users.noreply.github.com>
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
Co-authored-by: Jinlin Yang <86577891+jinlintt@users.noreply.github.com>
Co-authored-by: Navin Sharma <103643430+NavinSharma13@users.noreply.github.com>
Co-authored-by: Bogdan Antoniu <bogdan.antoniu@gmail.com>
Co-authored-by: Callum Richard Edwards <34392823+CallumREdwards@users.noreply.github.com>
Co-authored-by: Álvaro González <alvaro.gonzalez@auctane.com>
Co-authored-by: David Haglund <548769+daha@users.noreply.github.com>
Co-authored-by: Andrea Scarpino <andrea@scarpino.dev>
Co-authored-by: Nurlan Farajov <nur.rico@gmail.com>
Co-authored-by: david-leifker <114954101+david-leifker@users.noreply.github.com>
Co-authored-by: Dexter Lee <dexter@acryl.io>
Co-authored-by: Adam Price <adam.price@afreshtechnologies.com>
Co-authored-by: Chris Collins <chriscollins3456@gmail.com>
Co-authored-by: Mike <90835468+justmike1@users.noreply.github.com>
Co-authored-by: Matthijs van der Loos <33864538+matthijsvanderloos@users.noreply.github.com>
Co-authored-by: Matt Matravers <mattmatravers@hotmail.com>
Co-authored-by: Maiara Reinaldo <72740386+maiarareinaldo@users.noreply.github.com>
Co-authored-by: Justas Cernas <JustasCe@users.noreply.github.com>
Co-authored-by: Tomáš Novák <tom.nov96@gmail.com>
Co-authored-by: Upendra Vedullapalli <upendra.rao.vedullapalli@entur.org>
Co-authored-by: Kevin Meyer <meyerkev@users.noreply.github.com>
Co-authored-by: Indy Prentice <iprentic@users.noreply.github.com>
Co-authored-by: Indy Prentice <indy@Indys-MacBook-Pro.local>
Co-authored-by: KonstantinVishnivetskii <120646195+KonstantinVishnivetskii@users.noreply.github.com>
Co-authored-by: Jorrick Sleijster <jorricks3@gmail.com>
Co-authored-by: jorrick <jorrick.sleijster@adyen.com>
Co-authored-by: Matthijs van der Loos <matthijs.vanderloos@utopiamusic.com>
Co-authored-by: Max Pospischil <maxpospischil@gmail.com>
Co-authored-by: Sergio Gómez Villamor <sgomezvillamor@gmail.com>
Co-authored-by: TusharM <tushar.madhukar@gmail.com>
Co-authored-by: seokyun.ha <127274415+Seokyun-Ha@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Allow ElasticSearch passwords to be plain strings
2 participants