Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(cron): Adding more parameters to ingestion-cron #336

Merged
merged 3 commits into from
Jun 29, 2023

Conversation

miguelbirdie
Copy link
Contributor

@miguelbirdie miguelbirdie commented Jun 27, 2023

This PR enabling to users to set up different parameters of the cronjob/job in the ingestion process. In particular, we will be able to configure the following parameters:

  • concurrencyPolicy
  • successfulJobsHistoryLimit
  • failedJobsHistoryLimit
  • backoffLimit
  • restartPolicy

Checklist

  • The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
  • Links to related issues (if applicable)
  • Tests for the changes have been added/updated (if applicable)
  • Docs related to the changes have been added/updated (if applicable)

@benw-at-birdie
Copy link

Super!

@david-leifker
Copy link
Contributor

Thank you!!

@david-leifker
Copy link
Contributor

@miguelbirdie - Just looking to bump the version of the chart here

@miguelbirdie
Copy link
Contributor Author

@miguelbirdie - Just looking to bump the version of the chart here

Thanks @david-leifker! It's already merged 🎉

@david-leifker david-leifker merged commit 3a454ee into acryldata:master Jun 29, 2023
dim-ops pushed a commit to dim-ops/datahub-helm that referenced this pull request Jul 10, 2023
* feat(cron): Adding more parameters to ingestion-cron
---------

Co-authored-by: david-leifker <114954101+david-leifker@users.noreply.github.com>
ericnoam added a commit to ericnoam/datahub-helm that referenced this pull request Sep 15, 2023
* feat: values for PSQL db name, elastic prefix (acryldata#313)

* Values for PSQL database name, elastic prefix
---------

Co-authored-by: david-leifker <114954101+david-leifker@users.noreply.github.com>

* feat(search,schema-registry): updates for v0.10.3 release (acryldata#311)

* Update kafka chart to 22.1.3 for kafka 3.4.0 (acryldata#316)

* feat: add parameters to cleanupJob resources settings (acryldata#317)

* feat: Add ability to specify extraPodLabels per deployment (acryldata#310)

* feat: Add ability to specify extraPodLabels per deployment

* Update Chart.yaml

---------

Co-authored-by: jorrick <jorrick.sleijster@adyen.com>
Co-authored-by: david-leifker <114954101+david-leifker@users.noreply.github.com>

* fix: use common labels for (Cron)Jobs (acryldata#303)

* fix(datahub): use common labels for (Cron)Jobs

* chore: update version

---------

Co-authored-by: Matthijs van der Loos <matthijs.vanderloos@utopiamusic.com>
Co-authored-by: david-leifker <114954101+david-leifker@users.noreply.github.com>

* fix: add missing global values to subchart values (acryldata#302)

* fix(datahub): add missing global values to subchart values

* chore: update versions

---------

Co-authored-by: Matthijs van der Loos <matthijs.vanderloos@utopiamusic.com>
Co-authored-by: david-leifker <114954101+david-leifker@users.noreply.github.com>

* feat: allow pulling ebean username from secrets alongside password (acryldata#291)

* chore(secrets): use configurable refs instead of fixed names (acryldata#323)

* chore(secrets): use configurable refs instead of fixed names

* Update Chart.yaml

---------

Co-authored-by: david-leifker <114954101+david-leifker@users.noreply.github.com>

* Update Default version to v0.10.4 (acryldata#330)

* fix: Fixed indentation in  datahub-cleanup-job-template.yml (acryldata#328)

* feat(healthcheck): use new healthcheck endpoint for GMS (acryldata#331)

Co-authored-by: Indy Prentice <indy@Indys-MacBook-Pro.local>

* chore(version): version bump & indent (acryldata#324)

* feat: Default User Credentials (acryldata#321)

Co-authored-by: david-leifker <114954101+david-leifker@users.noreply.github.com>

* feat(cloud-sql-proxy): add support for running gcloud sql proxy as prerequisite (acryldata#332)

* feat: allow for overriding job annotations and adding init containers (acryldata#315)

* allow for overriding hook annotations
* allow for specifying init containers on all jobs
---------
Signed-off-by: David van der Spek <vanderspek.david@gmail.com>

* feat(cron): Adding more parameters to ingestion-cron (acryldata#336)

* feat(cron): Adding more parameters to ingestion-cron
---------

Co-authored-by: david-leifker <114954101+david-leifker@users.noreply.github.com>

* Update charts to include search and browse env variable flags (acryldata#337)

* Update charts to include search and browse env variable flags

* fix(config) Set search and browse flags default off (acryldata#339)

* fix(config) Set search and browse flags default off

* feat(cron): support nodeselector, affinity and toleration capabilities  (acryldata#342)

* feat : add tolerations parameter in datahub-ingestion-cron chart

---------

Co-authored-by: david-leifker <114954101+david-leifker@users.noreply.github.com>

* Helm update for 0.10.5 release (acryldata#346)

* Helm update for 0.10.5 release

* use latest point release for ingestion

* docs(readme): document secrets randomization (acryldata#350)

* docs(readme):  Add notes about randomized keys and credentials

* fix(auth-secrets): fix system update secrets (acryldata#351)

* fix(auth-secret): remove auth secret from common template, cannot be used by all jobs

* fix(datahub-system-update-job): enable `SPRING_KAFKA_PROPERTIES_AUTO_REGISTER_SCHEMAS` (acryldata#358)

* fix(datahub-system-update-job): enable SPRING_KAFKA_PROPERTIES_AUTO_REGISTER_SCHEMAS

* feat(session): add session duration configuration (acryldata#361)

* feat(session): add session duration configuration

* fix(ingestion-cron): fix indentation in ingestion cron template (acryldata#356)

* fix: remove unused and irrelevant sidecar configuration
* fix: correct sidecar configuration in cron job template
* fix: update ingestion cron sidecar parameter in README

---------

Co-authored-by: david-leifker <114954101+david-leifker@users.noreply.github.com>

* docs(ingestion-cron): add documentation for ingestion cron job values (acryldata#355)

* Helm changes for 0.11.0 release (acryldata#366)

Co-authored-by: Indy Prentice <indy@Indys-MacBook-Pro.local>

* Release v0.11.0 updates (acryldata#367)

* feat(release): updates for release v0.11.0

---------

Co-authored-by: KonstantinVishnivetskii <120646195+KonstantinVishnivetskii@users.noreply.github.com>
Co-authored-by: david-leifker <114954101+david-leifker@users.noreply.github.com>
Co-authored-by: Jinlin Yang <86577891+jinlintt@users.noreply.github.com>
Co-authored-by: Álvaro González <alvaro.gonzalez@auctane.com>
Co-authored-by: Jorrick Sleijster <jorricks3@gmail.com>
Co-authored-by: jorrick <jorrick.sleijster@adyen.com>
Co-authored-by: Matthijs van der Loos <33864538+matthijsvanderloos@users.noreply.github.com>
Co-authored-by: Matthijs van der Loos <matthijs.vanderloos@utopiamusic.com>
Co-authored-by: Max Pospischil <maxpospischil@gmail.com>
Co-authored-by: Sergio Gómez Villamor <sgomezvillamor@gmail.com>
Co-authored-by: Pedro Silva <pedro@acryl.io>
Co-authored-by: TusharM <tushar.madhukar@gmail.com>
Co-authored-by: Indy Prentice <iprentic@users.noreply.github.com>
Co-authored-by: Indy Prentice <indy@Indys-MacBook-Pro.local>
Co-authored-by: seokyun.ha <127274415+Seokyun-Ha@users.noreply.github.com>
Co-authored-by: Tony Ouyang <43738225+TonyOuyangGit@users.noreply.github.com>
Co-authored-by: David van der Spek <vanderspek.david@gmail.com>
Co-authored-by: miguelbirdie <86949255+miguelbirdie@users.noreply.github.com>
Co-authored-by: Chris Collins <chriscollins3456@gmail.com>
Co-authored-by: sachinsaju <33017477+sachinsaju@users.noreply.github.com>
Co-authored-by: Kohei Watanabe <littlewat.dev@gmail.com>
Co-authored-by: RyanHolstien <RyanHolstien@users.noreply.github.com>
@ericnoam ericnoam mentioned this pull request May 31, 2024
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants