fix(ingestion/scheduler): add extraArgs support for Ingestion Scheduler (e.g. for extra pip packages)

[Nelvin73]

The scheduled ingestion does not support extraArgs, like extra PIP packages.
Thats why Ingestions with PIP EXTRA PACKAGES or PIP EXTRA ARGS work with manual trigger, but do not work when run by scheduler.

Checklist

• The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
• Links to related issues (if applicable)
• Tests for the changes have been added/updated (if applicable)
• Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
• For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub

[pedro93]

Hello @Nelvin73,

Thank you for raising this PR. I don't see anything inherently wrong with it and I'm inclined to accept the change. Before I do however, could you elaborate on what you trying to achieve with this property?

As per our docs we allow specifying env vars, datahub plugins and extra pip packages all using this field already (it's an implementation detail) in the advanced config of ingestions.

[Nelvin73]

Hello @pedro93

the PR is actually exactly for those mentioned features (extra pip packages, extra pip args, ...) to work in scheduled mode.

When triggered manually (which triggers a GraphQL mutation query) this works perfectly and the venv is taking account of these extra args.
But unfortunately, the same ingestion source, when running in scheduled mode, it will ignore those settings, as the extraArgs are not written into the IngestionRequestInput aspect.

Looks like adding the extraArgs to Scheduler was forgotten somehow.
There are also some comments in Slack about that issue
e.g. https://datahubspace.slack.com/archives/C029A3M079U/p1703169843319679
but I didnt find a ISSUE ticket for it.

I compiled and tested locally and with the fix, the scheduled ingestion was taken care about the extraArgs.

If you have further questinos, let me know.

[Nelvin73]

@hsheth2 @pedro93
Is there something I can clarify or can be done, to get approval?
BR
Christian