Add support for custom containers while using dataproc serverless

[bveber]

resolves #642

Description

The current implementation only supports the standard spark runtime provided by dataproc serverless which does not allow installation of additional dependencies. The custom container can be defined in the runtime_config that is passed to the dataproc call. It expects an image that exists in Container Registry or Artifact Registry with a naming format like {hostname}/{project-id}/{image}:{tag}.

Checklist

• I have read the contributing guide and understand what's expected of me
• I have signed the CLA
• I have run this code in development and it appears to resolve the stated issue
• This PR includes tests, or tests are not required/relevant for this PR
• I have opened an issue to add/update docs, or docs changes are not required/relevant for this PR
• I have run changie new to create a changelog entry

[ghost]

This would very valuable for us in making dbt python models feasible in our stack. Are there any unmet requirements for this change to be reviewed? Anything I could contribute to help unblock this?

[bveber]

@Fleid @nathaniel-may Just bumping this since there seems to be interest in this enhancement beyond myself

[alyce-cherre]

Very eager for this to come out!!!

It'll help us make the case to productionize more of our ML models using dbt/python instead of using native GCP/AWS products

[thumarrushik]

+1

[fdiaspp]

+1

[DouglasNorrgard]

+1

Very eager. Implementation could be very similar to the new dataproc_batch-config?

[c0nn0rstevens]

Is there any progress on this? This is something my team would really benefit from. What is still needed for this? Can I help somehow?

[github-actions]

This PR has been marked as Stale because it has been open with no activity as of late. If you would like the PR to remain open, please comment on the PR or else it will be closed in 7 days.

[github-actions]

Although we are closing this PR as stale, it can still be reopened to continue development. Just add a comment to notify the maintainers.

[ShahbalKhan]

This feature is quite useful for avoiding managing custom clusters for our pyspark transformations. Would love to have it pushed. Also happy to contribute where required.