[BUG] Databricks job launcher bugs when adding maven dependencies

[loomlike]

Willingness to contribute

No. I cannot contribute a bug fix at this time.

Feathr version

0.10.4-rc1

System information

• OS Platform and Distribution (e.g., Linux Ubuntu 20.0):
• Python version: 3.8
• Spark version, if reporting runtime issue:

Describe the problem

in Databricks job launcher class, adding library dependencies to databricks job api config via index shouldn't be used:

submission_params['libraries'][0]['maven']
...
submission_params['libraries'][1]['maven']
...
submission_params["libraries"][0]["jar"]

since this will break the job api config.

E.g.
in our feature embedding example, we add pypi dependency of transformer package as follows:

"libraries": [
        {"jar": "FEATHR_FILL_IN"},
        # sentence-transformers pip package
        {"pypi": {"package": "sentence-transformers"}},
    ],

The new codes that are adding maven dependencies to the config will modify the config to be:

[
    {
        'jar': 'FEATHR_FILL_IN',
        'maven': {'coordinates': 'com.linkedin.feathr:feathr_2.12:0.10.4-rc1'}
    },
    {
        'pypi': {'package': 'sentence-transformers'},
        'maven': {'coordinates': 'com.github.everit-org.json-schema:org.everit.json.schema:1.9.1', 'repo': 'https://repository.mulesoft.org/nexus/content/repositories/public/'}
    }
]

which has a broken format and will cause unexpected behavior when creating the databricks job.

also, submission_params["libraries"][0]["jar"] is not safe to use. If the first dependency library value is not ['jar']: FEATHR_FILL, but something else like pypi or any custom maven dependency users may set, then "libraries"][0]["jar"] this logic will break the format too.

Tracking information

No response

Code to reproduce bug

No response

What component(s) does this bug affect?

• Python Client: This is the client users use to interact with most of our API. Mostly written in Python.
• Computation Engine: The computation engine that execute the actual feature join and generation work. Mostly in Scala and Spark.
• Feature Registry API: The frontend API layer supports SQL, Purview(Atlas) as storage. The API layer is in Python(FAST API)
• Feature Registry Web UI: The Web UI for feature registry. Written in React

[blrchen]

Specifiy 'maven': {'coordinates': 'com.github.everit-org.json-schema:org.everit.json.schema:1.9.1', 'repo': 'https://repository.mulesoft.org/nexus/content/repositories/public/'} is no longer needed on notebook side if user is using maven jar with verson greater than v0.10.4-rc5. everit.json.schema:1.9.1 is not available on maven and current cloud sparks can not pull packages from non maven central repos. Thus PR #1043 removes everit.json.schema:1.9.1' deps.

By default, python client uses same version of maven jar, so to get the fix for this issue, either

1. Wait till main branch pyhton client version is bumped to v0.10.4-rc5, this should happen soon if no new regressions are found on v0.10.4-rc5
2. Add a cell to force python client use a newer version maven jar os.environ['MAVEN_ARTIFACT_VERSION'] = "0.10.4-rc5" in notebook.