[BUG] Spark SQL Location Source e2e cannot pass

[Yuqing-cat]

Willingness to contribute

Yes. I can contribute a fix for this bug independently.

Feathr version

latest0.10.4-rc1

System information

• OS Platform and Distribution (e.g., Linux Ubuntu 20.0):
• Python version:
• Spark version, if reporting runtime issue:

Describe the problem

This test works in 0.10.4-rc1 but failed in recent main.

Must be some regression in recent PRs.

Tracking information

Based on investigation, it's caused by Spark SQL location treated as Simple Path.

Code to reproduce bug

No response

What component(s) does this bug affect?

• Python Client: This is the client users use to interact with most of our API. Mostly written in Python.
• Computation Engine: The computation engine that execute the actual feature join and generation work. Mostly in Scala and Spark.
• Feature Registry API: The frontend API layer supports SQL, Purview(Atlas) as storage. The API layer is in Python(FAST API)
• Feature Registry Web UI: The Web UI for feature registry. Written in React

[Yuqing-cat]

Source in Config file confirmed is correct:

[Yuqing-cat]

The real error log is hidden by retry message. It's not user friendly.

23/02/07 04:18:06 INFO BatchDataLoaderFactory: Creating spark data loader for path: SELECT * FROM green_tripdata_2020_04_with_index
23/02/07 04:18:06 INFO BatchDataLoader: Loading SimplePath(path=SELECT * FROM green_tripdata_2020_04_with_index) as DataFrame, using parameters Map(split.size -> )
23/02/07 04:18:06 WARN FileStreamSink: Assume no metadata directory. Error while looking for metadata directory in the path: SELECT * FROM green_tripdata_2020_04_with_index.
java.lang.IllegalArgumentException: Path must be absolute: SELECT * FROM green_tripdata_2020_04_with_index