-
Notifications
You must be signed in to change notification settings - Fork 93
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
get App Name using spark.app.name
#653
Comments
upvoted |
wajda
changed the title
get App Name using spark.app.name
get App Name using Apr 18, 2023
spark.app.name
This was referenced May 1, 2023
Closed
wajda
added
enhancement
dependency: Databricks
and removed
bug
Something isn't working
labels
May 11, 2023
thanks for contribution @uday1409 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
In databricks, a new SparkContext cannot be created. The spark application name extracted in harvester is done using sparkcontext (
ctx.session.sparkContext.appName
). With this approach, in Databricks, its always "Databricks Shell". This is limiting us to extract and persist the lineage if two different plans are extracted for same destination( file/table). We are using output of spline agent and have written a parser on top of it to store the lineage.Instead of relying on
ctx.session.sparkContext.appName
, can you please make a change to extract Application Name usingspark.conf.get("spark.app.name")
which should work both for OpenSource as well as databricks spark , considering sparkContext is also bit of legacy the way properties are set now ?Is there any other way to extract valid spark app Name or add additional config to the plan with codeless init if above proposed method does not work?
The text was updated successfully, but these errors were encountered: