Skip to content

[SPARK-3985] [Examples] fix file path using os.path.join #2834

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
adrian-wang wants to merge 1 commit into from
Closed

[SPARK-3985] [Examples] fix file path using os.path.join #2834

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion examples/src/main/python/sql.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@

# A JSON dataset is pointed to by path.
# The path can be either a single text file or a directory storing text files.
path = os.environ['SPARK_HOME'] + "examples/src/main/resources/people.json"
path = os.path.join(os.environ['SPARK_HOME'], "examples/src/main/resources/people.json")
Copy link
Member

@srowen srowen Oct 17, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it maybe simpler to just add a / ? This makes it look like the path is platform independent but it isn't as the following path always has /

Copy link
Contributor Author

@adrian-wang adrian-wang Oct 17, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was just trying to make this consistent with other code in python/pyspark/**.py. It is OK if you believe add a '/' is better.

Copy link
Member

@srowen srowen Oct 17, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consistency is good too. Leave it unless someone else thinks it should change.

Copy link
Contributor

@JoshRosen JoshRosen Oct 17, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the os.path.join is fine; I think it's more clear / less brittle. If it turns out that we do need to make changes to the path separator in order to support Windows, then using os.path.join makes it easier to spot where path construction is taking place in order to find the code that needs to be changed.

# Create a SchemaRDD from the file(s) pointed to by path
people = sqlContext.jsonFile(path)
# root
Expand Down