Skip to content

[SPARK-4764] Ensure that files are fetched atomically #2855

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
preaudc wants to merge 5 commits into from
Closed

[SPARK-4764] Ensure that files are fetched atomically #2855

Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
8ea871f
Ensure that files are fetched atomically
Oct 20, 2014
7456a33
Merge remote-tracking branch 'upstream/master'
Oct 23, 2014
c6a5590
Revert commit 8ea871f8130b2490f1bad7374a819bf56f0ccbbd
Nov 3, 2014
54419ae
Merge remote-tracking branch 'upstream/master'
Nov 3, 2014
9ba89ca
Ensure that files are fetched atomically
Nov 3, 2014
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 1 addition & 2 deletions core/src/main/scala/org/apache/spark/util/Utils.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -425,8 +425,7 @@ private[spark] object Utils extends Logging {
conf: SparkConf,
securityMgr: SecurityManager,
hadoopConf: Configuration) {
val tempDir = getLocalDir(conf)
val tempFile = File.createTempFile("fetchFileTemp", null, new File(tempDir))
val tempFile = File.createTempFile("fetchFileTemp", null, new File(targetDir.getAbsolutePath))
Copy link
Member

@zsxwing zsxwing Dec 31, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why need to create a new File here: new File(targetDir.getAbsolutePath)? Can't we use targetDir directly?

Copy link
Contributor

@JoshRosen JoshRosen Jan 4, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, we probably can just use targetDir directly; I guess I overlooked this when reviewing the patch.

val targetFile = new File(targetDir, filename)
val uri = new URI(url)
val fileOverwrite = conf.getBoolean("spark.files.overwrite", defaultValue = false)
Expand Down