[SPARK-17159] [streaming]: optimise check for new files in FileInputDStream by steveloughran · Pull Request #14731 · apache/spark

SPARK-17159: move filtering of directories and files out of glob/list filters and into filtering of the FileStatus instances returned in the results, so avoiding the need to create FileStatus intan…

steveloughran

committed

[SPARK-17159] Remove the fileModTime cache. Now that the modification time costs 0 to evaluate, caching it actually consumes memory and the time for a lookup.

steveloughran

committed

[SPARK-17159] inline FileStatus.getModificationTime; address style issues. Also note that 1s granularity is the resolution from HDFS; other filesystems may have a different resolution. The only one…

steveloughran

committed

[SPARK-17159] updates as discussed on PR: skip wildcards for non wildcarded listing; handle FNFE specially, add the docs

steveloughran

committed

[SPARK-17159] move glob operation into SparkHadoopUtils, alongside an existing/similar method. Add tests for the behaviour. Update docs with suggested fixes, and review/edit.

steveloughran

committed

[SPARK-17159] use a simpler pattern in the example docs

steveloughran

committed

[SPARK-17159] add directory rename test (taken from SPARK-7481 examples and made more robust)

steveloughran

committed

[SPARK-17159] method nested inside a sparktest test closure being mistaken for a public method and so needing to declare a return type.

steveloughran

committed

[SPARK-17159] method -> globToFileStatusIfNecessary

steveloughran

committed

[SPARK-17159] File input dstream: revert to directory list operation which doesn't shortcut on a non-wildcard operation

steveloughran

committed

[SPARK-17159] round out the file streaming text with the dirty details of how HDFS doesn't update file length or modtime until close or a block boundary is reached.

steveloughran

committed

SPARK-17159 Chris Nauroth of HDFS team clarified which operations update the mtime field; this is covered in the streaming section to emphasise why write + rename is the strategy for streaming in f…

steveloughran

committed

[SPARK-17159] rebase to master; verify new test still works; review & tighten documentation

steveloughran

committed

[SPARK-17159] review/tighten docs, move HDFS timestamp details out

steveloughran

committed

[SPARK-17159] ; address comments, move to withTempDir for tests with a temp dur. Docs now refer reader to the Hadoop FS spec for any details about what object stores do

steveloughran

committed

SPARK-17159 tail end of the rebase

steveloughran

committed

SPARK-17159 review of docs: quote paths to clearly show what is code and what is just punctuation

steveloughran

committed

SPARK-17159 address sean's review comments, and read over the object store text and update slightly to make things a bit clearer. The more I learn about object stores, the less they resemble file s…

steveloughran

committed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[SPARK-17159] [streaming]: optimise check for new files in FileInputDStream#14731

[SPARK-17159] [streaming]: optimise check for new files in FileInputDStream#14731
steveloughran wants to merge 18 commits intoapache:masterfrom
steveloughran:cloud/SPARK-17159-listfiles

Commits on Mar 21, 2017

SPARK-17159: move filtering of directories and files out of glob/list filters and into filtering of the FileStatus instances returned in the results, so avoiding the need to create FileStatus intan…

[SPARK-17159] Remove the fileModTime cache. Now that the modification time costs 0 to evaluate, caching it actually consumes memory and the time for a lookup.

[SPARK-17159] inline FileStatus.getModificationTime; address style issues. Also note that 1s granularity is the resolution from HDFS; other filesystems may have a different resolution. The only one…

[SPARK-17159] updates as discussed on PR: skip wildcards for non wildcarded listing; handle FNFE specially, add the docs

[SPARK-17159] move glob operation into SparkHadoopUtils, alongside an existing/similar method. Add tests for the behaviour. Update docs with suggested fixes, and review/edit.

[SPARK-17159] use a simpler pattern in the example docs

[SPARK-17159] add directory rename test (taken from SPARK-7481 examples and made more robust)

[SPARK-17159] method nested inside a sparktest test closure being mistaken for a public method and so needing to declare a return type.

[SPARK-17159] method -> globToFileStatusIfNecessary

[SPARK-17159] File input dstream: revert to directory list operation which doesn't shortcut on a non-wildcard operation

[SPARK-17159] round out the file streaming text with the dirty details of how HDFS doesn't update file length or modtime until close or a block boundary is reached.

SPARK-17159 Chris Nauroth of HDFS team clarified which operations update the mtime field; this is covered in the streaming section to emphasise why write + rename is the strategy for streaming in f…

[SPARK-17159] rebase to master; verify new test still works; review & tighten documentation

[SPARK-17159] review/tighten docs, move HDFS timestamp details out

[SPARK-17159] ; address comments, move to withTempDir for tests with a temp dur. Docs now refer reader to the Hadoop FS spec for any details about what object stores do

SPARK-17159 tail end of the rebase

SPARK-17159 review of docs: quote paths to clearly show what is code and what is just punctuation

SPARK-17159 address sean's review comments, and read over the object store text and update slightly to make things a bit clearer. The more I learn about object stores, the less they resemble file s…

[SPARK-17159] [streaming]: optimise check for new files in FileInputDStream#14731steveloughran wants to merge 18 commits intoapache:masterapache/spark:masterfrom steveloughran:cloud/SPARK-17159-listfilessteveloughran/spark:cloud/SPARK-17159-listfilesCopy head branch name to clipboard

Commits

Commits on Mar 21, 2017

[SPARK-17159] [streaming]: optimise check for new files in FileInputDStream#14731
steveloughran wants to merge 18 commits intoapache:masterfrom
steveloughran:cloud/SPARK-17159-listfiles