fix O(n^2) perf bug for std::io::fs::walk_dir #13720
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
We don't currently have a pressing reason to define a particular ordering of yielded directories from this function, so I'm ok leaving is unspecified. If we want to be conservative, perhaps the documentation could indicate that the order of traversal is subject to change? (not specified) |
The `walk_dir` iterator was simulating a queue using a vector (in particular, using `shift`), leading to O(n^2) performance. Since the order was not well-specified (see issue rust-lang#13411), the simplest fix is to use the vector as a stack (and thus yield a depth-first traversal). This patch does exactly that. It leaves the order as originally specified -- "some top-down order" -- and adds a test to ensure a top-down traversal. Note that the underlying `readdir` function does not specify any particular order, nor does the system call it uses. Closes rust-lang#13411.
|
@alexcrichton Updated the patch to revert the spec comment, and changed the test to only ensure some top-down order, rather than depth-first. r? |
bors
added a commit
that referenced
this pull request
Apr 24, 2014
The `walk_dir` iterator was simulating a queue using a vector (in particular, using `shift`), leading to O(n^2) performance. Since the order was not well-specified (see issue #13411), the simplest fix is to use the vector as a stack (and thus yield a depth-first traversal). This patch does exactly that, and adds a test checking for depth-first behavior. Note that the underlying `readdir` function does not specify any particular order, nor does the system call it uses. Closes #13411.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The
walk_diriterator was simulating a queue using a vector (in particular, usingshift),leading to O(n^2) performance. Since the order was not well-specified (see issue #13411),
the simplest fix is to use the vector as a stack (and thus yield a depth-first traversal).
This patch does exactly that, and adds a test checking for depth-first behavior.
Note that the underlying
readdirfunction does not specify any particular order, nordoes the system call it uses.
Closes #13411.