Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

jwalk silently swallows errors on read_dir, no guidance in docs #40

Open
untitaker opened this issue Jun 10, 2023 · 0 comments
Open

jwalk silently swallows errors on read_dir, no guidance in docs #40

untitaker opened this issue Jun 10, 2023 · 0 comments

Comments

@untitaker
Copy link

The du example swallows all errors when reading a directory, and it shows under massive parallelism as jwalk tends to run out of filedescriptors. Steps to reproduce:

  1. edit the example to have parallelism 500 instead of 4
  2. set ulimit -n 5 -- this step is sometimes not required, but may be required on systems with really large fd limit.
  3. launch cargo run --example du on a large folder repeatedly and observe indeterministic bytecount without any printed errors

Here is a fix for that:

diff --git a/examples/du.rs b/examples/du.rs
index 6e51703..b62dbe8 100644
--- a/examples/du.rs
+++ b/examples/du.rs
@@ -23,6 +23,10 @@ fn main() {
     {
         match dir_entry_result {
             Ok(dir_entry) => {
+                if let Some(ref err) = dir_entry.read_children_error {
+                    panic!("error: {}", err);
+                }
+
                 if let Some(len) = &dir_entry.client_state {
                     eprintln!("counting {:?}", dir_entry.path());
                     total += len;

This is a duplicate of #4 -- I don't think this is good API at all, there's a user in that issue confused by it and jwalk's own examples get its usage wrong.

Additionally, if I try to observe read_children_error in process_read_dir, it is always None regardless of whether jwalk ran into ulimits.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant