mdtest bugs in leaf-only mode

[rmn1]

Hello,
mdtest has serious bugs when run in leaf-only mode (-L option)

• stat phase gives invalid results
• stat phase processes only small/limited amount of files
• wrong number of files is calculated and reported (with -F option) for -I option

Following condition in mdtest.c mdtest_stat function is wrong, so only small amount of files (one directory usually) is tested:

    if( directory_loops != 1 || leaf_only ){
      stop_items = items_per_dir;
    }

But results are calculated using all items.

if (stat_only) {
        summary_table[iteration].rate[5] = items*size/(t[2] - t[1]);
        summary_table[iteration].time[5] = t[2] - t[1];
        summary_table[iteration].items[5] = items*size;
        summary_table[iteration].stonewall_last_item[5] = items;
}

Items calculations doesn't reflect leaf_only when run with -I option.

if (items_per_dir > 0) {
      if(items == 0){
        items = items_per_dir * num_dirs_in_tree;
      }else{
        num_dirs_in_tree_calc = num_dirs_in_tree;
      }
  } else {
      if (leaf_only) {
          if (branch_factor <= 1) {
              items_per_dir = items;
          } else {
              items_per_dir = (uint64_t) (items / pow(branch_factor, depth));
              items = items_per_dir * (uint64_t) pow(branch_factor, depth);
          }
      } else {
          items_per_dir = items / num_dirs_in_tree;
          items = items_per_dir * num_dirs_in_tree;
      }
}

I suggest reverting commit ed421bc and fixing items calculations.

[JulianKunkel]

Thanks for reporting. I confirm the bug appears when using subdirectories. $ /src/mdtest -L -F -b 10 -z 2 -n 100 -C => that leads to one file per directory $ /src/mdtest -L -F -b 10 -z 2 -n 100 -T => stats only one file. Unfortunately, the matter isn't that easy as another bug is existing: see: #112 Indeed, a little project is ongoing to disentangle the iteration of names from the execution which would resolve the matter. I'll keep diging. For the moment, it appears subdirectories cannot be used. Am Di., 14. Mai 2019 um 10:24 Uhr schrieb rmn1 <notifications@github.com>:

…

Hello, mdtest has serious bugs when run in leaf-only mode (-L option) - stat phase gives invalid results - stat phase processes only small/limited amount of files - wrong number of files is calculated and reported (with -F option) for -I option Following condition in mdtest.c mdtest_stat function is wrong, so only small amount of files (one directory usually) is tested: if( directory_loops != 1 || leaf_only ){ stop_items = items_per_dir; } But results are calculated using all items. if (stat_only) { summary_table[iteration].rate[5] = items*size/(t[2] - t[1]); summary_table[iteration].time[5] = t[2] - t[1]; summary_table[iteration].items[5] = items*size; summary_table[iteration].stonewall_last_item[5] = items; } Items calculations doesn't reflect leaf_only when run with -I option. if (items_per_dir > 0) { if(items == 0){ items = items_per_dir * num_dirs_in_tree; }else{ num_dirs_in_tree_calc = num_dirs_in_tree; } } else { if (leaf_only) { if (branch_factor <= 1) { items_per_dir = items; } else { items_per_dir = (uint64_t) (items / pow(branch_factor, depth)); items = items_per_dir * (uint64_t) pow(branch_factor, depth); } } else { items_per_dir = items / num_dirs_in_tree; items = items_per_dir * num_dirs_in_tree; } } I suggest reverting commit ed421bc <ed421bc> and fixing items calculations. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#147?email_source=notifications&email_token=ABGW5SUBLTBPYS74HT2EVNLPVKAOVA5CNFSM4HMX7JUKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4GTT6FNA>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABGW5SUYZ3QDB7HZOVAZ2KLPVKAOVANCNFSM4HMX7JUA> .

-- Dr. Julian Kunkel Lecturer, Department of Computer Science +44 (0) 118 378 8218 http://www.cs.reading.ac.uk/ https://hps.vi4io.org/ PGP Fingerprint: 1468 1A86 A908 D77E B40F 45D6 2B15 73A5 9D39 A28E

[rmn1]

Thank for quick response.
I think that ed421bc solves nothing and just breaks another functionality.
In my opinion #112 is caused by wrong items calculation for case when -I and -L options are used.

Reverting ed421bc and following short patch works for me and for #112.

diff --git a/src/mdtest.c b/src/mdtest.c
index b36dcea..fbfb1d4 100644
--- a/src/mdtest.c
+++ b/src/mdtest.c
@@ -2270,7 +2270,12 @@ mdtest_results_t * mdtest_run(int argc, char **argv, MPI_Comm world_com, FILE *
     }
     if (items_per_dir > 0) {
         if(items == 0){
-          items = items_per_dir * num_dirs_in_tree;
+          if (leaf_only) {
+              items = items_per_dir * (uint64_t) pow(branch_factor, depth);
+          }
+          else {
+              items = items_per_dir * num_dirs_in_tree;
+          }
         }else{
           num_dirs_in_tree_calc = num_dirs_in_tree;
         }

[JulianKunkel]

Great job, that indeed appears to works when debugging using strace. I committed it to master as this is of key importance. Am Di., 14. Mai 2019 um 11:42 Uhr schrieb rmn1 <notifications@github.com>:

…

Thank for quick response. I think that ed421bc <ed421bc> solves nothing and just breaks another functionality. In my opinion #112 <#112> is caused by wrong items calculation for case when -I and -L options are used. Reverting ed421bc <ed421bc> and following short patch works for me and for #112 <#112>. diff --git a/src/mdtest.c b/src/mdtest.c index b36dcea..fbfb1d4 100644 --- a/src/mdtest.c +++ b/src/mdtest.c @@ -2270,7 +2270,12 @@ mdtest_results_t * mdtest_run(int argc, char **argv, MPI_Comm world_com, FILE * } if (items_per_dir > 0) { if(items == 0){ - items = items_per_dir * num_dirs_in_tree; + if (leaf_only) { + items = items_per_dir * (uint64_t) pow(branch_factor, depth); + } + else { + items = items_per_dir * num_dirs_in_tree; + } }else{ num_dirs_in_tree_calc = num_dirs_in_tree; } — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#147?email_source=notifications&email_token=ABGW5SXYACV6YUU7T7CZI23PVKJS5A5CNFSM4HMX7JUKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODVLCQGI#issuecomment-492185625>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABGW5SUTZWH73RSGD7AZ5SLPVKJS5ANCNFSM4HMX7JUA> .

-- Dr. Julian Kunkel Lecturer, Department of Computer Science +44 (0) 118 378 8218 http://www.cs.reading.ac.uk/ https://hps.vi4io.org/ PGP Fingerprint: 1468 1A86 A908 D77E B40F 45D6 2B15 73A5 9D39 A28E

[glennklockwood]

Should we cut a 3.2.1 to address this and the other patch that’s been applied to the 3.2. branch? This seems like something worth releasing.

[JulianKunkel]

Yes, please do this. It indeed is a painful one. Am Di., 14. Mai 2019 um 14:59 Uhr schrieb Glenn K. Lockwood <notifications@github.com>:

…

Should we cut a 3.2.1 to address this and the other patch that’s been applied to the 3.2. branch? This seems like something worth releasing. — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

-- Dr. Julian Kunkel Lecturer, Department of Computer Science +44 (0) 118 378 8218 http://www.cs.reading.ac.uk/ https://hps.vi4io.org/ PGP Fingerprint: 1468 1A86 A908 D77E B40F 45D6 2B15 73A5 9D39 A28E