Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Failing unit tests on macos #392

Closed
hellkite500 opened this issue Mar 10, 2022 · 4 comments · Fixed by #394
Closed

Failing unit tests on macos #392

hellkite500 opened this issue Mar 10, 2022 · 4 comments · Fixed by #394

Comments

@hellkite500
Copy link
Member

A non-deterministic failure of unit tests has been introduced with #387 and/or possible #389.

Current behavior

The PR checks were successful on #387, but the post merge checks once again failed the macos bmi c tests.

[  FAILED  ] Bmi_C_Formulation_Test.GetOutputLineForTimestep_0_a
Returning unconverted value!
[  FAILED  ] Bmi_C_Formulation_Test.GetOutputLineForTimestep_1_a
Unit conversion error: 
[  FAILED  ] Bmi_C_Formulation_Test.GetOutputLineForTimestep_1_b

With the following errors reported for these tests

[ RUN      ] Bmi_C_Formulation_Test.GetOutputLineForTimestep_0_a
/Users/runner/work/ngen/ngen/test/realizations/catchments/Bmi_C_Formulation_Test.cpp:310: Failure
Value of: output
Expected: matches regular expression "-?0.000000,-?0.000000"
  Actual: "0.000000,-4.000000"
[  FAILED  ] Bmi_C_Formulation_Test.GetOutputLineForTimestep_0_a (65 ms)
[ RUN      ] Bmi_C_Formulation_Test.GetOutputLineForTimestep_1_a
/Users/runner/work/ngen/ngen/test/realizations/catchments/Bmi_C_Formulation_Test.cpp:326: Failure
Value of: output
Expected: matches regular expression "-?0.000000,-?0.000000"
  Actual: "-4.000000,0.000000"
[  FAILED  ] Bmi_C_Formulation_Test.GetOutputLineForTimestep_1_a (57 ms)
[ RUN      ] Bmi_C_Formulation_Test.GetOutputLineForTimestep_1_b
/Users/runner/work/ngen/ngen/test/realizations/catchments/Bmi_C_Formulation_Test.cpp:341: Failure
Value of: output
Expected: matches regular expression "-?0.000000,0.000001"
  Actual: "-4.000000,0.000001"
[  FAILED  ] Bmi_C_Formulation_Test.GetOutputLineForTimestep_1_b (133 ms)

Expected behavior

These three tests should return 0.000000 (or possibly -0.000000, though I'm starting to suspect that the -0.0 is related to the same bug???)

Steps to replicate behavior (include URLs)

  1. Seems non-deterministic, and doesn't fail consistently (local testing on various macbooks haven't been able to reproduce yet...)
@mattw-nws
Copy link
Contributor

uh....hmm. Those values are much larger than -0.000000.

@mattw-nws
Copy link
Contributor

Introduction of a nondeterministic bug in #389 doesn't seem possible--triggering of it maybe...of course it could have been around longer... that's the problem with randomness

@hellkite500
Copy link
Member Author

I think I have tracked down the issue with this (and why we didn't see it before, though the bug was still technically there...).

It is related to

for (auto & iter : bmi_var_names_map) {

There is a subtle bug here that actually makes #387 NOT work correctly in a non-multi BMI setting or, even in multi-bmi but when a module doesn't provide a certain variable...

I'll document that bug in more detail, and I have a patch that just needs cleaned up a bit and pushed.

@hellkite500
Copy link
Member Author

See #393 for details. I think the undefined behavior of the self referencing BMI variable is to blame of this failure. I removed the -0.0 allowance in the unit tests on #394 to check.

@hellkite500 hellkite500 mentioned this issue Mar 10, 2022
12 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants