Unit tests intermittently failing on `main` #197

isaac-io · 2022-10-23T10:27:39Z

The following tests sometimes fail on main:

$ ./db_compaction_test --gtest_filter=DBCompactionTestWithParam/DBCompactionTestWithParam.PartialCompactionFailure/1
Note: Google Test filter = DBCompactionTestWithParam/DBCompactionTestWithParam.PartialCompactionFailure/1
[==========] Running 1 test from 1 test case.
[----------] Global test environment set-up.
[----------] 1 test from DBCompactionTestWithParam/DBCompactionTestWithParam
[ RUN      ] DBCompactionTestWithParam/DBCompactionTestWithParam.PartialCompactionFailure/1
/home/isaac/projects/speedb-io/db/db_compaction_test.cc:2894: Failure
Value of: !dbfull()->TEST_WaitForCompact().ok()
  Actual: false
Expected: true
8:0
[  FAILED  ] DBCompactionTestWithParam/DBCompactionTestWithParam.PartialCompactionFailure/1, where GetParam() = (1, false) (290 ms)
[----------] 1 test from DBCompactionTestWithParam/DBCompactionTestWithParam (290 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test case ran. (290 ms total)
[  PASSED  ] 0 tests.
[  FAILED  ] 1 test, listed below:
[  FAILED  ] DBCompactionTestWithParam/DBCompactionTestWithParam.PartialCompactionFailure/1, where GetParam() = (1, false)

 1 FAILED TEST

$ ./deletefile_test --gtest_filter=DeleteFileTest.BackgroundPurgeCopyOptions
Note: Google Test filter = DeleteFileTest.BackgroundPurgeCopyOptions
[==========] Running 1 test from 1 test case.
[----------] Global test environment set-up.
[----------] 1 test from DeleteFileTest
[ RUN      ] DeleteFileTest.BackgroundPurgeCopyOptions
/home/isaac/projects/speedb-io/db/deletefile_test.cc:124: Failure
Expected equality of these values:
  required_sst
    Which is: 1
  sst_cnt
    Which is: 2
[  FAILED  ] DeleteFileTest.BackgroundPurgeCopyOptions (9732 ms)
[----------] 1 test from DeleteFileTest (9732 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test case ran. (9734 ms total)
[  PASSED  ] 0 tests.
[  FAILED  ] 1 test, listed below:
[  FAILED  ] DeleteFileTest.BackgroundPurgeCopyOptions

 1 FAILED TEST

This is caused by the changes introduced in #194, and there seems to be a timing aspect to the failures, since they weren't caught by me or QA testing when #194 was evaluated.

To Reproduce

Run the tests in a loop until they fail. Overloading the machine with activity seems to help the failure manifest quicker:

$ while ./db_compaction_test --gtest_filter=DBCompactionTestWithParam/DBCompactionTestWithParam.PartialCompactionFailure/1; do sleep 1; done

And

$ while ./deletefile_test --gtest_filter=DeleteFileTest.BackgroundPurgeCopyOptions; do sleep 1; done

Expected behavior

The tests pass successfully.

Additional context

N/A

The text was updated successfully, but these errors were encountered:

In #194 the default value for background compaction threads was changed to 8 (from 1). This caused some tests to fail, but they were fixed. Alas, the fixes seems to not be complete, as under stress some tests still fail. Make it so the tests are truly fixed now. This can be checked by running them in a loop with the machine overloaded: ``` $ while ./db_compaction_test --gtest_filter=DBCompactionTestWithParam/DBCompactionTestWithParam.PartialCompactionFailure/1; do sleep 1; done ``` And ``` $ while ./deletefile_test --gtest_filter=DeleteFileTest.BackgroundPurgeCopyOptions; do sleep 1; done ```

…#217) In #194 the default value for background compaction threads was changed to 8 (from 1). This caused some tests to fail and fixes were implemented as part of #197. Alas, it seems that the fixes weren't complete, as under stress a test still fail. Make it so the test is truly fixed now. This can be verified by running them in a loop with the machine overloaded: ``` $ while ./deletefile_test --gtest_filter=DeleteFileTest.BackgroundPurgeCFDropTest; do sleep 1; done ```