8 Flaky tests in the module extensions/throwingproviders

[harshith2000]

Found 8 flaky tests in the module extensions/throwingproviders

1. Tests that fail

Module: extensions/throwingproviders

Class Name1: com.google.inject.throwingproviders.CheckedProvidersTest,

Test Name: testUnsupportedMethods_otherMethod_throwsIllegalArgumentException

Class Name2: testUnsupportedMethods_otherMethod_throwsIllegalArgumentException

Tests Names: testCxtorWithManyExceptions, testCxtorWithSuperclassExceptionFails, testCxtorWithWrongException, testManyMethods, testProviderMethodWithManyExceptions, testProviderMethodWithSuperclassExceptionFails, testProviderMethodWithWrongException

2. Why these tests fail?

The test - testUnsupportedMethods_otherMethod_throwsIllegalArgumentException in CheckedProvidersTest.java has the getDeclaredMethods().
According to the official documentation, this returns the methods in a different order each time it is called, making this test flaky i.e. non-deterministic.
The tests testCxtorWithManyExceptions, testCxtorWithSuperclassExceptionFails, testCxtorWithWrongException, testManyMethods, testProviderMethodWithManyExceptions, testProviderMethodWithSuperclassExceptionFails, testProviderMethodWithWrongException in CheckedProviderTest.java has the getErrorMessages().
This returns the error messages in a different order each time it is called. For instance, sometimes it returns [RemoteException, BindException], but at other times, it returns [BindException, RemoteException].
Since there is no guarantee in the order of exception types returned, the order might change, resulting in the test failure.

3. How to reproduce?

I used an open-source tool called NonDex to detect the assumption by shuffling the order of returned exception types.
Running the following commands will test the aforementioned operation

Clone the Repo

https://github.com/google/guice

Compile the module

mvn install extensions/throwingproviders -am -DskipTests

(Optional) Run the unit test

mvn -pl throwingproviders test -Dtest=com.google.inject.throwingproviders.CheckedProvidersTest#testUnsupportedMethods_otherMethod_throwsIllegalArgumentException

Run the unit test using NonDex

mvn -pl throwingproviders edu.illinois:nondex-maven-plugin:2.1.1:nondex -Dtest=com.google.inject.throwingproviders.CheckedProvidersTest#testUnsupportedMethods_otherMethod_throwsIllegalArgumentException

4. How I fixed?

Since the order of the methods or error messages is not guaranteed, it is important to consider all possible orders. Fortunately, only two possible orders are possible for both methods and error messages. By checking if the actual message is equal to any of the orders, we can fix the flakiness.