Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Retrying failed tests with dataproviders #763

Closed
robotunicr0n opened this issue Jul 23, 2015 · 28 comments
Closed

Retrying failed tests with dataproviders #763

robotunicr0n opened this issue Jul 23, 2015 · 28 comments

Comments

@robotunicr0n
Copy link

I looked through all of the issues I could find on this project that related to dataproviders and retry logic and nothing seemed to be similar to what I'm about to ask. If I missed the appropriate existing topic feel free to merge this one into that one.

I'm running into an issue with our test suite where my tests that use dataproviders are not using the original data parameter from the first run of the test when they get retried. To illustrate:

  • Test 1 runs and the dataprovider generates a MockItem object that gets a dynamically generated ID from one of our web services.
  • Test 1 fails and it meets our conditions for a retry, so it triggers a retry. It runs the dataprovider again and generates a new MockItem object with a new dynamically generated ID

We'd like to use the MockItem object that is created and has a dynamically generated ID in step 1 during our retry in step 2. So instead of having MockItem (ID: 1234) and MockItem (ID: 1235, used for retry), we'd have MockItem (ID: 1234) and reuse MockItem (ID: 1234).

An extra and valuable note: currently we hook in a set of code that runs afterInvocation in the test suite that cleans up the MockItem object, so right now the original run's MockItem is deleted before the retry starts. If we skip deleting when we're retrying, will it automatically work? ie is there support in for this already and we're just messing the code up by deleting the original object?

@juherr
Copy link
Member

juherr commented Jul 23, 2015

Could you share a test case we can run? It will help us to fix the problem if any.

@robotunicr0n
Copy link
Author

It's a UI test for an ecommerce website.. I can share the code but I don't think it will run without our internal framework.

@rschmitt
Copy link
Contributor

You didn't specifically mention IRetryAnalyzer anywhere--is that how you're implementing retries, or are you somehow rolling your own?

@robotunicr0n
Copy link
Author

@rschmitt in our test framework which is a bunch of custom implementations over selenium for internal use we have retry code.. we've discussed building functionality into here that specifically uses the original run's parameter object but we wanted to make sure it wasn't something that could be used for everyone in testng, thus bringing it here.

public class RetryAnalyzer implements IRetryAnalyzer {

which has the retry method overwritten. If I weren't so useless wiht markdown I'd paste the whole thing but it just deals specifically with our conditions for running a retry

@robotunicr0n
Copy link
Author

@juherr sorry forgot to tag you previously. I work for a huge ecommerce company (not amazon though) and I don't think I'm allowed to share the code specifically :(

I might be able to draft up an example once I get home tonight that will illustrate my point

@juherr
Copy link
Member

juherr commented Jul 24, 2015

Ok. In this case, do you think it is possible to share an new test case based on your private tests?

@robotunicr0n
Copy link
Author

sure. on vacation now but will follow up on Monday

On Fri, Jul 24, 2015, 12:48 AM Julien Herr notifications@github.com wrote:

Ok. In this case, do you think it is possible to share an new test case
based on your private tests?


Reply to this email directly or view it on GitHub
#763 (comment).

@robotunicr0n
Copy link
Author

Hey guys, sorry for the delay on this. My coworker was fired and i'm doing 2 people's work now and don't have time to update this until after our next big release. My sincere apologies.

@robotunicr0n
Copy link
Author

Hey guys, just delivered the big release I mentioned previously a few weeks ago. After looking through this again and refreshing, I realized that we eliminated the issue that was occurring here by implementing hashcode and equals methods.

@bodaganj
Copy link

@robotunicr0n Hi Daniel, could You describe your solution regarding rerunning failed test with the same test data more clearly? As I have also faced with this issue.

@robotunicr0n
Copy link
Author

@bodaganj our solution was to override hashcode and equals to provide a way for the class to authenticate which object it's accessing. I believe this will help you out: https://en.wikipedia.org/wiki/Java_hashCode()

@bodaganj
Copy link

@robotunicr0n You've overrided hashcode and equals methods for your MockItem object, right?

@robotunicr0n
Copy link
Author

yes

Daniel Hensey 黄龍
+1 206-395-4466

On Sun, Nov 22, 2015 at 2:31 PM, bodaganj notifications@github.com wrote:

@robotunicr0n https://github.com/robotunicr0n You've overrided hashcode
and equals methods for your MockItem object, right?


Reply to this email directly or view it on GitHub
#763 (comment).

@bodaganj
Copy link

My objects' class contains overrided equals and hashcode methods. I've reproduced the same situation for dataprovider with Boolean objects (equals, hashcode are overrided as well) also and the same issue occurs. So i'm not sure that equals and hashcode methods really prevent our issue.

@robotunicr0n
Copy link
Author

Ah that's unfortunate to hear. Since this was the only step we took to
resolve our issue and haven't had any problems since, we didn't do any more
thinking/work toward a different solution so sadly I can't help you out. If
there's a way to transfer this issue to you then I'm happy to do so
otherwise you might find more help from making a new post

Daniel Hensey 黄龍
+1 206-395-4466

On Mon, Nov 23, 2015 at 1:35 AM, bodaganj notifications@github.com wrote:

My objects' class contains overrided equals and hashcode methods. I've
reproduced the same situation for dataprovider with Boolean objects
(equals, hashcode are overrided as well) also and the same issue occurs. So
i'm not sure that equals and hashcode methods really prevent our issue.


Reply to this email directly or view it on GitHub
#763 (comment).

@robotunicr0n
Copy link
Author

robotunicr0n commented May 18, 2016

Hey guys, I'm back after about a year. The equals and hashcode methods weren't actually resolving our problem and we're not sure why it went away temporarily but the issue is back so I'm going to reopen this. The retryanalyzer loops infinitely because it can't count the retries without matching the parameter data.

I'll reillustrate some of what I said before: mockitem gets a dummy id assigned to it at creation which happens in the dataprovider, users who write tests were configuring the product and then issuing a post command which updated the id from the dummy value to the actual id. If the test failed, the retry analyzer would have the actual id and try to compare it to the dummy id and it says "this isn't the same thing".

I've updated the API in such a way that users won't have to specify configuration and then POST inside of the test method, but I think the issue will still occur because the retryanalyzer calls the dataprovider again instead of reusing the test data. Due to this it will post new products and get new ids and we've run into the same problem again.

Here's an example of the method that our dataprovider would call:

public static List<String> getBasicTestProduct() {
        TestProduct product = new TestProduct();
        String sku = product.setTitle("Basic Test Product").registerProduct();
        return Lists.newArrayList(sku);
    }

And here is the dataprovider itself:

@DataProvider(parallel = true)
    public static Object[][] oneTestProduct() {
        assumeTrue(TestProductUtil.isTestProductEnabled());

        return new Object[][] {
                new Object[] {TestProductFactory.getBasicTestProduct()}
        };
    }

And here is an example of a test that would use the dataprovider:

public void testAddToCart(List<String> skus) {
        ProductPage productPage = openProductPageAndSelectSku(sku, false);
        ShoppingCartPage cartPage = productPage.addToCartNavigateToShoppingCart();
        cartPage.verifyAddition(sku, 1);
    }

The skus in that test method aren't 100% coded correctly because I'm in the process of updating how we're posting products, but assume that there is code that handles getting the sku out of the list and using it appropriately. Unfortunately I don't think there's a way you can run any of our tests, but maybe you can craft a small bit of code that does the same thing. I'm going to keep researching on my end to see if we can override the retry method in a way that allows us to use the same test data but in the mean time I look forward to your response.

@robotunicr0n robotunicr0n reopened this May 18, 2016
@robotunicr0n
Copy link
Author

Markup sucks and can't handle code... but you get the point

@juherr
Copy link
Member

juherr commented May 18, 2016

Sad to read it is still failing :(

Could you share a full and runnable project that shows the issue ?

About markdown, block highlighting is with triple `.

@robotunicr0n
Copy link
Author

Ah thanks for the tip on markdown, I'll see if that works.

I'm not sure this is actually a bug.. we've been moving pretty fast over here conversing about this and a few things come up. I think the main problem is that we're using the id in the hashcode/equals method when we don't necessarily need to. There's another required field that we could enforce uniqueness through and use that in the hashcode/equals. That gets around the problem of having dummy id vs actual id.

I can't really share something you could run, if you recall from earlier on in this issue we have a custom framework over selenium that I can't share and it would basically require giving you access to the codebase. I'm hoping I can sort out an issue today/tomorrow before we go into code freeze. I'll update on here as I figure stuff out.

Thank you for always being willing to help out @juherr it's very appreciated!

@robotunicr0n
Copy link
Author

I think, too, that i could probably bug our framework engineer to update our retry method to reuse the test parameters. I'm not sure that's possible as I don't have much experience working in the framework itself so am not overtly familiar with testng's innards.

@juherr
Copy link
Member

juherr commented May 19, 2016

I can't really share something you could run, if you recall from earlier on in this issue we have a custom framework over selenium that I can't share

No need to share your framework.
In fact, the question is more: are you able to make a project, as litle as possible, that reproduces the issue?

@LeHudon
Copy link

LeHudon commented Jan 30, 2020

Hi, I see this is very old but still Open and valid.
Confirmed on 7.0.0 and 7.1.0

Currently using iRetryAnalyzer as a Listener.
Issue I get is with using iRetryAnalyzer on a Testmethod that uses a Dataprovider with dynamic content. Im using a "maxRetries strategy" to increment between failures.
Each time it reaches the RetryLogic, it will increment the count from 0 to 1. Thus causing it to retry the same test indefintely.

Additional notes
• Occurs regardless of parallel threading vs sequential
• If i use a Dataprovider with static content, the retry works correctly,

    @DataProvider(name = "example")
    public Object[][] exampleDP() {
        return new Object[][]{
                {"someString"},
        };
    }

• Retry also works correctly with no Dataprovider

This is really specific to dynamic content in Dataproviders, Ex: Feeding a signup test with randomized users.

String timeStamp = new SimpleDateFormat("HH_mm_ss").format(new Date());
@DataProvider(name = "example")
    public Object[][] exampleDP() {
        return new Object[][]{
                {timeStamp},
        };
    }

I can get some mock-markup if needed. But it looks pretty clear cut with the above steps.

For the meantime my quick solution was to detect if a Failed test has Parameters and not-retry if so. Not Ideal. I might try to forcably update the DP object with the failed one's content to avoid this issue.

Contrary to some comments above, I would like to retain the ability to keep DP's content dynamic between each retry.
Maintaining the content of the dynamic DP between retries seems like a more complex fix from your end. And easier implemented on a per-framework basis

@juherr
Copy link
Member

juherr commented Jan 30, 2020

@LeHudon I'm not sure to understand what is working and what is not. Could you share a full sample?

@LeHudon
Copy link

LeHudon commented Mar 19, 2020

Stripped down to as little needed as possible to repro.
This retries indefinitely.

Change {randomCharacters()} to something static. And it will work fine

The behavior of this actually works fine on 7.0.0-beta1.

Test Class

'Test Class

public class test1 {

    WebDriver driver;

    @BeforeMethod
    public void setup(){
        System.setProperty("webdriver.chrome.driver", "./src/main/java/chromedriver.exe");
        driver = new ChromeDriver();
    }

    public String randomCharacters() {
        int leftLimit = 97; // letter 'a'
        int rightLimit = 122; // letter 'z'
        int targetStringLength = 4;
        Random random = new Random();
        StringBuilder buffer = new StringBuilder(targetStringLength);
        for (int i = 0; i < targetStringLength; i++) {
            int randomLimitedInt = leftLimit + (int)
                    (random.nextFloat() * (rightLimit - leftLimit + 1));
            buffer.append((char) randomLimitedInt);
        }
        return buffer.toString();
    }

    @DataProvider(name = "Test")
    public Object[][] dpTest() {
        return new Object[][] {
                {randomCharacters()}
        };
    }

    @Test(dataProvider = "Test")
    public void test(String string){
        driver.get("https://www.google.com");
        System.out.println(string);
        driver.findElement(By.cssSelector("asdfsadf"));
    }

    @AfterMethod
    public void teardown(){
        driver.quit();
    }
}

Retry Logic, boilerplate found Online

public class TestRetry extends test1 implements IRetryAnalyzer {

    private int count = 0;
    private static int maxTry = 2;

    @Override
    public boolean retry(ITestResult iTestResult)  {
        try {
            Throwable failureException = iTestResult.getThrowable();
            if (!(failureException instanceof AssertionError)) {
                if (!iTestResult.isSuccess()) {                      //Check if test did not succeed
                    if (count < maxTry) {                            //Check if maxtry count is reached
                        count++;                                     //Increase the maxTry count by 1
                        System.out.println("Retrying test " + iTestResult.getName() + " with status "
                                + getResultStatusName(iTestResult.getStatus()) + " for the " + (count) + " time(s).");
                        iTestResult.setStatus(ITestResult.FAILURE);  //Mark test as failed
                        return true;                                 //Tells TestNG to re-run the test
                    } else {
                        iTestResult.setStatus(ITestResult.FAILURE);  //If maxCount reached,test marked as failed
                    }
                } else {
                    iTestResult.setStatus(ITestResult.SUCCESS);      //If test passes, TestNG marks it as passed
                }
            }
        } catch (IllegalArgumentException e) {
            e.printStackTrace();
        }
        System.out.println("Test FAILED: "+ iTestResult.getMethod().getMethodName());
        return false;

    }
    private String getResultStatusName(int status) {
        String resultName = null;
        if (status == 1)
            resultName = "SUCCESS";
        if (status == 2)
            resultName = "FAILURE";
        if (status == 3)
            resultName = "SKIP";
        return resultName;
    }

The Test Listerner for the XML

public class RetryFailedTestsListener implements IAnnotationTransformer {


    @Override
    public void transform(ITestAnnotation annotation, Class testClass, Constructor testConstructor, Method testMethod) {
        annotation.setRetryAnalyzer(TestRetry.class);
    }

And then the XML

<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd" >
<suite name="Test" parallel="methods" thread-count="4">

    <listeners>
        <listener class-name="RetryFailedTestsListener"/>
    </listeners>

    <test verbose="100" name="Chrome">
        <classes>
            <class name="test1"/>
        </classes>
    </test>
</suite>

@krmahadevan krmahadevan self-assigned this Mar 25, 2020
@inderjohar
Copy link

Hey, is there any solution to this issue yet?

@krmahadevan
Copy link
Member

@inderjohar - Have you tried this using TestNG 7.3.0 ? If its still a problem, please feel free to share a sample

@vishuhanda
Copy link

vishuhanda commented Feb 22, 2021

yes , the issue still persists in testng 7.3.0 , the data provider is not running the test under @test annotation while using retry listener.
Q1 why data provider runs and fetch the data again from xls in case test case fails.
Q2 and after fetching why it did not run @test method in which data provider is given
Q3 once retry listener starts why it run ontestskip method of itestlistener as the test case got failed

Please check.

@krmahadevan
Copy link
Member

This is not an issue anymore in TestNG 7.8.0 (released version as of today). A data powered test that is being retried will get the same test data as the original and will continue to do so until it passes.

Here's a sample that shows this in action

import java.util.concurrent.atomic.AtomicInteger;
import org.testng.Assert;
import org.testng.IRetryAnalyzer;
import org.testng.ITestResult;
import org.testng.annotations.DataProvider;
import org.testng.annotations.Test;

public class SampleTestCase {

  private AtomicInteger counter = new AtomicInteger(0);

  @Test(dataProvider = "dp", retryAnalyzer = OnceAgain.class)
  public void sampleTestMethod(Player player) {
    if (counter.incrementAndGet() <= 2) {
      System.err.println("Test failed for Input: " + player + " in attempt " + counter.get());
      Assert.fail(player.toString());
    }
    System.err.println("Test passed for Input: " + player + " in attempt " + counter.get());
    counter = new AtomicInteger(0);
  }

  @DataProvider(name = "dp")
  public Object[][] getData() {
    System.err.println("Retrieving Test data");
    return new Object[][]{
        {new Player("Jack", 20)},
        {new Player("Daniel", 25)}
    };
  }

  public static class OnceAgain implements IRetryAnalyzer {

    private int count = 0;

    @Override
    public boolean retry(ITestResult result) {
      return count++ <= 3;
    }
  }


  public static class Player {

    private final String name;
    private final int age;

    public Player(String name, int age) {
      this.name = name;
      this.age = age;
    }

    @Override
    public String toString() {
      return "[name:" + name + ", age:" + age + "]";
    }
  }
}

Output

Retrieving Test data
Test failed for Input: [name:Jack, age:20] in attempt 1

Test ignored.
Test failed for Input: [name:Jack, age:20] in attempt 2

Test ignored.
Test passed for Input: [name:Jack, age:20] in attempt 3
Test failed for Input: [name:Daniel, age:25] in attempt 1

Test ignored.
Test failed for Input: [name:Daniel, age:25] in attempt 2

Test ignored.
Test passed for Input: [name:Daniel, age:25] in attempt 3

===============================================
Default Suite
Total tests run: 6, Passes: 2, Failures: 0, Skips: 0, Retries: 4
===============================================


Process finished with exit code 0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

8 participants