-
Notifications
You must be signed in to change notification settings - Fork 1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Retrying failed tests with dataproviders #763
Comments
Could you share a test case we can run? It will help us to fix the problem if any. |
It's a UI test for an ecommerce website.. I can share the code but I don't think it will run without our internal framework. |
You didn't specifically mention |
@rschmitt in our test framework which is a bunch of custom implementations over selenium for internal use we have retry code.. we've discussed building functionality into here that specifically uses the original run's parameter object but we wanted to make sure it wasn't something that could be used for everyone in testng, thus bringing it here.
which has the retry method overwritten. If I weren't so useless wiht markdown I'd paste the whole thing but it just deals specifically with our conditions for running a retry |
@juherr sorry forgot to tag you previously. I work for a huge ecommerce company (not amazon though) and I don't think I'm allowed to share the code specifically :( I might be able to draft up an example once I get home tonight that will illustrate my point |
Ok. In this case, do you think it is possible to share an new test case based on your private tests? |
sure. on vacation now but will follow up on Monday On Fri, Jul 24, 2015, 12:48 AM Julien Herr notifications@github.com wrote:
|
Hey guys, sorry for the delay on this. My coworker was fired and i'm doing 2 people's work now and don't have time to update this until after our next big release. My sincere apologies. |
Hey guys, just delivered the big release I mentioned previously a few weeks ago. After looking through this again and refreshing, I realized that we eliminated the issue that was occurring here by implementing hashcode and equals methods. |
@robotunicr0n Hi Daniel, could You describe your solution regarding rerunning failed test with the same test data more clearly? As I have also faced with this issue. |
@bodaganj our solution was to override hashcode and equals to provide a way for the class to authenticate which object it's accessing. I believe this will help you out: https://en.wikipedia.org/wiki/Java_hashCode() |
@robotunicr0n You've overrided hashcode and equals methods for your MockItem object, right? |
yes Daniel Hensey 黄龍 On Sun, Nov 22, 2015 at 2:31 PM, bodaganj notifications@github.com wrote:
|
My objects' class contains overrided equals and hashcode methods. I've reproduced the same situation for dataprovider with Boolean objects (equals, hashcode are overrided as well) also and the same issue occurs. So i'm not sure that equals and hashcode methods really prevent our issue. |
Ah that's unfortunate to hear. Since this was the only step we took to Daniel Hensey 黄龍 On Mon, Nov 23, 2015 at 1:35 AM, bodaganj notifications@github.com wrote:
|
Hey guys, I'm back after about a year. The equals and hashcode methods weren't actually resolving our problem and we're not sure why it went away temporarily but the issue is back so I'm going to reopen this. The retryanalyzer loops infinitely because it can't count the retries without matching the parameter data. I'll reillustrate some of what I said before: mockitem gets a dummy id assigned to it at creation which happens in the dataprovider, users who write tests were configuring the product and then issuing a post command which updated the id from the dummy value to the actual id. If the test failed, the retry analyzer would have the actual id and try to compare it to the dummy id and it says "this isn't the same thing". I've updated the API in such a way that users won't have to specify configuration and then POST inside of the test method, but I think the issue will still occur because the retryanalyzer calls the dataprovider again instead of reusing the test data. Due to this it will post new products and get new ids and we've run into the same problem again. Here's an example of the method that our dataprovider would call: public static List<String> getBasicTestProduct() {
TestProduct product = new TestProduct();
String sku = product.setTitle("Basic Test Product").registerProduct();
return Lists.newArrayList(sku);
} And here is the dataprovider itself: @DataProvider(parallel = true)
public static Object[][] oneTestProduct() {
assumeTrue(TestProductUtil.isTestProductEnabled());
return new Object[][] {
new Object[] {TestProductFactory.getBasicTestProduct()}
};
} And here is an example of a test that would use the dataprovider: public void testAddToCart(List<String> skus) {
ProductPage productPage = openProductPageAndSelectSku(sku, false);
ShoppingCartPage cartPage = productPage.addToCartNavigateToShoppingCart();
cartPage.verifyAddition(sku, 1);
} The skus in that test method aren't 100% coded correctly because I'm in the process of updating how we're posting products, but assume that there is code that handles getting the sku out of the list and using it appropriately. Unfortunately I don't think there's a way you can run any of our tests, but maybe you can craft a small bit of code that does the same thing. I'm going to keep researching on my end to see if we can override the retry method in a way that allows us to use the same test data but in the mean time I look forward to your response. |
Markup sucks and can't handle code... but you get the point |
Sad to read it is still failing :( Could you share a full and runnable project that shows the issue ? About markdown, block highlighting is with triple `. |
Ah thanks for the tip on markdown, I'll see if that works. I'm not sure this is actually a bug.. we've been moving pretty fast over here conversing about this and a few things come up. I think the main problem is that we're using the id in the hashcode/equals method when we don't necessarily need to. There's another required field that we could enforce uniqueness through and use that in the hashcode/equals. That gets around the problem of having dummy id vs actual id. I can't really share something you could run, if you recall from earlier on in this issue we have a custom framework over selenium that I can't share and it would basically require giving you access to the codebase. I'm hoping I can sort out an issue today/tomorrow before we go into code freeze. I'll update on here as I figure stuff out. Thank you for always being willing to help out @juherr it's very appreciated! |
I think, too, that i could probably bug our framework engineer to update our retry method to reuse the test parameters. I'm not sure that's possible as I don't have much experience working in the framework itself so am not overtly familiar with testng's innards. |
No need to share your framework. |
Hi, I see this is very old but still Open and valid. Currently using iRetryAnalyzer as a Listener. Additional notes
• Retry also works correctly with no Dataprovider This is really specific to dynamic content in Dataproviders, Ex: Feeding a signup test with randomized users.
I can get some mock-markup if needed. But it looks pretty clear cut with the above steps. For the meantime my quick solution was to detect if a Failed test has Parameters and not-retry if so. Not Ideal. I might try to forcably update the DP object with the failed one's content to avoid this issue. Contrary to some comments above, I would like to retain the ability to keep DP's content dynamic between each retry. |
@LeHudon I'm not sure to understand what is working and what is not. Could you share a full sample? |
Stripped down to as little needed as possible to repro. Change {randomCharacters()} to something static. And it will work fine The behavior of this actually works fine on 7.0.0-beta1. Test Class 'Test Class public class test1 {
WebDriver driver;
@BeforeMethod
public void setup(){
System.setProperty("webdriver.chrome.driver", "./src/main/java/chromedriver.exe");
driver = new ChromeDriver();
}
public String randomCharacters() {
int leftLimit = 97; // letter 'a'
int rightLimit = 122; // letter 'z'
int targetStringLength = 4;
Random random = new Random();
StringBuilder buffer = new StringBuilder(targetStringLength);
for (int i = 0; i < targetStringLength; i++) {
int randomLimitedInt = leftLimit + (int)
(random.nextFloat() * (rightLimit - leftLimit + 1));
buffer.append((char) randomLimitedInt);
}
return buffer.toString();
}
@DataProvider(name = "Test")
public Object[][] dpTest() {
return new Object[][] {
{randomCharacters()}
};
}
@Test(dataProvider = "Test")
public void test(String string){
driver.get("https://www.google.com");
System.out.println(string);
driver.findElement(By.cssSelector("asdfsadf"));
}
@AfterMethod
public void teardown(){
driver.quit();
}
} Retry Logic, boilerplate found Online public class TestRetry extends test1 implements IRetryAnalyzer {
private int count = 0;
private static int maxTry = 2;
@Override
public boolean retry(ITestResult iTestResult) {
try {
Throwable failureException = iTestResult.getThrowable();
if (!(failureException instanceof AssertionError)) {
if (!iTestResult.isSuccess()) { //Check if test did not succeed
if (count < maxTry) { //Check if maxtry count is reached
count++; //Increase the maxTry count by 1
System.out.println("Retrying test " + iTestResult.getName() + " with status "
+ getResultStatusName(iTestResult.getStatus()) + " for the " + (count) + " time(s).");
iTestResult.setStatus(ITestResult.FAILURE); //Mark test as failed
return true; //Tells TestNG to re-run the test
} else {
iTestResult.setStatus(ITestResult.FAILURE); //If maxCount reached,test marked as failed
}
} else {
iTestResult.setStatus(ITestResult.SUCCESS); //If test passes, TestNG marks it as passed
}
}
} catch (IllegalArgumentException e) {
e.printStackTrace();
}
System.out.println("Test FAILED: "+ iTestResult.getMethod().getMethodName());
return false;
}
private String getResultStatusName(int status) {
String resultName = null;
if (status == 1)
resultName = "SUCCESS";
if (status == 2)
resultName = "FAILURE";
if (status == 3)
resultName = "SKIP";
return resultName;
} The Test Listerner for the XML public class RetryFailedTestsListener implements IAnnotationTransformer {
@Override
public void transform(ITestAnnotation annotation, Class testClass, Constructor testConstructor, Method testMethod) {
annotation.setRetryAnalyzer(TestRetry.class);
} And then the XML <!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd" >
<suite name="Test" parallel="methods" thread-count="4">
<listeners>
<listener class-name="RetryFailedTestsListener"/>
</listeners>
<test verbose="100" name="Chrome">
<classes>
<class name="test1"/>
</classes>
</test>
</suite> |
Hey, is there any solution to this issue yet? |
@inderjohar - Have you tried this using TestNG |
yes , the issue still persists in testng 7.3.0 , the data provider is not running the test under @test annotation while using retry listener. Please check. |
This is not an issue anymore in TestNG Here's a sample that shows this in action import java.util.concurrent.atomic.AtomicInteger;
import org.testng.Assert;
import org.testng.IRetryAnalyzer;
import org.testng.ITestResult;
import org.testng.annotations.DataProvider;
import org.testng.annotations.Test;
public class SampleTestCase {
private AtomicInteger counter = new AtomicInteger(0);
@Test(dataProvider = "dp", retryAnalyzer = OnceAgain.class)
public void sampleTestMethod(Player player) {
if (counter.incrementAndGet() <= 2) {
System.err.println("Test failed for Input: " + player + " in attempt " + counter.get());
Assert.fail(player.toString());
}
System.err.println("Test passed for Input: " + player + " in attempt " + counter.get());
counter = new AtomicInteger(0);
}
@DataProvider(name = "dp")
public Object[][] getData() {
System.err.println("Retrieving Test data");
return new Object[][]{
{new Player("Jack", 20)},
{new Player("Daniel", 25)}
};
}
public static class OnceAgain implements IRetryAnalyzer {
private int count = 0;
@Override
public boolean retry(ITestResult result) {
return count++ <= 3;
}
}
public static class Player {
private final String name;
private final int age;
public Player(String name, int age) {
this.name = name;
this.age = age;
}
@Override
public String toString() {
return "[name:" + name + ", age:" + age + "]";
}
}
} Output Retrieving Test data
Test failed for Input: [name:Jack, age:20] in attempt 1
Test ignored.
Test failed for Input: [name:Jack, age:20] in attempt 2
Test ignored.
Test passed for Input: [name:Jack, age:20] in attempt 3
Test failed for Input: [name:Daniel, age:25] in attempt 1
Test ignored.
Test failed for Input: [name:Daniel, age:25] in attempt 2
Test ignored.
Test passed for Input: [name:Daniel, age:25] in attempt 3
===============================================
Default Suite
Total tests run: 6, Passes: 2, Failures: 0, Skips: 0, Retries: 4
===============================================
Process finished with exit code 0
|
I looked through all of the issues I could find on this project that related to dataproviders and retry logic and nothing seemed to be similar to what I'm about to ask. If I missed the appropriate existing topic feel free to merge this one into that one.
I'm running into an issue with our test suite where my tests that use dataproviders are not using the original data parameter from the first run of the test when they get retried. To illustrate:
We'd like to use the MockItem object that is created and has a dynamically generated ID in step 1 during our retry in step 2. So instead of having MockItem (ID: 1234) and MockItem (ID: 1235, used for retry), we'd have MockItem (ID: 1234) and reuse MockItem (ID: 1234).
An extra and valuable note: currently we hook in a set of code that runs afterInvocation in the test suite that cleans up the MockItem object, so right now the original run's MockItem is deleted before the retry starts. If we skip deleting when we're retrying, will it automatically work? ie is there support in for this already and we're just messing the code up by deleting the original object?
The text was updated successfully, but these errors were encountered: