How to use dataProvider and  only re-run failed test instances?

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

How to use dataProvider and  only re-run failed test instances?

sazz
I am using dataProvider to pass parameters to a test method. Say
dataProvider method passes 10 data sets and it fails 3 times. And now
I just want to run the test method only 3 times with the failed data,
am struggling a bit to make sure only that happens and does not run 10
times again. Please guide me in this.

In one single test method  I have both passed and failed test cases
present at a time hence when ever I invoke this perticular method I
have with me both the passed and failed report generated at the same
time. But the issue is, when ever I specifically want to run the
Failed test cases from the testng-failed.xml I come across again with
all the cases(passed as well as failed cases), which I do not want. I
only want to get the failed cases here only not the passed one even
though it is in the same  method.

For eg ::

public class BaseTest{
        String id;
        BaseTest(String id) {
                this.id = id;
        }

        @Test(dataProvider="CheckTest")
        public void testGenX(String testMe) {
                System.out.println("id=" + id +" testMe="+ testMe);
                if("A".equals(id))
                        Assert.assertEquals("james", testMe); //ListenerTest fails for
james
                else {
                        Assert.assertEquals("andrew", testMe); //AnotherTest fails for
andrew
                }
        }

        @DataProvider(name="CheckTest")
        public Object[][] permanentGiver() {

                return new Object[][] {
                                {"andrew"}, {"james"},
                                {"andrew"}, {"james"}
                };
        }
}


Result is::

===============================================
    Default test
    Tests run: 4, Failures: 2, Skips: 0
===============================================


now when I am running the testng-failure.xml for the fail result. I
again come across the same result(both the fail as well as the pass).
But I want only the fail result to execute. Kindly give a solution.

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.

Reply | Threaded
Open this post in threaded view
|

Re: How to use dataProvider and only re-run failed test instances?

Cédric Beust ♔-2


On Mon, Apr 25, 2011 at 1:57 AM, sazz <[hidden email]> wrote:
I am using dataProvider to pass parameters to a test method. Say
dataProvider method passes 10 data sets and it fails 3 times. And now
I just want to run the test method only 3 times with the failed data,
am struggling a bit to make sure only that happens and does not run 10
times again. Please guide me in this.

TestNG already does this automatically for you. Try it and take a look at the testng-failed.xml file that it generates, you will understand how this works.

--
Cédric


--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.
Reply | Threaded
Open this post in threaded view
|

Re: How to use dataProvider and only re-run failed test instances?

Raghuram Devarakonda-2


2011/4/25 Cédric Beust ♔ <[hidden email]>


On Mon, Apr 25, 2011 at 1:57 AM, sazz <[hidden email]> wrote:
I am using dataProvider to pass parameters to a test method. Say
dataProvider method passes 10 data sets and it fails 3 times. And now
I just want to run the test method only 3 times with the failed data,
am struggling a bit to make sure only that happens and does not run 10
times again. Please guide me in this.

TestNG already does this automatically for you. Try it and take a look at the testng-failed.xml file that it generates, you will understand how this works.




Very interesting. Out of curiosity, I wrote the following test method and it failed as expected for values 1 and 3.

----------
    @DataProvider
    public Object[][] dpEven() {
        return new Object[][] { {1}, {2}, {3}};
    }
   
    @Test(dataProvider="dpEven")
    public void testEven(int num) {
        Assert.assertEquals(num%2, 0);
    }
----------

Running with testng-failed.xml then ran the test for only failed parameters 1 and 3. So TestNG is actually saving all the parameters passed to the test method in testng-failed.xml (using "invocation-numbers" in this case). Is there a limitation on the types of parameters that can be supported this way?

Thanks,
Raghu





--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.
Reply | Threaded
Open this post in threaded view
|

Re: How to use dataProvider and only re-run failed test instances?

Cédric Beust ♔-2
Hi Raghuram,

On Mon, Apr 25, 2011 at 9:55 AM, Raghuram Devarakonda <[hidden email]> wrote:

Very interesting. Out of curiosity, I wrote the following test method and it failed as expected for values 1 and 3.

----------
    @DataProvider
    public Object[][] dpEven() {
        return new Object[][] { {1}, {2}, {3}};
    }
   
    @Test(dataProvider="dpEven")
    public void testEven(int num) {
        Assert.assertEquals(num%2, 0);
    }
----------

Running with testng-failed.xml then ran the test for only failed parameters 1 and 3. So TestNG is actually saving all the parameters passed to the test method in testng-failed.xml (using "invocation-numbers" in this case). Is there a limitation on the types of parameters that can be supported this way?

There is no limitation on this for a simple reason: testng-failed.xml only records the indices that failed, not the values. When you rerun your test, TestNG will call your data provider as usual but it will only invoke your test method with the indices in the XML file, and it will discard the others.

I can see why you're asking this question, though: in your example, the values you return happen to be integers that match the indices :-)

--
Cédric


--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.
Reply | Threaded
Open this post in threaded view
|

Re: How to use dataProvider and only re-run failed test instances?

Raghuram Devarakonda-2


2011/4/25 Cédric Beust ♔ <[hidden email]>

I can see why you're asking this question, though: in your example, the values you return happen to be integers that match the indices :-)

There is no limitation on this for a simple reason: testng-failed.xml only records the indices that failed, not the values. When you rerun your test, TestNG will call your data provider as usual but it will only invoke your test method with the indices in the XML file, and it will discard the others.--


Hi Cédric,

Nice solution to what I thought was a tricky problem. The user needs to be careful not to tinker with the DataProvider code in between the runs, though. Yes, my example test case did contribute to my question :-). 

My interest was piqued because coincidentally, a similar question was asked on nose forums today and the answer there was that it is not possible to run tests with only the parameters that failed. To be fair, DataProvider tests in TestNG have some differences with python generator based tests in nose.
    http://groups.google.com/group/nose-users/browse_thread/thread/7a27f571e9c56e36

Thanks,
Raghu

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.
Reply | Threaded
Open this post in threaded view
|

Re: How to use dataProvider and only re-run failed test instances?

Cédric Beust ♔-2
Hi Raguram,

2011/4/25 Raghuram Devarakonda <[hidden email]>

Hi Cédric,

Nice solution to what I thought was a tricky problem. The user needs to be careful not to tinker with the DataProvider code in between the runs, though.

Indeed, but if you break this assumption by modifying your tests in-between runs, then you should not be surprised if you get different results. And this is true whether you run testng.xml or testng-failed.xml.
 
Yes, my example test case did contribute to my question :-). 

My interest was piqued because coincidentally, a similar question was asked on nose forums today and the answer there was that it is not possible to run tests with only the parameters that failed. To be fair, DataProvider tests in TestNG have some differences with python generator based tests in nose.
    http://groups.google.com/group/nose-users/browse_thread/thread/7a27f571e9c56e36


I'm not familiar with Nose but I don't see any reason why it couldn't use the same trick I used with TestNG: just remember the number of the invocation that failed and discard the others when rerunning your tests...

--
Cédric


--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.
Reply | Threaded
Open this post in threaded view
|

Re: How to use dataProvider and only re-run failed test instances?

Raghuram Devarakonda-2


2011/4/25 Cédric Beust ♔ <[hidden email]>

I'm not familiar with Nose but I don't see any reason why it couldn't use the same trick I used with TestNG: just remember the number of the invocation that failed and discard the others when rerunning your tests...


Indeed. I have suggested the same on nose list.

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.
Reply | Threaded
Open this post in threaded view
|

Re: How to use dataProvider and only re-run failed test instances?

Frank Escobar
Fixed in version 6.10
https://mvnrepository.com/artifact/org.testng/testng/6.10
https://github.com/cbeust/testng/blob/master/CHANGES.txt#L95
https://github.com/cbeust/testng/issues/1250

On Monday, April 25, 2011 at 8:07:01 PM UTC+1, Raghuram Devarakonda wrote:


2011/4/25 Cédric Beust ♔ <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="dGLunjOaE2UJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">ced...@...>

I'm not familiar with Nose but I don't see any reason why it couldn't use the same trick I used with TestNG: just remember the number of the invocation that failed and discard the others when rerunning your tests...


Indeed. I have suggested the same on nose list.

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.