Retry Listener works once for @Factory test method instance

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Retry Listener works once for @Factory test method instance

pranav-9
List,
This is the situation :

1) I have a testclass with one test method
2) I create 4 instances of that test class using @Factory ( named
Test1, Test2, test3 and Test4 )
3) Odd number Test always fail ( Test 1 and Test3 will fail, but Test2
and Test4 will pass )
4) The test class has a retry listener implemented, with retry count as 2

Now after the test run I expect to see this


Failed tests:
  Test1(com.test.RetryTest): expected:<true> but was:<false>
  Test1(com.test.RetryTest): expected:<true> but was:<false>
  Test3(com.test.RetryTest): expected:<true> but was:<false>
Test3(com.test.RetryTest): expected:<true> but was:<false>

Tests run: 5, Failures: 3, Errors: 0, Skipped: 0



Test Run : 6, Failed :3, Skipped  15 ( coz 3 tests will fail, and as
they should get retried 5 times each, hence 3*5=15)

bit I got this

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.

Reply | Threaded
Open this post in threaded view
|

Re: Retry Listener works once for @Factory test method instance

pranav-9
Sorry for my earlier incomplete mail, Here I go again :

List,
This is the situation :

1) I have a testclass with one test method
2) I create 6 instances of that test class using @Factory ( named
Test1, Test2, Test3 ... Test6 )
3) Odd number Test always fail ( Test 1 , Test3 and Test5 will fail, but Test2
, Test4 and Test6 will pass )
4) The test class has a retry listener implemented, with retry count as 2

Now after the test run I expect to see this


Failed tests:
  Test1(com.test.RetryTest): expected:<true> but was:<false>
  Test1(com.test.RetryTest): expected:<true> but was:<false>
  Test3(com.test.RetryTest): expected:<true> but was:<false>
  Test3(com.test.RetryTest): expected:<true> but was:<false>
  Test5(com.test.RetryTest): expected:<true> but was:<false>
  Test5(com.test.RetryTest): expected:<true> but was:<false>

Tests run: 9, Failures: 6, Errors: 0, Skipped: 0


but I got this

Failed tests:
  Test1(com.test.RetryTest): expected:<true> but was:<false>
  Test1(com.test.RetryTest): expected:<true> but was:<false>
  Test3(com.test.RetryTest): expected:<true> but was:<false>
  Test5(com.test.RetryTest): expected:<true> but was:<false>

Tests run: 7, Failures: 4, Errors: 0, Skipped: 0

As you can see only Test1 was retried , rest all tests just failed
after first failure. My guess is, its happening because while retrying
TestNG consider all instance of one method ( created by @factory) as
one method, hence once the retry quota is over for first method, it
does not bother to run it for other instances of same method. Seems
like a bug , what do you guys say ?

I am attaching the sample code with this email, and I am TestNG 6.3.1.

best,
Pranav



On Wed, Dec 7, 2011 at 1:48 AM, Pranav Verma <[hidden email]> wrote:

> List,
> This is the situation :
>
> 1) I have a testclass with one test method
> 2) I create 4 instances of that test class using @Factory ( named
> Test1, Test2, test3 and Test4 )
> 3) Odd number Test always fail ( Test 1 and Test3 will fail, but Test2
> and Test4 will pass )
> 4) The test class has a retry listener implemented, with retry count as 2
>
> Now after the test run I expect to see this
>
>
> Failed tests:
>  Test1(com.test.RetryTest): expected:<true> but was:<false>
>  Test1(com.test.RetryTest): expected:<true> but was:<false>
>  Test3(com.test.RetryTest): expected:<true> but was:<false>
> Test3(com.test.RetryTest): expected:<true> but was:<false>
>
> Tests run: 5, Failures: 3, Errors: 0, Skipped: 0
>
>
>
> Test Run : 6, Failed :3, Skipped  15 ( coz 3 tests will fail, and as
> they should get retried 5 times each, hence 3*5=15)
>
> bit I got this
--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.


src.zip (5K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Retry Listener works once for @Factory test method instance

pranav-9
Cédric  , can you please have a look ...and let me know if I am doing
something wrong or its a bug ?

TIA,
Pranav

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.

Reply | Threaded
Open this post in threaded view
|

Re: Re: Retry Listener works once for @Factory test method instance

Cédric Beust ♔-2
Hi Pranav,

I'll try as soon as I get a chance, but I'm pretty busy right now...

-- 
Cédric




On Fri, Dec 9, 2011 at 2:03 AM, pranav <[hidden email]> wrote:
Cédric  , can you please have a look ...and let me know if I am doing
something wrong or its a bug ?

TIA,
Pranav

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.


--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.
Reply | Threaded
Open this post in threaded view
|

Re: Retry Listener works once for @Factory test method instance

pranav-9
Hey Cédric,


Happy New Year :), did you get a chance to have a look at this issue ?

Best,
Pranav

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.

Reply | Threaded
Open this post in threaded view
|

Re: Re: Retry Listener works once for @Factory test method instance

Cédric Beust ♔-2
Hi Pranav,

Sorry, not yet...

-- 
Cédric




On Mon, Jan 9, 2012 at 10:13 AM, pranav <[hidden email]> wrote:
Hey Cédric,


Happy New Year :), did you get a chance to have a look at this issue ?

Best,
Pranav

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.


--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.
Reply | Threaded
Open this post in threaded view
|

Re: Re: Retry Listener works once for @Factory test method instance

Krishnan Mahadevan
Pranav,

I dont find any issues with IRetryAnalyzer is TestNG and it works fine.

Here's a full fledged working sample that substantiates my claim.

public class RetryAnalyzerDemo implements ITest{
private String instanceName;
private int initialValue=1;
public RetryAnalyzerDemo(String instanceName, int i){
this.instanceName = instanceName;
this.initialValue = i;
}

@Override
public String getTestName() {
return "Test : " + this.instanceName ;
}

@Factory
public static Object[] getInstance(){
Object[] object = new Object[2];
object[0] = new RetryAnalyzerDemo("Java",1);
object[1] = new RetryAnalyzerDemo("Oracle", 3);
return object;
}
@Test(retryAnalyzer=MyGlobalListener.class)
public void testMethod(){
Assert.assertTrue(((this.initialValue++)%5)==0);
}
}


public class MyGlobalListener implements IRetryAnalyzer, ITestNGListener{
public ThreadLocal<Integer> retryCount = new ThreadLocal<Integer>();
public MyGlobalListener(){
System.out.println("RetryListener instantiated");
retryCount.set(new Integer(0));
}

@Override
public boolean retry(ITestResult result) {
int currentValue = 0;
if (retryCount.get() != null){
                      currentValue = retryCount.get().intValue();
}
System.out.println("Current Value = " + currentValue);
if (currentValue >= 5){
return false;
}
retryCount.set(new Integer(currentValue++));
System.out.println("Another iteration pending");
return true;
}
}

Output 

===============================================
    Default test
    Tests run: 8, Failures: 6, Skips: 0
===============================================


===============================================
BlueFin Test
Total tests run: 8, Failures: 6, Skips: 0
===============================================


As you can see, 
There were two instances that I created

The "Java" instance is supposed to fail 4 times, before it passes
The "Oracle" instance is supposed to fail 2 times, before it passes

which explains the result of 6 failures of which 2 passed.

I hard-coded the retry count to 5 times, but you can get creative and change that as well.

Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"


2012/1/10 Cédric Beust ♔ <[hidden email]>
Hi Pranav,

Sorry, not yet...

-- 
Cédric




On Mon, Jan 9, 2012 at 10:13 AM, pranav <[hidden email]> wrote:
Hey Cédric,


Happy New Year :), did you get a chance to have a look at this issue ?

Best,
Pranav

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.


--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.
Reply | Threaded
Open this post in threaded view
|

Re: Retry Listener works once for @Factory test method instance

pranav-9
Krishnan,

Thanks for trying to look into this issue, but your example is
different from my problem. Your example does not have a max limit for
retry , it will run tests as long as it does not pass, which I agree
works just fine. My problem is with fixing a max number of retries,
and reporting them properly .... please try the source zip attached in
my original post, and you will understand the problem better. Thanks
again.

Best,
Pranav

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.

Reply | Threaded
Open this post in threaded view
|

Re: Retry Listener works once for @Factory test method instance

shankarkc78@gmail.com
I think you are missing sth here.
@Override
public boolean retry(ITestResult result) {
int currentValue = 0;
if (retryCount.get() != null){
                      currentValue = retryCount.get().intValue();
}
System.out.println("Current Value = " + currentValue);
if (currentValue >= 5){
return false;
}
retryCount.set(new Integer(currentValue++));
System.out.println("Another iteration pending");
return true;
}
THis example has max limit.

On Tuesday, February 28, 2012 11:59:46 AM UTC+5:30, pranav wrote:
Krishnan,

Thanks for trying to look into this issue, but your example is
different from my problem. Your example does not have a max limit for
retry , it will run tests as long as it does not pass, which I agree
works just fine. My problem is with fixing a max number of retries,
and reporting them properly .... please try the source zip attached in
my original post, and you will understand the problem better. Thanks
again.

Best,
Pranav

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To view this discussion on the web visit https://groups.google.com/d/msg/testng-users/-/4GyeRqDKuuQJ.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.