Retrying failed test cases automatically via maven/custom listeners

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Retrying failed test cases automatically via maven/custom listeners

Rish-7
Hi All,

Not sure if this question has been answered already on the forums. We
have a suite of approx 250 UI test cases that runs overnight as a
Jenkins job, and provides us the result as we check into work. A lot
of UI failures are minor blips in system/network, and the team spend a
lot of times investigating these 'false' alarms.

 To counter the same, we added a custom listener that would retry a
failed test a set number of times before marking it as a 'failure'. If
it passes beforehand, it simply marks the earlier failed invocations
as 'skipped'. Also, we wrote a custom reporter, which extends the
default HTMLReporter class and added a method that looked like this:

private IResultMap removeIncorrectlyFailedTests(ITestContext test)
{
  List<ITestNGMethod> failsToRemove = new ArrayList<ITestNGMethod>();
  IResultMap returnValue = test.getFailedTests();

  for(ITestResult result : test.getFailedTests().getAllResults())
  {
    long failedResultTime = result.getEndMillis();

    for(ITestResult resultToCheck :
test.getSkippedTests().getAllResults())
    {
        if(failedResultTime == resultToCheck.getEndMillis())
        {
            failsToRemove.add(resultToCheck.getMethod());
            break;
        }
    }

    for(ITestResult resultToCheck :
test.getPassedTests().getAllResults())
    {
        if(failedResultTime == resultToCheck.getEndMillis())
        {
            failsToRemove.add(resultToCheck.getMethod());
            break;
        }
    }
  }

  for(ITestNGMethod method : failsToRemove)
  {
      returnValue.removeResult(method);
  }

  return returnValue;
}

This method essentially removes the duplicate entries while retaining
all other ones. The invocation point for this method was:

@Override
  public void onStart(ITestContext context) {

      //added this line to ensure unnecessary results are removed
      removeIncorrectlyFailedTests(m_testContext);
      m_testContext = context;
  }

The relevant section of the pom.xml looks like:

         <properties>
            <property>
              <name>usedefaultlisteners</name>
              <value>false</value>
            </property>
            <property>
              <name>listener</name>
              <value>com.inmobi.experiments.RetryTestListener</value>
            </property>
            <property>
              <name>reporter</name>
              <value>com.inmobi.experiments.CustomHTMLReporter</value>
            </property>
          </properties>

However, on running our test cases, and viewing the report, it seems
that the CustomHTMLReporter is not being utilized by testng for report
generation.

This is how the cmd-line looks like post-execution of a sample test
case:

Tests run: 4, Failures: 1, Errors: 0, Skipped: 3, Time elapsed: 0.191
sec <<< FAILURE!

and the surefire-reports folder is all empty :(

It would be really helpful if someone could point out where I am going
wrong. Also, is there a cleaner way to automatically run failed tests
without doing the above exercise (without referring to the xml )?

Apologies if this has already been answered on the forums, but I could
not spot something that would help.

Sincere Thanks,
Rishu



--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.

Reply | Threaded
Open this post in threaded view
|

Re: Retrying failed test cases automatically via maven/custom listeners

niharika varshney
When you say without referring to the xml, do you imply that you do not want to use the testng-failed xml is it?

Regards,
Niharika

On Wed, Feb 15, 2012 at 10:31 AM, Rish <[hidden email]> wrote:
Hi All,

Not sure if this question has been answered already on the forums. We
have a suite of approx 250 UI test cases that runs overnight as a
Jenkins job, and provides us the result as we check into work. A lot
of UI failures are minor blips in system/network, and the team spend a
lot of times investigating these 'false' alarms.

 To counter the same, we added a custom listener that would retry a
failed test a set number of times before marking it as a 'failure'. If
it passes beforehand, it simply marks the earlier failed invocations
as 'skipped'. Also, we wrote a custom reporter, which extends the
default HTMLReporter class and added a method that looked like this:

private IResultMap removeIncorrectlyFailedTests(ITestContext test)
{
 List<ITestNGMethod> failsToRemove = new ArrayList<ITestNGMethod>();
 IResultMap returnValue = test.getFailedTests();

 for(ITestResult result : test.getFailedTests().getAllResults())
 {
   long failedResultTime = result.getEndMillis();

   for(ITestResult resultToCheck :
test.getSkippedTests().getAllResults())
   {
       if(failedResultTime == resultToCheck.getEndMillis())
       {
           failsToRemove.add(resultToCheck.getMethod());
           break;
       }
   }

   for(ITestResult resultToCheck :
test.getPassedTests().getAllResults())
   {
       if(failedResultTime == resultToCheck.getEndMillis())
       {
           failsToRemove.add(resultToCheck.getMethod());
           break;
       }
   }
 }

 for(ITestNGMethod method : failsToRemove)
 {
     returnValue.removeResult(method);
 }

 return returnValue;
}

This method essentially removes the duplicate entries while retaining
all other ones. The invocation point for this method was:

@Override
 public void onStart(ITestContext context) {

     //added this line to ensure unnecessary results are removed
     removeIncorrectlyFailedTests(m_testContext);
     m_testContext = context;
 }

The relevant section of the pom.xml looks like:

        <properties>
           <property>
             <name>usedefaultlisteners</name>
             <value>false</value>
           </property>
           <property>
             <name>listener</name>
             <value>com.inmobi.experiments.RetryTestListener</value>
           </property>
           <property>
             <name>reporter</name>
             <value>com.inmobi.experiments.CustomHTMLReporter</value>
           </property>
         </properties>

However, on running our test cases, and viewing the report, it seems
that the CustomHTMLReporter is not being utilized by testng for report
generation.

This is how the cmd-line looks like post-execution of a sample test
case:

Tests run: 4, Failures: 1, Errors: 0, Skipped: 3, Time elapsed: 0.191
sec <<< FAILURE!

and the surefire-reports folder is all empty :(

It would be really helpful if someone could point out where I am going
wrong. Also, is there a cleaner way to automatically run failed tests
without doing the above exercise (without referring to the xml )?

Apologies if this has already been answered on the forums, but I could
not spot something that would help.

Sincere Thanks,
Rishu



--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.


--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.
Reply | Threaded
Open this post in threaded view
|

Re: Retrying failed test cases automatically via maven/custom listeners

Rish-7
Yes. We want to handle it pretty much within the execution itself. It
does that decently so far. However, the reporting is a bit of a
problem.

On Feb 15, 10:26 am, niharika varshney <[hidden email]>
wrote:

> When you say without referring to the xml, do you imply that you do not
> want to use the testng-failed xml is it?
>
> Regards,
> Niharika
>
> On Wed, Feb 15, 2012 at 10:31 AM, Rish <[hidden email]> wrote:
> > Hi All,
>
> > Not sure if this question has been answered already on the forums. We
> > have a suite of approx 250 UI test cases that runs overnight as a
> > Jenkins job, and provides us the result as we check into work. A lot
> > of UI failures are minor blips in system/network, and the team spend a
> > lot of times investigating these 'false' alarms.
>
> >  To counter the same, we added a custom listener that would retry a
> > failed test a set number of times before marking it as a 'failure'. If
> > it passes beforehand, it simply marks the earlier failed invocations
> > as 'skipped'. Also, we wrote a custom reporter, which extends the
> > default HTMLReporter class and added a method that looked like this:
>
> > private IResultMap removeIncorrectlyFailedTests(ITestContext test)
> > {
> >  List<ITestNGMethod> failsToRemove = new ArrayList<ITestNGMethod>();
> >  IResultMap returnValue = test.getFailedTests();
>
> >  for(ITestResult result : test.getFailedTests().getAllResults())
> >  {
> >    long failedResultTime = result.getEndMillis();
>
> >    for(ITestResult resultToCheck :
> > test.getSkippedTests().getAllResults())
> >    {
> >        if(failedResultTime == resultToCheck.getEndMillis())
> >        {
> >            failsToRemove.add(resultToCheck.getMethod());
> >            break;
> >        }
> >    }
>
> >    for(ITestResult resultToCheck :
> > test.getPassedTests().getAllResults())
> >    {
> >        if(failedResultTime == resultToCheck.getEndMillis())
> >        {
> >            failsToRemove.add(resultToCheck.getMethod());
> >            break;
> >        }
> >    }
> >  }
>
> >  for(ITestNGMethod method : failsToRemove)
> >  {
> >      returnValue.removeResult(method);
> >  }
>
> >  return returnValue;
> > }
>
> > This method essentially removes the duplicate entries while retaining
> > all other ones. The invocation point for this method was:
>
> > @Override
> >  public void onStart(ITestContext context) {
>
> >      //added this line to ensure unnecessary results are removed
> >      removeIncorrectlyFailedTests(m_testContext);
> >      m_testContext = context;
> >  }
>
> > The relevant section of the pom.xml looks like:
>
> >         <properties>
> >            <property>
> >              <name>usedefaultlisteners</name>
> >              <value>false</value>
> >            </property>
> >            <property>
> >              <name>listener</name>
> >              <value>com.inmobi.experiments.RetryTestListener</value>
> >            </property>
> >            <property>
> >              <name>reporter</name>
> >              <value>com.inmobi.experiments.CustomHTMLReporter</value>
> >            </property>
> >          </properties>
>
> > However, on running our test cases, and viewing the report, it seems
> > that the CustomHTMLReporter is not being utilized by testng for report
> > generation.
>
> > This is how the cmd-line looks like post-execution of a sample test
> > case:
>
> > Tests run: 4, Failures: 1, Errors: 0, Skipped: 3, Time elapsed: 0.191
> > sec <<< FAILURE!
>
> > and the surefire-reports folder is all empty :(
>
> > It would be really helpful if someone could point out where I am going
> > wrong. Also, is there a cleaner way to automatically run failed tests
> > without doing the above exercise (without referring to the xml )?
>
> > Apologies if this has already been answered on the forums, but I could
> > not spot something that would help.
>
> > Sincere Thanks,
> > Rishu
>
> > --
> > You received this message because you are subscribed to the Google Groups
> > "testng-users" group.
> > To post to this group, send email to [hidden email].
> > To unsubscribe from this group, send email to
> > [hidden email].
> > For more options, visit this group at
> >http://groups.google.com/group/testng-users?hl=en.

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.

Reply | Threaded
Open this post in threaded view
|

Re: Re: Retrying failed test cases automatically via maven/custom listeners

niharika varshney
Hmm...Can you move your custom reporter into the listener section and check..?

Regards,
Niharika

On Wed, Feb 15, 2012 at 10:58 AM, Rish <[hidden email]> wrote:
Yes. We want to handle it pretty much within the execution itself. It
does that decently so far. However, the reporting is a bit of a
problem.

On Feb 15, 10:26 am, niharika varshney <[hidden email]>
wrote:
> When you say without referring to the xml, do you imply that you do not
> want to use the testng-failed xml is it?
>
> Regards,
> Niharika
>
> On Wed, Feb 15, 2012 at 10:31 AM, Rish <[hidden email]> wrote:
> > Hi All,
>
> > Not sure if this question has been answered already on the forums. We
> > have a suite of approx 250 UI test cases that runs overnight as a
> > Jenkins job, and provides us the result as we check into work. A lot
> > of UI failures are minor blips in system/network, and the team spend a
> > lot of times investigating these 'false' alarms.
>
> >  To counter the same, we added a custom listener that would retry a
> > failed test a set number of times before marking it as a 'failure'. If
> > it passes beforehand, it simply marks the earlier failed invocations
> > as 'skipped'. Also, we wrote a custom reporter, which extends the
> > default HTMLReporter class and added a method that looked like this:
>
> > private IResultMap removeIncorrectlyFailedTests(ITestContext test)
> > {
> >  List<ITestNGMethod> failsToRemove = new ArrayList<ITestNGMethod>();
> >  IResultMap returnValue = test.getFailedTests();
>
> >  for(ITestResult result : test.getFailedTests().getAllResults())
> >  {
> >    long failedResultTime = result.getEndMillis();
>
> >    for(ITestResult resultToCheck :
> > test.getSkippedTests().getAllResults())
> >    {
> >        if(failedResultTime == resultToCheck.getEndMillis())
> >        {
> >            failsToRemove.add(resultToCheck.getMethod());
> >            break;
> >        }
> >    }
>
> >    for(ITestResult resultToCheck :
> > test.getPassedTests().getAllResults())
> >    {
> >        if(failedResultTime == resultToCheck.getEndMillis())
> >        {
> >            failsToRemove.add(resultToCheck.getMethod());
> >            break;
> >        }
> >    }
> >  }
>
> >  for(ITestNGMethod method : failsToRemove)
> >  {
> >      returnValue.removeResult(method);
> >  }
>
> >  return returnValue;
> > }
>
> > This method essentially removes the duplicate entries while retaining
> > all other ones. The invocation point for this method was:
>
> > @Override
> >  public void onStart(ITestContext context) {
>
> >      //added this line to ensure unnecessary results are removed
> >      removeIncorrectlyFailedTests(m_testContext);
> >      m_testContext = context;
> >  }
>
> > The relevant section of the pom.xml looks like:
>
> >         <properties>
> >            <property>
> >              <name>usedefaultlisteners</name>
> >              <value>false</value>
> >            </property>
> >            <property>
> >              <name>listener</name>
> >              <value>com.inmobi.experiments.RetryTestListener</value>
> >            </property>
> >            <property>
> >              <name>reporter</name>
> >              <value>com.inmobi.experiments.CustomHTMLReporter</value>
> >            </property>
> >          </properties>
>
> > However, on running our test cases, and viewing the report, it seems
> > that the CustomHTMLReporter is not being utilized by testng for report
> > generation.
>
> > This is how the cmd-line looks like post-execution of a sample test
> > case:
>
> > Tests run: 4, Failures: 1, Errors: 0, Skipped: 3, Time elapsed: 0.191
> > sec <<< FAILURE!
>
> > and the surefire-reports folder is all empty :(
>
> > It would be really helpful if someone could point out where I am going
> > wrong. Also, is there a cleaner way to automatically run failed tests
> > without doing the above exercise (without referring to the xml )?
>
> > Apologies if this has already been answered on the forums, but I could
> > not spot something that would help.
>
> > Sincere Thanks,
> > Rishu
>
> > --
> > You received this message because you are subscribed to the Google Groups
> > "testng-users" group.
> > To post to this group, send email to [hidden email].
> > To unsubscribe from this group, send email to
> > [hidden email].
> > For more options, visit this group at
> >http://groups.google.com/group/testng-users?hl=en.

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.


--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.
Reply | Threaded
Open this post in threaded view
|

Re: Retrying failed test cases automatically via maven/custom listeners

Rish-7
Did that. Here are the observations:

- The cmd-line option remains the same:

Tests run: 4, Failures: 1, Errors: 0, Skipped: 3, Time elapsed: 0.2
sec <<< FAILURE!

- The target folder just has 1 extra dir created as 'Command line
suite' which has an html file. The file shows all 4 runs of the test
case which is failing (1 failed, 3 skipped). As per the desired
output, it should have displayed only 1.

Will try to debug the latter point and double-check if it is removing
tests correctly.

Thanks for the input.
Regards,
Rishu

On Feb 15, 12:07 pm, niharika varshney <[hidden email]>
wrote:

> Hmm...Can you move your custom reporter into the listener section and
> check..?
>
> Regards,
> Niharika
>
> On Wed, Feb 15, 2012 at 10:58 AM, Rish <[hidden email]> wrote:
> > Yes. We want to handle it pretty much within the execution itself. It
> > does that decently so far. However, the reporting is a bit of a
> > problem.
>
> > On Feb 15, 10:26 am, niharika varshney <[hidden email]>
> > wrote:
> > > When you say without referring to the xml, do you imply that you do not
> > > want to use the testng-failed xml is it?
>
> > > Regards,
> > > Niharika
>
> > > On Wed, Feb 15, 2012 at 10:31 AM, Rish <[hidden email]>
> > wrote:
> > > > Hi All,
>
> > > > Not sure if this question has been answered already on the forums. We
> > > > have a suite of approx 250 UI test cases that runs overnight as a
> > > > Jenkins job, and provides us the result as we check into work. A lot
> > > > of UI failures are minor blips in system/network, and the team spend a
> > > > lot of times investigating these 'false' alarms.
>
> > > >  To counter the same, we added a custom listener that would retry a
> > > > failed test a set number of times before marking it as a 'failure'. If
> > > > it passes beforehand, it simply marks the earlier failed invocations
> > > > as 'skipped'. Also, we wrote a custom reporter, which extends the
> > > > default HTMLReporter class and added a method that looked like this:
>
> > > > private IResultMap removeIncorrectlyFailedTests(ITestContext test)
> > > > {
> > > >  List<ITestNGMethod> failsToRemove = new ArrayList<ITestNGMethod>();
> > > >  IResultMap returnValue = test.getFailedTests();
>
> > > >  for(ITestResult result : test.getFailedTests().getAllResults())
> > > >  {
> > > >    long failedResultTime = result.getEndMillis();
>
> > > >    for(ITestResult resultToCheck :
> > > > test.getSkippedTests().getAllResults())
> > > >    {
> > > >        if(failedResultTime == resultToCheck.getEndMillis())
> > > >        {
> > > >            failsToRemove.add(resultToCheck.getMethod());
> > > >            break;
> > > >        }
> > > >    }
>
> > > >    for(ITestResult resultToCheck :
> > > > test.getPassedTests().getAllResults())
> > > >    {
> > > >        if(failedResultTime == resultToCheck.getEndMillis())
> > > >        {
> > > >            failsToRemove.add(resultToCheck.getMethod());
> > > >            break;
> > > >        }
> > > >    }
> > > >  }
>
> > > >  for(ITestNGMethod method : failsToRemove)
> > > >  {
> > > >      returnValue.removeResult(method);
> > > >  }
>
> > > >  return returnValue;
> > > > }
>
> > > > This method essentially removes the duplicate entries while retaining
> > > > all other ones. The invocation point for this method was:
>
> > > > @Override
> > > >  public void onStart(ITestContext context) {
>
> > > >      //added this line to ensure unnecessary results are removed
> > > >      removeIncorrectlyFailedTests(m_testContext);
> > > >      m_testContext = context;
> > > >  }
>
> > > > The relevant section of the pom.xml looks like:
>
> > > >         <properties>
> > > >            <property>
> > > >              <name>usedefaultlisteners</name>
> > > >              <value>false</value>
> > > >            </property>
> > > >            <property>
> > > >              <name>listener</name>
> > > >              <value>com.inmobi.experiments.RetryTestListener</value>
> > > >            </property>
> > > >            <property>
> > > >              <name>reporter</name>
> > > >              <value>com.inmobi.experiments.CustomHTMLReporter</value>
> > > >            </property>
> > > >          </properties>
>
> > > > However, on running our test cases, and viewing the report, it seems
> > > > that the CustomHTMLReporter is not being utilized by testng for report
> > > > generation.
>
> > > > This is how the cmd-line looks like post-execution of a sample test
> > > > case:
>
> > > > Tests run: 4, Failures: 1, Errors: 0, Skipped: 3, Time elapsed: 0.191
> > > > sec <<< FAILURE!
>
> > > > and the surefire-reports folder is all empty :(
>
> > > > It would be really helpful if someone could point out where I am going
> > > > wrong. Also, is there a cleaner way to automatically run failed tests
> > > > without doing the above exercise (without referring to the xml )?
>
> > > > Apologies if this has already been answered on the forums, but I could
> > > > not spot something that would help.
>
> > > > Sincere Thanks,
> > > > Rishu
>
> > > > --
> > > > You received this message because you are subscribed to the Google
> > Groups
> > > > "testng-users" group.
> > > > To post to this group, send email to [hidden email].
> > > > To unsubscribe from this group, send email to
> > > > [hidden email].
> > > > For more options, visit this group at
> > > >http://groups.google.com/group/testng-users?hl=en.
>
> > --
> > You received this message because you are subscribed to the Google Groups
> > "testng-users" group.
> > To post to this group, send email to [hidden email].
> > To unsubscribe from this group, send email to
> > [hidden email].
> > For more options, visit this group at
> >http://groups.google.com/group/testng-users?hl=en.

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.

Reply | Threaded
Open this post in threaded view
|

Re: Retrying failed test cases automatically via maven/custom listeners

Rish-7
Ok...checked post debugging...There is some small problem with the
logic. However, that notwithstanding, why am i not getting the default
emailable report? Do i have to enable the default listeners too ?

Thanks,
Rishu

On Feb 15, 12:35 pm, Rish <[hidden email]> wrote:

> Did that. Here are the observations:
>
> - The cmd-line option remains the same:
>
> Tests run: 4, Failures: 1, Errors: 0, Skipped: 3, Time elapsed: 0.2
> sec <<< FAILURE!
>
> - The target folder just has 1 extra dir created as 'Command line
> suite' which has an html file. The file shows all 4 runs of the test
> case which is failing (1 failed, 3 skipped). As per the desired
> output, it should have displayed only 1.
>
> Will try to debug the latter point and double-check if it is removing
> tests correctly.
>
> Thanks for the input.
> Regards,
> Rishu
>
> On Feb 15, 12:07 pm, niharika varshney <[hidden email]>
> wrote:
>
> > Hmm...Can you move your custom reporter into the listener section and
> > check..?
>
> > Regards,
> > Niharika
>
> > On Wed, Feb 15, 2012 at 10:58 AM, Rish <[hidden email]> wrote:
> > > Yes. We want to handle it pretty much within the execution itself. It
> > > does that decently so far. However, the reporting is a bit of a
> > > problem.
>
> > > On Feb 15, 10:26 am, niharika varshney <[hidden email]>
> > > wrote:
> > > > When you say without referring to the xml, do you imply that you do not
> > > > want to use the testng-failed xml is it?
>
> > > > Regards,
> > > > Niharika
>
> > > > On Wed, Feb 15, 2012 at 10:31 AM, Rish <[hidden email]>
> > > wrote:
> > > > > Hi All,
>
> > > > > Not sure if this question has been answered already on the forums. We
> > > > > have a suite of approx 250 UI test cases that runs overnight as a
> > > > > Jenkins job, and provides us the result as we check into work. A lot
> > > > > of UI failures are minor blips in system/network, and the team spend a
> > > > > lot of times investigating these 'false' alarms.
>
> > > > >  To counter the same, we added a custom listener that would retry a
> > > > > failed test a set number of times before marking it as a 'failure'. If
> > > > > it passes beforehand, it simply marks the earlier failed invocations
> > > > > as 'skipped'. Also, we wrote a custom reporter, which extends the
> > > > > default HTMLReporter class and added a method that looked like this:
>
> > > > > private IResultMap removeIncorrectlyFailedTests(ITestContext test)
> > > > > {
> > > > >  List<ITestNGMethod> failsToRemove = new ArrayList<ITestNGMethod>();
> > > > >  IResultMap returnValue = test.getFailedTests();
>
> > > > >  for(ITestResult result : test.getFailedTests().getAllResults())
> > > > >  {
> > > > >    long failedResultTime = result.getEndMillis();
>
> > > > >    for(ITestResult resultToCheck :
> > > > > test.getSkippedTests().getAllResults())
> > > > >    {
> > > > >        if(failedResultTime == resultToCheck.getEndMillis())
> > > > >        {
> > > > >            failsToRemove.add(resultToCheck.getMethod());
> > > > >            break;
> > > > >        }
> > > > >    }
>
> > > > >    for(ITestResult resultToCheck :
> > > > > test.getPassedTests().getAllResults())
> > > > >    {
> > > > >        if(failedResultTime == resultToCheck.getEndMillis())
> > > > >        {
> > > > >            failsToRemove.add(resultToCheck.getMethod());
> > > > >            break;
> > > > >        }
> > > > >    }
> > > > >  }
>
> > > > >  for(ITestNGMethod method : failsToRemove)
> > > > >  {
> > > > >      returnValue.removeResult(method);
> > > > >  }
>
> > > > >  return returnValue;
> > > > > }
>
> > > > > This method essentially removes the duplicate entries while retaining
> > > > > all other ones. The invocation point for this method was:
>
> > > > > @Override
> > > > >  public void onStart(ITestContext context) {
>
> > > > >      //added this line to ensure unnecessary results are removed
> > > > >      removeIncorrectlyFailedTests(m_testContext);
> > > > >      m_testContext = context;
> > > > >  }
>
> > > > > The relevant section of the pom.xml looks like:
>
> > > > >         <properties>
> > > > >            <property>
> > > > >              <name>usedefaultlisteners</name>
> > > > >              <value>false</value>
> > > > >            </property>
> > > > >            <property>
> > > > >              <name>listener</name>
> > > > >              <value>com.inmobi.experiments.RetryTestListener</value>
> > > > >            </property>
> > > > >            <property>
> > > > >              <name>reporter</name>
> > > > >              <value>com.inmobi.experiments.CustomHTMLReporter</value>
> > > > >            </property>
> > > > >          </properties>
>
> > > > > However, on running our test cases, and viewing the report, it seems
> > > > > that the CustomHTMLReporter is not being utilized by testng for report
> > > > > generation.
>
> > > > > This is how the cmd-line looks like post-execution of a sample test
> > > > > case:
>
> > > > > Tests run: 4, Failures: 1, Errors: 0, Skipped: 3, Time elapsed: 0.191
> > > > > sec <<< FAILURE!
>
> > > > > and the surefire-reports folder is all empty :(
>
> > > > > It would be really helpful if someone could point out where I am going
> > > > > wrong. Also, is there a cleaner way to automatically run failed tests
> > > > > without doing the above exercise (without referring to the xml )?
>
> > > > > Apologies if this has already been answered on the forums, but I could
> > > > > not spot something that would help.
>
> > > > > Sincere Thanks,
> > > > > Rishu
>
> > > > > --
> > > > > You received this message because you are subscribed to the Google
> > > Groups
> > > > > "testng-users" group.
> > > > > To post to this group, send email to [hidden email].
> > > > > To unsubscribe from this group, send email to
> > > > > [hidden email].
> > > > > For more options, visit this group at
> > > > >http://groups.google.com/group/testng-users?hl=en.
>
> > > --
> > > You received this message because you are subscribed to the Google Groups
> > > "testng-users" group.
> > > To post to this group, send email to [hidden email].
> > > To unsubscribe from this group, send email to
> > > [hidden email].
> > > For more options, visit this group at
> > >http://groups.google.com/group/testng-users?hl=en.

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.

Reply | Threaded
Open this post in threaded view
|

Re: Retrying failed test cases automatically via maven/custom listeners

韩锐
I am actually trying to implement similar logic in side our test framework. We use the customized inside testng.xml not in the pom.xml if that could make any difference. 

In addition, I notice your xml you have set the defaultlisteners to be false, that actually turned off the default testng report. because testng report is a listener, the same goes to reportng if you are using it.

Henry


On Wednesday, 15 February 2012 07:45:06 UTC-8, Rish wrote:
Ok...checked post debugging...There is some small problem with the
logic. However, that notwithstanding, why am i not getting the default
emailable report? Do i have to enable the default listeners too ?

Thanks,
Rishu

On Feb 15, 12:35 pm, Rish <[hidden email]> wrote:

> Did that. Here are the observations:
>
> - The cmd-line option remains the same:
>
> Tests run: 4, Failures: 1, Errors: 0, Skipped: 3, Time elapsed: 0.2
> sec <<< FAILURE!
>
> - The target folder just has 1 extra dir created as 'Command line
> suite' which has an html file. The file shows all 4 runs of the test
> case which is failing (1 failed, 3 skipped). As per the desired
> output, it should have displayed only 1.
>
> Will try to debug the latter point and double-check if it is removing
> tests correctly.
>
> Thanks for the input.
> Regards,
> Rishu
>
> On Feb 15, 12:07 pm, niharika varshney <[hidden email]>
> wrote:
>
> > Hmm...Can you move your custom reporter into the listener section and
> > check..?
>
> > Regards,
> > Niharika
>
> > On Wed, Feb 15, 2012 at 10:58 AM, Rish <[hidden email]> wrote:
> > > Yes. We want to handle it pretty much within the execution itself. It
> > > does that decently so far. However, the reporting is a bit of a
> > > problem.
>
> > > On Feb 15, 10:26 am, niharika varshney <[hidden email]>
> > > wrote:
> > > > When you say without referring to the xml, do you imply that you do not
> > > > want to use the testng-failed xml is it?
>
> > > > Regards,
> > > > Niharika
>
> > > > On Wed, Feb 15, 2012 at 10:31 AM, Rish <[hidden email]>
> > > wrote:
> > > > > Hi All,
>
> > > > > Not sure if this question has been answered already on the forums. We
> > > > > have a suite of approx 250 UI test cases that runs overnight as a
> > > > > Jenkins job, and provides us the result as we check into work. A lot
> > > > > of UI failures are minor blips in system/network, and the team spend a
> > > > > lot of times investigating these 'false' alarms.
>
> > > > >  To counter the same, we added a custom listener that would retry a
> > > > > failed test a set number of times before marking it as a 'failure'. If
> > > > > it passes beforehand, it simply marks the earlier failed invocations
> > > > > as 'skipped'. Also, we wrote a custom reporter, which extends the
> > > > > default HTMLReporter class and added a method that looked like this:
>
> > > > > private IResultMap removeIncorrectlyFailedTests(ITestContext test)
> > > > > {
> > > > >  List<ITestNGMethod> failsToRemove = new ArrayList<ITestNGMethod>();
> > > > >  IResultMap returnValue = test.getFailedTests();
>
> > > > >  for(ITestResult result : test.getFailedTests().getAllResults())
> > > > >  {
> > > > >    long failedResultTime = result.getEndMillis();
>
> > > > >    for(ITestResult resultToCheck :
> > > > > test.getSkippedTests().getAllResults())
> > > > >    {
> > > > >        if(failedResultTime == resultToCheck.getEndMillis())
> > > > >        {
> > > > >            failsToRemove.add(resultToCheck.getMethod());
> > > > >            break;
> > > > >        }
> > > > >    }
>
> > > > >    for(ITestResult resultToCheck :
> > > > > test.getPassedTests().getAllResults())
> > > > >    {
> > > > >        if(failedResultTime == resultToCheck.getEndMillis())
> > > > >        {
> > > > >            failsToRemove.add(resultToCheck.getMethod());
> > > > >            break;
> > > > >        }
> > > > >    }
> > > > >  }
>
> > > > >  for(ITestNGMethod method : failsToRemove)
> > > > >  {
> > > > >      returnValue.removeResult(method);
> > > > >  }
>
> > > > >  return returnValue;
> > > > > }
>
> > > > > This method essentially removes the duplicate entries while retaining
> > > > > all other ones. The invocation point for this method was:
>
> > > > > @Override
> > > > >  public void onStart(ITestContext context) {
>
> > > > >      //added this line to ensure unnecessary results are removed
> > > > >      removeIncorrectlyFailedTests(m_testContext);
> > > > >      m_testContext = context;
> > > > >  }
>
> > > > > The relevant section of the pom.xml looks like:
>
> > > > >         <properties>
> > > > >            <property>
> > > > >              <name>usedefaultlisteners</name>
> > > > >              <value>false</value>
> > > > >            </property>
> > > > >            <property>
> > > > >              <name>listener</name>
> > > > >              <value>com.inmobi.experiments.RetryTestListener</value>
> > > > >            </property>
> > > > >            <property>
> > > > >              <name>reporter</name>
> > > > >              <value>com.inmobi.experiments.CustomHTMLReporter</value>
> > > > >            </property>
> > > > >          </properties>
>
> > > > > However, on running our test cases, and viewing the report, it seems
> > > > > that the CustomHTMLReporter is not being utilized by testng for report
> > > > > generation.
>
> > > > > This is how the cmd-line looks like post-execution of a sample test
> > > > > case:
>
> > > > > Tests run: 4, Failures: 1, Errors: 0, Skipped: 3, Time elapsed: 0.191
> > > > > sec <<< FAILURE!
>
> > > > > and the surefire-reports folder is all empty :(
>
> > > > > It would be really helpful if someone could point out where I am going
> > > > > wrong. Also, is there a cleaner way to automatically run failed tests
> > > > > without doing the above exercise (without referring to the xml )?
>
> > > > > Apologies if this has already been answered on the forums, but I could
> > > > > not spot something that would help.
>
> > > > > Sincere Thanks,
> > > > > Rishu
>
> > > > > --
> > > > > You received this message because you are subscribed to the Google
> > > Groups
> > > > > "testng-users" group.
> > > > > To post to this group, send email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="0BSYZC0Bnc0J">testng...@....
> > > > > To unsubscribe from this group, send email to
> > > > > <a href="javascript:" target="_blank" gdf-obfuscated-mailto="0BSYZC0Bnc0J">testng-users...@googlegroups.com.
> > > > > For more options, visit this group at
> > > > >http://groups.google.com/group/testng-users?hl=en.
>
> > > --
> > > You received this message because you are subscribed to the Google Groups
> > > "testng-users" group.
> > > To post to this group, send email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="0BSYZC0Bnc0J">testng...@....
> > > To unsubscribe from this group, send email to
> > > <a href="javascript:" target="_blank" gdf-obfuscated-mailto="0BSYZC0Bnc0J">testng-users...@googlegroups.com.
> > > For more options, visit this group at
> > >http://groups.google.com/group/testng-users?hl=en.

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at http://groups.google.com/group/testng-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|

Re: Retrying failed test cases automatically via maven/custom listeners

deepankur
This post has NOT been accepted by the mailing list yet.
In reply to this post by Rish-7
Hi Rishu,

I have the same problem. Hope you could help. I have classes( listeners) which run my tests "n" number of times before leaving them as failed tests. But, all these executions appear in my html report. I would like to change them to either skipped or removed or something custom. I was really excited looking at your solution but I think I have stumbled upon something here, see if you can help please.

Your two methods "removeIncorrectlyFailedTests(ITestContext test)" and "onStart(ITestContext context)"
- are both of these methods in the same class
- which class this class extends
- which interface this class implements#
- In the end you have a comment that there are some changes you made to the logic, would you be kind enough to share them.


Specifically, my implementation doesn't seem to like "onStart(ITestContext context)". After trying for a day, I thought it's best to ask you.

Thanks

Deepankur Tyagi


Reply | Threaded
Open this post in threaded view
|

Re: Retrying failed test cases automatically via maven/custom listeners

deepankur
In reply to this post by Rish-7
Hi Rishu,

I have the same problem. Hope you could help. I have classes( listeners) which run my tests "n" number of times before leaving them as failed tests. But, all these executions appear in my html report. I would like to change them to either skipped or removed or something custom. I was really excited looking at your solution but I think I have stumbled upon something here, see if you can help please.

Your two methods "removeIncorrectlyFailedTests(ITestContext test)" and "onStart(ITestContext context)"
- are both of these methods in the same class
- which class this class extends
- which interface this class implements#
- In the end you have a comment that there are some changes you made to the logic, would you be kind enough to share them.


Specifically, my implementation doesn't seem to like "onStart(ITestContext context)". After trying for a day, I thought it's best to ask you.

Thanks

Deepankur Tyagi