Running test despite the @AfterMethod failure

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

Running test despite the @AfterMethod failure

Jiri Mares-3

Hi,

I have tests using easymock. Beacause I don't want to put into each
test method the verify... I have common superclass to all tests that
has method verify with annotation @AfterMethod.

Everything is working well until the test are running. When there is
test when the verify method throws exception all following tests are
skipped.

And my question is: is there any way how to force TestNG to continue
in running following tests (alwaysRun doesn't help).

Jirka


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: Running test despite the @AfterMethod failure

Cédric Beust ♔
Can you post your code?

--
Cedric


On 3/2/07, Jiri Mares <[hidden email]> wrote:

Hi,

I have tests using easymock. Beacause I don't want to put into each
test method the verify... I have common superclass to all tests that
has method verify with annotation @AfterMethod.

Everything is working well until the test are running. When there is
test when the verify method throws exception all following tests are
skipped.

And my question is: is there any way how to force TestNG to continue
in running following tests (alwaysRun doesn't help).

Jirka

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: Running test despite the @AfterMethod failure

Jiri Mares-3

Here it is (very simplified, even all code is in one class):

package cz.svt.ce;

import java.lang.reflect.Method;

import org.easymock.EasyMock;
import org.testng.Assert;
import org.testng.annotations.AfterMethod;
import org.testng.annotations.BeforeClass;
import org.testng.annotations.BeforeMethod;
import org.testng.annotations.Test;

public class NGTest {
   protected A a;
   protected B b;

   @Test
   public void testA() {
      EasyMock.expect(a.a(1)).andReturn(2);
      EasyMock.expect(a.b(4)).andReturn(9); //this expect is never
fullfilled
      EasyMock.replay(a);

      Assert.assertEquals(b.a(1), 2);
   }

   @Test
   public void testB() {
      EasyMock.expect(a.a(1)).andReturn(2); //this expect is never
fullfilled
      EasyMock.expect(a.b(4)).andReturn(9);
      EasyMock.replay(a);

      Assert.assertEquals(b.b(4), 9);
   }

   @BeforeMethod(alwaysRun = true)
   protected void resetEasyMock(Method method) {
      EasyMock.reset(a);
   }

   @BeforeClass
   protected void initClass() {
      b = new B();
      b.setIA(a = EasyMock.createStrictMock(A.class));
   }

   @AfterMethod(alwaysRun = false)
   protected void verifyEasyMock(Method method) {
      EasyMock.verify(a);
   }

   public static interface A {
      int a(int a);

      int b(int b);
   }

   public static class B {
      protected A iA;

      public int a(int a) {
         return iA.a(a);
      }

      public int b(int b) {
         return iA.b(b);
      }

      public void setIA(A ia) {
         iA = ia;
      }

   }

}

The second test is never run because the verify method throws error.


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: Running test despite the @AfterMethod failure

Alexandru Popescu ☀

On 3/2/07, Jiri Mares <[hidden email]> wrote:

>
> Here it is (very simplified, even all code is in one class):
>
> package cz.svt.ce;
>
> import java.lang.reflect.Method;
>
> import org.easymock.EasyMock;
> import org.testng.Assert;
> import org.testng.annotations.AfterMethod;
> import org.testng.annotations.BeforeClass;
> import org.testng.annotations.BeforeMethod;
> import org.testng.annotations.Test;
>
> public class NGTest {
>    protected A a;
>    protected B b;
>
>    @Test
>    public void testA() {
>       EasyMock.expect(a.a(1)).andReturn(2);
>       EasyMock.expect(a.b(4)).andReturn(9); //this expect is never
> fullfilled
>       EasyMock.replay(a);
>
>       Assert.assertEquals(b.a(1), 2);
>    }
>
>    @Test
>    public void testB() {
>       EasyMock.expect(a.a(1)).andReturn(2); //this expect is never
> fullfilled
>       EasyMock.expect(a.b(4)).andReturn(9);
>       EasyMock.replay(a);
>
>       Assert.assertEquals(b.b(4), 9);
>    }
>
>    @BeforeMethod(alwaysRun = true)
>    protected void resetEasyMock(Method method) {
>       EasyMock.reset(a);
>    }
>
>    @BeforeClass
>    protected void initClass() {
>       b = new B();
>       b.setIA(a = EasyMock.createStrictMock(A.class));
>    }
>
>    @AfterMethod(alwaysRun = false)
>    protected void verifyEasyMock(Method method) {
>       EasyMock.verify(a);
>    }
>
>    public static interface A {
>       int a(int a);
>
>       int b(int b);
>    }
>
>    public static class B {
>       protected A iA;
>
>       public int a(int a) {
>          return iA.a(a);
>       }
>
>       public int b(int b) {
>          return iA.b(b);
>       }
>
>       public void setIA(A ia) {
>          iA = ia;
>       }
>
>    }
>
> }
>
> The second test is never run because the verify method throws error.
>
>

Hi Jiri!

This is the desgined behavior of configuration methods
(@Before/@After). The rationale behind it: most of the time a failure
in a configuration method is leaving the system in an unknown state
that may lead to completely unpredictable results (I agree that in
this particular case this will not happen, but f.e. moving the
mock.release call after mock.verify will lead to all methods failing).
Another aspect of this is that if 2 test methods are gonna fail, then
your report will be useless.

toodle oo,

./alex
--
.w( the_mindstorm )p.
  TestNG co-founder
EclipseTestNG Creator

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: Running test despite the @AfterMethod failure

Jiri Mares-3

Hi,

I know it is designed behaviour, I have been asking whether is there
any way how to change this behaviour. Or have it mean as request for
change.

At the moment the only way is to move the call to verify method to all
test methods and don't forget to put it there. This cause the code is
not as easy maintainable as it could be and also it is more error
prone. Or to leave it in the @AfterMethod method, and when one test is
wrong all following tests are skipped :-((

I can easily imagine adding the annotation @VerifyMethod (or parameter
verify="true" to @AfterMethod annotation) that will be called
immediately after the @Test method and verifies that everything went
well. And when this method goes wrong the test will continue. This is
suitable especially when using mock objects ...

Finally I don't understand your sentence: "Another aspect of this is
that if 2 test methods are gonna fail, then your report will be
useless."

Jirka


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: Running test despite the @AfterMethod failure

Alexandru Popescu ☀

On 3/2/07, Jiri Mares <[hidden email]> wrote:

>
> Hi,
>
> I know it is designed behaviour, I have been asking whether is there
> any way how to change this behaviour. Or have it mean as request for
> change.
>
> At the moment the only way is to move the call to verify method to all
> test methods and don't forget to put it there. This cause the code is
> not as easy maintainable as it could be and also it is more error
> prone. Or to leave it in the @AfterMethod method, and when one test is
> wrong all following tests are skipped :-((
>
> I can easily imagine adding the annotation @VerifyMethod (or parameter
> verify="true" to @AfterMethod annotation) that will be called
> immediately after the @Test method and verifies that everything went
> well. And when this method goes wrong the test will continue. This is
> suitable especially when using mock objects ...
>

Sorry for not making it clear enough. I agree with you that there are
no good options for you atm. Unfortunately, I don't think that the
annotation proposal or the attribute are clear enough and would have a
place in TestNG generic annotations. But, probably the additional of
an attribute to the @After annotations would make this behavior
possible -- something along continueAfterFailure or ... .

> Finally I don't understand your sentence: "Another aspect of this is
> that if 2 test methods are gonna fail, then your report will be
> useless."
>

Consider you have 3 @Test methods all of them being verified in an
@AfterMethod. Say 2 of them are failing. The report will tell you
about 2 failures of the @AfterMethod, but you will have no hints about
which of the 3 @Tests these failures belong to.

toodle oo

./alex
--
.w( the_mindstorm )p.
  TestNG co-founder
EclipseTestNG Creator

> Jirka
>
>
> >
>

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: Running test despite the @AfterMethod failure

Jiri Mares-3

Hi,

I thing the parameter to @AfterMethod annotation is very good
posibility ...

> Consider you have 3 @Test methods all of them being verified in an
> @AfterMethod. Say 2 of them are failing. The report will tell you
> about 2 failures of the @AfterMethod, but you will have no hints about
> which of the 3 @Tests these failures belong to.

Sorry, but you are not right, because to @AfterMethod method I can
pass parameter of type Method and this is populated during error
(contains the method after which was called).

Jirka


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: Running test despite the @AfterMethod failure

Alexandru Popescu ☀

On 3/6/07, Jiri Mares <[hidden email]> wrote:

>
> Hi,
>
> I thing the parameter to @AfterMethod annotation is very good
> posibility ...
>
> > Consider you have 3 @Test methods all of them being verified in an
> > @AfterMethod. Say 2 of them are failing. The report will tell you
> > about 2 failures of the @AfterMethod, but you will have no hints about
> > which of the 3 @Tests these failures belong to.
>
> Sorry, but you are not right, because to @AfterMethod method I can
> pass parameter of type Method and this is populated during error
> (contains the method after which was called).
>
> Jirka
>

... but for this scenario we will have to have a fix for all existing
reports... and that was what I initially meant.

./alex
--
.w( the_mindstorm )p.
  TestNG co-founder
EclipseTestNG Creator

>
> >
>

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: Running test despite the @AfterMethod failure

Jiri Mares-3

I Don't know, I'm using the eclipse plugin where the things are
working well. Alexandru, should I rise some feature request somewhere
or it make no sense?

Jirka


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: Running test despite the @AfterMethod failure

Alexandru Popescu ☀

On 3/6/07, Jiri Mares <[hidden email]> wrote:
>
> I Don't know, I'm using the eclipse plugin where the things are
> working well. Alexandru, should I rise some feature request somewhere
> or it make no sense?
>

Sure. Please open a JIRA Feature request. I will definitely think about it

./alex
--
.w( the_mindstorm )p.
  TestNG co-founder
EclipseTestNG Creator

> Jirka
>
>
> >
>

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: Running test despite the @AfterMethod failure

Jiri Mares-3

Opened ... I hope you find solution ...

Jirka


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en
-~----------~----~----~----~------~----~------~--~---