Exception in tearDown skips further tests

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Exception in tearDown skips further tests

tarun3kumar

Hi,

I've been porting a large test codebase from Junit to TestNG..I have a problem relating to the skipping of tests.

Many of our tests are designed so that an @AfterMethod tearDown() verifies that no illegal "garbage" was created in the database by the actual test. So a test should fail if the test method itself fails, or if it creates garbage in the database.

The problem is, that if the @AfterMethod detects an error and throws an exception, all subsequent testmethods will be skipped. Setting alwaysRun=true on the test methods and on the @AfterMethod doesn't seem to work. Consider the following example to trigger the behavior:

import org.testng.annotations.AfterMethod;
import org.testng.annotations.Test;

public class SkipTearDownNGTest
{
                private int m_counter = 0;
                private int m_failNumber = 3;
               
                @AfterMethod(alwaysRun = true)
                public void tearDown()
                {
                        log("tearDown called");
                        checkForGarbageInDatabase();
                }
               
                private void checkForGarbageInDatabase()
                {
                        m_counter++;
                        if (m_counter == m_failNumber)
                        {
                                log("Failing on counter " + m_counter);
                                throw new RuntimeException("The test produced garbage in database.");
                        }
                }
               
                @Test(alwaysRun = true)
                public void test1() { log("test1 called"); }

                @Test(alwaysRun = true)
                public void test2() { log("test2 called"); }

                @Test(alwaysRun = true)
                public void test3() { log("test3 called"); }

                @Test(alwaysRun = true)
                public void test4() { log("test4 called"); }

                @Test(alwaysRun = true)
                public void test5() { log("test5 called"); }

                private void log(String message)
                {
                        System.out.println(message);
                }
}


Now, I think I understand why this happens - all test() methods are depending on the tearDown method. If it fails, they should be skipped. The problem is, this differs from how Junit work, and the way our old tests are written.

Is there any way to get the desired behavior? That the tearDown method should mark the test as failed, but without leading to the other tests getting skipped?

Cheers,
Bjorn
---------------------------------------------------------------------
Posted via Jive Forums
http://forums.opensymphony.com/thread.jspa?threadID=57606&messageID=113604#113604


--~--~---------~--~----~------------~-------~--~----~
 You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: Exception in tearDown skips further tests

Cédric Beust ♔
Hi Bjorn,

I think the problem comes from the fact that no test methods depend on an @After method.

If you put the checkForGarbageInDatabase() call at the very beginning of one of your @Before methods, you should see the behavior you are expecting (all the tests that should be invoked after this method will be skipped if it throws an exception).

Can you try that and get back to us?

--
Cedric


On 1/9/07, Bjorn M <[hidden email]> wrote:

Hi,

I've been porting a large test codebase from Junit to TestNG..I have a problem relating to the skipping of tests.

Many of our tests are designed so that an @AfterMethod tearDown() verifies that no illegal "garbage" was created in the database by the actual test. So a test should fail if the test method itself fails, or if it creates garbage in the database.

The problem is, that if the @AfterMethod detects an error and throws an exception, all subsequent testmethods will be skipped. Setting alwaysRun=true on the test methods and on the @AfterMethod doesn't seem to work. Consider the following example to trigger the behavior:

import org.testng.annotations.AfterMethod;
import org.testng.annotations.Test;

public class SkipTearDownNGTest
{
                private int m_counter = 0;
                private int m_failNumber = 3;

                @AfterMethod(alwaysRun = true)
                public void tearDown()
                {
                        log("tearDown called");
                        checkForGarbageInDatabase();
                }

                private void checkForGarbageInDatabase()
                {
                        m_counter++;
                        if (m_counter == m_failNumber)
                        {
                                log("Failing on counter " + m_counter);
                                throw new RuntimeException("The test produced garbage in database.");
                        }
                }

                @Test(alwaysRun = true)
                public void test1() { log("test1 called"); }

                @Test(alwaysRun = true)
                public void test2() { log("test2 called"); }

                @Test(alwaysRun = true)
                public void test3() { log("test3 called"); }

                @Test(alwaysRun = true)
                public void test4() { log("test4 called"); }

                @Test(alwaysRun = true)
                public void test5() { log("test5 called"); }

                private void log(String message)
                {
                         System.out.println(message);
                }
}


Now, I think I understand why this happens - all test() methods are depending on the tearDown method. If it fails, they should be skipped. The problem is, this differs from how Junit work, and the way our old tests are written.

Is there any way to get the desired behavior? That the tearDown method should mark the test as failed, but without leading to the other tests getting skipped?

Cheers,
Bjorn
---------------------------------------------------------------------
Posted via Jive Forums
http://forums.opensymphony.com/thread.jspa?threadID=57606&messageID=113604#113604


--
Cédric
--~--~---------~--~----~------------~-------~--~----~
 You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: Exception in tearDown skips further tests

Alexandru Popescu ☀
On 1/9/07, Cédric Beust ♔ <[hidden email]> wrote:

> Hi Bjorn,
>
> I think the problem comes from the fact that no test methods depend on an
> @After method.
>
> If you put the checkForGarbageInDatabase() call at the very beginning of one
> of your @Before methods, you should see the behavior you are expecting (all
> the tests that should be invoked after this method will be skipped if it
> throws an exception).
>
> Can you try that and get back to us?
>

As far as I understand from Bjorn's original post is that he doesn't
want this behavior (if an @AfterMethod fail the next @Tests _are_
skipped) and he would like the test to continue.
So far I haven't figured out a why to trick this behavior - which imo
is very correct: if the cleanup is not correct, then the next tests
may fail due to this, so let's skip them - but I will try to figure
out if there is a way to get it working.

./alex
--
.w( the_mindstorm )p.
  TestNG co-founder
EclipseTestNG Creator

> --
> Cedric
>
>
>
> On 1/9/07, Bjorn M <[hidden email] > wrote:
> >
> >
> > Hi,
> >
> > I've been porting a large test codebase from Junit to TestNG..I have a
> problem relating to the skipping of tests.
> >
> > Many of our tests are designed so that an @AfterMethod tearDown() verifies
> that no illegal "garbage" was created in the database by the actual test. So
> a test should fail if the test method itself fails, or if it creates garbage
> in the database.
> >
> > The problem is, that if the @AfterMethod detects an error and throws an
> exception, all subsequent testmethods will be skipped. Setting
> alwaysRun=true on the test methods and on the @AfterMethod doesn't seem to
> work. Consider the following example to trigger the behavior:
> >
> > import org.testng.annotations.AfterMethod;
> > import org.testng.annotations.Test;
> >
> > public class SkipTearDownNGTest
> > {
> >                 private int m_counter = 0;
> >                 private int m_failNumber = 3;
> >
> >                 @AfterMethod(alwaysRun = true)
> >                 public void tearDown()
> >                 {
> >                         log("tearDown called");
> >                         checkForGarbageInDatabase();
> >                 }
> >
> >                 private void checkForGarbageInDatabase()
> >                 {
> >                         m_counter++;
> >                         if (m_counter == m_failNumber)
> >                         {
> >                                 log("Failing on counter "
> + m_counter);
> >                                 throw new
> RuntimeException("The test produced garbage in database.");
> >                         }
> >                 }
> >
> >                 @Test(alwaysRun = true)
> >                 public void test1() { log("test1 called"); }
> >
> >                 @Test(alwaysRun = true)
> >                 public void test2() { log("test2 called"); }
> >
> >                 @Test(alwaysRun = true)
> >                 public void test3() { log("test3 called"); }
> >
> >                 @Test(alwaysRun = true)
> >                 public void test4() { log("test4 called"); }
> >
> >                 @Test(alwaysRun = true)
> >                 public void test5() { log("test5 called"); }
> >
> >                 private void log(String message)
> >                 {
> >                          System.out.println(message);
> >                 }
> > }
> >
> >
> > Now, I think I understand why this happens - all test() methods are
> depending on the tearDown method. If it fails, they should be skipped. The
> problem is, this differs from how Junit work, and the way our old tests are
> written.
> >
> > Is there any way to get the desired behavior? That the tearDown method
> should mark the test as failed, but without leading to the other tests
> getting skipped?
> >
> > Cheers,
> > Bjorn
> >
> ---------------------------------------------------------------------
> > Posted via Jive Forums
> >
> http://forums.opensymphony.com/thread.jspa?threadID=57606&messageID=113604#113604
> >
> >
> > --
> > Cédric
> > > >
> >
>

--~--~---------~--~----~------------~-------~--~----~
 You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: Exception in tearDown skips further tests

tarun3kumar
In reply to this post by Cédric Beust ♔

Hi Cedric,

I'm afraid I want to achieve the opposite ;)

I _don't_ want exceptions in my @After method to lead to skips in subseqent test methods. Running the class in Junit gives the desired behavior:

test1 OK
test2 OK
test3 ERROR
test4 OK
test5 OK

This is what we want. But running it in TestNG gives:

test1 OK
test2 OK
test3 FAIL
test4 SKIP
test5 SKIP

I tried using the @Before method call as you hinted but it leads to skips.

Maybe an attribute on the @After or @Test tag saying neverSkip=true, stating that if the method fails, it should not lead to skipping dependant methods?

Cheers,
Bjorn
---------------------------------------------------------------------
Posted via Jive Forums
http://forums.opensymphony.com/thread.jspa?threadID=57606&messageID=113875#113875


--~--~---------~--~----~------------~-------~--~----~
 You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: Exception in tearDown skips further tests

cytoe
In reply to this post by Alexandru Popescu ☀
Hi, I also interested in this enhancement.

Thanks,
Ron
Alexandru Popescu-3 wrote
On 1/9/07, Cédric Beust ♔ <cbeust@google.com> wrote:
> Hi Bjorn,
>
> I think the problem comes from the fact that no test methods depend on an
> @After method.
>
> If you put the checkForGarbageInDatabase() call at the very beginning of one
> of your @Before methods, you should see the behavior you are expecting (all
> the tests that should be invoked after this method will be skipped if it
> throws an exception).
>
> Can you try that and get back to us?
>

As far as I understand from Bjorn's original post is that he doesn't
want this behavior (if an @AfterMethod fail the next @Tests _are_
skipped) and he would like the test to continue.
So far I haven't figured out a why to trick this behavior - which imo
is very correct: if the cleanup is not correct, then the next tests
may fail due to this, so let's skip them - but I will try to figure
out if there is a way to get it working.

./alex
--
.w( the_mindstorm )p.
  TestNG co-founder
EclipseTestNG Creator

> --
> Cedric
>
>
>
> On 1/9/07, Bjorn M <testng-users@opensymphony.com > wrote:
> >
> >
> > Hi,
> >
> > I've been porting a large test codebase from Junit to TestNG..I have a
> problem relating to the skipping of tests.
> >
> > Many of our tests are designed so that an @AfterMethod tearDown() verifies
> that no illegal "garbage" was created in the database by the actual test. So
> a test should fail if the test method itself fails, or if it creates garbage
> in the database.
> >
> > The problem is, that if the @AfterMethod detects an error and throws an
> exception, all subsequent testmethods will be skipped. Setting
> alwaysRun=true on the test methods and on the @AfterMethod doesn't seem to
> work. Consider the following example to trigger the behavior:
> >
> > import org.testng.annotations.AfterMethod;
> > import org.testng.annotations.Test;
> >
> > public class SkipTearDownNGTest
> > {
> >                 private int m_counter = 0;
> >                 private int m_failNumber = 3;
> >
> >                 @AfterMethod(alwaysRun = true)
> >                 public void tearDown()
> >                 {
> >                         log("tearDown called");
> >                         checkForGarbageInDatabase();
> >                 }
> >
> >                 private void checkForGarbageInDatabase()
> >                 {
> >                         m_counter++;
> >                         if (m_counter == m_failNumber)
> >                         {
> >                                 log("Failing on counter "
> + m_counter);
> >                                 throw new
> RuntimeException("The test produced garbage in database.");
> >                         }
> >                 }
> >
> >                 @Test(alwaysRun = true)
> >                 public void test1() { log("test1 called"); }
> >
> >                 @Test(alwaysRun = true)
> >                 public void test2() { log("test2 called"); }
> >
> >                 @Test(alwaysRun = true)
> >                 public void test3() { log("test3 called"); }
> >
> >                 @Test(alwaysRun = true)
> >                 public void test4() { log("test4 called"); }
> >
> >                 @Test(alwaysRun = true)
> >                 public void test5() { log("test5 called"); }
> >
> >                 private void log(String message)
> >                 {
> >                          System.out.println(message);
> >                 }
> > }
> >
> >
> > Now, I think I understand why this happens - all test() methods are
> depending on the tearDown method. If it fails, they should be skipped. The
> problem is, this differs from how Junit work, and the way our old tests are
> written.
> >
> > Is there any way to get the desired behavior? That the tearDown method
> should mark the test as failed, but without leading to the other tests
> getting skipped?
> >
> > Cheers,
> > Bjorn
> >
> ---------------------------------------------------------------------
> > Posted via Jive Forums
> >
> http://forums.opensymphony.com/thread.jspa?threadID=57606&messageID=113604#113604
> >
> >
> > --
> > Cédric
> > > >
> >
>

--~--~---------~--~----~------------~-------~--~----~
 You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to testng-users@googlegroups.com
To unsubscribe from this group, send email to testng-users-unsubscribe@googlegroups.com
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en
-~----------~----~----~----~------~----~------~--~---