Problem with @Factory, @BeforeClass failure and skipped tests

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Problem with @Factory, @BeforeClass failure and skipped tests

Jean-Francois Poilpret-2

Hello,

going further on my factory experiments (in order to proceed with
converting my JUnit3 tests to TestNG) I have discovered a strange
behavior that I believe is probably a bug. I am using TestNG 5.7 (from
maven2).

Let's consider the following snippet:

public class TestNGProblemFactoryTest
{
        @Factory static public Object[] create()
        {
                Object[] tests = new Object[5];
                for(int i = 0; i < tests.length; i++)
                {
                        tests[i] = new TestNGProblemTest(i);
                }
                return tests;
        }

        @Test(groups = {"utest"})
        static public class TestNGProblemTest implements ITest
        {
                public String getTestName()
                {
                        return "" + _num;
                }

                public TestNGProblemTest(int num)
                {
                        _num = num;
                }

                @BeforeClass public void init()
                {
                        System.out.println("init() #" + _num);
                        if (_num % 2 == 0)
                        {
                                fail("init() failed");
                        }
                }

                @Test public void test()
                {
                        System.out.println("test() #" + _num);
                }

                private final int _num;
        }
}

As you can see, on the 5 TestNGProblemTest instances created by the
create() factory method, 3 will have their @BeforeClass init() method
failing, while for the 2 other instances, that method will be OK.

Now if I run this test with TestNG, here is the results I get
(excerpted from testng-results.xml):

    test-method status="FAIL" signature="init()" name="init (4)"
    test-method status="SKIP" signature="init()" name="init (2)"
    test-method status="SKIP" signature="init()" name="init (0)"
    test-method status="SKIP" signature="test()" name="test (1)"
    test-method status="SKIP" signature="test()" name="test (4)"
    test-method status="SKIP" signature="init()" name="init (1)"
    test-method status="SKIP" signature="test()" name="test (2)"
    test-method status="SKIP" signature="test()" name="test (0)"
    test-method status="SKIP" signature="init()" name="init (3)"
    test-method status="SKIP" signature="test()" name="test (3)"

It turns out that TestNG as started executing instance #4 first, for
that instance the @BeforeMethod init() fails. This failure enforces
skipping test() of instance #4 (which is expected behavior) but also
skips ALL other methods of all other 4 instances, which is not
expected at all: @BeforeClass is supposed to work at instance level
(according to the doc), hence it should have an impact only on the
instance #4 on the example above, and TestNG should still try to run
at least init() on each instance and test() of all instances which
init() did not fail!

The expected result for the above snippet should look as follows
(maybe in a different order since we cannot determine the order in
which TestNG will run each instance methods, but the important info
below is the status value):

    test-method status="FAIL" signature="init()" name="init (4)"
    test-method status="SKIP" signature="test()" name="test (4)"
    test-method status="FAIL" signature="init()" name="init (2)"
    test-method status="SKIP" signature="test()" name="test (2)"
    test-method status="FAIL" signature="init()" name="init (0)"
    test-method status="SKIP" signature="test()" name="test (0)"
    test-method status="PASS" signature="test()" name="test (1)"
    test-method status="PASS" signature="init()" name="init (1)"
    test-method status="PASS" signature="init()" name="init (3)"
    test-method status="PASS" signature="test()" name="test (3)"

I.e. TestNG tries to run all init() methods, and test() only for
instances where init() has failed.

Is this a known problem (I could not find any JIRA for a similar
issue)?
Does TestNG 5.8 solves this issue? (I could not check since 5.8 is not
available on any maven2 repository, but the 5.8 changes don't seem to
cover this issue anyway).

This problem seems quite serious to me because my understanding is
that all factory-created instances of a test class are independent of
each other in terms of failure/skipping behavior.

Is there any workaround for this?

Cheerrs

Jean-Francois
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Problem with @Factory, @BeforeClass failure and skipped tests

Alexandru Popescu ☀

On Wed, Apr 9, 2008 at 4:02 PM, jfpoilpret <[hidden email]> wrote:

>
>  Hello,
>
>  going further on my factory experiments (in order to proceed with
>  converting my JUnit3 tests to TestNG) I have discovered a strange
>  behavior that I believe is probably a bug. I am using TestNG 5.7 (from
>  maven2).
>
>  Let's consider the following snippet:
>
>  public class TestNGProblemFactoryTest
>  {
>         @Factory static public Object[] create()
>         {
>                 Object[] tests = new Object[5];
>                 for(int i = 0; i < tests.length; i++)
>                 {
>                         tests[i] = new TestNGProblemTest(i);
>                 }
>                 return tests;
>         }
>
>         @Test(groups = {"utest"})
>         static public class TestNGProblemTest implements ITest
>         {
>                 public String   getTestName()
>                 {
>                         return "" + _num;
>                 }
>
>                 public TestNGProblemTest(int num)
>                 {
>                         _num = num;
>                 }
>
>                 @BeforeClass public void        init()
>                 {
>                         System.out.println("init() #" + _num);
>                         if (_num % 2 == 0)
>                         {
>                                 fail("init() failed");
>                         }
>                 }
>
>                 @Test public void       test()
>                 {
>                         System.out.println("test() #" + _num);
>                 }
>
>                 private final int       _num;
>         }
>  }
>
>  As you can see, on the 5 TestNGProblemTest instances created by the
>  create() factory method, 3 will have their @BeforeClass init() method
>  failing, while for the 2 other instances, that method will be OK.
>
>  Now if I run this test with TestNG, here is the results I get
>  (excerpted from testng-results.xml):
>
>     test-method status="FAIL" signature="init()" name="init (4)"
>     test-method status="SKIP" signature="init()" name="init (2)"
>     test-method status="SKIP" signature="init()" name="init (0)"
>     test-method status="SKIP" signature="test()" name="test (1)"
>     test-method status="SKIP" signature="test()" name="test (4)"
>     test-method status="SKIP" signature="init()" name="init (1)"
>     test-method status="SKIP" signature="test()" name="test (2)"
>     test-method status="SKIP" signature="test()" name="test (0)"
>     test-method status="SKIP" signature="init()" name="init (3)"
>     test-method status="SKIP" signature="test()" name="test (3)"
>
>  It turns out that TestNG as started executing instance #4 first, for
>  that instance the @BeforeMethod init() fails. This failure enforces
>  skipping test() of instance #4 (which is expected behavior) but also
>  skips ALL other methods of all other 4 instances, which is not
>  expected at all: @BeforeClass is supposed to work at instance level
>  (according to the doc), hence it should have an impact only on the
>  instance #4 on the example above, and TestNG should still try to run
>  at least init() on each instance and test() of all instances which
>  init() did not fail!
>
>  The expected result for the above snippet should look as follows
>  (maybe in a different order since we cannot determine the order in
>  which TestNG will run each instance methods, but the important info
>  below is the status value):
>
>     test-method status="FAIL" signature="init()" name="init (4)"
>     test-method status="SKIP" signature="test()" name="test (4)"
>     test-method status="FAIL" signature="init()" name="init (2)"
>     test-method status="SKIP" signature="test()" name="test (2)"
>     test-method status="FAIL" signature="init()" name="init (0)"
>     test-method status="SKIP" signature="test()" name="test (0)"
>     test-method status="PASS" signature="test()" name="test (1)"
>     test-method status="PASS" signature="init()" name="init (1)"
>     test-method status="PASS" signature="init()" name="init (3)"
>     test-method status="PASS" signature="test()" name="test (3)"
>
>  I.e. TestNG tries to run all init() methods, and test() only for
>  instances where init() has failed.
>
>  Is this a known problem (I could not find any JIRA for a similar
>  issue)?
>  Does TestNG 5.8 solves this issue? (I could not check since 5.8 is not
>  available on any maven2 repository, but the 5.8 changes don't seem to
>  cover this issue anyway).
>
>  This problem seems quite serious to me because my understanding is
>  that all factory-created instances of a test class are independent of
>  each other in terms of failure/skipping behavior.
>
>  Is there any workaround for this?
>
>  Cheerrs
>
>  Jean-Francois

As far as I know this is a limitation of the current versions.

./alex
--
.w( the_mindstorm )p.
  Alexandru Popescu
  TestNG co-founder
EclipseTestNG Creator

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Problem with @Factory, @BeforeClass failure and skipped tests

Jean-Francois Poilpret-2

Is there any plan to fix this some day?
Without such a fix, this makes the whole existence of @Factory quite
useless (except in the case where all tests run perfectly of course,
which is the ideal case, but not always possible, in particular at the
times those tests are first written or later extended).

Jean-Francois

On Apr 9, 11:17 pm, "Alexandru Popescu ☀"
<[hidden email]> wrote:

> On Wed, Apr 9, 2008 at 4:02 PM, jfpoilpret <[hidden email]> wrote:
>
> >  Hello,
>
> >  going further on my factory experiments (in order to proceed with
> >  converting my JUnit3 tests to TestNG) I have discovered a strange
> >  behavior that I believe is probably a bug. I am using TestNG 5.7 (from
> >  maven2).
>
> >  Let's consider the following snippet:
>
> >  public class TestNGProblemFactoryTest
> >  {
> >         @Factory static public Object[] create()
> >         {
> >                 Object[] tests = new Object[5];
> >                 for(int i = 0; i < tests.length; i++)
> >                 {
> >                         tests[i] = new TestNGProblemTest(i);
> >                 }
> >                 return tests;
> >         }
>
> >         @Test(groups = {"utest"})
> >         static public class TestNGProblemTest implements ITest
> >         {
> >                 public String   getTestName()
> >                 {
> >                         return "" + _num;
> >                 }
>
> >                 public TestNGProblemTest(int num)
> >                 {
> >                         _num = num;
> >                 }
>
> >                 @BeforeClass public void        init()
> >                 {
> >                         System.out.println("init() #" + _num);
> >                         if (_num % 2 == 0)
> >                         {
> >                                 fail("init() failed");
> >                         }
> >                 }
>
> >                 @Test public void       test()
> >                 {
> >                         System.out.println("test() #" + _num);
> >                 }
>
> >                 private final int       _num;
> >         }
> >  }
>
> >  As you can see, on the 5 TestNGProblemTest instances created by the
> >  create() factory method, 3 will have their @BeforeClass init() method
> >  failing, while for the 2 other instances, that method will be OK.
>
> >  Now if I run this test with TestNG, here is the results I get
> >  (excerpted from testng-results.xml):
>
> >     test-method status="FAIL" signature="init()" name="init (4)"
> >     test-method status="SKIP" signature="init()" name="init (2)"
> >     test-method status="SKIP" signature="init()" name="init (0)"
> >     test-method status="SKIP" signature="test()" name="test (1)"
> >     test-method status="SKIP" signature="test()" name="test (4)"
> >     test-method status="SKIP" signature="init()" name="init (1)"
> >     test-method status="SKIP" signature="test()" name="test (2)"
> >     test-method status="SKIP" signature="test()" name="test (0)"
> >     test-method status="SKIP" signature="init()" name="init (3)"
> >     test-method status="SKIP" signature="test()" name="test (3)"
>
> >  It turns out that TestNG as started executing instance #4 first, for
> >  that instance the @BeforeMethod init() fails. This failure enforces
> >  skipping test() of instance #4 (which is expected behavior) but also
> >  skips ALL other methods of all other 4 instances, which is not
> >  expected at all: @BeforeClass is supposed to work at instance level
> >  (according to the doc), hence it should have an impact only on the
> >  instance #4 on the example above, and TestNG should still try to run
> >  at least init() on each instance and test() of all instances which
> >  init() did not fail!
>
> >  The expected result for the above snippet should look as follows
> >  (maybe in a different order since we cannot determine the order in
> >  which TestNG will run each instance methods, but the important info
> >  below is the status value):
>
> >     test-method status="FAIL" signature="init()" name="init (4)"
> >     test-method status="SKIP" signature="test()" name="test (4)"
> >     test-method status="FAIL" signature="init()" name="init (2)"
> >     test-method status="SKIP" signature="test()" name="test (2)"
> >     test-method status="FAIL" signature="init()" name="init (0)"
> >     test-method status="SKIP" signature="test()" name="test (0)"
> >     test-method status="PASS" signature="test()" name="test (1)"
> >     test-method status="PASS" signature="init()" name="init (1)"
> >     test-method status="PASS" signature="init()" name="init (3)"
> >     test-method status="PASS" signature="test()" name="test (3)"
>
> >  I.e. TestNG tries to run all init() methods, and test() only for
> >  instances where init() has failed.
>
> >  Is this a known problem (I could not find any JIRA for a similar
> >  issue)?
> >  Does TestNG 5.8 solves this issue? (I could not check since 5.8 is not
> >  available on any maven2 repository, but the 5.8 changes don't seem to
> >  cover this issue anyway).
>
> >  This problem seems quite serious to me because my understanding is
> >  that all factory-created instances of a test class are independent of
> >  each other in terms of failure/skipping behavior.
>
> >  Is there any workaround for this?
>
> >  Cheerrs
>
> >  Jean-Francois
>
> As far as I know this is a limitation of the current versions.
>
> ./alex
> --
> .w( the_mindstorm )p.
>   Alexandru Popescu
>   TestNG co-founder
> EclipseTestNG Creator- Hide quoted text -
>
> - Show quoted text -
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Problem with @Factory, @BeforeClass failure and skipped tests

Alexandru Popescu ☀
On Wed, Apr 9, 2008 at 7:21 PM, jfpoilpret <[hidden email]> wrote:
>
>  Is there any plan to fix this some day?

For sure :-).

./alex
--
.w( the_mindstorm )p.
  Alexandru Popescu
  TestNG co-founder
EclipseTestNG Creator


>  Without such a fix, this makes the whole existence of @Factory quite
>  useless (except in the case where all tests run perfectly of course,
>  which is the ideal case, but not always possible, in particular at the
>  times those tests are first written or later extended).
>
>  Jean-Francois
>
>  On Apr 9, 11:17 pm, "Alexandru Popescu ☀"
>  <[hidden email]> wrote:
>
>
> > On Wed, Apr 9, 2008 at 4:02 PM, jfpoilpret <[hidden email]> wrote:
>  >
>  > >  Hello,
>  >
>  > >  going further on my factory experiments (in order to proceed with
>  > >  converting my JUnit3 tests to TestNG) I have discovered a strange
>  > >  behavior that I believe is probably a bug. I am using TestNG 5.7 (from
>  > >  maven2).
>  >
>  > >  Let's consider the following snippet:
>  >
>  > >  public class TestNGProblemFactoryTest
>  > >  {
>  > >         @Factory static public Object[] create()
>  > >         {
>  > >                 Object[] tests = new Object[5];
>  > >                 for(int i = 0; i < tests.length; i++)
>  > >                 {
>  > >                         tests[i] = new TestNGProblemTest(i);
>  > >                 }
>  > >                 return tests;
>  > >         }
>  >
>  > >         @Test(groups = {"utest"})
>  > >         static public class TestNGProblemTest implements ITest
>  > >         {
>  > >                 public String   getTestName()
>  > >                 {
>  > >                         return "" + _num;
>  > >                 }
>  >
>  > >                 public TestNGProblemTest(int num)
>  > >                 {
>  > >                         _num = num;
>  > >                 }
>  >
>  > >                 @BeforeClass public void        init()
>  > >                 {
>  > >                         System.out.println("init() #" + _num);
>  > >                         if (_num % 2 == 0)
>  > >                         {
>  > >                                 fail("init() failed");
>  > >                         }
>  > >                 }
>  >
>  > >                 @Test public void       test()
>  > >                 {
>  > >                         System.out.println("test() #" + _num);
>  > >                 }
>  >
>  > >                 private final int       _num;
>  > >         }
>  > >  }
>  >
>  > >  As you can see, on the 5 TestNGProblemTest instances created by the
>  > >  create() factory method, 3 will have their @BeforeClass init() method
>  > >  failing, while for the 2 other instances, that method will be OK.
>  >
>  > >  Now if I run this test with TestNG, here is the results I get
>  > >  (excerpted from testng-results.xml):
>  >
>  > >     test-method status="FAIL" signature="init()" name="init (4)"
>  > >     test-method status="SKIP" signature="init()" name="init (2)"
>  > >     test-method status="SKIP" signature="init()" name="init (0)"
>  > >     test-method status="SKIP" signature="test()" name="test (1)"
>  > >     test-method status="SKIP" signature="test()" name="test (4)"
>  > >     test-method status="SKIP" signature="init()" name="init (1)"
>  > >     test-method status="SKIP" signature="test()" name="test (2)"
>  > >     test-method status="SKIP" signature="test()" name="test (0)"
>  > >     test-method status="SKIP" signature="init()" name="init (3)"
>  > >     test-method status="SKIP" signature="test()" name="test (3)"
>  >
>  > >  It turns out that TestNG as started executing instance #4 first, for
>  > >  that instance the @BeforeMethod init() fails. This failure enforces
>  > >  skipping test() of instance #4 (which is expected behavior) but also
>  > >  skips ALL other methods of all other 4 instances, which is not
>  > >  expected at all: @BeforeClass is supposed to work at instance level
>  > >  (according to the doc), hence it should have an impact only on the
>  > >  instance #4 on the example above, and TestNG should still try to run
>  > >  at least init() on each instance and test() of all instances which
>  > >  init() did not fail!
>  >
>  > >  The expected result for the above snippet should look as follows
>  > >  (maybe in a different order since we cannot determine the order in
>  > >  which TestNG will run each instance methods, but the important info
>  > >  below is the status value):
>  >
>  > >     test-method status="FAIL" signature="init()" name="init (4)"
>  > >     test-method status="SKIP" signature="test()" name="test (4)"
>  > >     test-method status="FAIL" signature="init()" name="init (2)"
>  > >     test-method status="SKIP" signature="test()" name="test (2)"
>  > >     test-method status="FAIL" signature="init()" name="init (0)"
>  > >     test-method status="SKIP" signature="test()" name="test (0)"
>  > >     test-method status="PASS" signature="test()" name="test (1)"
>  > >     test-method status="PASS" signature="init()" name="init (1)"
>  > >     test-method status="PASS" signature="init()" name="init (3)"
>  > >     test-method status="PASS" signature="test()" name="test (3)"
>  >
>  > >  I.e. TestNG tries to run all init() methods, and test() only for
>  > >  instances where init() has failed.
>  >
>  > >  Is this a known problem (I could not find any JIRA for a similar
>  > >  issue)?
>  > >  Does TestNG 5.8 solves this issue? (I could not check since 5.8 is not
>  > >  available on any maven2 repository, but the 5.8 changes don't seem to
>  > >  cover this issue anyway).
>  >
>  > >  This problem seems quite serious to me because my understanding is
>  > >  that all factory-created instances of a test class are independent of
>  > >  each other in terms of failure/skipping behavior.
>  >
>  > >  Is there any workaround for this?
>  >
>  > >  Cheerrs
>  >
>  > >  Jean-Francois
>  >
>  > As far as I know this is a limitation of the current versions.
>  >
>  > ./alex
>  > --
>  > .w( the_mindstorm )p.
>  >   Alexandru Popescu
>  >   TestNG co-founder
>  > EclipseTestNG Creator- Hide quoted text -
>  >
>  > - Show quoted text -
>
>
> >
>

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Problem with @Factory, @BeforeClass failure and skipped tests

Nadav Nagel
Hi.
I am looking at this post thread, and I am having the same problem (using TestNG 6.9.11)
Is this issue was fixed?, and if not there are any planes to fix it?
This really blocking me, maybe you can give me some help on how to fix it?

On Wednesday, April 9, 2008 at 7:51:04 PM UTC+3, Alexandru wrote:
On Wed, Apr 9, 2008 at 7:21 PM, jfpoilpret <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="uUHy6DbwzDsJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">jfpoi...@...> wrote:
>
>  Is there any plan to fix this some day?

For sure :-).

./alex
--
.w( the_mindstorm )p.
  Alexandru Popescu
  TestNG co-founder
EclipseTestNG Creator


>  Without such a fix, this makes the whole existence of @Factory quite
>  useless (except in the case where all tests run perfectly of course,
>  which is the ideal case, but not always possible, in particular at the
>  times those tests are first written or later extended).
>
>  Jean-Francois
>
>  On Apr 9, 11:17 pm, "Alexandru Popescu ☀"
>  <the.mindstorm.mailingl...@gmail.com> wrote:
>
>
> > On Wed, Apr 9, 2008 at 4:02 PM, jfpoilpret <[hidden email]> wrote:
>  >
>  > >  Hello,
>  >
>  > >  going further on my factory experiments (in order to proceed with
>  > >  converting my JUnit3 tests to TestNG) I have discovered a strange
>  > >  behavior that I believe is probably a bug. I am using TestNG 5.7 (from
>  > >  maven2).
>  >
>  > >  Let's consider the following snippet:
>  >
>  > >  public class TestNGProblemFactoryTest
>  > >  {
>  > >         @Factory static public Object[] create()
>  > >         {
>  > >                 Object[] tests = new Object[5];
>  > >                 for(int i = 0; i < tests.length; i++)
>  > >                 {
>  > >                         tests[i] = new TestNGProblemTest(i);
>  > >                 }
>  > >                 return tests;
>  > >         }
>  >
>  > >         @Test(groups = {"utest"})
>  > >         static public class TestNGProblemTest implements ITest
>  > >         {
>  > >                 public String   getTestName()
>  > >                 {
>  > >                         return "" + _num;
>  > >                 }
>  >
>  > >                 public TestNGProblemTest(int num)
>  > >                 {
>  > >                         _num = num;
>  > >                 }
>  >
>  > >                 @BeforeClass public void        init()
>  > >                 {
>  > >                         System.out.println("init() #" + _num);
>  > >                         if (_num % 2 == 0)
>  > >                         {
>  > >                                 fail("init() failed");
>  > >                         }
>  > >                 }
>  >
>  > >                 @Test public void       test()
>  > >                 {
>  > >                         System.out.println("test() #" + _num);
>  > >                 }
>  >
>  > >                 private final int       _num;
>  > >         }
>  > >  }
>  >
>  > >  As you can see, on the 5 TestNGProblemTest instances created by the
>  > >  create() factory method, 3 will have their @BeforeClass init() method
>  > >  failing, while for the 2 other instances, that method will be OK.
>  >
>  > >  Now if I run this test with TestNG, here is the results I get
>  > >  (excerpted from testng-results.xml):
>  >
>  > >     test-method status="FAIL" signature="init()" name="init (4)"
>  > >     test-method status="SKIP" signature="init()" name="init (2)"
>  > >     test-method status="SKIP" signature="init()" name="init (0)"
>  > >     test-method status="SKIP" signature="test()" name="test (1)"
>  > >     test-method status="SKIP" signature="test()" name="test (4)"
>  > >     test-method status="SKIP" signature="init()" name="init (1)"
>  > >     test-method status="SKIP" signature="test()" name="test (2)"
>  > >     test-method status="SKIP" signature="test()" name="test (0)"
>  > >     test-method status="SKIP" signature="init()" name="init (3)"
>  > >     test-method status="SKIP" signature="test()" name="test (3)"
>  >
>  > >  It turns out that TestNG as started executing instance #4 first, for
>  > >  that instance the @BeforeMethod init() fails. This failure enforces
>  > >  skipping test() of instance #4 (which is expected behavior) but also
>  > >  skips ALL other methods of all other 4 instances, which is not
>  > >  expected at all: @BeforeClass is supposed to work at instance level
>  > >  (according to the doc), hence it should have an impact only on the
>  > >  instance #4 on the example above, and TestNG should still try to run
>  > >  at least init() on each instance and test() of all instances which
>  > >  init() did not fail!
>  >
>  > >  The expected result for the above snippet should look as follows
>  > >  (maybe in a different order since we cannot determine the order in
>  > >  which TestNG will run each instance methods, but the important info
>  > >  below is the status value):
>  >
>  > >     test-method status="FAIL" signature="init()" name="init (4)"
>  > >     test-method status="SKIP" signature="test()" name="test (4)"
>  > >     test-method status="FAIL" signature="init()" name="init (2)"
>  > >     test-method status="SKIP" signature="test()" name="test (2)"
>  > >     test-method status="FAIL" signature="init()" name="init (0)"
>  > >     test-method status="SKIP" signature="test()" name="test (0)"
>  > >     test-method status="PASS" signature="test()" name="test (1)"
>  > >     test-method status="PASS" signature="init()" name="init (1)"
>  > >     test-method status="PASS" signature="init()" name="init (3)"
>  > >     test-method status="PASS" signature="test()" name="test (3)"
>  >
>  > >  I.e. TestNG tries to run all init() methods, and test() only for
>  > >  instances where init() has failed.
>  >
>  > >  Is this a known problem (I could not find any JIRA for a similar
>  > >  issue)?
>  > >  Does TestNG 5.8 solves this issue? (I could not check since 5.8 is not
>  > >  available on any maven2 repository, but the 5.8 changes don't seem to
>  > >  cover this issue anyway).
>  >
>  > >  This problem seems quite serious to me because my understanding is
>  > >  that all factory-created instances of a test class are independent of
>  > >  each other in terms of failure/skipping behavior.
>  >
>  > >  Is there any workaround for this?
>  >
>  > >  Cheerrs
>  >
>  > >  Jean-Francois
>  >
>  > As far as I know this is a limitation of the current versions.
>  >
>  > ./alex
>  > --
>  > .w( the_mindstorm )p.
>  >   Alexandru Popescu
>  >   TestNG co-founder
>  > EclipseTestNG Creator- Hide quoted text -
>  >
>  > - Show quoted text -
>
>
> >
>

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Problem with @Factory, @BeforeClass failure and skipped tests

santhu ravikanti


On Sunday, May 22, 2016 at 7:14:55 PM UTC+5:30, Nadav Nagel wrote:
Hi.
I am looking at this post thread, and I am having the same problem (using TestNG 6.9.11)
Is this issue was fixed?, and if not there are any planes to fix it?
This really blocking me, maybe you can give me some help on how to fix it?

On Wednesday, April 9, 2008 at 7:51:04 PM UTC+3, Alexandru wrote:
On Wed, Apr 9, 2008 at 7:21 PM, jfpoilpret <[hidden email]> wrote:
>
>  Is there any plan to fix this some day?

For sure :-).

./alex
--
.w( the_mindstorm )p.
  Alexandru Popescu
  TestNG co-founder
EclipseTestNG Creator


>  Without such a fix, this makes the whole existence of @Factory quite
>  useless (except in the case where all tests run perfectly of course,
>  which is the ideal case, but not always possible, in particular at the
>  times those tests are first written or later extended).
>
>  Jean-Francois
>
>  On Apr 9, 11:17 pm, "Alexandru Popescu ☀"
>  <the.mindstorm.mailingl...@gmail.com> wrote:
>
>
> > On Wed, Apr 9, 2008 at 4:02 PM, jfpoilpret <[hidden email]> wrote:
>  >
>  > >  Hello,
>  >
>  > >  going further on my factory experiments (in order to proceed with
>  > >  converting my JUnit3 tests to TestNG) I have discovered a strange
>  > >  behavior that I believe is probably a bug. I am using TestNG 5.7 (from
>  > >  maven2).
>  >
>  > >  Let's consider the following snippet:
>  >
>  > >  public class TestNGProblemFactoryTest
>  > >  {
>  > >         @Factory static public Object[] create()
>  > >         {
>  > >                 Object[] tests = new Object[5];
>  > >                 for(int i = 0; i < tests.length; i++)
>  > >                 {
>  > >                         tests[i] = new TestNGProblemTest(i);
>  > >                 }
>  > >                 return tests;
>  > >         }
>  >
>  > >         @Test(groups = {"utest"})
>  > >         static public class TestNGProblemTest implements ITest
>  > >         {
>  > >                 public String   getTestName()
>  > >                 {
>  > >                         return "" + _num;
>  > >                 }
>  >
>  > >                 public TestNGProblemTest(int num)
>  > >                 {
>  > >                         _num = num;
>  > >                 }
>  >
>  > >                 @BeforeClass public void        init()
>  > >                 {
>  > >                         System.out.println("init() #" + _num);
>  > >                         if (_num % 2 == 0)
>  > >                         {
>  > >                                 fail("init() failed");
>  > >                         }
>  > >                 }
>  >
>  > >                 @Test public void       test()
>  > >                 {
>  > >                         System.out.println("test() #" + _num);
>  > >                 }
>  >
>  > >                 private final int       _num;
>  > >         }
>  > >  }
>  >
>  > >  As you can see, on the 5 TestNGProblemTest instances created by the
>  > >  create() factory method, 3 will have their @BeforeClass init() method
>  > >  failing, while for the 2 other instances, that method will be OK.
>  >
>  > >  Now if I run this test with TestNG, here is the results I get
>  > >  (excerpted from testng-results.xml):
>  >
>  > >     test-method status="FAIL" signature="init()" name="init (4)"
>  > >     test-method status="SKIP" signature="init()" name="init (2)"
>  > >     test-method status="SKIP" signature="init()" name="init (0)"
>  > >     test-method status="SKIP" signature="test()" name="test (1)"
>  > >     test-method status="SKIP" signature="test()" name="test (4)"
>  > >     test-method status="SKIP" signature="init()" name="init (1)"
>  > >     test-method status="SKIP" signature="test()" name="test (2)"
>  > >     test-method status="SKIP" signature="test()" name="test (0)"
>  > >     test-method status="SKIP" signature="init()" name="init (3)"
>  > >     test-method status="SKIP" signature="test()" name="test (3)"
>  >
>  > >  It turns out that TestNG as started executing instance #4 first, for
>  > >  that instance the @BeforeMethod init() fails. This failure enforces
>  > >  skipping test() of instance #4 (which is expected behavior) but also
>  > >  skips ALL other methods of all other 4 instances, which is not
>  > >  expected at all: @BeforeClass is supposed to work at instance level
>  > >  (according to the doc), hence it should have an impact only on the
>  > >  instance #4 on the example above, and TestNG should still try to run
>  > >  at least init() on each instance and test() of all instances which
>  > >  init() did not fail!
>  >
>  > >  The expected result for the above snippet should look as follows
>  > >  (maybe in a different order since we cannot determine the order in
>  > >  which TestNG will run each instance methods, but the important info
>  > >  below is the status value):
>  >
>  > >     test-method status="FAIL" signature="init()" name="init (4)"
>  > >     test-method status="SKIP" signature="test()" name="test (4)"
>  > >     test-method status="FAIL" signature="init()" name="init (2)"
>  > >     test-method status="SKIP" signature="test()" name="test (2)"
>  > >     test-method status="FAIL" signature="init()" name="init (0)"
>  > >     test-method status="SKIP" signature="test()" name="test (0)"
>  > >     test-method status="PASS" signature="test()" name="test (1)"
>  > >     test-method status="PASS" signature="init()" name="init (1)"
>  > >     test-method status="PASS" signature="init()" name="init (3)"
>  > >     test-method status="PASS" signature="test()" name="test (3)"
>  >
>  > >  I.e. TestNG tries to run all init() methods, and test() only for
>  > >  instances where init() has failed.
>  >
>  > >  Is this a known problem (I could not find any JIRA for a similar
>  > >  issue)?
>  > >  Does TestNG 5.8 solves this issue? (I could not check since 5.8 is not
>  > >  available on any maven2 repository, but the 5.8 changes don't seem to
>  > >  cover this issue anyway).
>  >
>  > >  This problem seems quite serious to me because my understanding is
>  > >  that all factory-created instances of a test class are independent of
>  > >  each other in terms of failure/skipping behavior.
>  >
>  > >  Is there any workaround for this?
>  >
>  > >  Cheerrs
>  >
>  > >  Jean-Francois
>  >
>  > As far as I know this is a limitation of the current versions.
>  >
>  > ./alex
>  > --
>  > .w( the_mindstorm )p.
>  >   Alexandru Popescu
>  >   TestNG co-founder
>  > EclipseTestNG Creator- Hide quoted text -
>  >
>  > - Show quoted text -
>
> is this issue fixed in latest version of test NG ?we are facing same in 
> >
>

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Problem with @Factory, @BeforeClass failure and skipped tests

santhu ravikanti
HI All,
we are facing same issue ,is this issue fixed in latest test ng version?

On Tuesday, April 4, 2017 at 9:57:30 PM UTC+5:30, santhu ravikanti wrote:


On Sunday, May 22, 2016 at 7:14:55 PM UTC+5:30, Nadav Nagel wrote:
Hi.
I am looking at this post thread, and I am having the same problem (using TestNG 6.9.11)
Is this issue was fixed?, and if not there are any planes to fix it?
This really blocking me, maybe you can give me some help on how to fix it?

On Wednesday, April 9, 2008 at 7:51:04 PM UTC+3, Alexandru wrote:
On Wed, Apr 9, 2008 at 7:21 PM, jfpoilpret <[hidden email]> wrote:
>
>  Is there any plan to fix this some day?

For sure :-).

./alex
--
.w( the_mindstorm )p.
  Alexandru Popescu
  TestNG co-founder
EclipseTestNG Creator


>  Without such a fix, this makes the whole existence of @Factory quite
>  useless (except in the case where all tests run perfectly of course,
>  which is the ideal case, but not always possible, in particular at the
>  times those tests are first written or later extended).
>
>  Jean-Francois
>
>  On Apr 9, 11:17 pm, "Alexandru Popescu ☀"
>  <the.mindstorm.mailingl...@gmail.com> wrote:
>
>
> > On Wed, Apr 9, 2008 at 4:02 PM, jfpoilpret <[hidden email]> wrote:
>  >
>  > >  Hello,
>  >
>  > >  going further on my factory experiments (in order to proceed with
>  > >  converting my JUnit3 tests to TestNG) I have discovered a strange
>  > >  behavior that I believe is probably a bug. I am using TestNG 5.7 (from
>  > >  maven2).
>  >
>  > >  Let's consider the following snippet:
>  >
>  > >  public class TestNGProblemFactoryTest
>  > >  {
>  > >         @Factory static public Object[] create()
>  > >         {
>  > >                 Object[] tests = new Object[5];
>  > >                 for(int i = 0; i < tests.length; i++)
>  > >                 {
>  > >                         tests[i] = new TestNGProblemTest(i);
>  > >                 }
>  > >                 return tests;
>  > >         }
>  >
>  > >         @Test(groups = {"utest"})
>  > >         static public class TestNGProblemTest implements ITest
>  > >         {
>  > >                 public String   getTestName()
>  > >                 {
>  > >                         return "" + _num;
>  > >                 }
>  >
>  > >                 public TestNGProblemTest(int num)
>  > >                 {
>  > >                         _num = num;
>  > >                 }
>  >
>  > >                 @BeforeClass public void        init()
>  > >                 {
>  > >                         System.out.println("init() #" + _num);
>  > >                         if (_num % 2 == 0)
>  > >                         {
>  > >                                 fail("init() failed");
>  > >                         }
>  > >                 }
>  >
>  > >                 @Test public void       test()
>  > >                 {
>  > >                         System.out.println("test() #" + _num);
>  > >                 }
>  >
>  > >                 private final int       _num;
>  > >         }
>  > >  }
>  >
>  > >  As you can see, on the 5 TestNGProblemTest instances created by the
>  > >  create() factory method, 3 will have their @BeforeClass init() method
>  > >  failing, while for the 2 other instances, that method will be OK.
>  >
>  > >  Now if I run this test with TestNG, here is the results I get
>  > >  (excerpted from testng-results.xml):
>  >
>  > >     test-method status="FAIL" signature="init()" name="init (4)"
>  > >     test-method status="SKIP" signature="init()" name="init (2)"
>  > >     test-method status="SKIP" signature="init()" name="init (0)"
>  > >     test-method status="SKIP" signature="test()" name="test (1)"
>  > >     test-method status="SKIP" signature="test()" name="test (4)"
>  > >     test-method status="SKIP" signature="init()" name="init (1)"
>  > >     test-method status="SKIP" signature="test()" name="test (2)"
>  > >     test-method status="SKIP" signature="test()" name="test (0)"
>  > >     test-method status="SKIP" signature="init()" name="init (3)"
>  > >     test-method status="SKIP" signature="test()" name="test (3)"
>  >
>  > >  It turns out that TestNG as started executing instance #4 first, for
>  > >  that instance the @BeforeMethod init() fails. This failure enforces
>  > >  skipping test() of instance #4 (which is expected behavior) but also
>  > >  skips ALL other methods of all other 4 instances, which is not
>  > >  expected at all: @BeforeClass is supposed to work at instance level
>  > >  (according to the doc), hence it should have an impact only on the
>  > >  instance #4 on the example above, and TestNG should still try to run
>  > >  at least init() on each instance and test() of all instances which
>  > >  init() did not fail!
>  >
>  > >  The expected result for the above snippet should look as follows
>  > >  (maybe in a different order since we cannot determine the order in
>  > >  which TestNG will run each instance methods, but the important info
>  > >  below is the status value):
>  >
>  > >     test-method status="FAIL" signature="init()" name="init (4)"
>  > >     test-method status="SKIP" signature="test()" name="test (4)"
>  > >     test-method status="FAIL" signature="init()" name="init (2)"
>  > >     test-method status="SKIP" signature="test()" name="test (2)"
>  > >     test-method status="FAIL" signature="init()" name="init (0)"
>  > >     test-method status="SKIP" signature="test()" name="test (0)"
>  > >     test-method status="PASS" signature="test()" name="test (1)"
>  > >     test-method status="PASS" signature="init()" name="init (1)"
>  > >     test-method status="PASS" signature="init()" name="init (3)"
>  > >     test-method status="PASS" signature="test()" name="test (3)"
>  >
>  > >  I.e. TestNG tries to run all init() methods, and test() only for
>  > >  instances where init() has failed.
>  >
>  > >  Is this a known problem (I could not find any JIRA for a similar
>  > >  issue)?
>  > >  Does TestNG 5.8 solves this issue? (I could not check since 5.8 is not
>  > >  available on any maven2 repository, but the 5.8 changes don't seem to
>  > >  cover this issue anyway).
>  >
>  > >  This problem seems quite serious to me because my understanding is
>  > >  that all factory-created instances of a test class are independent of
>  > >  each other in terms of failure/skipping behavior.
>  >
>  > >  Is there any workaround for this?
>  >
>  > >  Cheerrs
>  >
>  > >  Jean-Francois
>  >
>  > As far as I know this is a limitation of the current versions.
>  >
>  > ./alex
>  > --
>  > .w( the_mindstorm )p.
>  >   Alexandru Popescu
>  >   TestNG co-founder
>  > EclipseTestNG Creator- Hide quoted text -
>  >
>  > - Show quoted text -
>
> is this issue fixed in latest version of test NG ?we are facing same in 
> >
>

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.
Loading...