Wrong report about error when running tests

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Wrong report about error when running tests

Jiri Mares-3

Hi,

I'm using testNG 5.4 using the eclipse plugin. I have the following
test class:

class Test {

@Test(dataProvider="A")
public void testA(Exception err)
  ...
}

@DataProvider(name="A")
protected Object[][] dp() {
   return new Object[][] { {new IOException(), new SAXException() } };
}

@AfterMethod
protected void verify(Method method) {
   ....
   verifySomething();
   ...
}

}

I know that the problem of my test is wrong 2-dimensional array in data
provider that should be:

   return new Object[][] { {new IOException() }, { new SAXException() }
};

But the TestNG is by me wrongly reported the error, because shows error
that happen in verify method (is depended on code in the test,
switching easymock objects to replay stage) and doesn't tell anything
about IndexOutOfBoundsException thrown during applying parameters to
the test method, nor the test method that was going to be run.

Jirka


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: Wrong report about error when running tests

Alexandru Popescu ☀

On 1/16/07, Jiri Mares <[hidden email]> wrote:

>
> Hi,
>
> I'm using testNG 5.4 using the eclipse plugin. I have the following
> test class:
>
> class Test {
>
> @Test(dataProvider="A")
> public void testA(Exception err)
>   ...
> }
>
> @DataProvider(name="A")
> protected Object[][] dp() {
>    return new Object[][] { {new IOException(), new SAXException() } };
> }
>
> @AfterMethod
> protected void verify(Method method) {
>    ....
>    verifySomething();
>    ...
> }
>
> }
>
> I know that the problem of my test is wrong 2-dimensional array in data
> provider that should be:
>
>    return new Object[][] { {new IOException() }, { new SAXException() }
> };
>
> But the TestNG is by me wrongly reported the error, because shows error
> that happen in verify method (is depended on code in the test,
> switching easymock objects to replay stage) and doesn't tell anything
> about IndexOutOfBoundsException thrown during applying parameters to
> the test method, nor the test method that was going to be run.
>
> Jirka
>

Thanks for the report. I will try to reproduce the problem and keep
you posted about my findings.

BR,

./alex
--
.w( the_mindstorm )p.
  TestNG co-founder
EclipseTestNG Creator

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: Wrong report about error when running tests

Alexandru Popescu ☀
In reply to this post by Jiri Mares-3

With the current SVN trunk the output is:

java org.testng.TestNG -log 2 -testclass
test.expectedexceptions.ParametersExceptionTest

FAILED CONFIGURATION: @AfterMethod verify(public void
test.expectedexceptions.ParametersExceptionTest.testA(java.lang.Exception))
java.lang.AssertionError: forced failure expected:<true> but was:<false>
        at test.expectedexceptions.ParametersExceptionTest.verify(ParametersExceptionTest.java:29)
... Removed 26 stack frames
FAILED: testA(testA: Wrong number of arguments were passed by the Data
Provider: found 2 but expected 1)
java.lang.IllegalArgumentException: wrong number of arguments
... Removed 21 stack frames

===============================================
    Command line test
    Tests run: 1, Failures: 1, Skips: 0
    Configuration Failures: 1, Skips: 0
===============================================


===============================================
Command line suite
Total tests run: 1, Failures: 1, Skips: 0
Configuration Failures: 1, Skips: 0
===============================================

./alex
--
.w( the_mindstorm )p.
  TestNG co-founder
EclipseTestNG Creator

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: Wrong report about error when running tests

Jiri Mares-3

Ok, looks like been changed from the version 5.4, because I get the
following result:


 ===============================================
 Ant suite
 Total tests run: 2, Failures: 2, Skips: 0
 ===============================================

Result contains no detail message about error.

Have a nice day

Jirka

> FAILED CONFIGURATION: @AfterMethod verify(public void
> test.expectedexceptions.ParametersExceptionTest.testA(java.lang.Exception))
> java.lang.AssertionError: forced failure expected:<true> but was:<false>
>         at test.expectedexceptions.ParametersExceptionTest.verify(ParametersExceptionTest.java:29)
> ... Removed 26 stack frames
> FAILED: testA(testA: Wrong number of arguments were passed by the Data
> Provider: found 2 but expected 1)
> java.lang.IllegalArgumentException: wrong number of arguments
> ... Removed 21 stack frames
>
> ===============================================
>     Command line test
>     Tests run: 1, Failures: 1, Skips: 0
>     Configuration Failures: 1, Skips: 0
> ===============================================
>
> ===============================================
> Command line suite
> Total tests run: 1, Failures: 1, Skips: 0
> Configuration Failures: 1, Skips: 0
> ===============================================


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: Wrong report about error when running tests

Alexandru Popescu ☀

On 1/22/07, Jiri Mares <[hidden email]> wrote:

>
> Ok, looks like been changed from the version 5.4, because I get the
> following result:
>
>
>  ===============================================
>  Ant suite
>  Total tests run: 2, Failures: 2, Skips: 0
>  ===============================================
>
> Result contains no detail message about error.
>

By default the log level is 1 which prints out only this kind of
summary. If you need more details please set the log level to 2 (and
you should see the stacktraces).

HTH,

./alex
--
.w( the_mindstorm )p.
  TestNG co-founder
EclipseTestNG Creator

> Have a nice day
>
> Jirka
>
> > FAILED CONFIGURATION: @AfterMethod verify(public void
> > test.expectedexceptions.ParametersExceptionTest.testA(java.lang.Exception))
> > java.lang.AssertionError: forced failure expected:<true> but was:<false>
> >         at test.expectedexceptions.ParametersExceptionTest.verify(ParametersExceptionTest.java:29)
> > ... Removed 26 stack frames
> > FAILED: testA(testA: Wrong number of arguments were passed by the Data
> > Provider: found 2 but expected 1)
> > java.lang.IllegalArgumentException: wrong number of arguments
> > ... Removed 21 stack frames
> >
> > ===============================================
> >     Command line test
> >     Tests run: 1, Failures: 1, Skips: 0
> >     Configuration Failures: 1, Skips: 0
> > ===============================================
> >
> > ===============================================
> > Command line suite
> > Total tests run: 1, Failures: 1, Skips: 0
> > Configuration Failures: 1, Skips: 0
> > ===============================================
>
>
> >
>

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: Wrong report about error when running tests

Jiri Mares-3

I see .. because I don't know how to switch the log level using ant I
had a look at the html-results generated and there is the report okay,
same as yours. But when I run the same test inside eclipse I get the
wrong error report. I bundle the simple example demostrating the
problem. When running inside eclipse you get the error in verify method
and not in test ...

import static org.testng.Assert.*;

import org.testng.annotations.AfterMethod;
import org.testng.annotations.DataProvider;
import org.testng.annotations.Test;

public class ErrorTestCase {
   protected int a = 0;

   @DataProvider(name = "x")
   protected Object[][] dp() {
      return new Object[][] { { "a", "b" } };
   }

   @Test(dataProvider = "x")
   public void test(String arg) {
      System.out.println(arg);
      a = 1;
   }

   @AfterMethod
   protected void verify() {
      assertEquals(a, 1);
   }

}

Jirka


On Jan 22, 11:14 am, "Alexandru Popescu"
<[hidden email]> wrote:

> On 1/22/07, Jiri Mares <[hidden email]> wrote:
>
>
>
> > Ok, looks like been changed from the version 5.4, because I get the
> > following result:
>
> >  ===============================================
> >  Ant suite
> >  Total tests run: 2, Failures: 2, Skips: 0
> >  ===============================================
>
> > Result contains no detail message about error.By default the log level is 1 which prints out only this kind of
> summary. If you need more details please set the log level to 2 (and
> you should see the stacktraces).
>
> HTH,
>
> ./alex
> --
> .w( the_mindstorm )p.
>   TestNG co-founder
> EclipseTestNG Creator
>
> > Have a nice day
>
> > Jirka
>
> > > FAILED CONFIGURATION: @AfterMethod verify(public void
> > > test.expectedexceptions.ParametersExceptionTest.testA(java.lang.Exception))
> > > java.lang.AssertionError: forced failure expected:<true> but was:<false>
> > >         at test.expectedexceptions.ParametersExceptionTest.verify(ParametersExceptionTest.java:29)
> > > ... Removed 26 stack frames
> > > FAILED: testA(testA: Wrong number of arguments were passed by the Data
> > > Provider: found 2 but expected 1)
> > > java.lang.IllegalArgumentException: wrong number of arguments
> > > ... Removed 21 stack frames
>
> > > ===============================================
> > >     Command line test
> > >     Tests run: 1, Failures: 1, Skips: 0
> > >     Configuration Failures: 1, Skips: 0
> > > ===============================================
>
> > > ===============================================
> > > Command line suite
> > > Total tests run: 1, Failures: 1, Skips: 0
> > > Configuration Failures: 1, Skips: 0
> > > ===============================================


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: Wrong report about error when running tests

Alexandru Popescu ☀

Maybe this is due to a different version of plugin?

./alex
--
.w( the_mindstorm )p.
  TestNG co-founder
EclipseTestNG Creator


On 1/22/07, Jiri Mares <[hidden email]> wrote:

>
> I see .. because I don't know how to switch the log level using ant I
> had a look at the html-results generated and there is the report okay,
> same as yours. But when I run the same test inside eclipse I get the
> wrong error report. I bundle the simple example demostrating the
> problem. When running inside eclipse you get the error in verify method
> and not in test ...
>
> import static org.testng.Assert.*;
>
> import org.testng.annotations.AfterMethod;
> import org.testng.annotations.DataProvider;
> import org.testng.annotations.Test;
>
> public class ErrorTestCase {
>    protected int a = 0;
>
>    @DataProvider(name = "x")
>    protected Object[][] dp() {
>       return new Object[][] { { "a", "b" } };
>    }
>
>    @Test(dataProvider = "x")
>    public void test(String arg) {
>       System.out.println(arg);
>       a = 1;
>    }
>
>    @AfterMethod
>    protected void verify() {
>       assertEquals(a, 1);
>    }
>
> }
>
> Jirka
>
>
> On Jan 22, 11:14 am, "Alexandru Popescu"
> <[hidden email]> wrote:
> > On 1/22/07, Jiri Mares <[hidden email]> wrote:
> >
> >
> >
> > > Ok, looks like been changed from the version 5.4, because I get the
> > > following result:
> >
> > >  ===============================================
> > >  Ant suite
> > >  Total tests run: 2, Failures: 2, Skips: 0
> > >  ===============================================
> >
> > > Result contains no detail message about error.By default the log level is 1 which prints out only this kind of
> > summary. If you need more details please set the log level to 2 (and
> > you should see the stacktraces).
> >
> > HTH,
> >
> > ./alex
> > --
> > .w( the_mindstorm )p.
> >   TestNG co-founder
> > EclipseTestNG Creator
> >
> > > Have a nice day
> >
> > > Jirka
> >
> > > > FAILED CONFIGURATION: @AfterMethod verify(public void
> > > > test.expectedexceptions.ParametersExceptionTest.testA(java.lang.Exception))
> > > > java.lang.AssertionError: forced failure expected:<true> but was:<false>
> > > >         at test.expectedexceptions.ParametersExceptionTest.verify(ParametersExceptionTest.java:29)
> > > > ... Removed 26 stack frames
> > > > FAILED: testA(testA: Wrong number of arguments were passed by the Data
> > > > Provider: found 2 but expected 1)
> > > > java.lang.IllegalArgumentException: wrong number of arguments
> > > > ... Removed 21 stack frames
> >
> > > > ===============================================
> > > >     Command line test
> > > >     Tests run: 1, Failures: 1, Skips: 0
> > > >     Configuration Failures: 1, Skips: 0
> > > > ===============================================
> >
> > > > ===============================================
> > > > Command line suite
> > > > Total tests run: 1, Failures: 1, Skips: 0
> > > > Configuration Failures: 1, Skips: 0
> > > > ===============================================
>
>
> >
>

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: Wrong report about error when running tests

Jiri Mares-3

I have plugin version 5.4.0.0 and testng.jar is from package 5.4 for
jdk15 ...

You don't encounter the described problem?

On Jan 22, 12:32 pm, "Alexandru Popescu"
<[hidden email]> wrote:

> Maybe this is due to a different version of plugin?
>
> ./alex
> --
> .w( the_mindstorm )p.
>   TestNG co-founder
> EclipseTestNG Creator
>
> On 1/22/07, Jiri Mares <[hidden email]> wrote:
>
>
>
> > I see .. because I don't know how to switch the log level using ant I
> > had a look at the html-results generated and there is the report okay,
> > same as yours. But when I run the same test inside eclipse I get the
> > wrong error report. I bundle the simple example demostrating the
> > problem. When running inside eclipse you get the error in verify method
> > and not in test ...
>
> > import static org.testng.Assert.*;
>
> > import org.testng.annotations.AfterMethod;
> > import org.testng.annotations.DataProvider;
> > import org.testng.annotations.Test;
>
> > public class ErrorTestCase {
> >    protected int a = 0;
>
> >    @DataProvider(name = "x")
> >    protected Object[][] dp() {
> >       return new Object[][] { { "a", "b" } };
> >    }
>
> >    @Test(dataProvider = "x")
> >    public void test(String arg) {
> >       System.out.println(arg);
> >       a = 1;
> >    }
>
> >    @AfterMethod
> >    protected void verify() {
> >       assertEquals(a, 1);
> >    }
>
> > }
>
> > Jirka
>
> > On Jan 22, 11:14 am, "Alexandru Popescu"
> > <[hidden email]> wrote:
> > > On 1/22/07, Jiri Mares <[hidden email]> wrote:
>
> > > > Ok, looks like been changed from the version 5.4, because I get the
> > > > following result:
>
> > > >  ===============================================
> > > >  Ant suite
> > > >  Total tests run: 2, Failures: 2, Skips: 0
> > > >  ===============================================
>
> > > > Result contains no detail message about error.By default the log level is 1 which prints out only this kind of
> > > summary. If you need more details please set the log level to 2 (and
> > > you should see the stacktraces).
>
> > > HTH,
>
> > > ./alex
> > > --
> > > .w( the_mindstorm )p.
> > >   TestNG co-founder
> > > EclipseTestNG Creator
>
> > > > Have a nice day
>
> > > > Jirka
>
> > > > > FAILED CONFIGURATION: @AfterMethod verify(public void
> > > > > test.expectedexceptions.ParametersExceptionTest.testA(java.lang.Exception))
> > > > > java.lang.AssertionError: forced failure expected:<true> but was:<false>
> > > > >         at test.expectedexceptions.ParametersExceptionTest.verify(ParametersExceptionTest.java:29)
> > > > > ... Removed 26 stack frames
> > > > > FAILED: testA(testA: Wrong number of arguments were passed by the Data
> > > > > Provider: found 2 but expected 1)
> > > > > java.lang.IllegalArgumentException: wrong number of arguments
> > > > > ... Removed 21 stack frames
>
> > > > > ===============================================
> > > > >     Command line test
> > > > >     Tests run: 1, Failures: 1, Skips: 0
> > > > >     Configuration Failures: 1, Skips: 0
> > > > > ===============================================
>
> > > > > ===============================================
> > > > > Command line suite
> > > > > Total tests run: 1, Failures: 1, Skips: 0
> > > > > Configuration Failures: 1, Skips: 0
> > > > > ===============================================


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: Wrong report about error when running tests

Jiri Mares-3

Hi,

I've just upgraded to eclipse plugin 5.5.0.1 and still have the same
problem?? None other has this problem using eclipse-plugin?

Jirka


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en
-~----------~----~----~----~------~----~------~--~---