[testng-dev] BUG: JUnitReportReporter transposes errors and failures

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

[testng-dev] BUG: JUnitReportReporter transposes errors and failures

Thorson Little
Note: I reported this issue two months ago but had not done the deep dive to determine if this is a clear bug.  I am now clear that it is.  The original post is here: https://groups.google.com/forum/?fromgroups#!topic/testng-dev/M_oR3lifxMM

Documentation:
The following is from JUnit: A Cooks tour (http://junit.sourceforge.net/doc/cookstour/cookstour.htm)

JUnit distinguishes between failures and errors. The possibility of a failure is anticipated and checked for with assertions. Errors are unanticipated problems like an ArrayIndexOutOfBoundsException. Failures are signaled with an AssertionFailedError error. 

To Repro:

TestClass:

public class JunitExample {
  @Test
  public void firstTest() {
 assertTrue( true, "true assertion number 1" ); 
  }
  
  @Test(enabled=true)
  public void secondTest() {
 assertTrue( true, "Assertion 1/3 passes" );
 assertTrue(true);
 assertTrue( false, "Assertion 3/3 fails and whole test fails");
  }
  
  @Test (enabled=true)
  public void thirdTest() {
 assertTrue( true, "true assertion number 2" );
 
 fail( "This is a failure" );
  }
  
  
  @Test
  public void fourthTest() {
 assertTrue( true, "true assertion number 3" ); 
 assertTrue( true, "true assertion number 4" ); 
 throw new RuntimeException("watch this fail");
  }
  
}


Expected Results: 
1 Success(firstTest), 2 failures(secondTest & thirdTest) and 1 error(fourthTest)

Actual Results:
1 Success(firstTest), 2 error(secondTest & thirdTest) and 1 failure(fourthTest)

<?xml version="1.0" encoding="UTF-8"?>
<!-- Generated by org.testng.reporters.JUnitReportReporter -->
<testsuite hostname="tlittle-lt" name="com.example.pkg.JunitExample" tests="4" failures="1" timestamp="29 May 2013 19:14:17 GMT" time="0.011" errors="2">
  <testcase name="thirdTest" time="0.000" classname="com.example.pkg.JunitExample">
    <error type="java.lang.AssertionError" message="This is a failure">
      <![CDATA[java.lang.AssertionError: This is a failure
at org.testng.Assert.fail(Assert.java:94)
...
]]>
    </error>
  </testcase> <!-- thirdTest -->
  <testcase name="fourthTest" time="0.001" classname="com.example.pkg.JunitExample">
    <failure type="java.lang.RuntimeException" message="watch this fail">
      <![CDATA[java.lang.RuntimeException: watch this fail
at com.example.pkg.JunitExample.fourthTest(JunitExample.java:31)
        ...
]]>
    </failure>
  </testcase> <!-- fourthTest -->
  <testcase name="secondTest" time="0.001" classname="com.example.pkg.JunitExample">
    <error type="java.lang.AssertionError" message="Assertion 3/3 fails and whole test fails expected [true] but found [false]">
      <![CDATA[java.lang.AssertionError: Assertion 3/3 fails and whole test fails expected [true] but found [false]
at org.testng.Assert.fail(Assert.java:94)
...
]]>
    </error>
  </testcase> <!-- secondTest -->
  <testcase name="firstTest" time="0.009" classname="com.example.pkg.JunitExample"/>
</testsuite> <!-- com.example.pkg.JunitExample -->


Where To Find The Bug:
In JUnitReportReporter.java

lines 101-107:
       if (! isSuccess) {
          if (tr.getThrowable() instanceof AssertionError) {
            errors++;
          } else {
            failures++;
          }
        } 
line 126:
 testTag.errorTag = tr.getThrowable() instanceof AssertionError ? "error" : "failure";



--
You received this message because you are subscribed to the Google Groups "testng-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at http://groups.google.com/group/testng-dev?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|

[testng-dev] Re: BUG: JUnitReportReporter transposes errors and failures

Thorson Little
Attached environment


--
You received this message because you are subscribed to the Google Groups "testng-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at http://groups.google.com/group/testng-dev?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

playSpace.zip (94K) Download Attachment