Problems with surefire reports, missing information

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Problems with surefire reports, missing information

Henrik Lynggaard Hansen

Hi

I am testing testng with maven2 and I have run into a problem.

In the surefire report there is missing some information, both the
package list and class list is wrong.

The setup is ass following. I have created small project with two test
packages, each with one test class, which in turn each has 2 test
methods one that fails and one that succeeds.

The in the surefire report the package list contains 1 row with blank
package name claiming 4 run tests and 2 failures.

The Class list also only contains 1 row, class name TestSuite and
claiming 4 run tests and 2 failures

Both lists should contain 2 rows with 2 runs and 1 failure in each.

plugin versions:
testng 5.7
surefire-report: 2.4
[DEBUG]Adding managed dependencies for unknown:surefire-testng
[DEBUG]  org.apache.maven.surefire:surefire-api:jar:2.4
[DEBUG]  org.apache.maven.surefire:surefire-booter:jar:2.4
[DEBUG]  org.codehaus.plexus:plexus-utils:jar:1.4.9
[DEBUG]  org.apache.maven.surefire:surefire-testng:jar:2.4:test
(selected for test)

Am I doing something wrong oor have I found a bug ?

Best regards
Henrik
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: Problems with surefire reports, missing information

Andreas Guther-2
The information you have provided is not sufficient to make an assessment.  What about running the tests with 'mvn test'?  Does everything work as expected here?  What version of surefire are you using? 

Andreas

On Jan 16, 2008 3:42 AM, Henrik Lynggaard <[hidden email]> wrote:

Hi

I am testing testng with maven2 and I have run into a problem.

In the surefire report there is missing some information, both the
package list and class list is wrong.

The setup is ass following. I have created small project with two test
packages, each with one test class, which in turn each has 2 test
methods one that fails and one that succeeds.

The in the surefire report the package list contains 1 row with blank
package name claiming 4 run tests and 2 failures.

The Class list also only contains 1 row, class name TestSuite and
claiming 4 run tests and 2 failures

Both lists should contain 2 rows with 2 runs and 1 failure in each.

plugin versions:
testng 5.7
surefire-report: 2.4
[DEBUG]Adding managed dependencies for unknown:surefire-testng
[DEBUG]  org.apache.maven.surefire:surefire-api:jar:2.4
[DEBUG]  org.apache.maven.surefire:surefire-booter:jar:2.4
[DEBUG]   org.codehaus.plexus:plexus-utils:jar:1.4.9
[DEBUG]  org.apache.maven.surefire:surefire-testng:jar:2.4:test
(selected for test)

Am I doing something wrong oor have I found a bug ?

Best regards
Henrik --~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: Problems with surefire reports, missing information

Henrik Lynggaard Hansen

Hi

Please ask for all the information you need, and I will try to provide
it

Running the test using mvn test appears to work as expected and
genereted the following xml

testng-results.xml:
<testng-results>
  <reporter-output>
  </reporter-output>
  <suite name="Failed suite [Command line suite]">
    <groups>
    </groups>
    <test name="Command line test(failed)">
      <class name="net.apmoller.maventest.NgTest">
        <test-method status="FAIL" signature="testFailNg()"
name="testFailNg" duration-ms="4" started-at="2008-01-16T15:40:09Z"
finished-at="2008-01-16T15:40:09Z">
          <exception class="java.lang.AssertionError">
            <message>
              <![CDATA[expected:<true> but was:<false>]]>
            </message>
            <full-stacktrace>
              <![CDATA[java.lang.AssertionError: expected:&lt;true&gt;
but was:&lt;false&gt;
        at org.testng.Assert.fail(Assert.java:84)
        ..... clipped out....
]]>
            </full-stacktrace>
          </exception>
        </test-method>
      </class>
      <class name="net.apmoller.test2.NgTest">
        <test-method status="FAIL" signature="testFailNg()"
name="testFailNg" duration-ms="1" started-at="2008-01-16T15:40:10Z"
finished-at="2008-01-16T15:40:10Z">
          <exception class="java.lang.AssertionError">
            <message>
              <![CDATA[expected:<true> but was:<false>]]>
            </message>
            <full-stacktrace>
              <![CDATA[java.lang.AssertionError: expected:&lt;true&gt;
but was:&lt;false&gt;
        at org.testng.Assert.fail(Assert.java:84)
                ..... clipped out....
]]>
            </full-stacktrace>
          </exception>
        </test-method>
      </class>
    </test>
  </suite>
</testng-results>

The TEST-TestSuite.xml:
<?xml version="1.0" encoding="UTF-8" ?>
<testsuite failures="2" time="0.418" errors="0" skipped="0" tests="2"
name="TestSuite">
  <properties>
    <property name="java.runtime.name" value="Java(TM) SE Runtime
Environment"/>
    <property name="sun.boot.library.path" value="/opt/jdk1.6.0_02/jre/
lib/i386"/>
    <property name="java.vm.version" value="1.6.0_02-b05"/>
    <property name="java.vm.vendor" value="Sun Microsystems Inc."/>
    <property name="java.vendor.url" value="http://java.sun.com/"/>
    <property name="path.separator" value=":"/>
    <property name="java.vm.name" value="Java HotSpot(TM) Client VM"/>
    <property name="file.encoding.pkg" value="sun.io"/>
    <property name="user.country" value="DK"/>
    <property name="sun.java.launcher" value="SUN_STANDARD"/>
    <property name="sun.os.patch.level" value="unknown"/>
    <property name="java.vm.specification.name" value="Java Virtual
Machine Specification"/>
    <property name="user.dir" value="/home/hlyh/NetBeansProjects/
testproj"/>
    <property name="java.runtime.version" value="1.6.0_02-b05"/>
    <property name="java.awt.graphicsenv"
value="sun.awt.X11GraphicsEnvironment"/>
    <property name="basedir" value="/home/hlyh/NetBeansProjects/
testproj"/>
    <property name="java.endorsed.dirs" value="/opt/jdk1.6.0_02/jre/
lib/endorsed"/>
    <property name="os.arch" value="i386"/>
    <property name="java.io.tmpdir" value="/tmp"/>
    <property name="line.separator" value="
"/>
    <property name="java.vm.specification.vendor" value="Sun
Microsystems Inc."/>
    <property name="os.name" value="Linux"/>
    <property name="classworlds.conf" value="/opt/maven-2.0.8/bin/
m2.conf"/>
    <property name="sun.jnu.encoding" value="UTF-8"/>
    <property name="java.library.path" value="/opt/jdk1.6.0_02/jre/lib/
i386/client:/opt/jdk1.6.0_02/jre/lib/i386:/opt/jdk1.6.0_02/jre/../lib/
i386:/usr/java/packages/lib/i386:/lib:/usr/lib"/>
    <property name="surefire.test.class.path" value="/home/hlyh/
NetBeansProjects/testproj/target/test-classes:/home/hlyh/
NetBeansProjects/testproj/target/classes:/home/hlyh/.m2/repository/org/
testng/testng/5.7/testng-5.7-jdk15.jar:/home/hlyh/.m2/repository/junit/
junit/3.8.1/junit-3.8.1.jar:"/>
    <property name="java.specification.name" value="Java Platform API
Specification"/>
    <property name="java.class.version" value="50.0"/>
    <property name="sun.management.compiler" value="HotSpot Client
Compiler"/>
    <property name="os.version" value="2.6.22-14-generic"/>
    <property name="user.home" value="/home/hlyh"/>
    <property name="user.timezone" value="Europe/Copenhagen"/>
    <property name="java.awt.printerjob"
value="sun.print.PSPrinterJob"/>
    <property name="file.encoding" value="UTF-8"/>
    <property name="java.specification.version" value="1.6"/>
    <property name="user.name" value="hlyh"/>
    <property name="java.class.path" value="/opt/maven-2.0.8/boot/
classworlds-1.1.jar"/>
    <property name="java.vm.specification.version" value="1.0"/>
    <property name="sun.arch.data.model" value="32"/>
    <property name="java.home" value="/opt/jdk1.6.0_02/jre"/>
    <property name="java.specification.vendor" value="Sun Microsystems
Inc."/>
    <property name="user.language" value="en"/>
    <property name="java.vm.info" value="mixed mode, sharing"/>
    <property name="java.version" value="1.6.0_02"/>
    <property name="java.ext.dirs" value="/opt/jdk1.6.0_02/jre/lib/
ext:/usr/java/packages/lib/ext"/>
    <property name="sun.boot.class.path" value="/opt/jdk1.6.0_02/jre/
lib/resources.jar:/opt/jdk1.6.0_02/jre/lib/rt.jar:/opt/jdk1.6.0_02/jre/
lib/sunrsasign.jar:/opt/jdk1.6.0_02/jre/lib/jsse.jar:/opt/jdk1.6.0_02/
jre/lib/jce.jar:/opt/jdk1.6.0_02/jre/lib/charsets.jar:/opt/jdk1.6.0_02/
jre/classes"/>
    <property name="java.vendor" value="Sun Microsystems Inc."/>
    <property name="maven.home" value="/opt/maven-2.0.8"/>
    <property name="localRepository" value="/home/hlyh/.m2/repository"/
>
    <property name="file.separator" value="/"/>
    <property name="java.vendor.url.bug" value="http://java.sun.com/
cgi-bin/bugreport.cgi"/>
    <property name="sun.cpu.endian" value="little"/>
    <property name="sun.io.unicode.encoding" value="UnicodeLittle"/>
    <property name="sun.desktop" value="gnome"/>
    <property name="sun.cpu.isalist" value=""/>
  </properties>
  <testcase time="0.006" name="testFailNg">
    <failure message="expected:&amp;lt;true&amp;gt; but
was:&amp;lt;false&amp;gt;"
type="java.lang.AssertionError">java.lang.AssertionError:
expected:&amp;lt;true&amp;gt; but was:&amp;lt;false&amp;gt;
        at org.testng.Assert.fail(Assert.java:84)
        at org.testng.Assert.failNotEquals(Assert.java:438)
        at org.testng.Assert.assertTrue(Assert.java:32)
        at org.testng.Assert.assertTrue(Assert.java:42)
        at net.apmoller.maventest.NgTest.testFailNg(NgTest.java:23)
</failure>
  </testcase>
  <testcase time="0" name="testFailNg">
    <failure message="expected:&amp;lt;true&amp;gt; but
was:&amp;lt;false&amp;gt;"
type="java.lang.AssertionError">java.lang.AssertionError:
expected:&amp;lt;true&amp;gt; but was:&amp;lt;false&amp;gt;
        at org.testng.Assert.fail(Assert.java:84)
        at org.testng.Assert.failNotEquals(Assert.java:438)
        at org.testng.Assert.assertTrue(Assert.java:32)
        at org.testng.Assert.assertTrue(Assert.java:42)
        at net.apmoller.test2.NgTest.testFailNg(NgTest.java:23)
</failure>
  </testcase>
</testsuite>

If I run mvn site, then the resulting html contains this table for
package list
<table class="bodyTable"><tr class="a"><th>Package</th>
<th>Tests</th>
<th>Errors </th>
<th>Failures</th>
<th>Skipped</th>
<th>Success Rate</th>
<th>Time</th>
</tr>
<tr class="b"><td><a href="#"></a>
</td>
<td>2</td>
<td>0</td>
<td>2</td>
<td>0</td>
<td>0%</td>
<td>0.397</td>
</tr>
</table>

and this table for class list:
<table class="bodyTable"><tr class="a"><th></th>
<th>Class</th>
<th>Tests</th>
<th>Errors </th>
<th>Failures</th>
<th>Skipped</th>
<th>Success Rate</th>
<th>Time</th>
</tr>
<tr class="b"><td><a href="#TestSuite"><img src="images/
icon_warning_sml.gif" /></a>
</td>
<td><a href="#TestSuite">TestSuite</a>
</td>
<td>2</td>
<td>0</td>
<td>2</td>
<td>0</td>
<td>0%</td>
<td>0.397</td>
</tr>
</table>

I am running TestNG 5.7 and surefire 2.4, and using the following
testNg xml
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite thread-count="5" skipfailedinvocationCounts="false" verbose="0"
name="Failed suite [Command line suite]" junit="false"
annotations="JDK">
  <test name="Command line test(failed)" junit="false"
annotations="JDK">
    <classes>
      <class name="net.apmoller.test2.NgTest">
        <methods>
          <include name="testFailNg"/>
        </methods>
      </class>
      <class name="net.apmoller.maventest.NgTest">
        <methods>
          <include name="testFailNg"/>
        </methods>
      </class>
    </classes>
  </test>
</suite>

please note that I have removed the testcases which succeded

best regards
Henrik


On Jan 16, 3:23 pm, "Andreas Guther" <[hidden email]> wrote:

> The information you have provided is not sufficient to make an assessment.
> What about running the tests with 'mvn test'?  Does everything work as
> expected here?  What version of surefire are you using?
>
> Andreas
>
> On Jan 16, 2008 3:42 AM, Henrik Lynggaard <[hidden email]>
> wrote:
>
>
>
> > Hi
>
> > I am testing testng with maven2 and I have run into a problem.
>
> > In the surefire report there is missing some information, both the
> > package list and class list is wrong.
>
> > The setup is ass following. I have created small project with two test
> > packages, each with one test class, which in turn each has 2 test
> > methods one that fails and one that succeeds.
>
> > The in the surefire report the package list contains 1 row with blank
> > package name claiming 4 run tests and 2 failures.
>
> > The Class list also only contains 1 row, class name TestSuite and
> > claiming 4 run tests and 2 failures
>
> > Both lists should contain 2 rows with 2 runs and 1 failure in each.
>
> > plugin versions:
> > testng 5.7
> > surefire-report: 2.4
> > [DEBUG]Adding managed dependencies for unknown:surefire-testng
> > [DEBUG]  org.apache.maven.surefire:surefire-api:jar:2.4
> > [DEBUG]  org.apache.maven.surefire:surefire-booter:jar:2.4
> > [DEBUG]  org.codehaus.plexus:plexus-utils:jar:1.4.9
> > [DEBUG]  org.apache.maven.surefire:surefire-testng:jar:2.4:test
> > (selected for test)
>
> > Am I doing something wrong oor have I found a bug ?
>
> > Best regards
> > Henrik
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: Problems with surefire reports, missing information

Henrik Lynggaard Hansen
In reply to this post by Andreas Guther-2

Hi again

I have had a brief look at the surefire 2.4 reports plugin and it
appears that it doesn't contain any code to handle testNG's style of
test reports. And the format of the two are somewhat different.

The junit runner creates a TEST-*.xml for each class with a testcase
for each failing method. the name attribute of the testsuite is the
fully qualified classname

The testng runner creates a single TEST-*.xml file, with a testcase
for each failing method regardless of class. The name attribute of the
testsuite is "TestSuite"

If my observation is correct then the question is how to fix this:

1)Fix testNG to generate the same output as junit ?
2) Fix the surefire testng provider to execute the tests correctly?
3) fix the reporter plugin to handle testNG output e.g. by using
testng-result.xml as input ?

Can someone confirm my observation and clarify what the intended
behaviour is ?

Best regards
Henrik

On Jan 16, 3:23 pm, "Andreas Guther" <[hidden email]> wrote:

> The information you have provided is not sufficient to make an assessment.
> What about running the tests with 'mvn test'?  Does everything work as
> expected here?  What version of surefire are you using?
>
> Andreas
>
> On Jan 16, 2008 3:42 AM, Henrik Lynggaard <[hidden email]>
> wrote:
>
>
>
> > Hi
>
> > I am testing testng with maven2 and I have run into a problem.
>
> > In the surefire report there is missing some information, both the
> > package list and class list is wrong.
>
> > The setup is ass following. I have created small project with two test
> > packages, each with one test class, which in turn each has 2 test
> > methods one that fails and one that succeeds.
>
> > The in the surefire report the package list contains 1 row with blank
> > package name claiming 4 run tests and 2 failures.
>
> > The Class list also only contains 1 row, class name TestSuite and
> > claiming 4 run tests and 2 failures
>
> > Both lists should contain 2 rows with 2 runs and 1 failure in each.
>
> > plugin versions:
> > testng 5.7
> > surefire-report: 2.4
> > [DEBUG]Adding managed dependencies for unknown:surefire-testng
> > [DEBUG]  org.apache.maven.surefire:surefire-api:jar:2.4
> > [DEBUG]  org.apache.maven.surefire:surefire-booter:jar:2.4
> > [DEBUG]  org.codehaus.plexus:plexus-utils:jar:1.4.9
> > [DEBUG]  org.apache.maven.surefire:surefire-testng:jar:2.4:test
> > (selected for test)
>
> > Am I doing something wrong oor have I found a bug ?
>
> > Best regards
> > Henrik
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: Problems with surefire reports, missing information

dfabulich

Henrik Lynggaard wrote:

> If my observation is correct then the question is how to fix this:
>
> 1) Fix testNG to generate the same output as junit ?
> 2) Fix the surefire testng provider to execute the tests correctly?
> 3) fix the reporter plugin to handle testNG output e.g. by using
> testng-result.xml as input ?

In this case, while TestNG is generating its own testng-result.xml file,
the TEST-*.xml files are actually generated by Surefire, not by TestNG.

Basically, Surefire instructs TestNG not to generate JUnit XML, while
Surefire registers itself with TestNG to be "notified" of test results as
they happen.  Surefire then writes the XML file itself.

So, regardless how we ultimately fix it, this is a bug in Surefire.

Please file a bug here:
http://jira.codehaus.org/secure/CreateIssue!default.jspa

It would be very helpful if you were to attach a "reduced test case" in
the form of a minimal Maven project that demonstrates the problem.

Thanks!

-Dan

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: Problems with surefire reports, missing information

Henrik Lynggaard Hansen

Hello

Thanx for the clarification, with your help I actually think I found
the problem in the code.

I have created http://jira.codehaus.org/browse/SUREFIRE-433 covering
the issue including a test project

Best Regards
Henrik

On Jan 16, 11:09 pm, Dan Fabulich <[hidden email]> wrote:

> Henrik Lynggaard wrote:
> > If my observation is correct then the question is how to fix this:
>
> > 1) Fix testNG to generate the same output as junit ?
> > 2) Fix the surefire testng provider to execute the tests correctly?
> > 3) fix the reporter plugin to handle testNG output e.g. by using
> > testng-result.xml as input ?
>
> In this case, while TestNG is generating its own testng-result.xml file,
> the TEST-*.xml files are actually generated by Surefire, not by TestNG.
>
> Basically, Surefire instructs TestNG not to generate JUnit XML, while
> Surefire registers itself with TestNG to be "notified" of test results as
> they happen.  Surefire then writes the XML file itself.
>
> So, regardless how we ultimately fix it, this is a bug in Surefire.
>
> Please file a bug here:http://jira.codehaus.org/secure/CreateIssue!default.jspa
>
> It would be very helpful if you were to attach a "reduced test case" in
> the form of a minimal Maven project that demonstrates the problem.
>
> Thanks!
>
> -Dan
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en
-~----------~----~----~----~------~----~------~--~---