TestNG and CruiseControl

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

TestNG and CruiseControl

JonAzpiroz

Hi all,

I'm using Cruisecontrol 2.5 and TestNG 2.4, and I'm having a
frustrating problem. CruiseControl uses TestNG to run all out project
test and it does it properly. We are having troubles with the
presentation of the test results that CruiseControl is making. The main
page indicates that the build has been successful even if a test has
failed. If we use the project link we have the following page:

BUILD COMPLETE -  build.9
Date of build: 01/17/2007 17:35:11
Time to build: 8 seconds
Last changed: 01/17/2007 17:34:10
Last log entry:
Build Artifacts

 Errors/Warnings: (3)

Note: C:\Program
Files\CruiseControl\builds\checkout\dmsTestNG\test\com\pragprog\dms\selftest\ClasspathTests.java
uses unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
The usage of -target option is deprecated. Please use -annotations
instead

 Unit Tests: (2)
failure testTitle SuperTest

 Unit Test Error Details: (1)
Test: testTitle
Class: SuperTest

java.lang.AssertionError: expected:<Gettysburg Address> but was:<You
are going to fail>

        at com.pragprog.dms.DocumentTest.testTitle(Unknown Source)

        ... Removed 26 stack frames

As you can see, it says that the build is COMPLETE instead of FAILED,
but it also indicates that there is a Unit Test Error.

Does anyone have any ideas in how to fix this problem. Thanks.


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: TestNG and CruiseControl

Alexandru Popescu ☀

On 1/17/07, JonAzpiroz <[hidden email]> wrote:

>
> Hi all,
>
> I'm using Cruisecontrol 2.5 and TestNG 2.4, and I'm having a
> frustrating problem. CruiseControl uses TestNG to run all out project
> test and it does it properly. We are having troubles with the
> presentation of the test results that CruiseControl is making. The main
> page indicates that the build has been successful even if a test has
> failed. If we use the project link we have the following page:
>
> BUILD COMPLETE -  build.9
> Date of build:  01/17/2007 17:35:11
> Time to build:  8 seconds
> Last changed:   01/17/2007 17:34:10
> Last log entry:
> Build Artifacts
>
>  Errors/Warnings: (3)
>
> Note: C:\Program
> Files\CruiseControl\builds\checkout\dmsTestNG\test\com\pragprog\dms\selftest\ClasspathTests.java
> uses unchecked or unsafe operations.
> Note: Recompile with -Xlint:unchecked for details.
> The usage of -target option is deprecated. Please use -annotations
> instead
>
>  Unit Tests: (2)
> failure         testTitle       SuperTest
>
>  Unit Test Error Details: (1)
> Test:   testTitle
> Class:  SuperTest
>
> java.lang.AssertionError: expected:<Gettysburg Address> but was:<You
> are going to fail>
>
>         at com.pragprog.dms.DocumentTest.testTitle(Unknown Source)
>
>         ... Removed 26 stack frames
>
> As you can see, it says that the build is COMPLETE instead of FAILED,
> but it also indicates that there is a Unit Test Error.
>
> Does anyone have any ideas in how to fix this problem. Thanks.
>

Hi!

Are you sure your TestNG version is 2.4? It would be quite old
compaired with the last released (5.4), so my next question would be
if you ever considered upgrading? Also, I  am noticing some output
that I don't think was there in 2.4.

As per problem at hand, the build complete is not a message TestNG is
emitting but CC, so in the first place I would suspect a problem with
their communication. Is your CC triggering the test run through an Ant
build? If so, do you think you can show us the relevant pieces from
it?

BR,

./alex
--
.w( the_mindstorm )p.
  TestNG co-founder
EclipseTestNG Creator

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: TestNG and CruiseControl

JonAzpiroz

Hi,
Thanks for your quick response. I am going to explain a little bit more
our problem.

> Are you sure your TestNG version is 2.4?
Sorry, we are not using version 2.4, it was a mistake. we are using the
last released 5.4

> Is your CC triggering the test run through an Ant
> build? If so, do you think you can show us the relevant pieces from it?
CC uses an ant build to compile our classes and make the tests.

We first used JUnit to test our classes, and it worked fine with CC. CC
runs correctly the test classes. In the build.xml you have to use
<formatter type="xml" /> inside the junit tag, to have the test results
in an xml file:

<junit errorProperty="test.failed" failureProperty="test.failed">
      <classpath refid="project.classpath" />
      <formatter type="brief" usefile="false" />
      <formatter type="xml" />
      <batchtest todir="${test.xml.dir}">
        <fileset dir="${build.test.dir}"
          includes="**/*Test.class" />
      </batchtest>
</junit>

CC uses these xml files to publish the results in its web application.


But we want to migrate to TestNG because we want to run integration
test also. CC uses our new ant file where we have a testng tag to run
the TestNG tests:

 <target name="test" depends="compile-tests">
    <taskdef resource="testngtasks" classpathref="project.classpath"/>
    <delete dir="${test.xml.dir}"/>
    <mkdir dir="${test.xml.dir}"/>
    <testng
            classpathref="project.classpath"
            outputDir="${test.xml.dir}"
            target="1.5"
    >
         <xmlfileset dir="." includes="testng.xml"/>
    </testng>
  </target>

TestNG generates the test results as an XML, but the format is not the
same as the JUnit test results. Because of this difference, CC doesn't
publish the results correctly. CC first tells you if the build FAILED
or if its COMPLETED, and you can see the details of the build were you
can see the error messages and the time consumed by each test.  With
JUnit both pages (the main and the detailed one) are shown right, but
with TestNG, CC always thinks that the build is COMPLETED even if it
had errors. But if you see the detailed page you can see the error
messages (even if it first says that there was no errors). The problem
with this behaviour is that CC sends messages to the programmers if the
build failed, but because it thinks there are no errors, the messages
are not send.

I think that the entire problem is with the different XML test results
that JUnit and TestNG make. In this piece of code of the config.xml
file, you tell CC wherethe test results are:

<log>
            <merge dir="projects/${project.name}/target/test-results"/>
</log>

I think these are the relevant pieces of code, if you need other parts,
please tell me. Thanks a lot for your help.


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: TestNG and CruiseControl

Alexandru Popescu ☀

On 1/19/07, JonAzpiroz <[hidden email]> wrote:

>
> Hi,
> Thanks for your quick response. I am going to explain a little bit more
> our problem.
>
> > Are you sure your TestNG version is 2.4?
> Sorry, we are not using version 2.4, it was a mistake. we are using the
> last released 5.4
>
> > Is your CC triggering the test run through an Ant
> > build? If so, do you think you can show us the relevant pieces from it?
> CC uses an ant build to compile our classes and make the tests.
>
> We first used JUnit to test our classes, and it worked fine with CC. CC
> runs correctly the test classes. In the build.xml you have to use
> <formatter type="xml" /> inside the junit tag, to have the test results
> in an xml file:
>
> <junit errorProperty="test.failed" failureProperty="test.failed">
>       <classpath refid="project.classpath" />
>       <formatter type="brief" usefile="false" />
>       <formatter type="xml" />
>       <batchtest todir="${test.xml.dir}">
>         <fileset dir="${build.test.dir}"
>           includes="**/*Test.class" />
>       </batchtest>
> </junit>
>
> CC uses these xml files to publish the results in its web application.
>
>
> But we want to migrate to TestNG because we want to run integration
> test also. CC uses our new ant file where we have a testng tag to run
> the TestNG tests:
>
>  <target name="test" depends="compile-tests">
>     <taskdef resource="testngtasks" classpathref="project.classpath"/>
>     <delete dir="${test.xml.dir}"/>
>     <mkdir dir="${test.xml.dir}"/>
>     <testng
>             classpathref="project.classpath"
>             outputDir="${test.xml.dir}"
>             target="1.5"
>     >
>          <xmlfileset dir="." includes="testng.xml"/>
>     </testng>
>   </target>
>
> TestNG generates the test results as an XML, but the format is not the
> same as the JUnit test results. Because of this difference, CC doesn't
> publish the results correctly. CC first tells you if the build FAILED
> or if its COMPLETED, and you can see the details of the build were you
> can see the error messages and the time consumed by each test.  With
> JUnit both pages (the main and the detailed one) are shown right, but
> with TestNG, CC always thinks that the build is COMPLETED even if it
> had errors. But if you see the detailed page you can see the error
> messages (even if it first says that there was no errors). The problem
> with this behaviour is that CC sends messages to the programmers if the
> build failed, but because it thinks there are no errors, the messages
> are not send.
>
> I think that the entire problem is with the different XML test results
> that JUnit and TestNG make. In this piece of code of the config.xml
> file, you tell CC wherethe test results are:
>
> <log>
>             <merge dir="projects/${project.name}/target/test-results"/>
> </log>
>
> I think these are the relevant pieces of code, if you need other parts,
> please tell me. Thanks a lot for your help.
>

Thanks for the details. I guess that in this case what we need to
figure out are the differences between the XML output. Unfortunately,
I don't have JUnit XML output examples, so I would kindly ask you if
possible to submit (privately if you cannot make it public) the XML
reports generated by your prev JUnit run and your current TestNG XML
output. Or if this is not possible then try to create your own
ITestListener or IReporter that creates an XML matching exactly the
format CC and JUnit are using.

BR,

./alex
--
.w( the_mindstorm )p.
  TestNG co-founder
EclipseTestNG Creator

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: TestNG and CruiseControl

Anson-5
In reply to this post by JonAzpiroz

The build isn't failing because you didn't tell it to fail.  Based on
your testng tag above, you need to either:

1) add haltonfailure="true" as an attribute of the testng task
2) add failureProperty="test.failed" as an attribute of the testng
task, *and* a <fail if="test.failed"/> tag somewhere after its
invocation.

Either of these will make CruiseControl report a failed build.  The
format of the XML outputs has nothing to do with whether CruiseControl
reports successful or failed builds.

-aB


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: TestNG and CruiseControl

JonAzpiroz

Thanks a lot, problem has been fixed.
I show the test target that finally works:

<target name="test" depends="compile-tests">
    <taskdef resource="testngtasks" classpathref="project.classpath"/>
    <delete dir="${test.xml.dir}"/>
    <mkdir dir="${test.xml.dir}"/>
    <testng
            classpathref="project.classpath"
            outputDir="${test.xml.dir}"
            target="1.5"
            failureProperty="test.failed"
    >
         <xmlfileset dir="." includes="testng.xml"/>
    </testng>
    <fail message="Tests failed! Check test reports."
      if="test.failed" />
  </target>

I hope this can help somebody else.


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en
-~----------~----~----~----~------~----~------~--~---