Re: TestNG "haltonfailure" Ant task

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: TestNG "haltonfailure" Ant task

bklau

Alexandru Popescu wrote:

> On 11/21/06, BenLau <[hidden email]> wrote:
> >
> > So, Alex:
> >
> > In short, I need :
> >
> > 1) haltonfailures  to control failures originated from  running tests.
> > 2) haltonerrors to control any other errors/exception not originating
> > from tests themselves.
> >
> > So what do you think?.
> >
>
> Exactly what I was saying a couple of emails before:
>
> [quote]Nope it is not so clear. Can you show me an example pls?[/quote]

Alex:

Here's a sample log below to emphasize what I'm talking about.
I set the haltonfailure to false. So even if the TestNG runner fails or
test itself failed, I don't know which fails
because "haltonfailure" is a "catch all"  for both exception origins.
I want to  be able to take different course of action depending on the
origin of the failures.
The failure below was not due to running the tests itself but rather
due to a "ClassNotFound" exception while the runner initializes.
So its not a "proper failure" due to the tests itself but due to the
runner hosting the tests.
I need to be able to separate out failures orginated from the test
proper vs  other failures due to other cause.
For a developer running tests from an IDE, it might not matter, but for
an enterprise-level automation which runs thousands of tests from
different product group, it does matter to distinguish between the 2
failure mode.

That's what I mean.

-BK-

Sample log:
============================================================================================
2006-12-05 13:55:11,141[main] INFO        org.testng.TestNGAntTask  -
[TestNGAntTask] TESTNG PASSED
@C:\DOCUME~1\XYZ\LOCALS~1\Temp\testng14749 WHICH CONTAINS:
2006-12-05 13:55:11,156[main] INFO        org.testng.TestNGAntTask  -
[TestNGAntTask] -log 1 -d
G:\TestNG_5.1_Stage4\xute-0.3.0\testresults\org\junit\tests\JUnit-QuickTestWithSetup3.scen-XYZ-145
-sourcedir G:\TestNG_5.1_Stage4\xute-0.3.0 -listener
com.a.b.automation.xutex.runtime.Log4jListener -listener
com.a.b.automation.xutex.runtime.SuiteListener -suitename "Ant suite"
-testname "Ant test"
"G:\TestNG_5.1_Stage4\xute-0.3.0\tmp\suites\junit-suite-1.xml"
2006-12-05 13:55:11,188[main] DEBUG       org.testng.TestNGAntTask  -
Executing 'G:\TestNG_5.1_Stage4\xute-0.3.0\jdk1.4.2\jre\bin\java.exe'
with arguments:
'-Djava.library.path='
'-Dlog4j.configuration=log4j.test.xml'
'-Demma.rt.control.port=55555'
'-Demma.coverage.out.file=G:\TestNG_5.1_Stage4\xute-0.3.0\testresults\org\junit\tests\JUnit-QuickTestWithSetup3.scen-XYZ-145\coverage\result/coverage.ec'
'-Demma.coverage.out.merge=true'
'-Dtestng.show.stack.frames=true'
'-Xms32M'
'-Xmx128M'
'-ea'
'-Dxute.home=G:\TestNG_5.1_Stage4\xute-0.3.0'
'-Dxute.tai.log=xute.tai.log.xml'
'-classpath'
'G:\TestNG_5.1_Stage4\xute-0.3.0\tmp\instr\lib\junit.jar;G:\TestNG_5.1_Stage4\xute-0.3.0\tmp\instr\lib\junit.samples.money.jar;G:\TestNG_5.1_Stage4\xute-0.3.0\testcontents\org\junit\tests\junit.samples.money.jar;G:\TestNG_5.1_Stage4\xute-0.3.0\lib\apache\ant\lib\junit.jar;G:\TestNG_5.1_Stage4\xute-0.3.0\lib\framework\xute.jar;G:\TestNG_5.1_Stage4\xute-0.3.0\lib\apache\ant\lib\log4j-1.2.13.jar;G:\TestNG_5.1_Stage4\xute-0.3.0\run;G:\TestNG_5.1_Stage4\xute-0.3.0\lib\framework\testng-5.3-jdk14.jar'
'org.testng.TestNG'
'@C:\DOCUME~1\bklau\LOCALS~1\Temp\testng14749'

The ' characters around the executable and arguments are
not part of the command.
2006-12-05 13:55:11,953[Thread-5] WARN        org.testng.TestNGAntTask
- Exception in thread "main" java.lang.NoClassDefFoundError:
com/a/b/automation/logging/LoggerDispatcher
======================================================================================================

>
> ./alex
> --
> .w( the_mindstorm )p.
>   TestNG co-founder
> EclipseTestNG Creator
>
> > Thanks,
> >
> > BK
> > Alexandru Popescu wrote:
> > > On 11/15/06, BenLau <[hidden email]> wrote:
> > > >
> > > >
> > > > Alexandru Popescu wrote:
> > > > > On 11/15/06, BK Lau <[hidden email]> wrote:
> > > > > >
> > > > > > Hi :
> > > > > > The TestNG "haltonfailure" Ant property is defined in the TestNG doc as
> > > > > >  "Stop the build process if a failure has occurred during the test run.  No. Defaults to false ".
> > > > > >
> > > > > > I got to play with it a bit and noticed that it throws an exception if at least one of the tests failed. This assumes of course that the test got to to run in the first place... BUT this might not even get to the point of actually running the test.
> > > > > >
> > > > > > I have a different concern: Not of any tests failed , but if testng  failed to run for some reasons:
> > > > > > For example, if some erroreous parameters are passed to TestNG or testng couldn't find some classes in the classpath, then the testng Ant task would failed.
> > > > > > However,I would like it to failed gracefully instead of  failing the Ant Build script the TestNG Ant task is in.
> > > > >
> > > > > > This is to allow the Ant Build script to call another Ant target or Ant script to send out an email notification. If the Ant Build script the TestNG Ant task is in just failed, then an exception is thrown and the Build failed in the current TestNG 5.3
> > > > > >
> > > > >
> > > > > I am not quite sure I understand. haltonfailure must end the build
> > > > > execution. In case you want to continue with something else even if an
> > > > > failure occured use the corresponding property and trigger the email
> > > > > trigger task.
> > > > >
> > > > > <target name="test>
> > > > >    <testng ... failureProperty="test.failure" />
> > > > >     <antcall target="failure.email" />
> > > > > </target>
> > > > >
> > > > > <target name="failure.email" if="test.failure">
> > > > >    <!-- send email -->
> > > > > </target>
> > > > >
> > > >
> > > > Alex:
> > > > Let me explian more clearly:
> > > > The above code will only work if and only if TestNG runner itself does
> > > > encountered any errors during its initialization phase.
> > > > >From my perspective ,at least with JUnit tests running under TestNG,
> > > > is that haltonfailure="true" failureProperty="test.failure"
> > > > intended purpose is to stop the tests itself, not the test
> > > > infrastructure( i.e. TestNG runner) that runs the tests.
> > > > There is a distinction here. The tests failures vs test infrastructure
> > > > failures.
> > > > Tests failures are fairly normal in regular devs cycle, as we should
> > > > expect; devs want to bail out of further running tests in order to fix
> > > > a failed test first.
> > > > In a fully automated suite with  thousands of tests, with high setup
> > > > cost and time, typically, we don't want to do that. We picked thru
> > > > failed tests later on.
> > > > But test infrastructure failures are abnormal since it means the the
> > > > conditions for the TestNG runner has not been met and that we can't
> > > > even attempt to run
> > > > a test.
> > > >
> > > > In short failures due to TestNG runner itself directly/indirectly vs
> > > > failures in the tests themselves are separate issues.
> > > > Hope this is clear.
> > > >
> > >
> > > Nope it is not so clear. Can you show me an example pls?
> > >
> > > ./alex
> > > --
> > > .w( the_mindstorm )p.
> > >   TestNG co-founder
> > > EclipseTestNG Creator
> > >
> > > > > > So, if it not too much to ask if another "haltonerror" property and its associated "errorproperty"  can be implemented for this purposein next TestNG5.4 release?.
> > > > > >
> > > > >
> > > > > So, far we have avoided to re-introduce these 2 terms, used quite
> > > > > frequently in JUnit and that almost always got me confused.
> > > > > Considering the above approach I don't think this is really needed.
> > > > >
> > > > > HTH,
> > > > >
> > > > > ./alex
> > > > > --
> > > > > .w( the_mindstorm )p.
> > > > >   TestNG co-founder
> > > > > EclipseTestNG Creator
> > > > >
> > > > > >
> > > > > > -BK-
> > > > > > ---------------------------------------------------------------------
> > > > > > Posted via Jive Forums
> > > > > > http://forums.opensymphony.com/thread.jspa?threadID=50574&messageID=101719#101719
> > > > > >
> > > > > >
> > > > > > >
> > > > > >
> > > >
> > > >
> >
> >
> > >
> >


--~--~---------~--~----~------------~-------~--~----~
 You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Loading...