how to have my Jenkins job fail when TestNG configuration methods have failures

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

how to have my Jenkins job fail when TestNG configuration methods have failures

Todd Bradley
Hi, I could use some advice. We use Jenkins to run TestNG based tests via Gradle. When TestNG exits with a 0 status, Gradle considers that to be successful, and so my Jenkins job is considered a success, too.

However, if there is a failure in a configuration method (like @BeforeClass) that causes some tests to be skipped, TestNG still exits with 0. I want Gradle and Jenkins to notice that it's a problem when configuration methods fail, and flag the build as a failure. Do you have advice how to do this?


Thanks,
Todd.

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: how to have my Jenkins job fail when TestNG configuration methods have failures

Todd Bradley
Any ideas? How do other people who use TestNG with a CI system handle this?

On Friday, August 25, 2017 at 2:41:47 PM UTC-6, Todd Bradley wrote:
Hi, I could use some advice. We use Jenkins to run TestNG based tests via Gradle. When TestNG exits with a 0 status, Gradle considers that to be successful, and so my Jenkins job is considered a success, too.

However, if there is a failure in a configuration method (like @BeforeClass) that causes some tests to be skipped, TestNG still exits with 0. I want Gradle and Jenkins to notice that it's a problem when configuration methods fail, and flag the build as a failure. Do you have advice how to do this?


Thanks,
Todd.

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: how to have my Jenkins job fail when TestNG configuration methods have failures

⇜Krishnan Mahadevan⇝

Todd,

How are you stating that TestNG exits always with a 0 status?

To reproduce the problem, I created a simple maven project [ The project has both a pom.xml and a build.gradle file so that it can be run both by Maven and Gradle ].

 

In both the cases, when the configuration fails (I am explicitly throwing a RuntimeException within a @BeforeClass annotated method), the build fails. Please see below output:

 

Gradle output:

19:25 $ gradle clean test

 

> Task :test

 

com.rationaleemotions.AppTest.beforeClass FAILED

    java.lang.RuntimeException at AppTest.java:13

 

2 tests completed, 1 failed, 1 skipped

 

 

FAILURE: Build failed with an exception.

 

* What went wrong:

Execution failed for task ':test'.

> There were failing tests. See the report at: file:///Users/krmahadevan/temp/example/build/reports/tests/test/index.html

 

* Try:

Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.

 

* Get more help at https://help.gradle.org

 

BUILD FAILED in 1s

3 actionable tasks: 3 executed

 

 

Maven output:

 

19:26 $ mvn clean test

[INFO] Scanning for projects...

[INFO]

[INFO] ------------------------------------------------------------------------

[INFO] Building example 1.0-SNAPSHOT

[INFO] ------------------------------------------------------------------------

[INFO]

[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ example ---

[INFO] Deleting /Users/krmahadevan/temp/example/target

[INFO]

[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ example ---

[INFO] Using 'UTF-8' encoding to copy filtered resources.

[INFO] skip non existing resourceDirectory /Users/krmahadevan/temp/example/src/main/resources

[INFO]

[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ example ---

[INFO] No sources to compile

[INFO]

[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ example ---

[INFO] Using 'UTF-8' encoding to copy filtered resources.

[INFO] skip non existing resourceDirectory /Users/krmahadevan/temp/example/src/test/resources

[INFO]

[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ example ---

[INFO] Changes detected - recompiling the module!

[INFO] Compiling 1 source file to /Users/krmahadevan/temp/example/target/test-classes

[INFO]

[INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ example ---

[INFO] Surefire report directory: /Users/krmahadevan/temp/example/target/surefire-reports

 

-------------------------------------------------------

T E S T S

-------------------------------------------------------

Running com.rationaleemotions.AppTest

Configuring TestNG with: org.apache.maven.surefire.testng.conf.TestNG652Configurator@3b764bce

Tests run: 2, Failures: 1, Errors: 0, Skipped: 1, Time elapsed: 0.282 sec <<< FAILURE!

beforeClass(com.rationaleemotions.AppTest)  Time elapsed: 0.168 sec  <<< FAILURE!

java.lang.RuntimeException: Simulating a configuration failure

    at com.rationaleemotions.AppTest.beforeClass(AppTest.java:13)

 

 

Results :

 

Failed tests:   beforeClass(com.rationaleemotions.AppTest): Simulating a configuration failure

 

Tests run: 2, Failures: 1, Errors: 0, Skipped: 1

 

[INFO] ------------------------------------------------------------------------

[INFO] BUILD FAILURE

[INFO] ------------------------------------------------------------------------

[INFO] Total time: 2.623 s

[INFO] Finished at: 2017-08-29T19:27:25+05:30

[INFO] Final Memory: 16M/249M

[INFO] ------------------------------------------------------------------------

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.12.4:test (default-test) on project example: There are test failures.

[ERROR]

[ERROR] Please refer to /Users/krmahadevan/temp/example/target/surefire-reports for the individual test results.

[ERROR] -> [Help 1]

[ERROR]

[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.

[ERROR] Re-run Maven using the -X switch to enable full debug logging.

[ERROR]

[ERROR] For more information about the errors and possible solutions, please read the following articles:

[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException

 

 

 

Thanks & Regards

Krishnan Mahadevan

 

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"

My Scribblings @ http://wakened-cognition.blogspot.com/

My Technical Scribbings @ http://rationaleemotions.wordpress.com/

 

From: <[hidden email]> on behalf of Todd Bradley <[hidden email]>
Reply-To: <[hidden email]>
Date: Tuesday, August 29, 2017 at 3:50 AM
To: testng-users <[hidden email]>
Subject: [testng-users] Re: how to have my Jenkins job fail when TestNG configuration methods have failures

 

Any ideas? How do other people who use TestNG with a CI system handle this?

On Friday, August 25, 2017 at 2:41:47 PM UTC-6, Todd Bradley wrote:

Hi, I could use some advice. We use Jenkins to run TestNG based tests via Gradle. When TestNG exits with a 0 status, Gradle considers that to be successful, and so my Jenkins job is considered a success, too.

 

However, if there is a failure in a configuration method (like @BeforeClass) that causes some tests to be skipped, TestNG still exits with 0. I want Gradle and Jenkins to notice that it's a problem when configuration methods fail, and flag the build as a failure. Do you have advice how to do this?

 

 

Thanks,

Todd.

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.

example.zip (10K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: how to have my Jenkins job fail when TestNG configuration methods have failures

Todd Bradley-2
Ah, I see the difference in our approaches. You're using Maven's and Gradle's built-in TestNG runners. My Gradle task is of type JavaExec, which is equivalent to running "java ... org.testng.TestNG" on a command line. 

So when I say that TestNG exits with status 0, I mean that Java, when running TestNG's main method, exits with 0 when configuration methods fail, if no test methods fail. 

Tracing through the source for TestNG.java, you can see why this is the case. Or, if you want to reproduce this empirically, create a Gradle task like this in your build.gradle file:

task ('myTask', type: JavaExec) {
        main = 'org.testng.TestNG'
        classpath = sourceSets.main.runtimeClasspath + sourceSets.test.runtimeClasspath
        args = ["-testclass", "com.rationaleemotions.AppTest"]
}


And then do "gradle myTask". You'll see output like this:

===============================================
Command line suite
Total tests run: 1, Failures: 0, Skips: 1
Configuration Failures: 1, Skips: 0
===============================================

BUILD SUCCESSFUL
Total time: 1.025 secs

Does that help explain it?

Thanks,
Todd.





On Tue, Aug 29, 2017 at 8:07 AM, Krishnan Mahadevan <[hidden email]> wrote:

Todd,

How are you stating that TestNG exits always with a 0 status?

To reproduce the problem, I created a simple maven project [ The project has both a pom.xml and a build.gradle file so that it can be run both by Maven and Gradle ].

 

In both the cases, when the configuration fails (I am explicitly throwing a RuntimeException within a @BeforeClass annotated method), the build fails. Please see below output:

 

Gradle output:

19:25 $ gradle clean test

 

> Task :test

 

com.rationaleemotions.AppTest.beforeClass FAILED

    java.lang.RuntimeException at AppTest.java:13

 

2 tests completed, 1 failed, 1 skipped

 

 

FAILURE: Build failed with an exception.

 

* What went wrong:

Execution failed for task ':test'.

> There were failing tests. See the report at: file:///Users/krmahadevan/temp/example/build/reports/tests/test/index.html

 

* Try:

Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.

 

* Get more help at https://help.gradle.org

 

BUILD FAILED in 1s

3 actionable tasks: 3 executed

 

 

Maven output:

 

19:26 $ mvn clean test

[INFO] Scanning for projects...

[INFO]

[INFO] ------------------------------------------------------------------------

[INFO] Building example 1.0-SNAPSHOT

[INFO] ------------------------------------------------------------------------

[INFO]

[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ example ---

[INFO] Deleting /Users/krmahadevan/temp/example/target

[INFO]

[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ example ---

[INFO] Using 'UTF-8' encoding to copy filtered resources.

[INFO] skip non existing resourceDirectory /Users/krmahadevan/temp/example/src/main/resources

[INFO]

[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ example ---

[INFO] No sources to compile

[INFO]

[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ example ---

[INFO] Using 'UTF-8' encoding to copy filtered resources.

[INFO] skip non existing resourceDirectory /Users/krmahadevan/temp/example/src/test/resources

[INFO]

[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ example ---

[INFO] Changes detected - recompiling the module!

[INFO] Compiling 1 source file to /Users/krmahadevan/temp/example/target/test-classes

[INFO]

[INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ example ---

[INFO] Surefire report directory: /Users/krmahadevan/temp/example/target/surefire-reports

 

-------------------------------------------------------

T E S T S

-------------------------------------------------------

Running com.rationaleemotions.AppTest

Configuring TestNG with: org.apache.maven.surefire.testng.conf.TestNG652Configurator@3b764bce

Tests run: 2, Failures: 1, Errors: 0, Skipped: 1, Time elapsed: 0.282 sec <<< FAILURE!

beforeClass(com.rationaleemotions.AppTest)  Time elapsed: 0.168 sec  <<< FAILURE!

java.lang.RuntimeException: Simulating a configuration failure

    at com.rationaleemotions.AppTest.beforeClass(AppTest.java:13)

 

 

Results :

 

Failed tests:   beforeClass(com.rationaleemotions.AppTest): Simulating a configuration failure

 

Tests run: 2, Failures: 1, Errors: 0, Skipped: 1

 

[INFO] ------------------------------------------------------------------------

[INFO] BUILD FAILURE

[INFO] ------------------------------------------------------------------------

[INFO] Total time: 2.623 s

[INFO] Finished at: 2017-08-29T19:27:25+05:30

[INFO] Final Memory: 16M/249M

[INFO] ------------------------------------------------------------------------

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.12.4:test (default-test) on project example: There are test failures.

[ERROR]

[ERROR] Please refer to /Users/krmahadevan/temp/example/target/surefire-reports for the individual test results.

[ERROR] -> [Help 1]

[ERROR]

[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.

[ERROR] Re-run Maven using the -X switch to enable full debug logging.

[ERROR]

[ERROR] For more information about the errors and possible solutions, please read the following articles:

[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException

 

 

 

Thanks & Regards

Krishnan Mahadevan

 

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"

My Scribblings @ http://wakened-cognition.blogspot.com/

My Technical Scribbings @ http://rationaleemotions.wordpress.com/

 

From: <[hidden email]> on behalf of Todd Bradley <[hidden email]>
Reply-To: <[hidden email]>
Date: Tuesday, August 29, 2017 at 3:50 AM
To: testng-users <[hidden email]>
Subject: [testng-users] Re: how to have my Jenkins job fail when TestNG configuration methods have failures

 

Any ideas? How do other people who use TestNG with a CI system handle this?

On Friday, August 25, 2017 at 2:41:47 PM UTC-6, Todd Bradley wrote:

Hi, I could use some advice. We use Jenkins to run TestNG based tests via Gradle. When TestNG exits with a 0 status, Gradle considers that to be successful, and so my Jenkins job is considered a success, too.

 

However, if there is a failure in a configuration method (like @BeforeClass) that causes some tests to be skipped, TestNG still exits with 0. I want Gradle and Jenkins to notice that it's a problem when configuration methods fail, and flag the build as a failure. Do you have advice how to do this?

 

 

Thanks,

Todd.

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: how to have my Jenkins job fail when TestNG configuration methods have failures

⇜Krishnan Mahadevan⇝

Todd,

 

I think it’s a bug in TestNG for the way in which it is currently behaving. But there’s an easy workaround for you, which you can employ to get past this.

 

Please modify your Gradle task to as below:

 

task ('myTask', type: JavaExec) {

    main = 'org.testng.TestNG'

    classpath = sourceSets.main.runtimeClasspath + sourceSets.test.runtimeClasspath

    args = ["-testclass", "com.rationaleemotions.AppTest", "-listener", "org.testng.TestNG\$ExitCodeListener"]

}

 

 

This should start failing your build, when there are configuration failures.

 

The issue in TestNG is that the ExitCodeListener isn’t being wired in as a configuration listener as well, but is merely being wired in as a Test listener.

This causes TestNG to skip its onConfigurationFailure() implementation [ This is the part wherein the status gets set to non-zero value].

In this case, you will a status as “3” indicating a failure(1) and a skip(2)

 

I have filed this as an issue: https://github.com/cbeust/testng/issues/1517

 

 

Thanks & Regards

Krishnan Mahadevan

 

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"

My Scribblings @ http://wakened-cognition.blogspot.com/

My Technical Scribbings @ http://rationaleemotions.wordpress.com/

 

From: <[hidden email]> on behalf of Todd Bradley <[hidden email]>
Reply-To: <[hidden email]>
Date: Wednesday, August 30, 2017 at 3:33 AM
To: <[hidden email]>
Subject: Re: [testng-users] Re: how to have my Jenkins job fail when TestNG configuration methods have failures

 

Ah, I see the difference in our approaches. You're using Maven's and Gradle's built-in TestNG runners. My Gradle task is of type JavaExec, which is equivalent to running "java ... org.testng.TestNG" on a command line. 

 

So when I say that TestNG exits with status 0, I mean that Java, when running TestNG's main method, exits with 0 when configuration methods fail, if no test methods fail. 

 

Tracing through the source for TestNG.java, you can see why this is the case. Or, if you want to reproduce this empirically, create a Gradle task like this in your build.gradle file:

task ('myTask', type: JavaExec) {
        main = 'org.testng.TestNG'
        classpath = sourceSets.main.runtimeClasspath + sourceSets.test.runtimeClasspath
        args = ["-testclass", "com.rationaleemotions.AppTest"]
}



And then do "gradle myTask". You'll see output like this:

 

===============================================
Command line suite
Total tests run: 1, Failures: 0, Skips: 1
Configuration Failures: 1, Skips: 0
===============================================

BUILD SUCCESSFUL
Total time: 1.025 secs

Does that help explain it?

 

Thanks,

Todd.

 

 

 

 

 

On Tue, Aug 29, 2017 at 8:07 AM, Krishnan Mahadevan <[hidden email]> wrote:

Todd,

How are you stating that TestNG exits always with a 0 status?

To reproduce the problem, I created a simple maven project [ The project has both a pom.xml and a build.gradle file so that it can be run both by Maven and Gradle ].

 

In both the cases, when the configuration fails (I am explicitly throwing a RuntimeException within a @BeforeClass annotated method), the build fails. Please see below output:

 

Gradle output:

19:25 $ gradle clean test

 

> Task :test

 

com.rationaleemotions.AppTest.beforeClass FAILED

    java.lang.RuntimeException at AppTest.java:13

 

2 tests completed, 1 failed, 1 skipped

 

 

FAILURE: Build failed with an exception.

 

* What went wrong:

Execution failed for task ':test'.

> There were failing tests. See the report at: file:///Users/krmahadevan/temp/example/build/reports/tests/test/index.html

 

* Try:

Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.

 

* Get more help at https://help.gradle.org

 

BUILD FAILED in 1s

3 actionable tasks: 3 executed

 

 

Maven output:

 

19:26 $ mvn clean test

[INFO] Scanning for projects...

[INFO]

[INFO] ------------------------------------------------------------------------

[INFO] Building example 1.0-SNAPSHOT

[INFO] ------------------------------------------------------------------------

[INFO]

[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ example ---

[INFO] Deleting /Users/krmahadevan/temp/example/target

[INFO]

[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ example ---

[INFO] Using 'UTF-8' encoding to copy filtered resources.

[INFO] skip non existing resourceDirectory /Users/krmahadevan/temp/example/src/main/resources

[INFO]

[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ example ---

[INFO] No sources to compile

[INFO]

[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ example ---

[INFO] Using 'UTF-8' encoding to copy filtered resources.

[INFO] skip non existing resourceDirectory /Users/krmahadevan/temp/example/src/test/resources

[INFO]

[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ example ---

[INFO] Changes detected - recompiling the module!

[INFO] Compiling 1 source file to /Users/krmahadevan/temp/example/target/test-classes

[INFO]

[INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ example ---

[INFO] Surefire report directory: /Users/krmahadevan/temp/example/target/surefire-reports

 

-------------------------------------------------------

T E S T S

-------------------------------------------------------

Running com.rationaleemotions.AppTest

Configuring TestNG with: org.apache.maven.surefire.testng.conf.TestNG652Configurator@3b764bce

Tests run: 2, Failures: 1, Errors: 0, Skipped: 1, Time elapsed: 0.282 sec <<< FAILURE!

beforeClass(com.rationaleemotions.AppTest)  Time elapsed: 0.168 sec  <<< FAILURE!

java.lang.RuntimeException: Simulating a configuration failure

    at com.rationaleemotions.AppTest.beforeClass(AppTest.java:13)

 

 

Results :

 

Failed tests:   beforeClass(com.rationaleemotions.AppTest): Simulating a configuration failure

 

Tests run: 2, Failures: 1, Errors: 0, Skipped: 1

 

[INFO] ------------------------------------------------------------------------

[INFO] BUILD FAILURE

[INFO] ------------------------------------------------------------------------

[INFO] Total time: 2.623 s

[INFO] Finished at: 2017-08-29T19:27:25+05:30

[INFO] Final Memory: 16M/249M

[INFO] ------------------------------------------------------------------------

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.12.4:test (default-test) on project example: There are test failures.

[ERROR]

[ERROR] Please refer to /Users/krmahadevan/temp/example/target/surefire-reports for the individual test results.

[ERROR] -> [Help 1]

[ERROR]

[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.

[ERROR] Re-run Maven using the -X switch to enable full debug logging.

[ERROR]

[ERROR] For more information about the errors and possible solutions, please read the following articles:

[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException

 

 

 

Thanks & Regards

Krishnan Mahadevan

 

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"

My Scribblings @ http://wakened-cognition.blogspot.com/

My Technical Scribbings @ http://rationaleemotions.wordpress.com/

 

From: <[hidden email]> on behalf of Todd Bradley <[hidden email]>
Reply-To: <[hidden email]>
Date: Tuesday, August 29, 2017 at 3:50 AM
To: testng-users <[hidden email]>
Subject: [testng-users] Re: how to have my Jenkins job fail when TestNG configuration methods have failures

 

Any ideas? How do other people who use TestNG with a CI system handle this?

On Friday, August 25, 2017 at 2:41:47 PM UTC-6, Todd Bradley wrote:

Hi, I could use some advice. We use Jenkins to run TestNG based tests via Gradle. When TestNG exits with a 0 status, Gradle considers that to be successful, and so my Jenkins job is considered a success, too.

 

However, if there is a failure in a configuration method (like @BeforeClass) that causes some tests to be skipped, TestNG still exits with 0. I want Gradle and Jenkins to notice that it's a problem when configuration methods fail, and flag the build as a failure. Do you have advice how to do this?

 

 

Thanks,

Todd.

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.

 

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: how to have my Jenkins job fail when TestNG configuration methods have failures

Todd Bradley-2
Thank you, Krishnan. It works! So now I will start using TestNG.ExitCodeListener in my Gradle JavaExec tasks.

 
Regards,
Todd.

On Tue, Aug 29, 2017 at 10:02 PM, Krishnan Mahadevan <[hidden email]> wrote:

Todd,

 

I think it’s a bug in TestNG for the way in which it is currently behaving. But there’s an easy workaround for you, which you can employ to get past this.

 

Please modify your Gradle task to as below:

 

task ('myTask', type: JavaExec) {

    main = 'org.testng.TestNG'

    classpath = sourceSets.main.runtimeClasspath + sourceSets.test.runtimeClasspath

    args = ["-testclass", "com.rationaleemotions.AppTest", "-listener", "org.testng.TestNG\$ExitCodeListener"]

}

 

 

This should start failing your build, when there are configuration failures.

 

The issue in TestNG is that the ExitCodeListener isn’t being wired in as a configuration listener as well, but is merely being wired in as a Test listener.

This causes TestNG to skip its onConfigurationFailure() implementation [ This is the part wherein the status gets set to non-zero value].

In this case, you will a status as “3” indicating a failure(1) and a skip(2)

 

I have filed this as an issue: https://github.com/cbeust/testng/issues/1517

 

 

Thanks & Regards

Krishnan Mahadevan

 

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"

My Scribblings @ http://wakened-cognition.blogspot.com/

My Technical Scribbings @ http://rationaleemotions.wordpress.com/

 

From: <[hidden email]> on behalf of Todd Bradley <[hidden email]>
Reply-To: <[hidden email]>
Date: Wednesday, August 30, 2017 at 3:33 AM
To: <[hidden email]>
Subject: Re: [testng-users] Re: how to have my Jenkins job fail when TestNG configuration methods have failures

 

Ah, I see the difference in our approaches. You're using Maven's and Gradle's built-in TestNG runners. My Gradle task is of type JavaExec, which is equivalent to running "java ... org.testng.TestNG" on a command line. 

 

So when I say that TestNG exits with status 0, I mean that Java, when running TestNG's main method, exits with 0 when configuration methods fail, if no test methods fail. 

 

Tracing through the source for TestNG.java, you can see why this is the case. Or, if you want to reproduce this empirically, create a Gradle task like this in your build.gradle file:

task ('myTask', type: JavaExec) {
        main = 'org.testng.TestNG'
        classpath = sourceSets.main.runtimeClasspath + sourceSets.test.runtimeClasspath
        args = ["-testclass", "com.rationaleemotions.AppTest"]
}



And then do "gradle myTask". You'll see output like this:

 

===============================================
Command line suite
Total tests run: 1, Failures: 0, Skips: 1
Configuration Failures: 1, Skips: 0
===============================================

BUILD SUCCESSFUL
Total time: 1.025 secs

Does that help explain it?

 

Thanks,

Todd.

 

 

 

 

 

On Tue, Aug 29, 2017 at 8:07 AM, Krishnan Mahadevan <[hidden email]> wrote:

Todd,

How are you stating that TestNG exits always with a 0 status?

To reproduce the problem, I created a simple maven project [ The project has both a pom.xml and a build.gradle file so that it can be run both by Maven and Gradle ].

 

In both the cases, when the configuration fails (I am explicitly throwing a RuntimeException within a @BeforeClass annotated method), the build fails. Please see below output:

 

Gradle output:

19:25 $ gradle clean test

 

> Task :test

 

com.rationaleemotions.AppTest.beforeClass FAILED

    java.lang.RuntimeException at AppTest.java:13

 

2 tests completed, 1 failed, 1 skipped

 

 

FAILURE: Build failed with an exception.

 

* What went wrong:

Execution failed for task ':test'.

> There were failing tests. See the report at: file:///Users/krmahadevan/temp/example/build/reports/tests/test/index.html

 

* Try:

Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.

 

* Get more help at https://help.gradle.org

 

BUILD FAILED in 1s

3 actionable tasks: 3 executed

 

 

Maven output:

 

19:26 $ mvn clean test

[INFO] Scanning for projects...

[INFO]

[INFO] ------------------------------------------------------------------------

[INFO] Building example 1.0-SNAPSHOT

[INFO] ------------------------------------------------------------------------

[INFO]

[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ example ---

[INFO] Deleting /Users/krmahadevan/temp/example/target

[INFO]

[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ example ---

[INFO] Using 'UTF-8' encoding to copy filtered resources.

[INFO] skip non existing resourceDirectory /Users/krmahadevan/temp/example/src/main/resources

[INFO]

[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ example ---

[INFO] No sources to compile

[INFO]

[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ example ---

[INFO] Using 'UTF-8' encoding to copy filtered resources.

[INFO] skip non existing resourceDirectory /Users/krmahadevan/temp/example/src/test/resources

[INFO]

[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ example ---

[INFO] Changes detected - recompiling the module!

[INFO] Compiling 1 source file to /Users/krmahadevan/temp/example/target/test-classes

[INFO]

[INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ example ---

[INFO] Surefire report directory: /Users/krmahadevan/temp/example/target/surefire-reports

 

-------------------------------------------------------

T E S T S

-------------------------------------------------------

Running com.rationaleemotions.AppTest

Configuring TestNG with: org.apache.maven.surefire.testng.conf.TestNG652Configurator@3b764bce

Tests run: 2, Failures: 1, Errors: 0, Skipped: 1, Time elapsed: 0.282 sec <<< FAILURE!

beforeClass(com.rationaleemotions.AppTest)  Time elapsed: 0.168 sec  <<< FAILURE!

java.lang.RuntimeException: Simulating a configuration failure

    at com.rationaleemotions.AppTest.beforeClass(AppTest.java:13)

 

 

Results :

 

Failed tests:   beforeClass(com.rationaleemotions.AppTest): Simulating a configuration failure

 

Tests run: 2, Failures: 1, Errors: 0, Skipped: 1

 

[INFO] ------------------------------------------------------------------------

[INFO] BUILD FAILURE

[INFO] ------------------------------------------------------------------------

[INFO] Total time: 2.623 s

[INFO] Finished at: 2017-08-29T19:27:25+05:30

[INFO] Final Memory: 16M/249M

[INFO] ------------------------------------------------------------------------

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.12.4:test (default-test) on project example: There are test failures.

[ERROR]

[ERROR] Please refer to /Users/krmahadevan/temp/example/target/surefire-reports for the individual test results.

[ERROR] -> [Help 1]

[ERROR]

[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.

[ERROR] Re-run Maven using the -X switch to enable full debug logging.

[ERROR]

[ERROR] For more information about the errors and possible solutions, please read the following articles:

[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException

 

 

 

Thanks & Regards

Krishnan Mahadevan

 

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"

My Scribblings @ http://wakened-cognition.blogspot.com/

My Technical Scribbings @ http://rationaleemotions.wordpress.com/

 

From: <[hidden email]> on behalf of Todd Bradley <[hidden email]>
Reply-To: <[hidden email]>
Date: Tuesday, August 29, 2017 at 3:50 AM
To: testng-users <[hidden email]>
Subject: [testng-users] Re: how to have my Jenkins job fail when TestNG configuration methods have failures

 

Any ideas? How do other people who use TestNG with a CI system handle this?

On Friday, August 25, 2017 at 2:41:47 PM UTC-6, Todd Bradley wrote:

Hi, I could use some advice. We use Jenkins to run TestNG based tests via Gradle. When TestNG exits with a 0 status, Gradle considers that to be successful, and so my Jenkins job is considered a success, too.

 

However, if there is a failure in a configuration method (like @BeforeClass) that causes some tests to be skipped, TestNG still exits with 0. I want Gradle and Jenkins to notice that it's a problem when configuration methods fail, and flag the build as a failure. Do you have advice how to do this?

 

 

Thanks,

Todd.

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.

 

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: how to have my Jenkins job fail when TestNG configuration methods have failures

⇜Krishnan Mahadevan⇝

No problem Todd.. Oh btw, I have raised a PR to have this fixed in TestNG. [ https://github.com/cbeust/testng/pull/1518  ]

 

So starting TestNG 6.13, you shouldn’t be required to do it

 

 

Thanks & Regards

Krishnan Mahadevan

 

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"

My Scribblings @ http://wakened-cognition.blogspot.com/

My Technical Scribbings @ http://rationaleemotions.wordpress.com/

 

From: <[hidden email]> on behalf of Todd Bradley <[hidden email]>
Reply-To: <[hidden email]>
Date: Wednesday, August 30, 2017 at 10:24 PM
To: <[hidden email]>
Subject: Re: [testng-users] Re: how to have my Jenkins job fail when TestNG configuration methods have failures

 

Thank you, Krishnan. It works! So now I will start using TestNG.ExitCodeListener in my Gradle JavaExec tasks.

 

Regards,

Todd.

 

On Tue, Aug 29, 2017 at 10:02 PM, Krishnan Mahadevan <[hidden email]> wrote:

Todd,

 

I think it’s a bug in TestNG for the way in which it is currently behaving. But there’s an easy workaround for you, which you can employ to get past this.

 

Please modify your Gradle task to as below:

 

task ('myTask', type: JavaExec) {

    main = 'org.testng.TestNG'

    classpath = sourceSets.main.runtimeClasspath + sourceSets.test.runtimeClasspath

    args = ["-testclass", "com.rationaleemotions.AppTest", "-listener", "org.testng.TestNG\$ExitCodeListener"]

}

 

 

This should start failing your build, when there are configuration failures.

 

The issue in TestNG is that the ExitCodeListener isn’t being wired in as a configuration listener as well, but is merely being wired in as a Test listener.

This causes TestNG to skip its onConfigurationFailure() implementation [ This is the part wherein the status gets set to non-zero value].

In this case, you will a status as “3” indicating a failure(1) and a skip(2)

 

I have filed this as an issue: https://github.com/cbeust/testng/issues/1517

 

 

Thanks & Regards

Krishnan Mahadevan

 

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"

My Scribblings @ http://wakened-cognition.blogspot.com/

My Technical Scribbings @ http://rationaleemotions.wordpress.com/

 

From: <[hidden email]> on behalf of Todd Bradley <[hidden email]>
Reply-To: <[hidden email]>
Date: Wednesday, August 30, 2017 at 3:33 AM
To: <[hidden email]>
Subject: Re: [testng-users] Re: how to have my Jenkins job fail when TestNG configuration methods have failures

 

Ah, I see the difference in our approaches. You're using Maven's and Gradle's built-in TestNG runners. My Gradle task is of type JavaExec, which is equivalent to running "java ... org.testng.TestNG" on a command line. 

 

So when I say that TestNG exits with status 0, I mean that Java, when running TestNG's main method, exits with 0 when configuration methods fail, if no test methods fail. 

 

Tracing through the source for TestNG.java, you can see why this is the case. Or, if you want to reproduce this empirically, create a Gradle task like this in your build.gradle file:

task ('myTask', type: JavaExec) {
        main = 'org.testng.TestNG'
        classpath = sourceSets.main.runtimeClasspath + sourceSets.test.runtimeClasspath
        args = ["-testclass", "com.rationaleemotions.AppTest"]
}



And then do "gradle myTask". You'll see output like this:

 

===============================================
Command line suite
Total tests run: 1, Failures: 0, Skips: 1
Configuration Failures: 1, Skips: 0
===============================================

BUILD SUCCESSFUL
Total time: 1.025 secs

Does that help explain it?

 

Thanks,

Todd.

 

 

 

 

 

On Tue, Aug 29, 2017 at 8:07 AM, Krishnan Mahadevan <[hidden email]> wrote:

Todd,

How are you stating that TestNG exits always with a 0 status?

To reproduce the problem, I created a simple maven project [ The project has both a pom.xml and a build.gradle file so that it can be run both by Maven and Gradle ].

 

In both the cases, when the configuration fails (I am explicitly throwing a RuntimeException within a @BeforeClass annotated method), the build fails. Please see below output:

 

Gradle output:

19:25 $ gradle clean test

 

> Task :test

 

com.rationaleemotions.AppTest.beforeClass FAILED

    java.lang.RuntimeException at AppTest.java:13

 

2 tests completed, 1 failed, 1 skipped

 

 

FAILURE: Build failed with an exception.

 

* What went wrong:

Execution failed for task ':test'.

> There were failing tests. See the report at: file:///Users/krmahadevan/temp/example/build/reports/tests/test/index.html

 

* Try:

Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.

 

* Get more help at https://help.gradle.org

 

BUILD FAILED in 1s

3 actionable tasks: 3 executed

 

 

Maven output:

 

19:26 $ mvn clean test

[INFO] Scanning for projects...

[INFO]

[INFO] ------------------------------------------------------------------------

[INFO] Building example 1.0-SNAPSHOT

[INFO] ------------------------------------------------------------------------

[INFO]

[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ example ---

[INFO] Deleting /Users/krmahadevan/temp/example/target

[INFO]

[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ example ---

[INFO] Using 'UTF-8' encoding to copy filtered resources.

[INFO] skip non existing resourceDirectory /Users/krmahadevan/temp/example/src/main/resources

[INFO]

[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ example ---

[INFO] No sources to compile

[INFO]

[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ example ---

[INFO] Using 'UTF-8' encoding to copy filtered resources.

[INFO] skip non existing resourceDirectory /Users/krmahadevan/temp/example/src/test/resources

[INFO]

[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ example ---

[INFO] Changes detected - recompiling the module!

[INFO] Compiling 1 source file to /Users/krmahadevan/temp/example/target/test-classes

[INFO]

[INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ example ---

[INFO] Surefire report directory: /Users/krmahadevan/temp/example/target/surefire-reports

 

-------------------------------------------------------

T E S T S

-------------------------------------------------------

Running com.rationaleemotions.AppTest

Configuring TestNG with: org.apache.maven.surefire.testng.conf.TestNG652Configurator@3b764bce

Tests run: 2, Failures: 1, Errors: 0, Skipped: 1, Time elapsed: 0.282 sec <<< FAILURE!

beforeClass(com.rationaleemotions.AppTest)  Time elapsed: 0.168 sec  <<< FAILURE!

java.lang.RuntimeException: Simulating a configuration failure

    at com.rationaleemotions.AppTest.beforeClass(AppTest.java:13)

 

 

Results :

 

Failed tests:   beforeClass(com.rationaleemotions.AppTest): Simulating a configuration failure

 

Tests run: 2, Failures: 1, Errors: 0, Skipped: 1

 

[INFO] ------------------------------------------------------------------------

[INFO] BUILD FAILURE

[INFO] ------------------------------------------------------------------------

[INFO] Total time: 2.623 s

[INFO] Finished at: 2017-08-29T19:27:25+05:30

[INFO] Final Memory: 16M/249M

[INFO] ------------------------------------------------------------------------

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.12.4:test (default-test) on project example: There are test failures.

[ERROR]

[ERROR] Please refer to /Users/krmahadevan/temp/example/target/surefire-reports for the individual test results.

[ERROR] -> [Help 1]

[ERROR]

[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.

[ERROR] Re-run Maven using the -X switch to enable full debug logging.

[ERROR]

[ERROR] For more information about the errors and possible solutions, please read the following articles:

[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException

 

 

 

Thanks & Regards

Krishnan Mahadevan

 

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"

My Scribblings @ http://wakened-cognition.blogspot.com/

My Technical Scribbings @ http://rationaleemotions.wordpress.com/

 

From: <[hidden email]> on behalf of Todd Bradley <[hidden email]>
Reply-To: <[hidden email]>
Date: Tuesday, August 29, 2017 at 3:50 AM
To: testng-users <[hidden email]>
Subject: [testng-users] Re: how to have my Jenkins job fail when TestNG configuration methods have failures

 

Any ideas? How do other people who use TestNG with a CI system handle this?

On Friday, August 25, 2017 at 2:41:47 PM UTC-6, Todd Bradley wrote:

Hi, I could use some advice. We use Jenkins to run TestNG based tests via Gradle. When TestNG exits with a 0 status, Gradle considers that to be successful, and so my Jenkins job is considered a success, too.

 

However, if there is a failure in a configuration method (like @BeforeClass) that causes some tests to be skipped, TestNG still exits with 0. I want Gradle and Jenkins to notice that it's a problem when configuration methods fail, and flag the build as a failure. Do you have advice how to do this?

 

 

Thanks,

Todd.

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.

 

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.

 

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.