TestNG testsuite fails undeterministically on java8, test.testng387.TestNG387

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

TestNG testsuite fails undeterministically on java8, test.testng387.TestNG387

Gábor Boskovits
Dear TestNG community,

we are having problems with the TestNG testsuite undeterministically failing on java8.

The message we sometimes get is:

--8<---------------cut here---------------start------------->8---
===============================================
TestNG
Total tests run: 1630, Failures: 1, Skips: 0
===============================================

Failures in <suite> :TestNG, <test> :Regression2
test.testng387.TestNG387.testInvocationCounterIsCorrectForMethodWithDataProvider()
StackTrace:
 java.lang.AssertionError: Arrays do not have the same size:3 != 4
        at test.testng387.TestNG387.testInvocationCounterIsCorrectForMethodWithDataProvider(Unknown Source)
... Removed 27 stack frames

This is testng version 6.14.2.

Can this be related to https://github.com/cbeust/testng/issues/582, or is it totally unrelated?
Is it safe to disable this test, or does it actually mean problem?



--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: TestNG testsuite fails undeterministically on java8, test.testng387.TestNG387

Krishnan Mahadevan

Gabor,

 

What tests are you referring to ? Are you talking about the unit tests that reside within TestNG codebase or something else ?

 

TestNG still has its official compatibility set to JDK7.

 

The issue that you quoted was closed almost 3 years back.

 

I tried running the testng build with JDK8 compatibility and I don’t see the problem you are mentioning.

 

Thanks & Regards

Krishnan Mahadevan

 

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"

My Scribblings @ http://wakened-cognition.blogspot.com/

My Technical Scribbings @ http://rationaleemotions.wordpress.com/

Reply-To: <[hidden email]>
Date: Wednesday, February 21, 2018 at 2:54 PM
To: testng-users <[hidden email]>
Subject: [testng-users] TestNG testsuite fails undeterministically on java8, test.testng387.TestNG387

 

Dear TestNG community,

 

we are having problems with the TestNG testsuite undeterministically failing on java8.

 

The message we sometimes get is:

 

--8<---------------cut here---------------start------------->8---
===============================================
TestNG
Total tests run: 1630, Failures: 1, Skips: 0
===============================================

Failures in <suite> :TestNG, <test> :Regression2
test.testng387.TestNG387.testInvocationCounterIsCorrectForMethodWithDataProvider()
StackTrace:
 java.lang.AssertionError: Arrays do not have the same size:3 != 4
        at test.testng387.TestNG387.testInvocationCounterIsCorrectForMethodWithDataProvider(Unknown Source)
... Removed 27 stack frames

 

This is testng version 6.14.2.

 

Can this be related to https://github.com/cbeust/testng/issues/582, or is it totally unrelated?

Is it safe to disable this test, or does it actually mean problem?

 

 

 

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to
[hidden email].
To post to this group, send email to
[hidden email].
Visit this group at
https://groups.google.com/group/testng-users.
For more options, visit
https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: TestNG testsuite fails undeterministically on java8, test.testng387.TestNG387

Gábor Boskovits


On Friday, February 23, 2018 at 4:56:12 AM UTC+1, Krishnan Mahadevan wrote:

Gabor,

 

What tests are you referring to ? Are you talking about the unit tests that reside within TestNG codebase or something else ?



Yes, it is in the TestNG codebase. TestNG version is:6.14.2.
Test file path: testng/src/test/java/test/testng387/TestNG387.java.

 

 

TestNG still has its official compatibility set to JDK7.

 

The issue that you quoted was closed almost 3 years back.

 

I tried running the testng build with JDK8 compatibility and I don’t see the problem you are mentioning.

 



This problem is not deterministic, I could trigger it only a few times from about 100 builds.
Only this one test is failing, which seems to be counting invocations.
I suspect this might be caused by a race condition.
I will soon report back with concrete statistics.
 

Thanks & Regards

Krishnan Mahadevan

 

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"

My Scribblings @ <a href="http://wakened-cognition.blogspot.com/" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fwakened-cognition.blogspot.com%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHzOdYJCASIDF_28vQtkp9gnKAYSQ&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fwakened-cognition.blogspot.com%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHzOdYJCASIDF_28vQtkp9gnKAYSQ&#39;;return true;">http://wakened-cognition.blogspot.com/

My Technical Scribbings @ <a href="http://rationaleemotions.wordpress.com/" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Frationaleemotions.wordpress.com%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEcBOKyRn0lpL8LDbBuyAlKdwgXwQ&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Frationaleemotions.wordpress.com%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEcBOKyRn0lpL8LDbBuyAlKdwgXwQ&#39;;return true;">http://rationaleemotions.wordpress.com/

Reply-To: <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="Tz8bRImRAQAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">testng...@...>
Date: Wednesday, February 21, 2018 at 2:54 PM
To: testng-users <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="Tz8bRImRAQAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">testng...@...>
Subject: [testng-users] TestNG testsuite fails undeterministically on java8, test.testng387.TestNG387

 

Dear TestNG community,

 

we are having problems with the TestNG testsuite undeterministically failing on java8.

 

The message we sometimes get is:

 

--8<---------------cut here---------------start------------->8---
===============================================
TestNG
Total tests run: 1630, Failures: 1, Skips: 0
===============================================

Failures in <suite> :TestNG, <test> :Regression2
test.testng387.TestNG387.testInvocationCounterIsCorrectForMethodWithDataProvider()
StackTrace:
 java.lang.AssertionError: Arrays do not have the same size:3 != 4
        at test.testng387.TestNG387.testInvocationCounterIsCorrectForMethodWithDataProvider(Unknown Source)
... Removed 27 stack frames

 

This is testng version 6.14.2.

 

Can this be related to <a href="https://github.com/cbeust/testng/issues/582" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fcbeust%2Ftestng%2Fissues%2F582\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFzQXH5BgqYGAd4fPp-Z0uMygcPZA&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fcbeust%2Ftestng%2Fissues%2F582\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFzQXH5BgqYGAd4fPp-Z0uMygcPZA&#39;;return true;">https://github.com/cbeust/testng/issues/582, or is it totally unrelated?

Is it safe to disable this test, or does it actually mean problem?

 

 

 

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to
<a href="javascript:" target="_blank" gdf-obfuscated-mailto="Tz8bRImRAQAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">testng-users+unsubscribe@googlegroups.com.
To post to this group, send email to
<a href="javascript:" target="_blank" gdf-obfuscated-mailto="Tz8bRImRAQAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">testng-users@....
Visit this group at
<a href="https://groups.google.com/group/testng-users" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/group/testng-users&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/group/testng-users&#39;;return true;">https://groups.google.com/group/testng-users.
For more options, visit
<a href="https://groups.google.com/d/optout" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: TestNG testsuite fails undeterministically on java8, test.testng387.TestNG387

Gábor Boskovits
On Friday, February 23, 2018 at 12:14:34 PM UTC+1, Gábor Boskovits wrote:


It seems, that this issue is resolved in version 6.14.3. Thank you very much!

 
On Friday, February 23, 2018 at 4:56:12 AM UTC+1, Krishnan Mahadevan wrote:

Gabor,

 

What tests are you referring to ? Are you talking about the unit tests that reside within TestNG codebase or something else ?



Yes, it is in the TestNG codebase. TestNG version is:6.14.2.
Test file path: <a href="https://github.com/cbeust/testng" style="background-color:transparent;color:rgb(3,102,214)" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fcbeust%2Ftestng\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFFDR5zzPkrmHE2lZfN86z9hVtONQ&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fcbeust%2Ftestng\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFFDR5zzPkrmHE2lZfN86z9hVtONQ&#39;;return true;">testng/<a href="https://github.com/cbeust/testng/tree/master/src" style="background-color:transparent;color:rgb(3,102,214)" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fcbeust%2Ftestng%2Ftree%2Fmaster%2Fsrc\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEIrUka_In870Nh781riX4pzhQ1eA&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fcbeust%2Ftestng%2Ftree%2Fmaster%2Fsrc\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEIrUka_In870Nh781riX4pzhQ1eA&#39;;return true;">src/<a href="https://github.com/cbeust/testng/tree/master/src/test" style="background-color:transparent;color:rgb(3,102,214)" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fcbeust%2Ftestng%2Ftree%2Fmaster%2Fsrc%2Ftest\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEemmF_IWa8WlnBKusIaSGtCNiAIA&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fcbeust%2Ftestng%2Ftree%2Fmaster%2Fsrc%2Ftest\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEemmF_IWa8WlnBKusIaSGtCNiAIA&#39;;return true;">test/<a href="https://github.com/cbeust/testng/tree/master/src/test/java" style="background-color:transparent;color:rgb(3,102,214)" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fcbeust%2Ftestng%2Ftree%2Fmaster%2Fsrc%2Ftest%2Fjava\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHnK0VBCjrYErBbbkqe2Oj5LgeiDA&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fcbeust%2Ftestng%2Ftree%2Fmaster%2Fsrc%2Ftest%2Fjava\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHnK0VBCjrYErBbbkqe2Oj5LgeiDA&#39;;return true;">java/<a href="https://github.com/cbeust/testng/tree/master/src/test/java/test" style="background-color:transparent;color:rgb(3,102,214)" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fcbeust%2Ftestng%2Ftree%2Fmaster%2Fsrc%2Ftest%2Fjava%2Ftest\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNGNSoiUz1DtAe5Njua9uM5TEStNQg&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fcbeust%2Ftestng%2Ftree%2Fmaster%2Fsrc%2Ftest%2Fjava%2Ftest\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNGNSoiUz1DtAe5Njua9uM5TEStNQg&#39;;return true;">test/<a href="https://github.com/cbeust/testng/tree/master/src/test/java/test/testng387" style="background-color:transparent;color:rgb(3,102,214)" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fcbeust%2Ftestng%2Ftree%2Fmaster%2Fsrc%2Ftest%2Fjava%2Ftest%2Ftestng387\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH1uXX7vIiSNcZlI0wo5G6af-RSNA&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fcbeust%2Ftestng%2Ftree%2Fmaster%2Fsrc%2Ftest%2Fjava%2Ftest%2Ftestng387\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH1uXX7vIiSNcZlI0wo5G6af-RSNA&#39;;return true;">testng387/TestNG387.java.

 

 

TestNG still has its official compatibility set to JDK7.

 

The issue that you quoted was closed almost 3 years back.

 

I tried running the testng build with JDK8 compatibility and I don’t see the problem you are mentioning.

 



This problem is not deterministic, I could trigger it only a few times from about 100 builds.
Only this one test is failing, which seems to be counting invocations.
I suspect this might be caused by a race condition.
I will soon report back with concrete statistics.
 

Thanks & Regards

Krishnan Mahadevan

 

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"

My Scribblings @ <a href="http://wakened-cognition.blogspot.com/" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fwakened-cognition.blogspot.com%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHzOdYJCASIDF_28vQtkp9gnKAYSQ&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fwakened-cognition.blogspot.com%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHzOdYJCASIDF_28vQtkp9gnKAYSQ&#39;;return true;">http://wakened-cognition.blogspot.com/

My Technical Scribbings @ <a href="http://rationaleemotions.wordpress.com/" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Frationaleemotions.wordpress.com%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEcBOKyRn0lpL8LDbBuyAlKdwgXwQ&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Frationaleemotions.wordpress.com%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEcBOKyRn0lpL8LDbBuyAlKdwgXwQ&#39;;return true;">http://rationaleemotions.wordpress.com/

Reply-To: <[hidden email]>
Date: Wednesday, February 21, 2018 at 2:54 PM
To: testng-users <[hidden email]>
Subject: [testng-users] TestNG testsuite fails undeterministically on java8, test.testng387.TestNG387

 

Dear TestNG community,

 

we are having problems with the TestNG testsuite undeterministically failing on java8.

 

The message we sometimes get is:

 

--8<---------------cut here---------------start------------->8---
===============================================
TestNG
Total tests run: 1630, Failures: 1, Skips: 0
===============================================

Failures in <suite> :TestNG, <test> :Regression2
test.testng387.TestNG387.testInvocationCounterIsCorrectForMethodWithDataProvider()
StackTrace:
 java.lang.AssertionError: Arrays do not have the same size:3 != 4
        at test.testng387.TestNG387.testInvocationCounterIsCorrectForMethodWithDataProvider(Unknown Source)
... Removed 27 stack frames

 

This is testng version 6.14.2.

 

Can this be related to <a href="https://github.com/cbeust/testng/issues/582" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fcbeust%2Ftestng%2Fissues%2F582\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFzQXH5BgqYGAd4fPp-Z0uMygcPZA&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fcbeust%2Ftestng%2Fissues%2F582\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFzQXH5BgqYGAd4fPp-Z0uMygcPZA&#39;;return true;">https://github.com/cbeust/testng/issues/582, or is it totally unrelated?

Is it safe to disable this test, or does it actually mean problem?

 

 

 

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to
testng-users+unsubscribe@googlegroups.com.
To post to this group, send email to
[hidden email].
Visit this group at
<a href="https://groups.google.com/group/testng-users" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/group/testng-users&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/group/testng-users&#39;;return true;">https://groups.google.com/group/testng-users.
For more options, visit
<a href="https://groups.google.com/d/optout" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.