Quantcast

Tests seem to vanish when running in parallel - How can I troubleshoot this?

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Tests seem to vanish when running in parallel - How can I troubleshoot this?

Matthew Sheppard
TLDR - When I run a large-ish selenium test suite with parallel enabled, a bunch of test methods 'vanish' - They are no longer reported as passed, failed or skipped, just seemingly gone. I'm hoping someone might be able to provide me with some pointers on how to troubleshoot the problem.


When I run my test set normally, it takes about 10 hours (ergh), with just under 800 individual tests. It's normally run through maven and surefire, with a suite.xml file something like this...

<suite name="MainSuitetime-out="1800000verbose="10">
<listeners>
<listener class-name="com.example.selenium2webdriver.testng.listener.StatsdSuiteListener"/>
</listeners>
<test name="AllTests">
<groups>
<run>
<exclude name="windowsonly"/>
<exclude name="upgrade"/>
</run>
</groups>
<packages>
<package name="com.example.selenium.*"/>
<package name="com.example.selenium2webdriver.tests.*"/>
</packages>
</test>
</suite>

and I get a surefire-reports/testng-results.xml file which starts with something like this...

<testng-results skipped="7failed="8total="789passed="774">
<reporter-output></reporter-output>
<suite name="MainSuiteduration-ms="35693671started-at="2016-12-28T08:49:22Zfinished-at="2016-12-28T18:44:16Z">


Now, the problem. If I add parallel="methods" thread-count="2" to the suite declaration there in the hope of speeding up the overall runtime, I now see the following at the top of my surefire-reports/testng-results.xml file...

<testng-results skipped="5failed="9total="45passed="31">
<reporter-output></reporter-output>
<suite name="MainSuiteduration-ms="1818150started-at="2017-01-08T08:50:25Zfinished-at="2017-01-08T09:20:43Z">

Sure, it's lots faster, but suddenly I'm only seeing about 5% of my tests being reported (and looking at the target machine suggests that the missing ones are in fact not being run).


So, my question, after all that, is 'how can I ensure all my tests are still run when I enable the parallel mode'? Or, failing a simple answer, how can I troubleshoot this further to either pin down the problem in my setup, or distill it down to a reproducible bug which could be reported.


Many thanks for any help!
Matt


----

A few random other pieces of background/info in case they're relevant/helpful.


I'm using TestNG 6.9.10, but I'm about to bump it to 6.10 just in case that helps (I'll report back either way).


In the hope of being able to provide a more reproducible example, I used javaparser to stub out all the actual test method content (leaving the annotations etc in place). The resulting test set ran everything in just a few seconds, and no longer seemed to suffer the problem regardless of the parallel configuration. I imagine that suggests that either the problem is timing specific, or that something in the tests themselves triggers the problem.


The specific tests which are run in parallel are not consistent, nor is the count. Over a few runs, I've seen total hit around 100 at most.


Looking closer at the surefire-reports/testng-results.xml file, /testng-results/suite/groups/group does seem to list ever test method...For example, I see...

<group name="DeleteCollectionCLI">
<method signature="DeleteCollectionCLI.deleteCollectionfromCLI()[pri:0, instance:com.example.selenium.cli.DeleteCollectionCLI@e3c0e40]name="deleteCollectionfromCLI"class="com.example.selenium.cli.DeleteCollectionCLI"/>
</group>
<!-- DeleteCollectionCLI -->

...but neither that class nor that method are not represented under the /testng-results/suite/test section of the results

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Tests seem to vanish when running in parallel - How can I troubleshoot this?

⇜Krishnan Mahadevan⇝
Matthew,

Since there seem to be a lot of moving parts, why not start off with building a debug listener (one which implements an IInvokedMethodListener) wherein you start logging the test that is about to start and the test that just ran to completion inside its beforeInvocation() and afterInvocation(). If you used a logger to log this information into a file, you will have a good way of starting with your debugging to see what tests are being executed.

And just to ensure that you absolutely get to know the list of methods that are going to be executed, you can log them from within an implementation of org.testng.IAnnotationTransformer via a logger. 

Once you inject both these listeners into your suite and try running your suite in parallel, you should be able to start getting some insight into what is getting chosen for execution [ the IAnnotationTransformer output ] and what exactly was executed [ The IInvokedMethodListener output ]


Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @ http://wakened-cognition.blogspot.com/
My Technical Scribbings @ http://rationaleemotions.wordpress.com/

On Mon, Jan 9, 2017 at 6:00 AM, Matthew Sheppard <[hidden email]> wrote:
TLDR - When I run a large-ish selenium test suite with parallel enabled, a bunch of test methods 'vanish' - They are no longer reported as passed, failed or skipped, just seemingly gone. I'm hoping someone might be able to provide me with some pointers on how to troubleshoot the problem.


When I run my test set normally, it takes about 10 hours (ergh), with just under 800 individual tests. It's normally run through maven and surefire, with a suite.xml file something like this...

<suite name="MainSuitetime-out="1800000verbose="10">
<listeners>
<listener class-name="com.example.selenium2webdriver.testng.listener.StatsdSuiteListener"/>
</listeners>
<test name="AllTests">
<groups>
<run>
<exclude name="windowsonly"/>
<exclude name="upgrade"/>
</run>
</groups>
<packages>
<package name="com.example.selenium.*"/>
<package name="com.example.selenium2webdriver.tests.*"/>
</packages>
</test>
</suite>

and I get a surefire-reports/testng-results.xml file which starts with something like this...

<testng-results skipped="7failed="8total="789passed="774">
<reporter-output></reporter-output>
<suite name="MainSuiteduration-ms="35693671started-at="2016-12-28T08:49:22Zfinished-at="2016-12-28T18:44:16Z">


Now, the problem. If I add parallel="methods" thread-count="2" to the suite declaration there in the hope of speeding up the overall runtime, I now see the following at the top of my surefire-reports/testng-results.xml file...

<testng-results skipped="5failed="9total="45passed="31">
<reporter-output></reporter-output>
<suite name="MainSuiteduration-ms="1818150started-at="2017-01-08T08:50:25Zfinished-at="2017-01-08T09:20:43Z">

Sure, it's lots faster, but suddenly I'm only seeing about 5% of my tests being reported (and looking at the target machine suggests that the missing ones are in fact not being run).


So, my question, after all that, is 'how can I ensure all my tests are still run when I enable the parallel mode'? Or, failing a simple answer, how can I troubleshoot this further to either pin down the problem in my setup, or distill it down to a reproducible bug which could be reported.


Many thanks for any help!
Matt


----

A few random other pieces of background/info in case they're relevant/helpful.


I'm using TestNG 6.9.10, but I'm about to bump it to 6.10 just in case that helps (I'll report back either way).


In the hope of being able to provide a more reproducible example, I used javaparser to stub out all the actual test method content (leaving the annotations etc in place). The resulting test set ran everything in just a few seconds, and no longer seemed to suffer the problem regardless of the parallel configuration. I imagine that suggests that either the problem is timing specific, or that something in the tests themselves triggers the problem.


The specific tests which are run in parallel are not consistent, nor is the count. Over a few runs, I've seen total hit around 100 at most.


Looking closer at the surefire-reports/testng-results.xml file, /testng-results/suite/groups/group does seem to list ever test method...For example, I see...

<group name="DeleteCollectionCLI">
<method signature="DeleteCollectionCLI.deleteCollectionfromCLI()[pri:0, instance:com.example.selenium.cli.DeleteCollectionCLI@e3c0e40]name="deleteCollectionfromCLI"class="com.example.selenium.cli.DeleteCollectionCLI"/>
</group>
<!-- DeleteCollectionCLI -->

...but neither that class nor that method are not represented under the /testng-results/suite/test section of the results

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Tests seem to vanish when running in parallel - How can I troubleshoot this?

Matthew Sheppard
Just to follow up - I did test with 6.10 and the problem still occurs.

Turns out I do already have an IInvokedMethodListener in there which we use to log the test order, so I can confirm that the beforeInvocation/afterInvocation methods in there are being called only for the tests which actually appear to run (i.e. the vanished tests are not mentioned).

I'll take a look at the IAnnotationTransformer now.

Thanks for your assistance!
Matt

On Monday, 9 January 2017 14:58:50 UTC+11, Krishnan wrote:
Matthew,

Since there seem to be a lot of moving parts, why not start off with building a debug listener (one which implements an IInvokedMethodListener) wherein you start logging the test that is about to start and the test that just ran to completion inside its beforeInvocation() and afterInvocation(). If you used a logger to log this information into a file, you will have a good way of starting with your debugging to see what tests are being executed.

And just to ensure that you absolutely get to know the list of methods that are going to be executed, you can log them from within an implementation of org.testng.IAnnotationTransformer via a logger. 

Once you inject both these listeners into your suite and try running your suite in parallel, you should be able to start getting some insight into what is getting chosen for execution [ the IAnnotationTransformer output ] and what exactly was executed [ The IInvokedMethodListener output ]


Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @ <a href="http://wakened-cognition.blogspot.com/" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fwakened-cognition.blogspot.com%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHzOdYJCASIDF_28vQtkp9gnKAYSQ&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fwakened-cognition.blogspot.com%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHzOdYJCASIDF_28vQtkp9gnKAYSQ&#39;;return true;">http://wakened-cognition.blogspot.com/
My Technical Scribbings @ <a href="http://rationaleemotions.wordpress.com/" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Frationaleemotions.wordpress.com%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEcBOKyRn0lpL8LDbBuyAlKdwgXwQ&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Frationaleemotions.wordpress.com%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEcBOKyRn0lpL8LDbBuyAlKdwgXwQ&#39;;return true;">http://rationaleemotions.wordpress.com/

On Mon, Jan 9, 2017 at 6:00 AM, Matthew Sheppard <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="c_yDdU1RFQAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">matthew....@...> wrote:
TLDR - When I run a large-ish selenium test suite with parallel enabled, a bunch of test methods 'vanish' - They are no longer reported as passed, failed or skipped, just seemingly gone. I'm hoping someone might be able to provide me with some pointers on how to troubleshoot the problem.


When I run my test set normally, it takes about 10 hours (ergh), with just under 800 individual tests. It's normally run through maven and surefire, with a suite.xml file something like this...

<suite name="MainSuitetime-out="1800000verbose="10">
<listeners>
<listener class-name="com.example.selenium2webdriver.testng.listener.StatsdSuiteListener"/>
</listeners>
<test name="AllTests">
<groups>
<run>
<exclude name="windowsonly"/>
<exclude name="upgrade"/>
</run>
</groups>
<packages>
<package name="com.example.selenium.*"/>
<package name="com.example.selenium2webdriver.tests.*"/>
</packages>
</test>
</suite>

and I get a surefire-reports/testng-results.xml file which starts with something like this...

<testng-results skipped="7failed="8total="789passed="774">
<reporter-output></reporter-output>
<suite name="MainSuiteduration-ms="35693671started-at="2016-12-28T08:49:22Zfinished-at="2016-12-28T18:44:16Z">


Now, the problem. If I add parallel="methods" thread-count="2" to the suite declaration there in the hope of speeding up the overall runtime, I now see the following at the top of my surefire-reports/testng-results.xml file...

<testng-results skipped="5failed="9total="45passed="31">
<reporter-output></reporter-output>
<suite name="MainSuiteduration-ms="1818150started-at="2017-01-08T08:50:25Zfinished-at="2017-01-08T09:20:43Z">

Sure, it's lots faster, but suddenly I'm only seeing about 5% of my tests being reported (and looking at the target machine suggests that the missing ones are in fact not being run).


So, my question, after all that, is 'how can I ensure all my tests are still run when I enable the parallel mode'? Or, failing a simple answer, how can I troubleshoot this further to either pin down the problem in my setup, or distill it down to a reproducible bug which could be reported.


Many thanks for any help!
Matt


----

A few random other pieces of background/info in case they're relevant/helpful.


I'm using TestNG 6.9.10, but I'm about to bump it to 6.10 just in case that helps (I'll report back either way).


In the hope of being able to provide a more reproducible example, I used javaparser to stub out all the actual test method content (leaving the annotations etc in place). The resulting test set ran everything in just a few seconds, and no longer seemed to suffer the problem regardless of the parallel configuration. I imagine that suggests that either the problem is timing specific, or that something in the tests themselves triggers the problem.


The specific tests which are run in parallel are not consistent, nor is the count. Over a few runs, I've seen total hit around 100 at most.


Looking closer at the surefire-reports/testng-results.xml file, /testng-results/suite/groups/group does seem to list ever test method...For example, I see...

<group name="DeleteCollectionCLI">
<method signature="DeleteCollectionCLI.deleteCollectionfromCLI()[pri:0, instance:com.example.selenium.cli.DeleteCollectionCLI@e3c0e40]name="deleteCollectionfromCLI"class="com.example.selenium.cli.DeleteCollectionCLI"/>
</group>
<!-- DeleteCollectionCLI -->

...but neither that class nor that method are not represented under the /testng-results/suite/test section of the results

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="c_yDdU1RFQAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">testng-users...@googlegroups.com.
To post to this group, send email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="c_yDdU1RFQAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">testng...@....
Visit this group at <a href="https://groups.google.com/group/testng-users" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/group/testng-users&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/group/testng-users&#39;;return true;">https://groups.google.com/group/testng-users.
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Tests seem to vanish when running in parallel - How can I troubleshoot this?

Matthew Sheppard
So, having run both in parallel and non-parallel with an IAnnotationTransformer logging everything it sees go past, I see 1066 invocations of the transform method, and after sorting the lines, they appear to be identical between the two runs. At a glance, I'm looking at the ~800 actual test methods plus calls for the classes containing tests.

Hmmm...

...and, that's when I noticed that timeout on the suite...which looking closer is set to 30 minutes...presumably someone added it there thinking it should apply per-test, but when running in parallel we do indeed seem to stop after about 30 minutes of running tests...though I see no reason why it should be applied only when running in parallel...

...Hmmm...

Will report back once I've tested that out. I'll be quite disappointed if that's the explanation :)

Matt

On Wednesday, 11 January 2017 15:04:16 UTC+11, Matthew Sheppard wrote:
Just to follow up - I did test with 6.10 and the problem still occurs.

Turns out I do already have an IInvokedMethodListener in there which we use to log the test order, so I can confirm that the beforeInvocation/afterInvocation methods in there are being called only for the tests which actually appear to run (i.e. the vanished tests are not mentioned).

I'll take a look at the IAnnotationTransformer now.

Thanks for your assistance!
Matt

On Monday, 9 January 2017 14:58:50 UTC+11, Krishnan wrote:
Matthew,

Since there seem to be a lot of moving parts, why not start off with building a debug listener (one which implements an IInvokedMethodListener) wherein you start logging the test that is about to start and the test that just ran to completion inside its beforeInvocation() and afterInvocation(). If you used a logger to log this information into a file, you will have a good way of starting with your debugging to see what tests are being executed.

And just to ensure that you absolutely get to know the list of methods that are going to be executed, you can log them from within an implementation of org.testng.IAnnotationTransformer via a logger. 

Once you inject both these listeners into your suite and try running your suite in parallel, you should be able to start getting some insight into what is getting chosen for execution [ the IAnnotationTransformer output ] and what exactly was executed [ The IInvokedMethodListener output ]


Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @ <a href="http://wakened-cognition.blogspot.com/" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fwakened-cognition.blogspot.com%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHzOdYJCASIDF_28vQtkp9gnKAYSQ&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fwakened-cognition.blogspot.com%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHzOdYJCASIDF_28vQtkp9gnKAYSQ&#39;;return true;">http://wakened-cognition.blogspot.com/
My Technical Scribbings @ <a href="http://rationaleemotions.wordpress.com/" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Frationaleemotions.wordpress.com%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEcBOKyRn0lpL8LDbBuyAlKdwgXwQ&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Frationaleemotions.wordpress.com%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEcBOKyRn0lpL8LDbBuyAlKdwgXwQ&#39;;return true;">http://rationaleemotions.wordpress.com/

On Mon, Jan 9, 2017 at 6:00 AM, Matthew Sheppard <[hidden email]> wrote:
TLDR - When I run a large-ish selenium test suite with parallel enabled, a bunch of test methods 'vanish' - They are no longer reported as passed, failed or skipped, just seemingly gone. I'm hoping someone might be able to provide me with some pointers on how to troubleshoot the problem.


When I run my test set normally, it takes about 10 hours (ergh), with just under 800 individual tests. It's normally run through maven and surefire, with a suite.xml file something like this...

<suite name="MainSuitetime-out="1800000verbose="10">
<listeners>
<listener class-name="com.example.selenium2webdriver.testng.listener.StatsdSuiteListener"/>
</listeners>
<test name="AllTests">
<groups>
<run>
<exclude name="windowsonly"/>
<exclude name="upgrade"/>
</run>
</groups>
<packages>
<package name="com.example.selenium.*"/>
<package name="com.example.selenium2webdriver.tests.*"/>
</packages>
</test>
</suite>

and I get a surefire-reports/testng-results.xml file which starts with something like this...

<testng-results skipped="7failed="8total="789passed="774">
<reporter-output></reporter-output>
<suite name="MainSuiteduration-ms="35693671started-at="2016-12-28T08:49:22Zfinished-at="2016-12-28T18:44:16Z">


Now, the problem. If I add parallel="methods" thread-count="2" to the suite declaration there in the hope of speeding up the overall runtime, I now see the following at the top of my surefire-reports/testng-results.xml file...

<testng-results skipped="5failed="9total="45passed="31">
<reporter-output></reporter-output>
<suite name="MainSuiteduration-ms="1818150started-at="2017-01-08T08:50:25Zfinished-at="2017-01-08T09:20:43Z">

Sure, it's lots faster, but suddenly I'm only seeing about 5% of my tests being reported (and looking at the target machine suggests that the missing ones are in fact not being run).


So, my question, after all that, is 'how can I ensure all my tests are still run when I enable the parallel mode'? Or, failing a simple answer, how can I troubleshoot this further to either pin down the problem in my setup, or distill it down to a reproducible bug which could be reported.


Many thanks for any help!
Matt


----

A few random other pieces of background/info in case they're relevant/helpful.


I'm using TestNG 6.9.10, but I'm about to bump it to 6.10 just in case that helps (I'll report back either way).


In the hope of being able to provide a more reproducible example, I used javaparser to stub out all the actual test method content (leaving the annotations etc in place). The resulting test set ran everything in just a few seconds, and no longer seemed to suffer the problem regardless of the parallel configuration. I imagine that suggests that either the problem is timing specific, or that something in the tests themselves triggers the problem.


The specific tests which are run in parallel are not consistent, nor is the count. Over a few runs, I've seen total hit around 100 at most.


Looking closer at the surefire-reports/testng-results.xml file, /testng-results/suite/groups/group does seem to list ever test method...For example, I see...

<group name="DeleteCollectionCLI">
<method signature="DeleteCollectionCLI.deleteCollectionfromCLI()[pri:0, instance:com.example.selenium.cli.DeleteCollectionCLI@e3c0e40]name="deleteCollectionfromCLI"class="com.example.selenium.cli.DeleteCollectionCLI"/>
</group>
<!-- DeleteCollectionCLI -->

...but neither that class nor that method are not represented under the /testng-results/suite/test section of the results

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to testng-users...@googlegroups.com.
To post to this group, send email to [hidden email].
Visit this group at <a href="https://groups.google.com/group/testng-users" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/group/testng-users&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/group/testng-users&#39;;return true;">https://groups.google.com/group/testng-users.
For more options, visit <a href="https://groups.google.com/d/optout" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Tests seem to vanish when running in parallel - How can I troubleshoot this?

Matthew Sheppard
Confirmed that removing the timeout seems to eliminate the problem as described.

I guess that suggests the possibility that it's a bug that the timeout didn't apply when in non-parallel mode - If that's the case, it ought to be easy enough to reproduce, so I'll try to do that, and report it as a bug if so.

Thanks for the help!
Matt

On Wednesday, 11 January 2017 16:18:08 UTC+11, Matthew Sheppard wrote:
So, having run both in parallel and non-parallel with an IAnnotationTransformer logging everything it sees go past, I see 1066 invocations of the transform method, and after sorting the lines, they appear to be identical between the two runs. At a glance, I'm looking at the ~800 actual test methods plus calls for the classes containing tests.

Hmmm...

...and, that's when I noticed that timeout on the suite...which looking closer is set to 30 minutes...presumably someone added it there thinking it should apply per-test, but when running in parallel we do indeed seem to stop after about 30 minutes of running tests...though I see no reason why it should be applied only when running in parallel...

...Hmmm...

Will report back once I've tested that out. I'll be quite disappointed if that's the explanation :)

Matt

On Wednesday, 11 January 2017 15:04:16 UTC+11, Matthew Sheppard wrote:
Just to follow up - I did test with 6.10 and the problem still occurs.

Turns out I do already have an IInvokedMethodListener in there which we use to log the test order, so I can confirm that the beforeInvocation/afterInvocation methods in there are being called only for the tests which actually appear to run (i.e. the vanished tests are not mentioned).

I'll take a look at the IAnnotationTransformer now.

Thanks for your assistance!
Matt

On Monday, 9 January 2017 14:58:50 UTC+11, Krishnan wrote:
Matthew,

Since there seem to be a lot of moving parts, why not start off with building a debug listener (one which implements an IInvokedMethodListener) wherein you start logging the test that is about to start and the test that just ran to completion inside its beforeInvocation() and afterInvocation(). If you used a logger to log this information into a file, you will have a good way of starting with your debugging to see what tests are being executed.

And just to ensure that you absolutely get to know the list of methods that are going to be executed, you can log them from within an implementation of org.testng.IAnnotationTransformer via a logger. 

Once you inject both these listeners into your suite and try running your suite in parallel, you should be able to start getting some insight into what is getting chosen for execution [ the IAnnotationTransformer output ] and what exactly was executed [ The IInvokedMethodListener output ]


Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @ <a href="http://wakened-cognition.blogspot.com/" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fwakened-cognition.blogspot.com%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHzOdYJCASIDF_28vQtkp9gnKAYSQ&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fwakened-cognition.blogspot.com%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHzOdYJCASIDF_28vQtkp9gnKAYSQ&#39;;return true;">http://wakened-cognition.blogspot.com/
My Technical Scribbings @ <a href="http://rationaleemotions.wordpress.com/" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Frationaleemotions.wordpress.com%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEcBOKyRn0lpL8LDbBuyAlKdwgXwQ&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Frationaleemotions.wordpress.com%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEcBOKyRn0lpL8LDbBuyAlKdwgXwQ&#39;;return true;">http://rationaleemotions.wordpress.com/

On Mon, Jan 9, 2017 at 6:00 AM, Matthew Sheppard <[hidden email]> wrote:
TLDR - When I run a large-ish selenium test suite with parallel enabled, a bunch of test methods 'vanish' - They are no longer reported as passed, failed or skipped, just seemingly gone. I'm hoping someone might be able to provide me with some pointers on how to troubleshoot the problem.


When I run my test set normally, it takes about 10 hours (ergh), with just under 800 individual tests. It's normally run through maven and surefire, with a suite.xml file something like this...

<suite name="MainSuitetime-out="1800000verbose="10">
<listeners>
<listener class-name="com.example.selenium2webdriver.testng.listener.StatsdSuiteListener"/>
</listeners>
<test name="AllTests">
<groups>
<run>
<exclude name="windowsonly"/>
<exclude name="upgrade"/>
</run>
</groups>
<packages>
<package name="com.example.selenium.*"/>
<package name="com.example.selenium2webdriver.tests.*"/>
</packages>
</test>
</suite>

and I get a surefire-reports/testng-results.xml file which starts with something like this...

<testng-results skipped="7failed="8total="789passed="774">
<reporter-output></reporter-output>
<suite name="MainSuiteduration-ms="35693671started-at="2016-12-28T08:49:22Zfinished-at="2016-12-28T18:44:16Z">


Now, the problem. If I add parallel="methods" thread-count="2" to the suite declaration there in the hope of speeding up the overall runtime, I now see the following at the top of my surefire-reports/testng-results.xml file...

<testng-results skipped="5failed="9total="45passed="31">
<reporter-output></reporter-output>
<suite name="MainSuiteduration-ms="1818150started-at="2017-01-08T08:50:25Zfinished-at="2017-01-08T09:20:43Z">

Sure, it's lots faster, but suddenly I'm only seeing about 5% of my tests being reported (and looking at the target machine suggests that the missing ones are in fact not being run).


So, my question, after all that, is 'how can I ensure all my tests are still run when I enable the parallel mode'? Or, failing a simple answer, how can I troubleshoot this further to either pin down the problem in my setup, or distill it down to a reproducible bug which could be reported.


Many thanks for any help!
Matt


----

A few random other pieces of background/info in case they're relevant/helpful.


I'm using TestNG 6.9.10, but I'm about to bump it to 6.10 just in case that helps (I'll report back either way).


In the hope of being able to provide a more reproducible example, I used javaparser to stub out all the actual test method content (leaving the annotations etc in place). The resulting test set ran everything in just a few seconds, and no longer seemed to suffer the problem regardless of the parallel configuration. I imagine that suggests that either the problem is timing specific, or that something in the tests themselves triggers the problem.


The specific tests which are run in parallel are not consistent, nor is the count. Over a few runs, I've seen total hit around 100 at most.


Looking closer at the surefire-reports/testng-results.xml file, /testng-results/suite/groups/group does seem to list ever test method...For example, I see...

<group name="DeleteCollectionCLI">
<method signature="DeleteCollectionCLI.deleteCollectionfromCLI()[pri:0, instance:com.example.selenium.cli.DeleteCollectionCLI@e3c0e40]name="deleteCollectionfromCLI"class="com.example.selenium.cli.DeleteCollectionCLI"/>
</group>
<!-- DeleteCollectionCLI -->

...but neither that class nor that method are not represented under the /testng-results/suite/test section of the results

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to testng-users...@googlegroups.com.
To post to this group, send email to [hidden email].
Visit this group at <a href="https://groups.google.com/group/testng-users" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/group/testng-users&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/group/testng-users&#39;;return true;">https://groups.google.com/group/testng-users.
For more options, visit <a href="https://groups.google.com/d/optout" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.
Loading...