[testng-dev] testng slow to start running tests

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

[testng-dev] testng slow to start running tests

Michael Ondrejko
We are using testng with a large source base. With testng 6.3.2 and 6.4 there is a long pause before tests start running. It seems that this is the time that testng is scanning classes for testng annotations. For our source base, this requires 15+ minutes to complete. Unfortunately, our test classes are in the same package structure as the product classes themselves and it is not possible to change this organization. I have looked for a way to limit that classes (and jars) that testng is scanning, but it seems to pick up the test classes, everything in the classpath that matches the package definition specified in testng.xml are scanned.
 
Is it possible to further limit this scanning or otherwise speed up the startup time for testng.

--
You received this message because you are subscribed to the Google Groups "testng-dev" group.
To view this discussion on the web visit https://groups.google.com/d/msg/testng-dev/-/iVXlAbeoN3MJ.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/testng-dev?hl=en.
Reply | Threaded
Open this post in threaded view
|

Re: [testng-dev] testng slow to start running tests

Cédric Beust ♔-2
Hi Michael,

It's a bit hard to comment without any data, do you have the output of a profiler to share?

Having said that, you are right that specifying packages is expensive, consider specifying classes explicitly or only running your tests with a minimal classpath.

-- 
Cédric




On Tue, Jun 12, 2012 at 3:09 PM, Michael Ondrejko <[hidden email]> wrote:
We are using testng with a large source base. With testng 6.3.2 and 6.4 there is a long pause before tests start running. It seems that this is the time that testng is scanning classes for testng annotations. For our source base, this requires 15+ minutes to complete. Unfortunately, our test classes are in the same package structure as the product classes themselves and it is not possible to change this organization. I have looked for a way to limit that classes (and jars) that testng is scanning, but it seems to pick up the test classes, everything in the classpath that matches the package definition specified in testng.xml are scanned.
 
Is it possible to further limit this scanning or otherwise speed up the startup time for testng.

--
You received this message because you are subscribed to the Google Groups "testng-dev" group.
To view this discussion on the web visit https://groups.google.com/d/msg/testng-dev/-/iVXlAbeoN3MJ.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/testng-dev?hl=en.

--
You received this message because you are subscribed to the Google Groups "testng-dev" group.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/testng-dev?hl=en.
Reply | Threaded
Open this post in threaded view
|

[testng-dev] Re: testng slow to start running tests

Michael Ondrejko
In reply to this post by Michael Ondrejko
There are thousands of test classes, so specifying each individual test class is not feasible. Previously, with testng 6.0.1m the project team used an include sub-element in the package specification too filter specific tests. For example,
 
  <package name="com.company.*">
    <include name=".*Test"/>
  </package>
 
To process only classes that end in "Test" in the "com.company" package hierarchy. However, this now results in all tests being excluded as the include here only filters based upon package name. I tried to use wildcards in the class element for testng.xml; however, class specification does not seem to support wildcards.
 
Is there a way in testng.xml to specify a pattern for the classes to scan for annotations within a package structure?

--
You received this message because you are subscribed to the Google Groups "testng-dev" group.
To view this discussion on the web visit https://groups.google.com/d/msg/testng-dev/-/TyAEPfuPMK0J.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/testng-dev?hl=en.
Reply | Threaded
Open this post in threaded view
|

Re: [testng-dev] Re: testng slow to start running tests

Cédric Beust ♔-2

On Wed, Jun 13, 2012 at 6:54 AM, Michael Ondrejko <[hidden email]> wrote:
Is there a way in testng.xml to specify a pattern for the classes to scan for annotations within a package structure?

This won't make a difference, the problem is that TestNG has to look up all the classes and packages found on the class path, if only to apply that filter.

But again, we're just guessing at this point until you run a profiler to find out exactly where the performance problems are coming from.

-- 
Cédric

--
You received this message because you are subscribed to the Google Groups "testng-dev" group.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/testng-dev?hl=en.
Reply | Threaded
Open this post in threaded view
|

Re: [testng-dev] Re: testng slow to start running tests

Alexandru Baxanean
When using testng inside maven, is there any way to tell testng to scan for classes only in test-classes directory ?
That's to avoid the useless scanning of hundreds of jar files and the classes directory where source (not test) classes are located.

On Wednesday, June 13, 2012 at 8:26:00 AM UTC-7, Cédric Beust ♔ wrote:

On Wed, Jun 13, 2012 at 6:54 AM, Michael Ondrejko <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="LxV5lROi9cIJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">mic...@...> wrote:
Is there a way in testng.xml to specify a pattern for the classes to scan for annotations within a package structure?

This won't make a difference, the problem is that TestNG has to look up all the classes and packages found on the class path, if only to apply that filter.

But again, we're just guessing at this point until you run a profiler to find out exactly where the performance problems are coming from.

-- 
Cédric

--
You received this message because you are subscribed to the Google Groups "testng-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-dev.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

[testng-dev] Re: testng slow to start running tests

g_cai
In reply to this post by Michael Ondrejko
We have 15 suite-file and use gradle to lunch the test, and we observed that test class objects are created and then paused for ~15 minutes to start running the actual test methods....very poor performance.

On Tuesday, June 12, 2012 at 3:09:42 PM UTC-7, Michael Ondrejko wrote:
We are using testng with a large source base. With testng 6.3.2 and 6.4 there is a long pause before tests start running. It seems that this is the time that testng is scanning classes for testng annotations. For our source base, this requires 15+ minutes to complete. Unfortunately, our test classes are in the same package structure as the product classes themselves and it is not possible to change this organization. I have looked for a way to limit that classes (and jars) that testng is scanning, but it seems to pick up the test classes, everything in the classpath that matches the package definition specified in testng.xml are scanned.
 
Is it possible to further limit this scanning or otherwise speed up the startup time for testng.

--
You received this message because you are subscribed to the Google Groups "testng-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-dev.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [testng-dev] Re: testng slow to start running tests

Julien Herr
Did you try recent releases like 6.11 or 6.12? Is it better?

Le ven. 18 août 2017 à 18:25, <[hidden email]> a écrit :
We have 15 suite-file and use gradle to lunch the test, and we observed that test class objects are created and then paused for ~15 minutes to start running the actual test methods....very poor performance.

On Tuesday, June 12, 2012 at 3:09:42 PM UTC-7, Michael Ondrejko wrote:
We are using testng with a large source base. With testng 6.3.2 and 6.4 there is a long pause before tests start running. It seems that this is the time that testng is scanning classes for testng annotations. For our source base, this requires 15+ minutes to complete. Unfortunately, our test classes are in the same package structure as the product classes themselves and it is not possible to change this organization. I have looked for a way to limit that classes (and jars) that testng is scanning, but it seems to pick up the test classes, everything in the classpath that matches the package definition specified in testng.xml are scanned.
 
Is it possible to further limit this scanning or otherwise speed up the startup time for testng.

--
You received this message because you are subscribed to the Google Groups "testng-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-dev.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "testng-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-dev.
For more options, visit https://groups.google.com/d/optout.