Performance issue running code in TestNG?

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Performance issue running code in TestNG?

Rich Taylor
We have run into a severe performance issue running some of our code in TestNG - happens in JUnit also.

We have a library that can convert MS Word documents to PDF.  With some documents there is a severe
slowdown when the code is run via TestNG or JUnit.  Timings look like this:
     Call library directly                          5-6 seconds
     Call library from TestNG         220-280 seconds
     Call library from JUnit                  ~300 seconds

I have tried with different JRE versions (1.7, 1.7), architectures (32-bit, 64-bit) and memory sizes
(default -> -Xmx3072m).  No differences.

This doesn't happen with all of our code - just with this library and some specific files. (We've been happily
using TestNG for regression testing for a while now.)  But, the slowdown is severe and is the difference
between 2 minutes or 2 hours for testing a set of files.

Based on the symptoms our code has something to do with the problem.  I've spent a few days chasing
that without success (grrr).

Can anyone illuminate how TestNG may be interacting with our code to cause this performance issue?

Thanks for any help!


Rich Taylor
Sr. Developer
Qoppa Software (www.qoppa.com)


--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Performance issue running code in TestNG?

Cédric Beust ♔-2
Hi Rich,

Well, since the problem happens with both TestNG and JUnit, the problem seems to be with your code somewhere.

Can you run the tests with a profiler and see if anything comes up?

Such a difference in time should be easy to track down.


-- 
Cédric


On Thu, Jun 22, 2017 at 1:55 PM, Rich Taylor <[hidden email]> wrote:
We have run into a severe performance issue running some of our code in TestNG - happens in JUnit also.

We have a library that can convert MS Word documents to PDF.  With some documents there is a severe
slowdown when the code is run via TestNG or JUnit.  Timings look like this:
     Call library directly                          5-6 seconds
     Call library from TestNG         220-280 seconds
     Call library from JUnit                  ~300 seconds

I have tried with different JRE versions (1.7, 1.7), architectures (32-bit, 64-bit) and memory sizes
(default -> -Xmx3072m).  No differences.

This doesn't happen with all of our code - just with this library and some specific files. (We've been happily
using TestNG for regression testing for a while now.)  But, the slowdown is severe and is the difference
between 2 minutes or 2 hours for testing a set of files.

Based on the symptoms our code has something to do with the problem.  I've spent a few days chasing
that without success (grrr).

Can anyone illuminate how TestNG may be interacting with our code to cause this performance issue?

Thanks for any help!


Rich Taylor
Sr. Developer
Qoppa Software (www.qoppa.com)


--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Performance issue running code in TestNG?

Rich Taylor
Well...  ultimately I expect to identify some part of our code that is a factor for the problem.
However, since it _only_ happens in the context of a unit testing framework...  I expect to find
some of the problem there too...

We're just running our usual code - where we aren't calling into TestNG or JUnit.
Yet, simply the context of running as a TestNG test has slowed down our code
by an incredible factor of 60x (or so).

Can you shine light on what is different in the environment when running 
code in a TestNG test?

-= Rich


On Thursday, June 22, 2017 at 5:21:30 PM UTC-4, Cédric Beust ♔ wrote:
Hi Rich,

Well, since the problem happens with both TestNG and JUnit, the problem seems to be with your code somewhere.

Can you run the tests with a profiler and see if anything comes up?

Such a difference in time should be easy to track down.


-- 
Cédric


On Thu, Jun 22, 2017 at 1:55 PM, Rich Taylor <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="2Gbt-nFJBAAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">rtayl...@...> wrote:
We have run into a severe performance issue running some of our code in TestNG - happens in JUnit also.

We have a library that can convert MS Word documents to PDF.  With some documents there is a severe
slowdown when the code is run via TestNG or JUnit.  Timings look like this:
     Call library directly                          5-6 seconds
     Call library from TestNG         220-280 seconds
     Call library from JUnit                  ~300 seconds

I have tried with different JRE versions (1.7, 1.7), architectures (32-bit, 64-bit) and memory sizes
(default -> -Xmx3072m).  No differences.

This doesn't happen with all of our code - just with this library and some specific files. (We've been happily
using TestNG for regression testing for a while now.)  But, the slowdown is severe and is the difference
between 2 minutes or 2 hours for testing a set of files.

Based on the symptoms our code has something to do with the problem.  I've spent a few days chasing
that without success (grrr).

Can anyone illuminate how TestNG may be interacting with our code to cause this performance issue?

Thanks for any help!


Rich Taylor
Sr. Developer
Qoppa Software (<a href="http://www.qoppa.com" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fwww.qoppa.com\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHKaNyJxDH8ILY3kW4yQEA2kw0C7g&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fwww.qoppa.com\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHKaNyJxDH8ILY3kW4yQEA2kw0C7g&#39;;return true;">www.qoppa.com)


--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="2Gbt-nFJBAAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">testng-users...@googlegroups.com.
To post to this group, send email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="2Gbt-nFJBAAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">testng...@....
Visit this group at <a href="https://groups.google.com/group/testng-users" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/group/testng-users&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/group/testng-users&#39;;return true;">https://groups.google.com/group/testng-users.
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Performance issue running code in TestNG?

Cédric Beust ♔-2
You are definitely right that the fact this is only happening when run from a test environment is mystifying, and why I'm really curious to get to the bottom of this. I think the answer will make us all chuckle.

As far as TestNG is concerned, test methods are invoked through simple reflection so it's hard for me to imagine that a piece of code that's invoked from a `main()` would run more slowly when invoked by TestNG through a `@Test`.

Can you do this profiler run? It would help a lot.

If not, you could try to do some binary searching and progressively disable test methods until you find the one causing the problem?


-- 
Cédric


On Thu, Jun 22, 2017 at 5:47 PM, Rich Taylor <[hidden email]> wrote:
Well...  ultimately I expect to identify some part of our code that is a factor for the problem.
However, since it _only_ happens in the context of a unit testing framework...  I expect to find
some of the problem there too...

We're just running our usual code - where we aren't calling into TestNG or JUnit.
Yet, simply the context of running as a TestNG test has slowed down our code
by an incredible factor of 60x (or so).

Can you shine light on what is different in the environment when running 
code in a TestNG test?

-= Rich


On Thursday, June 22, 2017 at 5:21:30 PM UTC-4, Cédric Beust ♔ wrote:
Hi Rich,

Well, since the problem happens with both TestNG and JUnit, the problem seems to be with your code somewhere.

Can you run the tests with a profiler and see if anything comes up?

Such a difference in time should be easy to track down.


-- 
Cédric


On Thu, Jun 22, 2017 at 1:55 PM, Rich Taylor <[hidden email]> wrote:
We have run into a severe performance issue running some of our code in TestNG - happens in JUnit also.

We have a library that can convert MS Word documents to PDF.  With some documents there is a severe
slowdown when the code is run via TestNG or JUnit.  Timings look like this:
     Call library directly                          5-6 seconds
     Call library from TestNG         220-280 seconds
     Call library from JUnit                  ~300 seconds

I have tried with different JRE versions (1.7, 1.7), architectures (32-bit, 64-bit) and memory sizes
(default -> -Xmx3072m).  No differences.

This doesn't happen with all of our code - just with this library and some specific files. (We've been happily
using TestNG for regression testing for a while now.)  But, the slowdown is severe and is the difference
between 2 minutes or 2 hours for testing a set of files.

Based on the symptoms our code has something to do with the problem.  I've spent a few days chasing
that without success (grrr).

Can anyone illuminate how TestNG may be interacting with our code to cause this performance issue?

Thanks for any help!


Rich Taylor
Sr. Developer
Qoppa Software (www.qoppa.com)


--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to testng-users...@googlegroups.com.
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Performance issue running code in TestNG?

Rich Taylor
Do you have a preferred/recommended profiler?

Fortunately I haven't needed to become intimate with such a tool recently...

-= Rich

On Thursday, June 22, 2017 at 8:52:00 PM UTC-4, Cédric Beust ♔ wrote:
You are definitely right that the fact this is only happening when run from a test environment is mystifying, and why I'm really curious to get to the bottom of this. I think the answer will make us all chuckle.

As far as TestNG is concerned, test methods are invoked through simple reflection so it's hard for me to imagine that a piece of code that's invoked from a `main()` would run more slowly when invoked by TestNG through a `@Test`.

Can you do this profiler run? It would help a lot.

If not, you could try to do some binary searching and progressively disable test methods until you find the one causing the problem?


-- 
Cédric


On Thu, Jun 22, 2017 at 5:47 PM, Rich Taylor <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="XHgfu-5UBAAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">rtayl...@...> wrote:
Well...  ultimately I expect to identify some part of our code that is a factor for the problem.
However, since it _only_ happens in the context of a unit testing framework...  I expect to find
some of the problem there too...

We're just running our usual code - where we aren't calling into TestNG or JUnit.
Yet, simply the context of running as a TestNG test has slowed down our code
by an incredible factor of 60x (or so).

Can you shine light on what is different in the environment when running 
code in a TestNG test?

-= Rich


On Thursday, June 22, 2017 at 5:21:30 PM UTC-4, Cédric Beust ♔ wrote:
Hi Rich,

Well, since the problem happens with both TestNG and JUnit, the problem seems to be with your code somewhere.

Can you run the tests with a profiler and see if anything comes up?

Such a difference in time should be easy to track down.


-- 
Cédric


On Thu, Jun 22, 2017 at 1:55 PM, Rich Taylor <[hidden email]> wrote:
We have run into a severe performance issue running some of our code in TestNG - happens in JUnit also.

We have a library that can convert MS Word documents to PDF.  With some documents there is a severe
slowdown when the code is run via TestNG or JUnit.  Timings look like this:
     Call library directly                          5-6 seconds
     Call library from TestNG         220-280 seconds
     Call library from JUnit                  ~300 seconds

I have tried with different JRE versions (1.7, 1.7), architectures (32-bit, 64-bit) and memory sizes
(default -> -Xmx3072m).  No differences.

This doesn't happen with all of our code - just with this library and some specific files. (We've been happily
using TestNG for regression testing for a while now.)  But, the slowdown is severe and is the difference
between 2 minutes or 2 hours for testing a set of files.

Based on the symptoms our code has something to do with the problem.  I've spent a few days chasing
that without success (grrr).

Can anyone illuminate how TestNG may be interacting with our code to cause this performance issue?

Thanks for any help!


Rich Taylor
Sr. Developer
Qoppa Software (<a href="http://www.qoppa.com" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fwww.qoppa.com\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHKaNyJxDH8ILY3kW4yQEA2kw0C7g&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fwww.qoppa.com\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHKaNyJxDH8ILY3kW4yQEA2kw0C7g&#39;;return true;">www.qoppa.com)


--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to testng-users...@googlegroups.com.
To post to this group, send email to [hidden email].
Visit this group at <a href="https://groups.google.com/group/testng-users" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/group/testng-users&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/group/testng-users&#39;;return true;">https://groups.google.com/group/testng-users.
For more options, visit <a href="https://groups.google.com/d/optout" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="XHgfu-5UBAAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">testng-users...@googlegroups.com.
To post to this group, send email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="XHgfu-5UBAAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">testng...@....
Visit this group at <a href="https://groups.google.com/group/testng-users" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/group/testng-users&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/group/testng-users&#39;;return true;">https://groups.google.com/group/testng-users.
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Performance issue running code in TestNG?

Rich Taylor
I'm now working with VisualVM for profiling.

Most of the delay is happening in SocketInputStream.read(byte b[], int off, int length, int timeout)
which is at the bottom (top?) of a stack started from BaseMessageSender$ReaderThread.run()

I am not net debugging w/TestNG source, so I can't say much.  But, looks like network
communication being used for something is getting slowed down radically.

Does that tell us anything?  (while I continue digging...)

-= Rich



On Thursday, June 22, 2017 at 9:29:02 PM UTC-4, Rich Taylor wrote:
Do you have a preferred/recommended profiler?

Fortunately I haven't needed to become intimate with such a tool recently...

-= Rich

On Thursday, June 22, 2017 at 8:52:00 PM UTC-4, Cédric Beust ♔ wrote:
You are definitely right that the fact this is only happening when run from a test environment is mystifying, and why I'm really curious to get to the bottom of this. I think the answer will make us all chuckle.

As far as TestNG is concerned, test methods are invoked through simple reflection so it's hard for me to imagine that a piece of code that's invoked from a `main()` would run more slowly when invoked by TestNG through a `@Test`.

Can you do this profiler run? It would help a lot.

If not, you could try to do some binary searching and progressively disable test methods until you find the one causing the problem?


-- 
Cédric


On Thu, Jun 22, 2017 at 5:47 PM, Rich Taylor <[hidden email]> wrote:
Well...  ultimately I expect to identify some part of our code that is a factor for the problem.
However, since it _only_ happens in the context of a unit testing framework...  I expect to find
some of the problem there too...

We're just running our usual code - where we aren't calling into TestNG or JUnit.
Yet, simply the context of running as a TestNG test has slowed down our code
by an incredible factor of 60x (or so).

Can you shine light on what is different in the environment when running 
code in a TestNG test?

-= Rich


On Thursday, June 22, 2017 at 5:21:30 PM UTC-4, Cédric Beust ♔ wrote:
Hi Rich,

Well, since the problem happens with both TestNG and JUnit, the problem seems to be with your code somewhere.

Can you run the tests with a profiler and see if anything comes up?

Such a difference in time should be easy to track down.


-- 
Cédric


On Thu, Jun 22, 2017 at 1:55 PM, Rich Taylor <[hidden email]> wrote:
We have run into a severe performance issue running some of our code in TestNG - happens in JUnit also.

We have a library that can convert MS Word documents to PDF.  With some documents there is a severe
slowdown when the code is run via TestNG or JUnit.  Timings look like this:
     Call library directly                          5-6 seconds
     Call library from TestNG         220-280 seconds
     Call library from JUnit                  ~300 seconds

I have tried with different JRE versions (1.7, 1.7), architectures (32-bit, 64-bit) and memory sizes
(default -> -Xmx3072m).  No differences.

This doesn't happen with all of our code - just with this library and some specific files. (We've been happily
using TestNG for regression testing for a while now.)  But, the slowdown is severe and is the difference
between 2 minutes or 2 hours for testing a set of files.

Based on the symptoms our code has something to do with the problem.  I've spent a few days chasing
that without success (grrr).

Can anyone illuminate how TestNG may be interacting with our code to cause this performance issue?

Thanks for any help!


Rich Taylor
Sr. Developer
Qoppa Software (<a href="http://www.qoppa.com" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fwww.qoppa.com\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHKaNyJxDH8ILY3kW4yQEA2kw0C7g&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fwww.qoppa.com\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHKaNyJxDH8ILY3kW4yQEA2kw0C7g&#39;;return true;">www.qoppa.com)


--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to testng-users...@googlegroups.com.
To post to this group, send email to [hidden email].
Visit this group at <a href="https://groups.google.com/group/testng-users" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/group/testng-users&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/group/testng-users&#39;;return true;">https://groups.google.com/group/testng-users.
For more options, visit <a href="https://groups.google.com/d/optout" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to testng-users...@googlegroups.com.
To post to this group, send email to [hidden email].
Visit this group at <a href="https://groups.google.com/group/testng-users" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/group/testng-users&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/group/testng-users&#39;;return true;">https://groups.google.com/group/testng-users.
For more options, visit <a href="https://groups.google.com/d/optout" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Performance issue running code in TestNG?

Cédric Beust ♔-2
Sounds like you're running TestNG from your IDE if these classes are involved.

Can you try running your tests with TestNG from the command line?


-- 
Cédric


On Fri, Jun 23, 2017 at 10:43 AM, Rich Taylor <[hidden email]> wrote:
I'm now working with VisualVM for profiling.

Most of the delay is happening in SocketInputStream.read(byte b[], int off, int length, int timeout)
which is at the bottom (top?) of a stack started from BaseMessageSender$ReaderThread.run()

I am not net debugging w/TestNG source, so I can't say much.  But, looks like network
communication being used for something is getting slowed down radically.

Does that tell us anything?  (while I continue digging...)

-= Rich



On Thursday, June 22, 2017 at 9:29:02 PM UTC-4, Rich Taylor wrote:
Do you have a preferred/recommended profiler?

Fortunately I haven't needed to become intimate with such a tool recently...

-= Rich

On Thursday, June 22, 2017 at 8:52:00 PM UTC-4, Cédric Beust ♔ wrote:
You are definitely right that the fact this is only happening when run from a test environment is mystifying, and why I'm really curious to get to the bottom of this. I think the answer will make us all chuckle.

As far as TestNG is concerned, test methods are invoked through simple reflection so it's hard for me to imagine that a piece of code that's invoked from a `main()` would run more slowly when invoked by TestNG through a `@Test`.

Can you do this profiler run? It would help a lot.

If not, you could try to do some binary searching and progressively disable test methods until you find the one causing the problem?


-- 
Cédric


On Thu, Jun 22, 2017 at 5:47 PM, Rich Taylor <[hidden email]> wrote:
Well...  ultimately I expect to identify some part of our code that is a factor for the problem.
However, since it _only_ happens in the context of a unit testing framework...  I expect to find
some of the problem there too...

We're just running our usual code - where we aren't calling into TestNG or JUnit.
Yet, simply the context of running as a TestNG test has slowed down our code
by an incredible factor of 60x (or so).

Can you shine light on what is different in the environment when running 
code in a TestNG test?

-= Rich


On Thursday, June 22, 2017 at 5:21:30 PM UTC-4, Cédric Beust ♔ wrote:
Hi Rich,

Well, since the problem happens with both TestNG and JUnit, the problem seems to be with your code somewhere.

Can you run the tests with a profiler and see if anything comes up?

Such a difference in time should be easy to track down.


-- 
Cédric


On Thu, Jun 22, 2017 at 1:55 PM, Rich Taylor <[hidden email]> wrote:
We have run into a severe performance issue running some of our code in TestNG - happens in JUnit also.

We have a library that can convert MS Word documents to PDF.  With some documents there is a severe
slowdown when the code is run via TestNG or JUnit.  Timings look like this:
     Call library directly                          5-6 seconds
     Call library from TestNG         220-280 seconds
     Call library from JUnit                  ~300 seconds

I have tried with different JRE versions (1.7, 1.7), architectures (32-bit, 64-bit) and memory sizes
(default -> -Xmx3072m).  No differences.

This doesn't happen with all of our code - just with this library and some specific files. (We've been happily
using TestNG for regression testing for a while now.)  But, the slowdown is severe and is the difference
between 2 minutes or 2 hours for testing a set of files.

Based on the symptoms our code has something to do with the problem.  I've spent a few days chasing
that without success (grrr).

Can anyone illuminate how TestNG may be interacting with our code to cause this performance issue?

Thanks for any help!


Rich Taylor
Sr. Developer
Qoppa Software (www.qoppa.com)


--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email]om.
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email]om.
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Performance issue running code in TestNG?

Rich Taylor
I ran from command line & (quickly) attached the VisualVM profiler.
Same results - almost all time is in SocketInputStream.socketRead() in a thread named "RMI TCP Connection"

In Eclipse I set a breakpoint in socketRead() and I see it getting called from 

REF_WordImages.REF_wordToImage_embed [TestNG]
org.testng.remote.RemoteTestNG at localhost:63360
Thread [main] (Running)
Thread [ReaderThread] (Suspended (breakpoint at line 116 in SocketInputStream))
owns: InputStreamReader  (id=18)
SocketInputStream.socketRead(FileDescriptor, byte[], int, int, int) line: 116
SocketInputStream.read(byte[], int, int, int) line: 170
SocketInputStream.read(byte[], int, int) line: 141
StreamDecoder.readBytes() line: 284
StreamDecoder.implRead(char[], int, int) line: 326
StreamDecoder.read(char[], int, int) line: 178
InputStreamReader.read(char[], int, int) line: 184
BufferedReader.fill() line: 161
BufferedReader.readLine(boolean) line: 324
BufferedReader.readLine() line: 389
BaseMessageSender$ReaderThread.run() line: 232
C:\Program Files\Java\jdk1.8.0_92\bin\javaw.exe (Jun 23, 2017, 4:31:01 PM)

When running command line & with profiling I see massive delays - so, not related to Eclipse I assume.

I'm not sure where to look next.

-= Rich Taylor

On Friday, June 23, 2017 at 2:26:57 PM UTC-4, Cédric Beust ♔ wrote:
Sounds like you're running TestNG from your IDE if these classes are involved.

Can you try running your tests with TestNG from the command line?


-- 
Cédric


On Fri, Jun 23, 2017 at 10:43 AM, Rich Taylor <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="czEWKoCOBAAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">rtayl...@...> wrote:
I'm now working with VisualVM for profiling.

Most of the delay is happening in SocketInputStream.read(byte b[], int off, int length, int timeout)
which is at the bottom (top?) of a stack started from BaseMessageSender$ReaderThread.run()

I am not net debugging w/TestNG source, so I can't say much.  But, looks like network
communication being used for something is getting slowed down radically.

Does that tell us anything?  (while I continue digging...)

-= Rich



On Thursday, June 22, 2017 at 9:29:02 PM UTC-4, Rich Taylor wrote:
Do you have a preferred/recommended profiler?

Fortunately I haven't needed to become intimate with such a tool recently...

-= Rich

On Thursday, June 22, 2017 at 8:52:00 PM UTC-4, Cédric Beust ♔ wrote:
You are definitely right that the fact this is only happening when run from a test environment is mystifying, and why I'm really curious to get to the bottom of this. I think the answer will make us all chuckle.

As far as TestNG is concerned, test methods are invoked through simple reflection so it's hard for me to imagine that a piece of code that's invoked from a `main()` would run more slowly when invoked by TestNG through a `@Test`.

Can you do this profiler run? It would help a lot.

If not, you could try to do some binary searching and progressively disable test methods until you find the one causing the problem?


-- 
Cédric


On Thu, Jun 22, 2017 at 5:47 PM, Rich Taylor <[hidden email]> wrote:
Well...  ultimately I expect to identify some part of our code that is a factor for the problem.
However, since it _only_ happens in the context of a unit testing framework...  I expect to find
some of the problem there too...

We're just running our usual code - where we aren't calling into TestNG or JUnit.
Yet, simply the context of running as a TestNG test has slowed down our code
by an incredible factor of 60x (or so).

Can you shine light on what is different in the environment when running 
code in a TestNG test?

-= Rich


On Thursday, June 22, 2017 at 5:21:30 PM UTC-4, Cédric Beust ♔ wrote:
Hi Rich,

Well, since the problem happens with both TestNG and JUnit, the problem seems to be with your code somewhere.

Can you run the tests with a profiler and see if anything comes up?

Such a difference in time should be easy to track down.


-- 
Cédric


On Thu, Jun 22, 2017 at 1:55 PM, Rich Taylor <[hidden email]> wrote:
We have run into a severe performance issue running some of our code in TestNG - happens in JUnit also.

We have a library that can convert MS Word documents to PDF.  With some documents there is a severe
slowdown when the code is run via TestNG or JUnit.  Timings look like this:
     Call library directly                          5-6 seconds
     Call library from TestNG         220-280 seconds
     Call library from JUnit                  ~300 seconds

I have tried with different JRE versions (1.7, 1.7), architectures (32-bit, 64-bit) and memory sizes
(default -> -Xmx3072m).  No differences.

This doesn't happen with all of our code - just with this library and some specific files. (We've been happily
using TestNG for regression testing for a while now.)  But, the slowdown is severe and is the difference
between 2 minutes or 2 hours for testing a set of files.

Based on the symptoms our code has something to do with the problem.  I've spent a few days chasing
that without success (grrr).

Can anyone illuminate how TestNG may be interacting with our code to cause this performance issue?

Thanks for any help!


Rich Taylor
Sr. Developer
Qoppa Software (<a href="http://www.qoppa.com" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fwww.qoppa.com\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHKaNyJxDH8ILY3kW4yQEA2kw0C7g&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fwww.qoppa.com\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHKaNyJxDH8ILY3kW4yQEA2kw0C7g&#39;;return true;">www.qoppa.com)


--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to testng-users...@googlegroups.com.
To post to this group, send email to [hidden email].
Visit this group at <a href="https://groups.google.com/group/testng-users" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/group/testng-users&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/group/testng-users&#39;;return true;">https://groups.google.com/group/testng-users.
For more options, visit <a href="https://groups.google.com/d/optout" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to testng-users...@googlegroups.com.
To post to this group, send email to [hidden email].
Visit this group at <a href="https://groups.google.com/group/testng-users" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/group/testng-users&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/group/testng-users&#39;;return true;">https://groups.google.com/group/testng-users.
For more options, visit <a href="https://groups.google.com/d/optout" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="czEWKoCOBAAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">testng-users...@googlegroups.com.
To post to this group, send email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="czEWKoCOBAAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">testng...@....
Visit this group at <a href="https://groups.google.com/group/testng-users" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/group/testng-users&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/group/testng-users&#39;;return true;">https://groups.google.com/group/testng-users.
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.
Loading...