Quantcast

Simplified report?

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Simplified report?

Jeff-351
I have a test class with 2 test methods I am running via Maven using TestNG 6.10.

Both test methods are multi-threaded (50 threads) and fed by DataProviders that pull information from a database.  

While the test seems to run okay (10 minutes or less), it takes 90 minutes to generate the report and it's driving me crazy and is up to 1.6GB RAM used.  I'm not even close to the scale I want either.

I've tried disabling the default listeners via the surefire plugin configuration and only using the XMLReporter (assuming this is more minimalist).  I've also tried to use the ReportNG methods, but the POM changes (see below) didn't seem to have any effect.  Is there something else I should or need to do or is the issue not in the reporter?

Thanks!

            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <configuration>
                    <argLine>-Xmx2048m</argLine>
                    <property>
                        <name>usedefaultlisteners</name>
                        <value>false</value>
                    </property>
                    <property>
                        <name>listener</name>
                        <value>org.testng.reporters.XMLReporter</value>
                    </property>
                </configuration>
            </plugin>

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Simplified report?

Cédric Beust ♔-2

The issue is probably in the reporter.

If you run your tests with verbose="2", you should see timings for the reporters:

[TestNG] Time taken by org.testng.reporters.XMLReporter@631330c: 53 ms
[TestNG] Time taken by org.testng.reporters.SuiteHTMLReporter@1a1d6a08: 19 ms
[TestNG] Time taken by org.testng.reporters.jq.Main@74ad1f1f: 52 ms
[TestNG] Time taken by [FailedReporter passed=0 failed=0 skipped=0]: 0 ms
[TestNG] Time taken by org.testng.reporters.JUnitReportReporter@69b0fd6f: 26 ms
[TestNG] Time taken by org.testng.reporters.EmailableReporter2@52f759d7: 22 ms

Can you share this?


-- 
Cédric


On Wed, Feb 15, 2017 at 10:31 AM, Jeff <[hidden email]> wrote:
I have a test class with 2 test methods I am running via Maven using TestNG 6.10.

Both test methods are multi-threaded (50 threads) and fed by DataProviders that pull information from a database.  

While the test seems to run okay (10 minutes or less), it takes 90 minutes to generate the report and it's driving me crazy and is up to 1.6GB RAM used.  I'm not even close to the scale I want either.

I've tried disabling the default listeners via the surefire plugin configuration and only using the XMLReporter (assuming this is more minimalist).  I've also tried to use the ReportNG methods, but the POM changes (see below) didn't seem to have any effect.  Is there something else I should or need to do or is the issue not in the reporter?

Thanks!

            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <configuration>
                    <argLine>-Xmx2048m</argLine>
                    <property>
                        <name>usedefaultlisteners</name>
                        <value>false</value>
                    </property>
                    <property>
                        <name>listener</name>
                        <value>org.testng.reporters.XMLReporter</value>
                    </property>
                </configuration>
            </plugin>

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Simplified report?

Jeff-351

On Wed, Feb 15, 2017 at 11:39 AM, Cédric Beust ♔ <[hidden email]> wrote:
verbose="2"

Yes, I'm working on getting this info.  Can I somehow set the verbose flag from the maven command-line or do I need a testng.xml file (I don't normally use XML files)?  I tried -Dverbose=2 but it didn't appear to work but I might have been too impatient.

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Simplified report?

Cédric Beust ♔-2

On the TestNG command line, it would be -verbose 2, but I don’t know about Maven. Look up Surefire, pretty sure it’s supported.


-- 
Cédric


On Wed, Feb 15, 2017 at 1:40 PM, Jeff <[hidden email]> wrote:

On Wed, Feb 15, 2017 at 11:39 AM, Cédric Beust ♔ <[hidden email]> wrote:
verbose="2"

Yes, I'm working on getting this info.  Can I somehow set the verbose flag from the maven command-line or do I need a testng.xml file (I don't normally use XML files)?  I tried -Dverbose=2 but it didn't appear to work but I might have been too impatient.

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Simplified report?

Jeff-351
Finally got it working.  In doing so, I found the main issue with how I was specifying the option in my POM file.  Now can actually see and compare timings and change reporters.  If I disable all but the XMLReporter, it does complete much more quickly, but now that I can actually swap in ReportNG, I'll probably keep using it.

Default reporters(53K tests):
[TestNG] Time taken by [FailedReporter passed=0 failed=0 skipped=0]: 0 ms
[TestNG] Time taken by org.testng.reporters.XMLReporter@534243e4: 292556 ms
[TestNG] Time taken by org.testng.reporters.SuiteHTMLReporter@697446d4: 1401 ms
[TestNG] Time taken by org.testng.reporters.jq.Main@3f4f9acd: 235518 ms
[TestNG] Time taken by org.testng.reporters.EmailableReporter2@3e58d65e: 651 ms
[TestNG] Time taken by org.testng.reporters.JUnitReportReporter@7e0b9178: 5649 ms
Tests run: 53000, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 762.772 sec

ReportNG (53K tests):
[TestNG] Time taken by org.uncommons.reportng.HTMLReporter@2c779e5: 889 ms
[TestNG] Time taken by org.uncommons.reportng.JUnitXMLReporter@588ffeb: 277 ms
Tests run: 53000, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 222.183 sec

For the sake of completeness, the surefire plugin in my POM now looks like (I was initially missing the enclosing <properties\> tags (oops):

            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <configuration>
                    <argLine>-Xmx2048m</argLine>
                    <properties>
                        <property>
                            <name>surefire.testng.verbose</name>
                            <value>2</value>
                        </property>
                        <property>
                            <name>usedefaultlisteners</name>
                            <value>false</value>
                        </property>
                        <property>
                            <name>listener</name>
                            <value>org.uncommons.reportng.HTMLReporter, org.uncommons.reportng.JUnitXMLReporter</value>
                        </property>
                    </properties>
                </configuration>
            </plugin>

Question 1:
As part of my investigation, I discovered that I could get the XmlSuite object instance in a @BeforeSuite method, but setting the verbose value it didn't seem to take effect.  Why would that be?

    @BeforeSuite
    public void setupSuite(ITestContext ctx) throws InitializationException {
        ctx.getSuite().getXmlSuite().setVerbose(2);
    }

Question 2:
Also, the way maven-surefire is reporting the results is odd.  The console output shows only 10001 tests with 1 failure:

Inline image 2

But all test report files (including XML, HTML and TXT files) show the correct totals.  Why the difference? Any thoughts?

    From target/surefire-reports/html/index.html:
Inline image 1

    From target/surefire-reports/TestIntegration_BCISFeedValidation.txt:
Inline image 3

    From target/surefire-reports/TEST-TestIntegration_BCISFeedValidation.xml:
Inline image 4

    From target/surefire-reports/xml/TestIntegration_BCISFeedValidation_results.xml:
Inline image 5

Thanks!

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Simplified report?

Cédric Beust ♔-2
Thanks for the investigation. There is definitely room for improvement in performances in these two reporters.


-- 
Cédric


On Thu, Feb 16, 2017 at 2:57 PM, Jeff <[hidden email]> wrote:
Finally got it working.  In doing so, I found the main issue with how I was specifying the option in my POM file.  Now can actually see and compare timings and change reporters.  If I disable all but the XMLReporter, it does complete much more quickly, but now that I can actually swap in ReportNG, I'll probably keep using it.

Default reporters(53K tests):
[TestNG] Time taken by [FailedReporter passed=0 failed=0 skipped=0]: 0 ms
[TestNG] Time taken by org.testng.reporters.XMLReporter@534243e4: 292556 ms
[TestNG] Time taken by org.testng.reporters.SuiteHTMLReporter@697446d4: 1401 ms
[TestNG] Time taken by org.testng.reporters.jq.Main@3f4f9acd: 235518 ms
[TestNG] Time taken by org.testng.reporters.EmailableReporter2@3e58d65e: 651 ms
[TestNG] Time taken by org.testng.reporters.JUnitReportReporter@7e0b9178: 5649 ms
Tests run: 53000, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 762.772 sec

ReportNG (53K tests):
[TestNG] Time taken by org.uncommons.reportng.HTMLReporter@2c779e5: 889 ms
[TestNG] Time taken by org.uncommons.reportng.JUnitXMLReporter@588ffeb: 277 ms
Tests run: 53000, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 222.183 sec

For the sake of completeness, the surefire plugin in my POM now looks like (I was initially missing the enclosing <properties\> tags (oops):

            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <configuration>
                    <argLine>-Xmx2048m</argLine>
                    <properties>
                        <property>
                            <name>surefire.testng.verbose</name>
                            <value>2</value>
                        </property>
                        <property>
                            <name>usedefaultlisteners</name>
                            <value>false</value>
                        </property>
                        <property>
                            <name>listener</name>
                            <value>org.uncommons.reportng.HTMLReporter, org.uncommons.reportng.JUnitXMLReporter</value>
                        </property>
                    </properties>
                </configuration>
            </plugin>

Question 1:
As part of my investigation, I discovered that I could get the XmlSuite object instance in a @BeforeSuite method, but setting the verbose value it didn't seem to take effect.  Why would that be?

    @BeforeSuite
    public void setupSuite(ITestContext ctx) throws InitializationException {
        ctx.getSuite().getXmlSuite().setVerbose(2);
    }

Question 2:
Also, the way maven-surefire is reporting the results is odd.  The console output shows only 10001 tests with 1 failure:

Inline image 2

But all test report files (including XML, HTML and TXT files) show the correct totals.  Why the difference? Any thoughts?

    From target/surefire-reports/html/index.html:
Inline image 1

    From target/surefire-reports/TestIntegration_BCISFeedValidation.txt:
Inline image 3

    From target/surefire-reports/TEST-TestIntegration_BCISFeedValidation.xml:
Inline image 4

    From target/surefire-reports/xml/TestIntegration_BCISFeedValidation_results.xml:
Inline image 5

Thanks!

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Simplified report?

Julien Herr
> Question 1:
> As part of my investigation, I discovered that I could get the XmlSuite object instance in a @BeforeSuite method, but setting the verbose value it didn't seem to take effect.  Why would that be?

I'm not sure Xml* objects are designed to be modified at runtime into configuration methods like @BeforeSuite (Cédric, agree?).
ITestContext should be the way to do it instead, but `setVerbose` doesn't exist there :p

> Question 2:
> Also, the way maven-surefire is reporting the results is odd.  The console output shows only 10001 tests with 1 failure:

It looks like a surefire issue. If you are able to reproduce it easily, you should open an issue on https://issues.apache.org/jira/browse/SUREFIRE/

>  There is definitely room for improvement in performances in these two reporters.

Or we can consider integrating ReportNG which is no more supported: https://github.com/dwdyer/reportng

Le vendredi 17 février 2017 00:41:27 UTC+1, Cédric Beust ♔ a écrit :
Thanks for the investigation. There is definitely room for improvement in performances in these two reporters.


-- 
Cédric


On Thu, Feb 16, 2017 at 2:57 PM, Jeff <[hidden email]> wrote:
Finally got it working.  In doing so, I found the main issue with how I was specifying the option in my POM file.  Now can actually see and compare timings and change reporters.  If I disable all but the XMLReporter, it does complete much more quickly, but now that I can actually swap in ReportNG, I'll probably keep using it.

Default reporters(53K tests):
[TestNG] Time taken by [FailedReporter passed=0 failed=0 skipped=0]: 0 ms
[TestNG] Time taken by org.testng.reporters.XMLReporter@534243e4: 292556 ms
[TestNG] Time taken by org.testng.reporters.SuiteHTMLReporter@697446d4: 1401 ms
[TestNG] Time taken by org.testng.reporters.jq.Main@3f4f9acd: 235518 ms
[TestNG] Time taken by org.testng.reporters.EmailableReporter2@3e58d65e: 651 ms
[TestNG] Time taken by org.testng.reporters.JUnitReportReporter@7e0b9178: 5649 ms
Tests run: 53000, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 762.772 sec

ReportNG (53K tests):
[TestNG] Time taken by org.uncommons.reportng.HTMLReporter@2c779e5: 889 ms
[TestNG] Time taken by org.uncommons.reportng.JUnitXMLReporter@588ffeb: 277 ms
Tests run: 53000, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 222.183 sec

For the sake of completeness, the surefire plugin in my POM now looks like (I was initially missing the enclosing <properties\> tags (oops):

            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <configuration>
                    <argLine>-Xmx2048m</argLine>
                    <properties>
                        <property>
                            <name>surefire.testng.verbose</name>
                            <value>2</value>
                        </property>
                        <property>
                            <name>usedefaultlisteners</name>
                            <value>false</value>
                        </property>
                        <property>
                            <name>listener</name>
                            <value>org.uncommons.reportng.HTMLReporter, org.uncommons.reportng.JUnitXMLReporter</value>
                        </property>
                    </properties>
                </configuration>
            </plugin>

Question 1:
As part of my investigation, I discovered that I could get the XmlSuite object instance in a @BeforeSuite method, but setting the verbose value it didn't seem to take effect.  Why would that be?

    @BeforeSuite
    public void setupSuite(ITestContext ctx) throws InitializationException {
        ctx.getSuite().getXmlSuite().setVerbose(2);
    }

Question 2:
Also, the way maven-surefire is reporting the results is odd.  The console output shows only 10001 tests with 1 failure:

Inline image 2

But all test report files (including XML, HTML and TXT files) show the correct totals.  Why the difference? Any thoughts?

    From target/surefire-reports/html/index.html:
Inline image 1

    From target/surefire-reports/TestIntegration_BCISFeedValidation.txt:
Inline image 3

    From target/surefire-reports/TEST-TestIntegration_BCISFeedValidation.xml:
Inline image 4

    From target/surefire-reports/xml/TestIntegration_BCISFeedValidation_results.xml:
Inline image 5

Thanks!

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.
Loading...