Issue with testng-failed.xml file with 6.9.10

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Issue with testng-failed.xml file with 6.9.10

Nagesh
I updated the version in my project from 6.9.9 to 6.9.10 yesterday and started noticing failures in our CI runs. The issue is that the testng-failed.xml file that gets generated at the end of the first run, still contains the same tests as the original suite file. The expectation is that it contains only the failed test cases. Also if all the tests passed in the first run with the original suite file, a testng-failed.xml file still gets generated. Prior to 6.9.10 the failed xml file would not get generated if all tests in the run passed.

I have now reverted back to 6.9.9. Has this been discussed previously? I looked here and in GitHub but didn't find a thread regarding this issue.

Here is the original suite file -

<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="Turbo WebDriver Tests" parallel="classes" thread-count="5">
<listeners>
<listener class-name="com.axiom.turbo.webdriver.TestListener" />
<listener class-name="com.axiom.turbo.webdriver.TestAnnotationTransformer" />
</listeners>
        <test name = "Classes with a single test method">
<groups>
<run>
<exclude name="KnownIssues" />
<exclude name="CatalogsCache" />
<exclude name="SlowRunning" />
<exclude name="Defunct" />
<exclude name="FeatureParity" />
</run>
</groups>
<classes>
<class name = "com.axiom.turbo.webdriver.tests.privacypolicy.PrivacyPolicyPageWebDriverTests" />
                        <class name = "com.axiom.turbo.webdriver.tests.labstests.LabsPageWebDriverTests" />
<class name = "com.axiom.turbo.webdriver.tests.SuccessTestClass" />
<class name = "com.axiom.turbo.webdriver.tests.SkipTestClass" />
<class name = "com.axiom.turbo.webdriver.tests.FailTestClass" />
</classes>
</test>
</suite>


Here is the testng-failed.xml file that gets generated if all or some tests passed with all the tests as the original suite file -

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite guice-stage="DEVELOPMENT" verbose="0" name="Failed suite [Turbo WebDriver Tests]" parallel="classes">
  <listeners>
    <listener class-name="com.axiom.turbo.webdriver.TestListener"/>
    <listener class-name="com.axiom.turbo.webdriver.TestAnnotationTransformer"/>
  </listeners>
  <test name="Classes with a single test method">
    <groups>
      <run>
        <exclude name="KnownIssues"/>
        <exclude name="CatalogsCache"/>
        <exclude name="SlowRunning"/>
        <exclude name="Defunct"/>
        <exclude name="FeatureParity"/>
      </run>
    </groups>
    <classes>
      <class name="com.axiom.turbo.webdriver.tests.privacypolicy.PrivacyPolicyPageWebDriverTests"/>
      <class name="com.axiom.turbo.webdriver.tests.SuccessTestClass"/>
      <class name="com.axiom.turbo.webdriver.tests.SkipTestClass"/>
      <class name="com.axiom.turbo.webdriver.tests.FailTestClass"/>
    </classes>
  </test> <!-- Classes with a single test method -->
</suite> <!-- Failed suite [Turbo WebDriver Tests] -->


Thanks!

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Issue with testng-failed.xml file with 6.9.10

Nagesh
I took out the custom listeners from the suite file to see if that helps narrow down the problem, but I am still seeing an incorrect testng-failed.xml file getting generated at the end of the run. I took a look at the upcoming changes and I don't know if it's being addressed with the next version.

Can someone try it out and shed some light on this issue? As I said earlier, this functionality has worked as expected until 6.9.9, but has started failing with latest 6.9.10. I will be happy to help debug/provide more info if necessary.

Greatly appreciate your inputs.

-Nagesh

On Wednesday, January 13, 2016 at 4:38:04 PM UTC-8, Nagesh wrote:
I updated the version in my project from 6.9.9 to 6.9.10 yesterday and started noticing failures in our CI runs. The issue is that the testng-failed.xml file that gets generated at the end of the first run, still contains the same tests as the original suite file. The expectation is that it contains only the failed test cases. Also if all the tests passed in the first run with the original suite file, a testng-failed.xml file still gets generated. Prior to 6.9.10 the failed xml file would not get generated if all tests in the run passed.

I have now reverted back to 6.9.9. Has this been discussed previously? I looked here and in GitHub but didn't find a thread regarding this issue.

Here is the original suite file -

<!DOCTYPE suite SYSTEM "<a href="http://testng.org/testng-1.0.dtd" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\75http%3A%2F%2Ftestng.org%2Ftestng-1.0.dtd\46sa\75D\46sntz\0751\46usg\75AFQjCNFHihXzQ1YlwcxcDehnw97TuniFLA&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\75http%3A%2F%2Ftestng.org%2Ftestng-1.0.dtd\46sa\75D\46sntz\0751\46usg\75AFQjCNFHihXzQ1YlwcxcDehnw97TuniFLA&#39;;return true;">http://testng.org/testng-1.0.dtd">
<suite name="Turbo WebDriver Tests" parallel="classes" thread-count="5">
<listeners>
<listener class-name="com.axiom.turbo.webdriver.TestListener" />
<listener class-name="com.axiom.turbo.webdriver.TestAnnotationTransformer" />
</listeners>
        <test name = "Classes with a single test method">
<groups>
<run>
<exclude name="KnownIssues" />
<exclude name="CatalogsCache" />
<exclude name="SlowRunning" />
<exclude name="Defunct" />
<exclude name="FeatureParity" />
</run>
</groups>
<classes>
<class name = "com.axiom.turbo.webdriver.tests.privacypolicy.PrivacyPolicyPageWebDriverTests" />
                        <class name = "com.axiom.turbo.webdriver.tests.labstests.LabsPageWebDriverTests" />
<class name = "com.axiom.turbo.webdriver.tests.SuccessTestClass" />
<class name = "com.axiom.turbo.webdriver.tests.SkipTestClass" />
<class name = "com.axiom.turbo.webdriver.tests.FailTestClass" />
</classes>
</test>
</suite>


Here is the testng-failed.xml file that gets generated if all or some tests passed with all the tests as the original suite file -

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "<a href="http://testng.org/testng-1.0.dtd" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\75http%3A%2F%2Ftestng.org%2Ftestng-1.0.dtd\46sa\75D\46sntz\0751\46usg\75AFQjCNFHihXzQ1YlwcxcDehnw97TuniFLA&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\75http%3A%2F%2Ftestng.org%2Ftestng-1.0.dtd\46sa\75D\46sntz\0751\46usg\75AFQjCNFHihXzQ1YlwcxcDehnw97TuniFLA&#39;;return true;">http://testng.org/testng-1.0.dtd">
<suite guice-stage="DEVELOPMENT" verbose="0" name="Failed suite [Turbo WebDriver Tests]" parallel="classes">
  <listeners>
    <listener class-name="com.axiom.turbo.webdriver.TestListener"/>
    <listener class-name="com.axiom.turbo.webdriver.TestAnnotationTransformer"/>
  </listeners>
  <test name="Classes with a single test method">
    <groups>
      <run>
        <exclude name="KnownIssues"/>
        <exclude name="CatalogsCache"/>
        <exclude name="SlowRunning"/>
        <exclude name="Defunct"/>
        <exclude name="FeatureParity"/>
      </run>
    </groups>
    <classes>
      <class name="com.axiom.turbo.webdriver.tests.privacypolicy.PrivacyPolicyPageWebDriverTests"/>
      <class name="com.axiom.turbo.webdriver.tests.SuccessTestClass"/>
      <class name="com.axiom.turbo.webdriver.tests.SkipTestClass"/>
      <class name="com.axiom.turbo.webdriver.tests.FailTestClass"/>
    </classes>
  </test> <!-- Classes with a single test method -->
</suite> <!-- Failed suite [Turbo WebDriver Tests] -->


Thanks!

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Issue with testng-failed.xml file with 6.9.10

Cédric Beust ♔-2
Krishnan, could you take a look at this?


-- 
Cédric


On Sat, Jan 16, 2016 at 11:45 PM, Nagesh <[hidden email]> wrote:
I took out the custom listeners from the suite file to see if that helps narrow down the problem, but I am still seeing an incorrect testng-failed.xml file getting generated at the end of the run. I took a look at the upcoming changes and I don't know if it's being addressed with the next version.

Can someone try it out and shed some light on this issue? As I said earlier, this functionality has worked as expected until 6.9.9, but has started failing with latest 6.9.10. I will be happy to help debug/provide more info if necessary.

Greatly appreciate your inputs.

-Nagesh


On Wednesday, January 13, 2016 at 4:38:04 PM UTC-8, Nagesh wrote:
I updated the version in my project from 6.9.9 to 6.9.10 yesterday and started noticing failures in our CI runs. The issue is that the testng-failed.xml file that gets generated at the end of the first run, still contains the same tests as the original suite file. The expectation is that it contains only the failed test cases. Also if all the tests passed in the first run with the original suite file, a testng-failed.xml file still gets generated. Prior to 6.9.10 the failed xml file would not get generated if all tests in the run passed.

I have now reverted back to 6.9.9. Has this been discussed previously? I looked here and in GitHub but didn't find a thread regarding this issue.

Here is the original suite file -

<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="Turbo WebDriver Tests" parallel="classes" thread-count="5">
<listeners>
<listener class-name="com.axiom.turbo.webdriver.TestListener" />
<listener class-name="com.axiom.turbo.webdriver.TestAnnotationTransformer" />
</listeners>
        <test name = "Classes with a single test method">
<groups>
<run>
<exclude name="KnownIssues" />
<exclude name="CatalogsCache" />
<exclude name="SlowRunning" />
<exclude name="Defunct" />
<exclude name="FeatureParity" />
</run>
</groups>
<classes>
<class name = "com.axiom.turbo.webdriver.tests.privacypolicy.PrivacyPolicyPageWebDriverTests" />
                        <class name = "com.axiom.turbo.webdriver.tests.labstests.LabsPageWebDriverTests" />
<class name = "com.axiom.turbo.webdriver.tests.SuccessTestClass" />
<class name = "com.axiom.turbo.webdriver.tests.SkipTestClass" />
<class name = "com.axiom.turbo.webdriver.tests.FailTestClass" />
</classes>
</test>
</suite>


Here is the testng-failed.xml file that gets generated if all or some tests passed with all the tests as the original suite file -

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite guice-stage="DEVELOPMENT" verbose="0" name="Failed suite [Turbo WebDriver Tests]" parallel="classes">
  <listeners>
    <listener class-name="com.axiom.turbo.webdriver.TestListener"/>
    <listener class-name="com.axiom.turbo.webdriver.TestAnnotationTransformer"/>
  </listeners>
  <test name="Classes with a single test method">
    <groups>
      <run>
        <exclude name="KnownIssues"/>
        <exclude name="CatalogsCache"/>
        <exclude name="SlowRunning"/>
        <exclude name="Defunct"/>
        <exclude name="FeatureParity"/>
      </run>
    </groups>
    <classes>
      <class name="com.axiom.turbo.webdriver.tests.privacypolicy.PrivacyPolicyPageWebDriverTests"/>
      <class name="com.axiom.turbo.webdriver.tests.SuccessTestClass"/>
      <class name="com.axiom.turbo.webdriver.tests.SkipTestClass"/>
      <class name="com.axiom.turbo.webdriver.tests.FailTestClass"/>
    </classes>
  </test> <!-- Classes with a single test method -->
</suite> <!-- Failed suite [Turbo WebDriver Tests] -->


Thanks!

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Issue with testng-failed.xml file with 6.9.10

Krishnan Mahadevan
Sure Cedric. I will take a look at this and get back. It was a long weekend and trying to get over Monday morning blues :)

Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @ http://wakened-cognition.blogspot.com/
My Technical Scribbings @ http://rationaleemotions.wordpress.com/

On Sun, Jan 17, 2016 at 1:29 PM, Cédric Beust ♔ <[hidden email]> wrote:
Krishnan, could you take a look at this?


-- 
Cédric


On Sat, Jan 16, 2016 at 11:45 PM, Nagesh <[hidden email]> wrote:
I took out the custom listeners from the suite file to see if that helps narrow down the problem, but I am still seeing an incorrect testng-failed.xml file getting generated at the end of the run. I took a look at the upcoming changes and I don't know if it's being addressed with the next version.

Can someone try it out and shed some light on this issue? As I said earlier, this functionality has worked as expected until 6.9.9, but has started failing with latest 6.9.10. I will be happy to help debug/provide more info if necessary.

Greatly appreciate your inputs.

-Nagesh


On Wednesday, January 13, 2016 at 4:38:04 PM UTC-8, Nagesh wrote:
I updated the version in my project from 6.9.9 to 6.9.10 yesterday and started noticing failures in our CI runs. The issue is that the testng-failed.xml file that gets generated at the end of the first run, still contains the same tests as the original suite file. The expectation is that it contains only the failed test cases. Also if all the tests passed in the first run with the original suite file, a testng-failed.xml file still gets generated. Prior to 6.9.10 the failed xml file would not get generated if all tests in the run passed.

I have now reverted back to 6.9.9. Has this been discussed previously? I looked here and in GitHub but didn't find a thread regarding this issue.

Here is the original suite file -

<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="Turbo WebDriver Tests" parallel="classes" thread-count="5">
<listeners>
<listener class-name="com.axiom.turbo.webdriver.TestListener" />
<listener class-name="com.axiom.turbo.webdriver.TestAnnotationTransformer" />
</listeners>
        <test name = "Classes with a single test method">
<groups>
<run>
<exclude name="KnownIssues" />
<exclude name="CatalogsCache" />
<exclude name="SlowRunning" />
<exclude name="Defunct" />
<exclude name="FeatureParity" />
</run>
</groups>
<classes>
<class name = "com.axiom.turbo.webdriver.tests.privacypolicy.PrivacyPolicyPageWebDriverTests" />
                        <class name = "com.axiom.turbo.webdriver.tests.labstests.LabsPageWebDriverTests" />
<class name = "com.axiom.turbo.webdriver.tests.SuccessTestClass" />
<class name = "com.axiom.turbo.webdriver.tests.SkipTestClass" />
<class name = "com.axiom.turbo.webdriver.tests.FailTestClass" />
</classes>
</test>
</suite>


Here is the testng-failed.xml file that gets generated if all or some tests passed with all the tests as the original suite file -

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite guice-stage="DEVELOPMENT" verbose="0" name="Failed suite [Turbo WebDriver Tests]" parallel="classes">
  <listeners>
    <listener class-name="com.axiom.turbo.webdriver.TestListener"/>
    <listener class-name="com.axiom.turbo.webdriver.TestAnnotationTransformer"/>
  </listeners>
  <test name="Classes with a single test method">
    <groups>
      <run>
        <exclude name="KnownIssues"/>
        <exclude name="CatalogsCache"/>
        <exclude name="SlowRunning"/>
        <exclude name="Defunct"/>
        <exclude name="FeatureParity"/>
      </run>
    </groups>
    <classes>
      <class name="com.axiom.turbo.webdriver.tests.privacypolicy.PrivacyPolicyPageWebDriverTests"/>
      <class name="com.axiom.turbo.webdriver.tests.SuccessTestClass"/>
      <class name="com.axiom.turbo.webdriver.tests.SkipTestClass"/>
      <class name="com.axiom.turbo.webdriver.tests.FailTestClass"/>
    </classes>
  </test> <!-- Classes with a single test method -->
</suite> <!-- Failed suite [Turbo WebDriver Tests] -->


Thanks!

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.


--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Issue with testng-failed.xml file with 6.9.10

Krishnan Mahadevan
Cedric,

I figured out the root cause of the problem.

The issue is occurring in 6.9.10 because of this change : Fixed: GITHUB-866 / GITHUB-869 : Some attributes were not cloned when XmlSuite#clone was used (Virender Singh)

This pull request : https://github.com/cbeust/testng/pull/869 basically ended up introducing a deep copy on the XmlSuite object.

Because of the deep copy, when we do a 

we end up copying all of the XmlTest objects as well and thus 


    if(null != failedSuite.getTests() && failedSuite.getTests().size() > 0) {
      Utils.writeUtf8File(outputDir, TESTNG_FAILED_XML, failedSuite.toXml());
      Utils.writeUtf8File(suite.getOutputDirectory(), TESTNG_FAILED_XML, failedSuite.toXml());
    }

is now always evaluating to true and thus causing the problem.

The fix for this issue IMO would be basically get rid of this line : https://github.com/cbeust/testng/blob/master/src/main/java/org/testng/xml/XmlSuite.java#L658

result.setTests(getTests());

I am yet to test out this fix because at work, I cannot work on public repositories codebases [ internal restriction ].

I will investigate a bit more to see if that fixes the problem and if it does and if you are fine with the suggested fix, I will raise a pull request as well.

Please let me know.


Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @ http://wakened-cognition.blogspot.com/
My Technical Scribbings @ http://rationaleemotions.wordpress.com/

On Mon, Jan 18, 2016 at 10:35 AM, Krishnan Mahadevan <[hidden email]> wrote:
Sure Cedric. I will take a look at this and get back. It was a long weekend and trying to get over Monday morning blues :)

Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @ http://wakened-cognition.blogspot.com/
My Technical Scribbings @ http://rationaleemotions.wordpress.com/

On Sun, Jan 17, 2016 at 1:29 PM, Cédric Beust ♔ <[hidden email]> wrote:
Krishnan, could you take a look at this?


-- 
Cédric


On Sat, Jan 16, 2016 at 11:45 PM, Nagesh <[hidden email]> wrote:
I took out the custom listeners from the suite file to see if that helps narrow down the problem, but I am still seeing an incorrect testng-failed.xml file getting generated at the end of the run. I took a look at the upcoming changes and I don't know if it's being addressed with the next version.

Can someone try it out and shed some light on this issue? As I said earlier, this functionality has worked as expected until 6.9.9, but has started failing with latest 6.9.10. I will be happy to help debug/provide more info if necessary.

Greatly appreciate your inputs.

-Nagesh


On Wednesday, January 13, 2016 at 4:38:04 PM UTC-8, Nagesh wrote:
I updated the version in my project from 6.9.9 to 6.9.10 yesterday and started noticing failures in our CI runs. The issue is that the testng-failed.xml file that gets generated at the end of the first run, still contains the same tests as the original suite file. The expectation is that it contains only the failed test cases. Also if all the tests passed in the first run with the original suite file, a testng-failed.xml file still gets generated. Prior to 6.9.10 the failed xml file would not get generated if all tests in the run passed.

I have now reverted back to 6.9.9. Has this been discussed previously? I looked here and in GitHub but didn't find a thread regarding this issue.

Here is the original suite file -

<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="Turbo WebDriver Tests" parallel="classes" thread-count="5">
<listeners>
<listener class-name="com.axiom.turbo.webdriver.TestListener" />
<listener class-name="com.axiom.turbo.webdriver.TestAnnotationTransformer" />
</listeners>
        <test name = "Classes with a single test method">
<groups>
<run>
<exclude name="KnownIssues" />
<exclude name="CatalogsCache" />
<exclude name="SlowRunning" />
<exclude name="Defunct" />
<exclude name="FeatureParity" />
</run>
</groups>
<classes>
<class name = "com.axiom.turbo.webdriver.tests.privacypolicy.PrivacyPolicyPageWebDriverTests" />
                        <class name = "com.axiom.turbo.webdriver.tests.labstests.LabsPageWebDriverTests" />
<class name = "com.axiom.turbo.webdriver.tests.SuccessTestClass" />
<class name = "com.axiom.turbo.webdriver.tests.SkipTestClass" />
<class name = "com.axiom.turbo.webdriver.tests.FailTestClass" />
</classes>
</test>
</suite>


Here is the testng-failed.xml file that gets generated if all or some tests passed with all the tests as the original suite file -

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite guice-stage="DEVELOPMENT" verbose="0" name="Failed suite [Turbo WebDriver Tests]" parallel="classes">
  <listeners>
    <listener class-name="com.axiom.turbo.webdriver.TestListener"/>
    <listener class-name="com.axiom.turbo.webdriver.TestAnnotationTransformer"/>
  </listeners>
  <test name="Classes with a single test method">
    <groups>
      <run>
        <exclude name="KnownIssues"/>
        <exclude name="CatalogsCache"/>
        <exclude name="SlowRunning"/>
        <exclude name="Defunct"/>
        <exclude name="FeatureParity"/>
      </run>
    </groups>
    <classes>
      <class name="com.axiom.turbo.webdriver.tests.privacypolicy.PrivacyPolicyPageWebDriverTests"/>
      <class name="com.axiom.turbo.webdriver.tests.SuccessTestClass"/>
      <class name="com.axiom.turbo.webdriver.tests.SkipTestClass"/>
      <class name="com.axiom.turbo.webdriver.tests.FailTestClass"/>
    </classes>
  </test> <!-- Classes with a single test method -->
</suite> <!-- Failed suite [Turbo WebDriver Tests] -->


Thanks!

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.



--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Issue with testng-failed.xml file with 6.9.10

Cédric Beust ♔-2
That's awesome, thanks Krishnan! A pull request would be great.

-- 
Cédric


On Mon, Jan 18, 2016 at 12:33 AM, Krishnan Mahadevan <[hidden email]> wrote:
Cedric,

I figured out the root cause of the problem.

The issue is occurring in 6.9.10 because of this change : Fixed: GITHUB-866 / GITHUB-869 : Some attributes were not cloned when XmlSuite#clone was used (Virender Singh)

This pull request : https://github.com/cbeust/testng/pull/869 basically ended up introducing a deep copy on the XmlSuite object.

Because of the deep copy, when we do a 

we end up copying all of the XmlTest objects as well and thus 


    if(null != failedSuite.getTests() && failedSuite.getTests().size() > 0) {
      Utils.writeUtf8File(outputDir, TESTNG_FAILED_XML, failedSuite.toXml());
      Utils.writeUtf8File(suite.getOutputDirectory(), TESTNG_FAILED_XML, failedSuite.toXml());
    }

is now always evaluating to true and thus causing the problem.

The fix for this issue IMO would be basically get rid of this line : https://github.com/cbeust/testng/blob/master/src/main/java/org/testng/xml/XmlSuite.java#L658

result.setTests(getTests());

I am yet to test out this fix because at work, I cannot work on public repositories codebases [ internal restriction ].

I will investigate a bit more to see if that fixes the problem and if it does and if you are fine with the suggested fix, I will raise a pull request as well.

Please let me know.


Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @ http://wakened-cognition.blogspot.com/
My Technical Scribbings @ http://rationaleemotions.wordpress.com/

On Mon, Jan 18, 2016 at 10:35 AM, Krishnan Mahadevan <[hidden email]> wrote:
Sure Cedric. I will take a look at this and get back. It was a long weekend and trying to get over Monday morning blues :)

Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @ http://wakened-cognition.blogspot.com/
My Technical Scribbings @ http://rationaleemotions.wordpress.com/

On Sun, Jan 17, 2016 at 1:29 PM, Cédric Beust ♔ <[hidden email]> wrote:
Krishnan, could you take a look at this?


-- 
Cédric


On Sat, Jan 16, 2016 at 11:45 PM, Nagesh <[hidden email]> wrote:
I took out the custom listeners from the suite file to see if that helps narrow down the problem, but I am still seeing an incorrect testng-failed.xml file getting generated at the end of the run. I took a look at the upcoming changes and I don't know if it's being addressed with the next version.

Can someone try it out and shed some light on this issue? As I said earlier, this functionality has worked as expected until 6.9.9, but has started failing with latest 6.9.10. I will be happy to help debug/provide more info if necessary.

Greatly appreciate your inputs.

-Nagesh


On Wednesday, January 13, 2016 at 4:38:04 PM UTC-8, Nagesh wrote:
I updated the version in my project from 6.9.9 to 6.9.10 yesterday and started noticing failures in our CI runs. The issue is that the testng-failed.xml file that gets generated at the end of the first run, still contains the same tests as the original suite file. The expectation is that it contains only the failed test cases. Also if all the tests passed in the first run with the original suite file, a testng-failed.xml file still gets generated. Prior to 6.9.10 the failed xml file would not get generated if all tests in the run passed.

I have now reverted back to 6.9.9. Has this been discussed previously? I looked here and in GitHub but didn't find a thread regarding this issue.

Here is the original suite file -

<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="Turbo WebDriver Tests" parallel="classes" thread-count="5">
<listeners>
<listener class-name="com.axiom.turbo.webdriver.TestListener" />
<listener class-name="com.axiom.turbo.webdriver.TestAnnotationTransformer" />
</listeners>
        <test name = "Classes with a single test method">
<groups>
<run>
<exclude name="KnownIssues" />
<exclude name="CatalogsCache" />
<exclude name="SlowRunning" />
<exclude name="Defunct" />
<exclude name="FeatureParity" />
</run>
</groups>
<classes>
<class name = "com.axiom.turbo.webdriver.tests.privacypolicy.PrivacyPolicyPageWebDriverTests" />
                        <class name = "com.axiom.turbo.webdriver.tests.labstests.LabsPageWebDriverTests" />
<class name = "com.axiom.turbo.webdriver.tests.SuccessTestClass" />
<class name = "com.axiom.turbo.webdriver.tests.SkipTestClass" />
<class name = "com.axiom.turbo.webdriver.tests.FailTestClass" />
</classes>
</test>
</suite>


Here is the testng-failed.xml file that gets generated if all or some tests passed with all the tests as the original suite file -

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite guice-stage="DEVELOPMENT" verbose="0" name="Failed suite [Turbo WebDriver Tests]" parallel="classes">
  <listeners>
    <listener class-name="com.axiom.turbo.webdriver.TestListener"/>
    <listener class-name="com.axiom.turbo.webdriver.TestAnnotationTransformer"/>
  </listeners>
  <test name="Classes with a single test method">
    <groups>
      <run>
        <exclude name="KnownIssues"/>
        <exclude name="CatalogsCache"/>
        <exclude name="SlowRunning"/>
        <exclude name="Defunct"/>
        <exclude name="FeatureParity"/>
      </run>
    </groups>
    <classes>
      <class name="com.axiom.turbo.webdriver.tests.privacypolicy.PrivacyPolicyPageWebDriverTests"/>
      <class name="com.axiom.turbo.webdriver.tests.SuccessTestClass"/>
      <class name="com.axiom.turbo.webdriver.tests.SkipTestClass"/>
      <class name="com.axiom.turbo.webdriver.tests.FailTestClass"/>
    </classes>
  </test> <!-- Classes with a single test method -->
</suite> <!-- Failed suite [Turbo WebDriver Tests] -->


Thanks!

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.




--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Issue with testng-failed.xml file with 6.9.10

Krishnan Mahadevan
Cedric,

I raised a github issue to track this problem : https://github.com/cbeust/testng/issues/960

The corresponding pull request that fixes this problem : https://github.com/cbeust/testng/pull/961

Just to ensure that this problem doesn't surface again, I added up a couple of unit tests which check for the presence/absence of the testng-failed.xml file based on whether there were any failures or not.

Please help review the PR.


Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @ http://wakened-cognition.blogspot.com/
My Technical Scribbings @ http://rationaleemotions.wordpress.com/

On Mon, Jan 18, 2016 at 9:46 PM, Cédric Beust ♔ <[hidden email]> wrote:
That's awesome, thanks Krishnan! A pull request would be great.

-- 
Cédric


On Mon, Jan 18, 2016 at 12:33 AM, Krishnan Mahadevan <[hidden email]> wrote:
Cedric,

I figured out the root cause of the problem.

The issue is occurring in 6.9.10 because of this change : Fixed: GITHUB-866 / GITHUB-869 : Some attributes were not cloned when XmlSuite#clone was used (Virender Singh)

This pull request : https://github.com/cbeust/testng/pull/869 basically ended up introducing a deep copy on the XmlSuite object.

Because of the deep copy, when we do a 

we end up copying all of the XmlTest objects as well and thus 


    if(null != failedSuite.getTests() && failedSuite.getTests().size() > 0) {
      Utils.writeUtf8File(outputDir, TESTNG_FAILED_XML, failedSuite.toXml());
      Utils.writeUtf8File(suite.getOutputDirectory(), TESTNG_FAILED_XML, failedSuite.toXml());
    }

is now always evaluating to true and thus causing the problem.

The fix for this issue IMO would be basically get rid of this line : https://github.com/cbeust/testng/blob/master/src/main/java/org/testng/xml/XmlSuite.java#L658

result.setTests(getTests());

I am yet to test out this fix because at work, I cannot work on public repositories codebases [ internal restriction ].

I will investigate a bit more to see if that fixes the problem and if it does and if you are fine with the suggested fix, I will raise a pull request as well.

Please let me know.


Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @ http://wakened-cognition.blogspot.com/
My Technical Scribbings @ http://rationaleemotions.wordpress.com/

On Mon, Jan 18, 2016 at 10:35 AM, Krishnan Mahadevan <[hidden email]> wrote:
Sure Cedric. I will take a look at this and get back. It was a long weekend and trying to get over Monday morning blues :)

Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @ http://wakened-cognition.blogspot.com/
My Technical Scribbings @ http://rationaleemotions.wordpress.com/

On Sun, Jan 17, 2016 at 1:29 PM, Cédric Beust ♔ <[hidden email]> wrote:
Krishnan, could you take a look at this?


-- 
Cédric


On Sat, Jan 16, 2016 at 11:45 PM, Nagesh <[hidden email]> wrote:
I took out the custom listeners from the suite file to see if that helps narrow down the problem, but I am still seeing an incorrect testng-failed.xml file getting generated at the end of the run. I took a look at the upcoming changes and I don't know if it's being addressed with the next version.

Can someone try it out and shed some light on this issue? As I said earlier, this functionality has worked as expected until 6.9.9, but has started failing with latest 6.9.10. I will be happy to help debug/provide more info if necessary.

Greatly appreciate your inputs.

-Nagesh


On Wednesday, January 13, 2016 at 4:38:04 PM UTC-8, Nagesh wrote:
I updated the version in my project from 6.9.9 to 6.9.10 yesterday and started noticing failures in our CI runs. The issue is that the testng-failed.xml file that gets generated at the end of the first run, still contains the same tests as the original suite file. The expectation is that it contains only the failed test cases. Also if all the tests passed in the first run with the original suite file, a testng-failed.xml file still gets generated. Prior to 6.9.10 the failed xml file would not get generated if all tests in the run passed.

I have now reverted back to 6.9.9. Has this been discussed previously? I looked here and in GitHub but didn't find a thread regarding this issue.

Here is the original suite file -

<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="Turbo WebDriver Tests" parallel="classes" thread-count="5">
<listeners>
<listener class-name="com.axiom.turbo.webdriver.TestListener" />
<listener class-name="com.axiom.turbo.webdriver.TestAnnotationTransformer" />
</listeners>
        <test name = "Classes with a single test method">
<groups>
<run>
<exclude name="KnownIssues" />
<exclude name="CatalogsCache" />
<exclude name="SlowRunning" />
<exclude name="Defunct" />
<exclude name="FeatureParity" />
</run>
</groups>
<classes>
<class name = "com.axiom.turbo.webdriver.tests.privacypolicy.PrivacyPolicyPageWebDriverTests" />
                        <class name = "com.axiom.turbo.webdriver.tests.labstests.LabsPageWebDriverTests" />
<class name = "com.axiom.turbo.webdriver.tests.SuccessTestClass" />
<class name = "com.axiom.turbo.webdriver.tests.SkipTestClass" />
<class name = "com.axiom.turbo.webdriver.tests.FailTestClass" />
</classes>
</test>
</suite>


Here is the testng-failed.xml file that gets generated if all or some tests passed with all the tests as the original suite file -

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite guice-stage="DEVELOPMENT" verbose="0" name="Failed suite [Turbo WebDriver Tests]" parallel="classes">
  <listeners>
    <listener class-name="com.axiom.turbo.webdriver.TestListener"/>
    <listener class-name="com.axiom.turbo.webdriver.TestAnnotationTransformer"/>
  </listeners>
  <test name="Classes with a single test method">
    <groups>
      <run>
        <exclude name="KnownIssues"/>
        <exclude name="CatalogsCache"/>
        <exclude name="SlowRunning"/>
        <exclude name="Defunct"/>
        <exclude name="FeatureParity"/>
      </run>
    </groups>
    <classes>
      <class name="com.axiom.turbo.webdriver.tests.privacypolicy.PrivacyPolicyPageWebDriverTests"/>
      <class name="com.axiom.turbo.webdriver.tests.SuccessTestClass"/>
      <class name="com.axiom.turbo.webdriver.tests.SkipTestClass"/>
      <class name="com.axiom.turbo.webdriver.tests.FailTestClass"/>
    </classes>
  </test> <!-- Classes with a single test method -->
</suite> <!-- Failed suite [Turbo WebDriver Tests] -->


Thanks!

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.





--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Issue with testng-failed.xml file with 6.9.10

Krishnan Mahadevan
Nagesh,

FYI, The fix is now merged. You can perhaps try out using the SNAPSHOT version of TestNG.

Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @ http://wakened-cognition.blogspot.com/
My Technical Scribbings @ http://rationaleemotions.wordpress.com/

On Tue, Jan 19, 2016 at 8:34 AM, Krishnan Mahadevan <[hidden email]> wrote:
Cedric,

I raised a github issue to track this problem : https://github.com/cbeust/testng/issues/960

The corresponding pull request that fixes this problem : https://github.com/cbeust/testng/pull/961

Just to ensure that this problem doesn't surface again, I added up a couple of unit tests which check for the presence/absence of the testng-failed.xml file based on whether there were any failures or not.

Please help review the PR.


Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @ http://wakened-cognition.blogspot.com/
My Technical Scribbings @ http://rationaleemotions.wordpress.com/

On Mon, Jan 18, 2016 at 9:46 PM, Cédric Beust ♔ <[hidden email]> wrote:
That's awesome, thanks Krishnan! A pull request would be great.

-- 
Cédric


On Mon, Jan 18, 2016 at 12:33 AM, Krishnan Mahadevan <[hidden email]> wrote:
Cedric,

I figured out the root cause of the problem.

The issue is occurring in 6.9.10 because of this change : Fixed: GITHUB-866 / GITHUB-869 : Some attributes were not cloned when XmlSuite#clone was used (Virender Singh)

This pull request : https://github.com/cbeust/testng/pull/869 basically ended up introducing a deep copy on the XmlSuite object.

Because of the deep copy, when we do a 

we end up copying all of the XmlTest objects as well and thus 


    if(null != failedSuite.getTests() && failedSuite.getTests().size() > 0) {
      Utils.writeUtf8File(outputDir, TESTNG_FAILED_XML, failedSuite.toXml());
      Utils.writeUtf8File(suite.getOutputDirectory(), TESTNG_FAILED_XML, failedSuite.toXml());
    }

is now always evaluating to true and thus causing the problem.

The fix for this issue IMO would be basically get rid of this line : https://github.com/cbeust/testng/blob/master/src/main/java/org/testng/xml/XmlSuite.java#L658

result.setTests(getTests());

I am yet to test out this fix because at work, I cannot work on public repositories codebases [ internal restriction ].

I will investigate a bit more to see if that fixes the problem and if it does and if you are fine with the suggested fix, I will raise a pull request as well.

Please let me know.


Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @ http://wakened-cognition.blogspot.com/
My Technical Scribbings @ http://rationaleemotions.wordpress.com/

On Mon, Jan 18, 2016 at 10:35 AM, Krishnan Mahadevan <[hidden email]> wrote:
Sure Cedric. I will take a look at this and get back. It was a long weekend and trying to get over Monday morning blues :)

Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @ http://wakened-cognition.blogspot.com/
My Technical Scribbings @ http://rationaleemotions.wordpress.com/

On Sun, Jan 17, 2016 at 1:29 PM, Cédric Beust ♔ <[hidden email]> wrote:
Krishnan, could you take a look at this?


-- 
Cédric


On Sat, Jan 16, 2016 at 11:45 PM, Nagesh <[hidden email]> wrote:
I took out the custom listeners from the suite file to see if that helps narrow down the problem, but I am still seeing an incorrect testng-failed.xml file getting generated at the end of the run. I took a look at the upcoming changes and I don't know if it's being addressed with the next version.

Can someone try it out and shed some light on this issue? As I said earlier, this functionality has worked as expected until 6.9.9, but has started failing with latest 6.9.10. I will be happy to help debug/provide more info if necessary.

Greatly appreciate your inputs.

-Nagesh


On Wednesday, January 13, 2016 at 4:38:04 PM UTC-8, Nagesh wrote:
I updated the version in my project from 6.9.9 to 6.9.10 yesterday and started noticing failures in our CI runs. The issue is that the testng-failed.xml file that gets generated at the end of the first run, still contains the same tests as the original suite file. The expectation is that it contains only the failed test cases. Also if all the tests passed in the first run with the original suite file, a testng-failed.xml file still gets generated. Prior to 6.9.10 the failed xml file would not get generated if all tests in the run passed.

I have now reverted back to 6.9.9. Has this been discussed previously? I looked here and in GitHub but didn't find a thread regarding this issue.

Here is the original suite file -

<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="Turbo WebDriver Tests" parallel="classes" thread-count="5">
<listeners>
<listener class-name="com.axiom.turbo.webdriver.TestListener" />
<listener class-name="com.axiom.turbo.webdriver.TestAnnotationTransformer" />
</listeners>
        <test name = "Classes with a single test method">
<groups>
<run>
<exclude name="KnownIssues" />
<exclude name="CatalogsCache" />
<exclude name="SlowRunning" />
<exclude name="Defunct" />
<exclude name="FeatureParity" />
</run>
</groups>
<classes>
<class name = "com.axiom.turbo.webdriver.tests.privacypolicy.PrivacyPolicyPageWebDriverTests" />
                        <class name = "com.axiom.turbo.webdriver.tests.labstests.LabsPageWebDriverTests" />
<class name = "com.axiom.turbo.webdriver.tests.SuccessTestClass" />
<class name = "com.axiom.turbo.webdriver.tests.SkipTestClass" />
<class name = "com.axiom.turbo.webdriver.tests.FailTestClass" />
</classes>
</test>
</suite>


Here is the testng-failed.xml file that gets generated if all or some tests passed with all the tests as the original suite file -

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite guice-stage="DEVELOPMENT" verbose="0" name="Failed suite [Turbo WebDriver Tests]" parallel="classes">
  <listeners>
    <listener class-name="com.axiom.turbo.webdriver.TestListener"/>
    <listener class-name="com.axiom.turbo.webdriver.TestAnnotationTransformer"/>
  </listeners>
  <test name="Classes with a single test method">
    <groups>
      <run>
        <exclude name="KnownIssues"/>
        <exclude name="CatalogsCache"/>
        <exclude name="SlowRunning"/>
        <exclude name="Defunct"/>
        <exclude name="FeatureParity"/>
      </run>
    </groups>
    <classes>
      <class name="com.axiom.turbo.webdriver.tests.privacypolicy.PrivacyPolicyPageWebDriverTests"/>
      <class name="com.axiom.turbo.webdriver.tests.SuccessTestClass"/>
      <class name="com.axiom.turbo.webdriver.tests.SkipTestClass"/>
      <class name="com.axiom.turbo.webdriver.tests.FailTestClass"/>
    </classes>
  </test> <!-- Classes with a single test method -->
</suite> <!-- Failed suite [Turbo WebDriver Tests] -->


Thanks!

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.






--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Issue with testng-failed.xml file with 6.9.10

Nagesh
Thanks a lot Krishnan! Verified your fix with the snapshot version. Looks good to me now.

-Nagesh

On Tuesday, January 19, 2016 at 9:32:21 PM UTC-8, Krishnan wrote:
Nagesh,

FYI, The fix is now merged. You can perhaps try out using the SNAPSHOT version of TestNG.

Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @ <a href="http://wakened-cognition.blogspot.com/" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\75http%3A%2F%2Fwakened-cognition.blogspot.com%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNHzOdYJCASIDF_28vQtkp9gnKAYSQ&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\75http%3A%2F%2Fwakened-cognition.blogspot.com%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNHzOdYJCASIDF_28vQtkp9gnKAYSQ&#39;;return true;">http://wakened-cognition.blogspot.com/
My Technical Scribbings @ <a href="http://rationaleemotions.wordpress.com/" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\75http%3A%2F%2Frationaleemotions.wordpress.com%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNEcBOKyRn0lpL8LDbBuyAlKdwgXwQ&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\75http%3A%2F%2Frationaleemotions.wordpress.com%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNEcBOKyRn0lpL8LDbBuyAlKdwgXwQ&#39;;return true;">http://rationaleemotions.wordpress.com/

On Tue, Jan 19, 2016 at 8:34 AM, Krishnan Mahadevan <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="GR7_jU6ZDgAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">krishnan.ma...@gmail.com> wrote:
Cedric,

I raised a github issue to track this problem : <a href="https://github.com/cbeust/testng/issues/960" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\75https%3A%2F%2Fgithub.com%2Fcbeust%2Ftestng%2Fissues%2F960\46sa\75D\46sntz\0751\46usg\75AFQjCNFgK3WcPC2-xOzsA9-wg-OHQJlE-g&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\75https%3A%2F%2Fgithub.com%2Fcbeust%2Ftestng%2Fissues%2F960\46sa\75D\46sntz\0751\46usg\75AFQjCNFgK3WcPC2-xOzsA9-wg-OHQJlE-g&#39;;return true;">https://github.com/cbeust/testng/issues/960

The corresponding pull request that fixes this problem : <a href="https://github.com/cbeust/testng/pull/961" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\75https%3A%2F%2Fgithub.com%2Fcbeust%2Ftestng%2Fpull%2F961\46sa\75D\46sntz\0751\46usg\75AFQjCNGZugiYAzenbuVP-1sPE-45xp1BHA&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\75https%3A%2F%2Fgithub.com%2Fcbeust%2Ftestng%2Fpull%2F961\46sa\75D\46sntz\0751\46usg\75AFQjCNGZugiYAzenbuVP-1sPE-45xp1BHA&#39;;return true;">https://github.com/cbeust/testng/pull/961

Just to ensure that this problem doesn't surface again, I added up a couple of unit tests which check for the presence/absence of the testng-failed.xml file based on whether there were any failures or not.

Please help review the PR.


Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @ <a href="http://wakened-cognition.blogspot.com/" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\75http%3A%2F%2Fwakened-cognition.blogspot.com%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNHzOdYJCASIDF_28vQtkp9gnKAYSQ&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\75http%3A%2F%2Fwakened-cognition.blogspot.com%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNHzOdYJCASIDF_28vQtkp9gnKAYSQ&#39;;return true;">http://wakened-cognition.blogspot.com/
My Technical Scribbings @ <a href="http://rationaleemotions.wordpress.com/" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\75http%3A%2F%2Frationaleemotions.wordpress.com%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNEcBOKyRn0lpL8LDbBuyAlKdwgXwQ&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\75http%3A%2F%2Frationaleemotions.wordpress.com%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNEcBOKyRn0lpL8LDbBuyAlKdwgXwQ&#39;;return true;">http://rationaleemotions.wordpress.com/

On Mon, Jan 18, 2016 at 9:46 PM, Cédric Beust ♔ <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="GR7_jU6ZDgAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">ced...@...> wrote:
That's awesome, thanks Krishnan! A pull request would be great.

-- 
Cédric


On Mon, Jan 18, 2016 at 12:33 AM, Krishnan Mahadevan <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="GR7_jU6ZDgAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">krishnan.ma...@gmail.com> wrote:
Cedric,

I figured out the root cause of the problem.

The issue is occurring in 6.9.10 because of this change : Fixed: GITHUB-866 / GITHUB-869 : Some attributes were not cloned when XmlSuite#clone was used (Virender Singh)

This pull request : <a href="https://github.com/cbeust/testng/pull/869" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\75https%3A%2F%2Fgithub.com%2Fcbeust%2Ftestng%2Fpull%2F869\46sa\75D\46sntz\0751\46usg\75AFQjCNHRle2TdcuHrG1dVvZY0CLMn0nYGg&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\75https%3A%2F%2Fgithub.com%2Fcbeust%2Ftestng%2Fpull%2F869\46sa\75D\46sntz\0751\46usg\75AFQjCNHRle2TdcuHrG1dVvZY0CLMn0nYGg&#39;;return true;">https://github.com/cbeust/testng/pull/869 basically ended up introducing a deep copy on the XmlSuite object.

Because of the deep copy, when we do a 
XmlSuite failedSuite = (XmlSuite) xmlSuite.clone(); [ <a href="https://github.com/cbeust/testng/blob/6.9.9/src/main/java/org/testng/reporters/FailedReporter.java#L52" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\75https%3A%2F%2Fgithub.com%2Fcbeust%2Ftestng%2Fblob%2F6.9.9%2Fsrc%2Fmain%2Fjava%2Forg%2Ftestng%2Freporters%2FFailedReporter.java%23L52\46sa\75D\46sntz\0751\46usg\75AFQjCNFhRLAqkICPBnKk1ks2dGd_ReoLQA&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\75https%3A%2F%2Fgithub.com%2Fcbeust%2Ftestng%2Fblob%2F6.9.9%2Fsrc%2Fmain%2Fjava%2Forg%2Ftestng%2Freporters%2FFailedReporter.java%23L52\46sa\75D\46sntz\0751\46usg\75AFQjCNFhRLAqkICPBnKk1ks2dGd_ReoLQA&#39;;return true;">https://github.com/cbeust/testng/blob/6.9.9/src/main/java/org/testng/reporters/FailedReporter.java#L52 ]

we end up copying all of the XmlTest objects as well and thus 

the below condition [ <a href="https://github.com/cbeust/testng/blob/6.9.9/src/main/java/org/testng/reporters/FailedReporter.java#L52" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\75https%3A%2F%2Fgithub.com%2Fcbeust%2Ftestng%2Fblob%2F6.9.9%2Fsrc%2Fmain%2Fjava%2Forg%2Ftestng%2Freporters%2FFailedReporter.java%23L52\46sa\75D\46sntz\0751\46usg\75AFQjCNFhRLAqkICPBnKk1ks2dGd_ReoLQA&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\75https%3A%2F%2Fgithub.com%2Fcbeust%2Ftestng%2Fblob%2F6.9.9%2Fsrc%2Fmain%2Fjava%2Forg%2Ftestng%2Freporters%2FFailedReporter.java%23L52\46sa\75D\46sntz\0751\46usg\75AFQjCNFhRLAqkICPBnKk1ks2dGd_ReoLQA&#39;;return true;">https://github.com/cbeust/testng/blob/6.9.9/src/main/java/org/testng/reporters/FailedReporter.java#L52 ]

    if(null != failedSuite.getTests() && failedSuite.getTests().size() > 0) {
      Utils.writeUtf8File(outputDir, TESTNG_FAILED_XML, failedSuite.toXml());
      Utils.writeUtf8File(suite.getOutputDirectory(), TESTNG_FAILED_XML, failedSuite.toXml());
    }

is now always evaluating to true and thus causing the problem.

The fix for this issue IMO would be basically get rid of this line : <a href="https://github.com/cbeust/testng/blob/master/src/main/java/org/testng/xml/XmlSuite.java#L658" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\75https%3A%2F%2Fgithub.com%2Fcbeust%2Ftestng%2Fblob%2Fmaster%2Fsrc%2Fmain%2Fjava%2Forg%2Ftestng%2Fxml%2FXmlSuite.java%23L658\46sa\75D\46sntz\0751\46usg\75AFQjCNGJNWcpuaX27Igku3E44t_DjXylNQ&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\75https%3A%2F%2Fgithub.com%2Fcbeust%2Ftestng%2Fblob%2Fmaster%2Fsrc%2Fmain%2Fjava%2Forg%2Ftestng%2Fxml%2FXmlSuite.java%23L658\46sa\75D\46sntz\0751\46usg\75AFQjCNGJNWcpuaX27Igku3E44t_DjXylNQ&#39;;return true;">https://github.com/cbeust/testng/blob/master/src/main/java/org/testng/xml/XmlSuite.java#L658

result.setTests(getTests());

I am yet to test out this fix because at work, I cannot work on public repositories codebases [ internal restriction ].

I will investigate a bit more to see if that fixes the problem and if it does and if you are fine with the suggested fix, I will raise a pull request as well.

Please let me know.


Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @ <a href="http://wakened-cognition.blogspot.com/" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\75http%3A%2F%2Fwakened-cognition.blogspot.com%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNHzOdYJCASIDF_28vQtkp9gnKAYSQ&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\75http%3A%2F%2Fwakened-cognition.blogspot.com%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNHzOdYJCASIDF_28vQtkp9gnKAYSQ&#39;;return true;">http://wakened-cognition.blogspot.com/
My Technical Scribbings @ <a href="http://rationaleemotions.wordpress.com/" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\75http%3A%2F%2Frationaleemotions.wordpress.com%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNEcBOKyRn0lpL8LDbBuyAlKdwgXwQ&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\75http%3A%2F%2Frationaleemotions.wordpress.com%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNEcBOKyRn0lpL8LDbBuyAlKdwgXwQ&#39;;return true;">http://rationaleemotions.wordpress.com/

On Mon, Jan 18, 2016 at 10:35 AM, Krishnan Mahadevan <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="GR7_jU6ZDgAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">krishnan.ma...@gmail.com> wrote:
Sure Cedric. I will take a look at this and get back. It was a long weekend and trying to get over Monday morning blues :)

Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @ <a href="http://wakened-cognition.blogspot.com/" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\75http%3A%2F%2Fwakened-cognition.blogspot.com%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNHzOdYJCASIDF_28vQtkp9gnKAYSQ&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\75http%3A%2F%2Fwakened-cognition.blogspot.com%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNHzOdYJCASIDF_28vQtkp9gnKAYSQ&#39;;return true;">http://wakened-cognition.blogspot.com/
My Technical Scribbings @ <a href="http://rationaleemotions.wordpress.com/" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\75http%3A%2F%2Frationaleemotions.wordpress.com%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNEcBOKyRn0lpL8LDbBuyAlKdwgXwQ&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\75http%3A%2F%2Frationaleemotions.wordpress.com%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNEcBOKyRn0lpL8LDbBuyAlKdwgXwQ&#39;;return true;">http://rationaleemotions.wordpress.com/

On Sun, Jan 17, 2016 at 1:29 PM, Cédric Beust ♔ <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="GR7_jU6ZDgAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">ced...@...> wrote:
Krishnan, could you take a look at this?


-- 
Cédric


On Sat, Jan 16, 2016 at 11:45 PM, Nagesh <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="GR7_jU6ZDgAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">nagesh....@...> wrote:
I took out the custom listeners from the suite file to see if that helps narrow down the problem, but I am still seeing an incorrect testng-failed.xml file getting generated at the end of the run. I took a look at the upcoming changes and I don't know if it's being addressed with the next version.

Can someone try it out and shed some light on this issue? As I said earlier, this functionality has worked as expected until 6.9.9, but has started failing with latest 6.9.10. I will be happy to help debug/provide more info if necessary.

Greatly appreciate your inputs.

-Nagesh


On Wednesday, January 13, 2016 at 4:38:04 PM UTC-8, Nagesh wrote:
I updated the version in my project from 6.9.9 to 6.9.10 yesterday and started noticing failures in our CI runs. The issue is that the testng-failed.xml file that gets generated at the end of the first run, still contains the same tests as the original suite file. The expectation is that it contains only the failed test cases. Also if all the tests passed in the first run with the original suite file, a testng-failed.xml file still gets generated. Prior to 6.9.10 the failed xml file would not get generated if all tests in the run passed.

I have now reverted back to 6.9.9. Has this been discussed previously? I looked here and in GitHub but didn't find a thread regarding this issue.

Here is the original suite file -

<!DOCTYPE suite SYSTEM "<a href="http://testng.org/testng-1.0.dtd" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\75http%3A%2F%2Ftestng.org%2Ftestng-1.0.dtd\46sa\75D\46sntz\0751\46usg\75AFQjCNFHihXzQ1YlwcxcDehnw97TuniFLA&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\75http%3A%2F%2Ftestng.org%2Ftestng-1.0.dtd\46sa\75D\46sntz\0751\46usg\75AFQjCNFHihXzQ1YlwcxcDehnw97TuniFLA&#39;;return true;">http://testng.org/testng-1.0.dtd">
<suite name="Turbo WebDriver Tests" parallel="classes" thread-count="5">
<listeners>
<listener class-name="com.axiom.turbo.webdriver.TestListener" />
<listener class-name="com.axiom.turbo.webdriver.TestAnnotationTransformer" />
</listeners>
        <test name = "Classes with a single test method">
<groups>
<run>
<exclude name="KnownIssues" />
<exclude name="CatalogsCache" />
<exclude name="SlowRunning" />
<exclude name="Defunct" />
<exclude name="FeatureParity" />
</run>
</groups>
<classes>
<class name = "com.axiom.turbo.webdriver.tests.privacypolicy.PrivacyPolicyPageWebDriverTests" />
                        <class name = "com.axiom.turbo.webdriver.tests.labstests.LabsPageWebDriverTests" />
<class name = "com.axiom.turbo.webdriver.tests.SuccessTestClass" />
<class name = "com.axiom.turbo.webdriver.tests.SkipTestClass" />
<class name = "com.axiom.turbo.webdriver.tests.FailTestClass" />
</classes>
</test>
</suite>


Here is the testng-failed.xml file that gets generated if all or some tests passed with all the tests as the original suite file -

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "<a href="http://testng.org/testng-1.0.dtd" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\75http%3A%2F%2Ftestng.org%2Ftestng-1.0.dtd\46sa\75D\46sntz\0751\46usg\75AFQjCNFHihXzQ1YlwcxcDehnw97TuniFLA&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\75http%3A%2F%2Ftestng.org%2Ftestng-1.0.dtd\46sa\75D\46sntz\0751\46usg\75AFQjCNFHihXzQ1YlwcxcDehnw97TuniFLA&#39;;return true;">http://testng.org/testng-1.0.dtd">
<suite guice-stage="DEVELOPMENT" verbose="0" name="Failed suite [Turbo WebDriver Tests]" parallel="classes">
  <listeners>
    <listener class-name="com.axiom.turbo.webdriver.TestListener"/>
    <listener class-name="com.axiom.turbo.webdriver.TestAnnotationTransformer"/>
  </listeners>
  <test name="Classes with a single test method">
    <groups>
      <run>
        <exclude name="KnownIssues"/>
        <exclude name="CatalogsCache"/>
        <exclude name="SlowRunning"/>
        <exclude name="Defunct"/>
        <exclude name="FeatureParity"/>
      </run>
    </groups>
    <classes>
      <class name="com.axiom.turbo.webdriver.tests.privacypolicy.PrivacyPolicyPageWebDriverTests"/>
      <class name="com.axiom.turbo.webdriver.tests.SuccessTestClass"/>
      <class name="com.axiom.turbo.webdriver.tests.SkipTestClass"/>
      <class name="com.axiom.turbo.webdriver.tests.FailTestClass"/>
    </classes>
  </test> <!-- Classes with a single test method -->
</suite> <!-- Failed suite [Turbo WebDriver Tests] -->


Thanks!

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="GR7_jU6ZDgAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">testng-users...@googlegroups.com.
To post to this group, send email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="GR7_jU6ZDgAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">testng...@....
Visit this group at <a href="https://groups.google.com/group/testng-users" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/group/testng-users&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/group/testng-users&#39;;return true;">https://groups.google.com/group/testng-users.
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.






--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.