Running only failed tests with DataProvider and XML fed parameters

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Running only failed tests with DataProvider and XML fed parameters

edwolb
We use XML parameters to feed runtime parameters to our tests, and then use DataProviders to further iterate through different test scenarios.  We've setup our framework such that a CSV file can be passed in through the XML file and used to run the test against specific data.  The xml looks like this:

<suite name="Test Suite" verbose="1" parallel="tests" thread-count="10">
  <test name="Test" >
    <classes>
  <parameter name="csvFile" value="path/to/file.csv" />
           <class name="com.example.automation.Test.java" />
    </classes>
  </test>
</suite>

This works well for us, as it means we can use the same test against different sets of data, for different databases or one file for smoke testing and one for full regression.  We have a generic DataProvider that we reuse that always looks for the "csvFile" parameter and creates a set of data to pass to the test using it.

The problem is that when tests fail for specific rows of data, and we try to replay only failed tests, the csvFile parameter goes missing.  This results in a total failure, and the only way we can replay certain scenarios is by doing the entire test, which kills us for heavy data driven tests.  How can we ensure that parameters are passed into the failed test XML files?

Thanks

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To view this discussion on the web visit https://groups.google.com/d/msg/testng-users/-/dusvJ6H6hhcJ.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.
Reply | Threaded
Open this post in threaded view
|

Re: Running only failed tests with DataProvider and XML fed parameters

Cédric Beust ♔-2
Hi Ed,

Probably an oversight in how I generate testng-failed.xml (I don't believe I look up the <parameter> tag in the original testng.xml file).

I'll look into it.

-- 
Cédric




On Mon, May 14, 2012 at 8:07 AM, edwolb <[hidden email]> wrote:
We use XML parameters to feed runtime parameters to our tests, and then use DataProviders to further iterate through different test scenarios.  We've setup our framework such that a CSV file can be passed in through the XML file and used to run the test against specific data.  The xml looks like this:

<suite name="Test Suite" verbose="1" parallel="tests" thread-count="10">
  <test name="Test" >
    <classes>
  <parameter name="csvFile" value="path/to/file.csv" />
           <class name="com.example.automation.Test.java" />
    </classes>
  </test>
</suite>

This works well for us, as it means we can use the same test against different sets of data, for different databases or one file for smoke testing and one for full regression.  We have a generic DataProvider that we reuse that always looks for the "csvFile" parameter and creates a set of data to pass to the test using it.

The problem is that when tests fail for specific rows of data, and we try to replay only failed tests, the csvFile parameter goes missing.  This results in a total failure, and the only way we can replay certain scenarios is by doing the entire test, which kills us for heavy data driven tests.  How can we ensure that parameters are passed into the failed test XML files?

Thanks

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To view this discussion on the web visit https://groups.google.com/d/msg/testng-users/-/dusvJ6H6hhcJ.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.
Reply | Threaded
Open this post in threaded view
|

Re: Running only failed tests with DataProvider and XML fed parameters

edwolb
Thanks Cédric, let me know if I can provide any more input.


On Monday, 14 May 2012 14:23:13 UTC-4, Cédric Beust ♔ wrote:
Hi Ed,

Probably an oversight in how I generate testng-failed.xml (I don't believe I look up the <parameter> tag in the original testng.xml file).

I'll look into it.

-- 
Cédric




On Mon, May 14, 2012 at 8:07 AM, edwolb <> wrote:
We use XML parameters to feed runtime parameters to our tests, and then use DataProviders to further iterate through different test scenarios.  We've setup our framework such that a CSV file can be passed in through the XML file and used to run the test against specific data.  The xml looks like this:

<suite name="Test Suite" verbose="1" parallel="tests" thread-count="10">
  <test name="Test" >
    <classes>
  <parameter name="csvFile" value="path/to/file.csv" />
           <class name="com.example.automation.Test.java" />
    </classes>
  </test>
</suite>

This works well for us, as it means we can use the same test against different sets of data, for different databases or one file for smoke testing and one for full regression.  We have a generic DataProvider that we reuse that always looks for the "csvFile" parameter and creates a set of data to pass to the test using it.

The problem is that when tests fail for specific rows of data, and we try to replay only failed tests, the csvFile parameter goes missing.  This results in a total failure, and the only way we can replay certain scenarios is by doing the entire test, which kills us for heavy data driven tests.  How can we ensure that parameters are passed into the failed test XML files?

Thanks

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To view this discussion on the web visit https://groups.google.com/d/msg/testng-users/-/dusvJ6H6hhcJ.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To view this discussion on the web visit https://groups.google.com/d/msg/testng-users/-/dU-C5Ff8XugJ.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.
Reply | Threaded
Open this post in threaded view
|

Re: Running only failed tests with DataProvider and XML fed parameters

Cédric Beust ♔-2
Hi Ed,

I'm seeing these parameters being carried over to testng-failed.xml, although not exactly where they should be. Still, they are here and you should see them:

<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="Failed suite [Suite]">
  <parameter name="n" value="42"/>
  <parameter name="o" value="43"/>
  <test name="Test(failed)" preserve-order="true">
    <parameter name="n" value="42"/>
    <parameter name="o" value="43"/>
    <classes>
      <class name="test.reports.FailedSampleTest">
        <methods>
          <include name="failed"/>
        </methods>
      </class> <!-- test.reports.FailedSampleTest -->
    </classes>
  </test> <!-- Test(failed) -->
</suite> <!-- Failed suite [Suite] -->

Can you post a small testng.xml and then the testng-failed.xml you are getting?

Thanks.

-- 
Cédric




On Mon, May 14, 2012 at 1:20 PM, edwolb <[hidden email]> wrote:
Thanks Cédric, let me know if I can provide any more input.


On Monday, 14 May 2012 14:23:13 UTC-4, Cédric Beust ♔ wrote:
Hi Ed,

Probably an oversight in how I generate testng-failed.xml (I don't believe I look up the <parameter> tag in the original testng.xml file).

I'll look into it.

-- 
Cédric




On Mon, May 14, 2012 at 8:07 AM, edwolb <> wrote:
We use XML parameters to feed runtime parameters to our tests, and then use DataProviders to further iterate through different test scenarios.  We've setup our framework such that a CSV file can be passed in through the XML file and used to run the test against specific data.  The xml looks like this:

<suite name="Test Suite" verbose="1" parallel="tests" thread-count="10">
  <test name="Test" >
    <classes>
  <parameter name="csvFile" value="path/to/file.csv" />
           <class name="com.example.automation.Test.java" />
    </classes>
  </test>
</suite>

This works well for us, as it means we can use the same test against different sets of data, for different databases or one file for smoke testing and one for full regression.  We have a generic DataProvider that we reuse that always looks for the "csvFile" parameter and creates a set of data to pass to the test using it.

The problem is that when tests fail for specific rows of data, and we try to replay only failed tests, the csvFile parameter goes missing.  This results in a total failure, and the only way we can replay certain scenarios is by doing the entire test, which kills us for heavy data driven tests.  How can we ensure that parameters are passed into the failed test XML files?

Thanks

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To view this discussion on the web visit https://groups.google.com/d/msg/testng-users/-/dusvJ6H6hhcJ.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To view this discussion on the web visit https://groups.google.com/d/msg/testng-users/-/dU-C5Ff8XugJ.

To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.
Reply | Threaded
Open this post in threaded view
|

Re: Running only failed tests with DataProvider and XML fed parameters

Cédric Beust ♔-2
Hi Ed,

I just pushed a fix in the beta that should fix this, can you try it and report back?


Run with verbose="2" and make sure the version is at least 6.6beta 20120616_1332

Thanks.

-- 
Cédric




On Sat, Jun 16, 2012 at 10:43 AM, Cédric Beust ♔ <[hidden email]> wrote:
Hi Ed,

I'm seeing these parameters being carried over to testng-failed.xml, although not exactly where they should be. Still, they are here and you should see them:

<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="Failed suite [Suite]">
  <parameter name="n" value="42"/>
  <parameter name="o" value="43"/>
  <test name="Test(failed)" preserve-order="true">
    <parameter name="n" value="42"/>
    <parameter name="o" value="43"/>
    <classes>
      <class name="test.reports.FailedSampleTest">
        <methods>
          <include name="failed"/>
        </methods>
      </class> <!-- test.reports.FailedSampleTest -->
    </classes>
  </test> <!-- Test(failed) -->
</suite> <!-- Failed suite [Suite] -->

Can you post a small testng.xml and then the testng-failed.xml you are getting?

Thanks.

-- 
Cédric




On Mon, May 14, 2012 at 1:20 PM, edwolb <[hidden email]> wrote:
Thanks Cédric, let me know if I can provide any more input.


On Monday, 14 May 2012 14:23:13 UTC-4, Cédric Beust ♔ wrote:
Hi Ed,

Probably an oversight in how I generate testng-failed.xml (I don't believe I look up the <parameter> tag in the original testng.xml file).

I'll look into it.

-- 
Cédric




On Mon, May 14, 2012 at 8:07 AM, edwolb <> wrote:
We use XML parameters to feed runtime parameters to our tests, and then use DataProviders to further iterate through different test scenarios.  We've setup our framework such that a CSV file can be passed in through the XML file and used to run the test against specific data.  The xml looks like this:

<suite name="Test Suite" verbose="1" parallel="tests" thread-count="10">
  <test name="Test" >
    <classes>
  <parameter name="csvFile" value="path/to/file.csv" />
           <class name="com.example.automation.Test.java" />
    </classes>
  </test>
</suite>

This works well for us, as it means we can use the same test against different sets of data, for different databases or one file for smoke testing and one for full regression.  We have a generic DataProvider that we reuse that always looks for the "csvFile" parameter and creates a set of data to pass to the test using it.

The problem is that when tests fail for specific rows of data, and we try to replay only failed tests, the csvFile parameter goes missing.  This results in a total failure, and the only way we can replay certain scenarios is by doing the entire test, which kills us for heavy data driven tests.  How can we ensure that parameters are passed into the failed test XML files?

Thanks

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To view this discussion on the web visit https://groups.google.com/d/msg/testng-users/-/dusvJ6H6hhcJ.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To view this discussion on the web visit https://groups.google.com/d/msg/testng-users/-/dU-C5Ff8XugJ.

To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.


--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.
Reply | Threaded
Open this post in threaded view
|

Re: Running only failed tests with DataProvider and XML fed parameters

edwolb
Hi Cédric,

It looks like the test successfully ran again, however instead of only rerunning the one failed instance provided by the data provider, it ran all instances again.  So you've gotten the 1 test out of the suite to run, but we're still running all scenarios, even those that previously passed.  Is this intended?

Also, from what I can tell, the parameters are now being included twice, once at the suite level, and once at the test level, even though the original XML file only included the parameters at the class level.  I know the DTD may only recommend that parameters be included at the test level, and not the class level.  Could this be causing the issue?  Including it at the class level seems to be working otherwise.

Here's the failed xml file:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="Failed suite [suite]">
  <parameter name="csvFile" value="com/automation/tests/orders/TC56.csv"/>
  <parameter name="browser" value="chrome"/>
  <test name="TC56_verifyOrder(failed)" preserve-order="true">
    <parameter name="csvFile" value=" com/automation/tests/orders/TC56.csv "/>
    <parameter name="browser" value="chrome"/>
    <classes>
      <class name="com.automation.tests.orders.TC56">
        <methods>
          <include name="setupNavigation"/>
          <include name="closeDrivers"/>
          <include name="login"/>
          <include name="navigateToOrders"/>
          <include name="cleanup"/>
          <include name="DCorder" invocation-numbers="0"/>
          <include name="setLoginProfile"/>
          <include name="setupDriver"/>
        </methods>
      </class> <!-- com.automation.tests.orders.TC56 -->
    </classes>
  </test> <!-- TC56_verifyDiscontinueAndStikeOutActiveOrder(failed) -->
</suite> <!-- Failed suite [suite] -->

and the original:

<suite name="suite">
  <test name="TC56">
    <classes>
  <parameter name="csvFile" value="com/automation/tests/orders/TC56.csv" />
  <parameter name="browser" value="chrome" />
       <class name="com.automation.tests.orders.TC56" />

    </classes>
  </test>
</suite>


Thanks

--
Chris

On Sat, Jun 16, 2012 at 4:40 PM, Cédric Beust ♔ <[hidden email]> wrote:
Hi Ed,

I just pushed a fix in the beta that should fix this, can you try it and report back?


Run with verbose="2" and make sure the version is at least 6.6beta 20120616_1332

Thanks.

-- 
Cédric




On Sat, Jun 16, 2012 at 10:43 AM, Cédric Beust ♔ <[hidden email]> wrote:
Hi Ed,

I'm seeing these parameters being carried over to testng-failed.xml, although not exactly where they should be. Still, they are here and you should see them:

<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="Failed suite [Suite]">
  <parameter name="n" value="42"/>
  <parameter name="o" value="43"/>
  <test name="Test(failed)" preserve-order="true">
    <parameter name="n" value="42"/>
    <parameter name="o" value="43"/>
    <classes>
      <class name="test.reports.FailedSampleTest">
        <methods>
          <include name="failed"/>
        </methods>
      </class> <!-- test.reports.FailedSampleTest -->
    </classes>
  </test> <!-- Test(failed) -->
</suite> <!-- Failed suite [Suite] -->

Can you post a small testng.xml and then the testng-failed.xml you are getting?

Thanks.

-- 
Cédric




On Mon, May 14, 2012 at 1:20 PM, edwolb <[hidden email]> wrote:
Thanks Cédric, let me know if I can provide any more input.


On Monday, 14 May 2012 14:23:13 UTC-4, Cédric Beust ♔ wrote:
Hi Ed,

Probably an oversight in how I generate testng-failed.xml (I don't believe I look up the <parameter> tag in the original testng.xml file).

I'll look into it.

-- 
Cédric




On Mon, May 14, 2012 at 8:07 AM, edwolb <> wrote:
We use XML parameters to feed runtime parameters to our tests, and then use DataProviders to further iterate through different test scenarios.  We've setup our framework such that a CSV file can be passed in through the XML file and used to run the test against specific data.  The xml looks like this:

<suite name="Test Suite" verbose="1" parallel="tests" thread-count="10">
  <test name="Test" >
    <classes>
  <parameter name="csvFile" value="path/to/file.csv" />
           <class name="com.example.automation.Test.java" />
    </classes>
  </test>
</suite>

This works well for us, as it means we can use the same test against different sets of data, for different databases or one file for smoke testing and one for full regression.  We have a generic DataProvider that we reuse that always looks for the "csvFile" parameter and creates a set of data to pass to the test using it.

The problem is that when tests fail for specific rows of data, and we try to replay only failed tests, the csvFile parameter goes missing.  This results in a total failure, and the only way we can replay certain scenarios is by doing the entire test, which kills us for heavy data driven tests.  How can we ensure that parameters are passed into the failed test XML files?

Thanks

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To view this discussion on the web visit https://groups.google.com/d/msg/testng-users/-/dusvJ6H6hhcJ.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To view this discussion on the web visit https://groups.google.com/d/msg/testng-users/-/dU-C5Ff8XugJ.

To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.


--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.
Reply | Threaded
Open this post in threaded view
|

Re: Running only failed tests with DataProvider and XML fed parameters

Frank Escobar
Fixed in version 6.10
https://mvnrepository.com/artifact/org.testng/testng/6.10
https://github.com/cbeust/testng/blob/master/CHANGES.txt#L95
https://github.com/cbeust/testng/issues/1250

On Sunday, June 17, 2012 at 1:41:48 AM UTC+1, edwolb wrote:
Hi Cédric,

It looks like the test successfully ran again, however instead of only rerunning the one failed instance provided by the data provider, it ran all instances again.  So you've gotten the 1 test out of the suite to run, but we're still running all scenarios, even those that previously passed.  Is this intended?

Also, from what I can tell, the parameters are now being included twice, once at the suite level, and once at the test level, even though the original XML file only included the parameters at the class level.  I know the DTD may only recommend that parameters be included at the test level, and not the class level.  Could this be causing the issue?  Including it at the class level seems to be working otherwise.

Here's the failed xml file:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "<a href="http://testng.org/testng-1.0.dtd" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Ftestng.org%2Ftestng-1.0.dtd\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFHihXzQ1YlwcxcDehnw97TuniFLA&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Ftestng.org%2Ftestng-1.0.dtd\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFHihXzQ1YlwcxcDehnw97TuniFLA&#39;;return true;">http://testng.org/testng-1.0.dtd">
<suite name="Failed suite [suite]">
  <parameter name="csvFile" value="com/automation/tests/orders/TC56.csv"/>
  <parameter name="browser" value="chrome"/>
  <test name="TC56_verifyOrder(failed)" preserve-order="true">
    <parameter name="csvFile" value=" com/automation/tests/orders/TC56.csv "/>
    <parameter name="browser" value="chrome"/>
    <classes>
      <class name="com.automation.tests.orders.TC56">
        <methods>
          <include name="setupNavigation"/>
          <include name="closeDrivers"/>
          <include name="login"/>
          <include name="navigateToOrders"/>
          <include name="cleanup"/>
          <include name="DCorder" invocation-numbers="0"/>
          <include name="setLoginProfile"/>
          <include name="setupDriver"/>
        </methods>
      </class> <!-- com.automation.tests.orders.TC56 -->
    </classes>
  </test> <!-- TC56_verifyDiscontinueAndStikeOutActiveOrder(failed) -->
</suite> <!-- Failed suite [suite] -->

and the original:

<suite name="suite">
  <test name="TC56">
    <classes>
  <parameter name="csvFile" value="com/automation/tests/orders/TC56.csv" />
  <parameter name="browser" value="chrome" />
       <class name="com.automation.tests.orders.TC56" />

    </classes>
  </test>
</suite>


Thanks

--
Chris

On Sat, Jun 16, 2012 at 4:40 PM, Cédric Beust ♔ <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="LR5GlBWB4I8J" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">ced...@...> wrote:
Hi Ed,

I just pushed a fix in the beta that should fix this, can you try it and report back?

<a href="http://testng.org/beta" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Ftestng.org%2Fbeta\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFSKY3ppx5vP2n-3VnX8yAfmyev7Q&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Ftestng.org%2Fbeta\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFSKY3ppx5vP2n-3VnX8yAfmyev7Q&#39;;return true;">http://testng.org/beta

Run with verbose="2" and make sure the version is at least 6.6beta 20120616_1332

Thanks.

-- 
Cédric




On Sat, Jun 16, 2012 at 10:43 AM, Cédric Beust ♔ <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="LR5GlBWB4I8J" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">ced...@...> wrote:
Hi Ed,

I'm seeing these parameters being carried over to testng-failed.xml, although not exactly where they should be. Still, they are here and you should see them:

<!DOCTYPE suite SYSTEM "<a href="http://testng.org/testng-1.0.dtd" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Ftestng.org%2Ftestng-1.0.dtd\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFHihXzQ1YlwcxcDehnw97TuniFLA&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Ftestng.org%2Ftestng-1.0.dtd\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFHihXzQ1YlwcxcDehnw97TuniFLA&#39;;return true;">http://testng.org/testng-1.0.dtd">
<suite name="Failed suite [Suite]">
  <parameter name="n" value="42"/>
  <parameter name="o" value="43"/>
  <test name="Test(failed)" preserve-order="true">
    <parameter name="n" value="42"/>
    <parameter name="o" value="43"/>
    <classes>
      <class name="test.reports.FailedSampleTest">
        <methods>
          <include name="failed"/>
        </methods>
      </class> <!-- test.reports.FailedSampleTest -->
    </classes>
  </test> <!-- Test(failed) -->
</suite> <!-- Failed suite [Suite] -->

Can you post a small testng.xml and then the testng-failed.xml you are getting?

Thanks.

-- 
Cédric




On Mon, May 14, 2012 at 1:20 PM, edwolb <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="LR5GlBWB4I8J" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">edw...@...> wrote:
Thanks Cédric, let me know if I can provide any more input.


On Monday, 14 May 2012 14:23:13 UTC-4, Cédric Beust ♔ wrote:
Hi Ed,

Probably an oversight in how I generate testng-failed.xml (I don't believe I look up the <parameter> tag in the original testng.xml file).

I'll look into it.

-- 
Cédric




On Mon, May 14, 2012 at 8:07 AM, edwolb <> wrote:
We use XML parameters to feed runtime parameters to our tests, and then use DataProviders to further iterate through different test scenarios.  We've setup our framework such that a CSV file can be passed in through the XML file and used to run the test against specific data.  The xml looks like this:

<suite name="Test Suite" verbose="1" parallel="tests" thread-count="10">
  <test name="Test" >
    <classes>
  <parameter name="csvFile" value="path/to/file.csv" />
           <class name="com.example.automation.Test.java" />
    </classes>
  </test>
</suite>

This works well for us, as it means we can use the same test against different sets of data, for different databases or one file for smoke testing and one for full regression.  We have a generic DataProvider that we reuse that always looks for the "csvFile" parameter and creates a set of data to pass to the test using it.

The problem is that when tests fail for specific rows of data, and we try to replay only failed tests, the csvFile parameter goes missing.  This results in a total failure, and the only way we can replay certain scenarios is by doing the entire test, which kills us for heavy data driven tests.  How can we ensure that parameters are passed into the failed test XML files?

Thanks

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To view this discussion on the web visit <a href="https://groups.google.com/d/msg/testng-users/-/dusvJ6H6hhcJ" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/msg/testng-users/-/dusvJ6H6hhcJ&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/msg/testng-users/-/dusvJ6H6hhcJ&#39;;return true;">https://groups.google.com/d/msg/testng-users/-/dusvJ6H6hhcJ.
To post to this group, send email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="LR5GlBWB4I8J" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">testng...@....
To unsubscribe from this group, send email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="LR5GlBWB4I8J" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">testng-users...@googlegroups.com.
For more options, visit this group at <a href="http://groups.google.com/group/testng-users?hl=en" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://groups.google.com/group/testng-users?hl\x3den&#39;;return true;" onclick="this.href=&#39;http://groups.google.com/group/testng-users?hl\x3den&#39;;return true;">http://groups.google.com/group/testng-users?hl=en.

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To view this discussion on the web visit <a href="https://groups.google.com/d/msg/testng-users/-/dU-C5Ff8XugJ" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/msg/testng-users/-/dU-C5Ff8XugJ&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/msg/testng-users/-/dU-C5Ff8XugJ&#39;;return true;">https://groups.google.com/d/msg/testng-users/-/dU-C5Ff8XugJ.

To post to this group, send email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="LR5GlBWB4I8J" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">testng...@....
To unsubscribe from this group, send email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="LR5GlBWB4I8J" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">testng-users...@googlegroups.com.
For more options, visit this group at <a href="http://groups.google.com/group/testng-users?hl=en" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://groups.google.com/group/testng-users?hl\x3den&#39;;return true;" onclick="this.href=&#39;http://groups.google.com/group/testng-users?hl\x3den&#39;;return true;">http://groups.google.com/group/testng-users?hl=en.


--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="LR5GlBWB4I8J" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">testng...@....
To unsubscribe from this group, send email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="LR5GlBWB4I8J" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">testng-users...@googlegroups.com.
For more options, visit this group at <a href="http://groups.google.com/group/testng-users?hl=en" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://groups.google.com/group/testng-users?hl\x3den&#39;;return true;" onclick="this.href=&#39;http://groups.google.com/group/testng-users?hl\x3den&#39;;return true;">http://groups.google.com/group/testng-users?hl=en.

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.