DataProvider, test dependency (but for the same data)

classic Classic list List threaded Threaded
19 messages Options
Reply | Threaded
Open this post in threaded view
|

DataProvider, test dependency (but for the same data)

hithwen
I'm just starting with TestNG and Selenium

I want to perform a registration test in a collection of webpages.
I've writen a Register class with the following methods

@Test( dataProvider = "WebSites", groups = "launchSite")
public void launchSite(WeboSite webSite)

@Test( dataProvider = "WebSites", groups = "goToRegPage",
       dependsOnGroups = "launchSite")
public void openRegisterPage(WeboSite webSite)

@Test( dataProvider = "WebSites", groups = "register",
   dependsOnGroups = "goToRegPage")
public void enterRegistrationData(BingoSite bingoSite)

So I've made each test dependant on the previous (obviously if you
cannot enter the registration page you cannot register)

What I want now is each webpage result be independant from the
others.

Now openRegisterPage is run for every webpage and if it fails in one
website enterRegistrationData is not run for any of them but should be
run for the rest of websites that passed openRegisterPage

What is the correct way to do this?

Thanks a lot

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.

Reply | Threaded
Open this post in threaded view
|

Re: DataProvider, test dependency (but for the same data)

Cédric Beust ♔-2
Hi Julia,

You're coming across a behavior that has been the standard in TestNG but that I've been meaning to change for a while. The only thing that was holding me back is the fact that it might break existing tests, but I think this behavior makes more sense so I'm just going to go ahead an implement it.

The short answer to your question: please try the beta (http://testng.org/beta ), it should fix your problem.

The longer answer now...

So far, TestNG's dependency mechanism was acting at the class level instead of the instance level. In other words, if method f2() depends on f1() and f1() fails, then f2() is going to be skipped across all the instances. This seemed reasonable to me at the time but more recently, I started realizing that just because a method such as f1() fails on a certain instance doesn't mean it will fail on other instances, so it shouldn't take down all the other classes on the other instances in the process.

So, what could break with this change?

If you are relying on the current behavior, you might start seeing tests that used to fail start passing. Fundamentally, I think these new tests suddenly passing are not lying to you: all the methods and the methods they depend on are successful, so there is little reason to flag any failure there.

I would love to get some feedback on this change, just to make sure I didn't miss anything.

Thanks.

-- 
Cédric




On Wed, Jun 8, 2011 at 4:41 AM, Julia S.S <[hidden email]> wrote:
I'm just starting with TestNG and Selenium

I want to perform a registration test in a collection of webpages.
I've writen a Register class with the following methods

@Test( dataProvider = "WebSites", groups = "launchSite")
public void launchSite(WeboSite webSite)

@Test( dataProvider = "WebSites", groups = "goToRegPage",
      dependsOnGroups = "launchSite")
public void openRegisterPage(WeboSite webSite)

@Test( dataProvider = "WebSites", groups = "register",
  dependsOnGroups = "goToRegPage")
public void enterRegistrationData(BingoSite bingoSite)

So I've made each test dependant on the previous (obviously if you
cannot enter the registration page you cannot register)

What I want now is each webpage result be independant from the
others.

Now openRegisterPage is run for every webpage and if it fails in one
website enterRegistrationData is not run for any of them but should be
run for the rest of websites that passed openRegisterPage

What is the correct way to do this?

Thanks a lot

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.


--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.
Reply | Threaded
Open this post in threaded view
|

Re: DataProvider, test dependency (but for the same data)

Nalin Makar
Cedric,

Isn't this the same as the behavior controlled using "configfailurepolicy" option?

-nalin



2011/6/8 Cédric Beust ♔ <[hidden email]>
Hi Julia,

You're coming across a behavior that has been the standard in TestNG but that I've been meaning to change for a while. The only thing that was holding me back is the fact that it might break existing tests, but I think this behavior makes more sense so I'm just going to go ahead an implement it.

The short answer to your question: please try the beta (http://testng.org/beta ), it should fix your problem.

The longer answer now...

So far, TestNG's dependency mechanism was acting at the class level instead of the instance level. In other words, if method f2() depends on f1() and f1() fails, then f2() is going to be skipped across all the instances. This seemed reasonable to me at the time but more recently, I started realizing that just because a method such as f1() fails on a certain instance doesn't mean it will fail on other instances, so it shouldn't take down all the other classes on the other instances in the process.

So, what could break with this change?

If you are relying on the current behavior, you might start seeing tests that used to fail start passing. Fundamentally, I think these new tests suddenly passing are not lying to you: all the methods and the methods they depend on are successful, so there is little reason to flag any failure there.

I would love to get some feedback on this change, just to make sure I didn't miss anything.

Thanks.

-- 
Cédric




On Wed, Jun 8, 2011 at 4:41 AM, Julia S.S <[hidden email]> wrote:
I'm just starting with TestNG and Selenium

I want to perform a registration test in a collection of webpages.
I've writen a Register class with the following methods

@Test( dataProvider = "WebSites", groups = "launchSite")
public void launchSite(WeboSite webSite)

@Test( dataProvider = "WebSites", groups = "goToRegPage",
      dependsOnGroups = "launchSite")
public void openRegisterPage(WeboSite webSite)

@Test( dataProvider = "WebSites", groups = "register",
  dependsOnGroups = "goToRegPage")
public void enterRegistrationData(BingoSite bingoSite)

So I've made each test dependant on the previous (obviously if you
cannot enter the registration page you cannot register)

What I want now is each webpage result be independant from the
others.

Now openRegisterPage is run for every webpage and if it fails in one
website enterRegistrationData is not run for any of them but should be
run for the rest of websites that passed openRegisterPage

What is the correct way to do this?

Thanks a lot

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.


--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.
Reply | Threaded
Open this post in threaded view
|

Re: DataProvider, test dependency (but for the same data)

Cédric Beust ♔-2
Nalin,

On Wed, Jun 8, 2011 at 11:01 AM, Nalin Makar <[hidden email]> wrote:
Cedric,

Isn't this the same as the behavior controlled using "configfailurepolicy" option?

I was under the impression that configfailurepolicy was not looking at instances, it was still an "all-or-nothing" skip or continue, am I wrong?

-- 
Cédric

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.
Reply | Threaded
Open this post in threaded view
|

Re: DataProvider, test dependency (but for the same data)

Nalin Makar
That is an all or nothing approach for skipping vs continuing. 

I figured you were talking about the same. How is this different? You'll also be now looking at each instance and continue running tests for one instance even when a dependency failed for another.

-nalin



2011/6/8 Cédric Beust ♔ <[hidden email]>
Nalin,

On Wed, Jun 8, 2011 at 11:01 AM, Nalin Makar <[hidden email]> wrote:
Cedric,

Isn't this the same as the behavior controlled using "configfailurepolicy" option?

I was under the impression that configfailurepolicy was not looking at instances, it was still an "all-or-nothing" skip or continue, am I wrong?

-- 
Cédric

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.
Reply | Threaded
Open this post in threaded view
|

Re: DataProvider, test dependency (but for the same data)

François Reynaud
Hi Cedric,

I'm not sure I understand the change, but i believe that will impact my tests a lot.

can you clarify what will happen for a test designed that way ?


class Demo {
@Test
public void m1(){
checkServerIsUp()
}

@DataProvider(name=user)
public Object[][] users(){
// returns some users
}

@Test(depednsOnMethos="m1", dataProvider=user)
public void m2(User u){
// test}


@Test(dependsOnMethods=m2)
public void m3(){
//
}


How would a class like that work ? How will testng map instances of the methods with dataprovider and instances without 

thanks
François
On Wed, Jun 8, 2011 at 8:57 PM, Nalin Makar <[hidden email]> wrote:
That is an all or nothing approach for skipping vs continuing. 

I figured you were talking about the same. How is this different? You'll also be now looking at each instance and continue running tests for one instance even when a dependency failed for another.

-nalin



2011/6/8 Cédric Beust ♔ <[hidden email]>
Nalin,

On Wed, Jun 8, 2011 at 11:01 AM, Nalin Makar <[hidden email]> wrote:
Cedric,

Isn't this the same as the behavior controlled using "configfailurepolicy" option?

I was under the impression that configfailurepolicy was not looking at instances, it was still an "all-or-nothing" skip or continue, am I wrong?

-- 
Cédric

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.
Reply | Threaded
Open this post in threaded view
|

Re: DataProvider, test dependency (but for the same data)

Cédric Beust ♔-2
Hi François,
Hi François,

Can you try your tests with the current beta and let me know?

This is a beta, so it's definitely not set in stone and if we catch problems with this change, I'll be happy to hide it behind a configuration parameter.

-- 
Cédric




2011/6/8 François Reynaud <[hidden email]>
Hi Cedric,

I'm not sure I understand the change, but i believe that will impact my tests a lot.

can you clarify what will happen for a test designed that way ?


class Demo {
@Test
public void m1(){
checkServerIsUp()
}

@DataProvider(name=user)
public Object[][] users(){
// returns some users
}

@Test(depednsOnMethos="m1", dataProvider=user)
public void m2(User u){
// test}


@Test(dependsOnMethods=m2)
public void m3(){
//
}


How would a class like that work ? How will testng map instances of the methods with dataprovider and instances without 

thanks
François

On Wed, Jun 8, 2011 at 8:57 PM, Nalin Makar <[hidden email]> wrote:
That is an all or nothing approach for skipping vs continuing. 

I figured you were talking about the same. How is this different? You'll also be now looking at each instance and continue running tests for one instance even when a dependency failed for another.

-nalin



2011/6/8 Cédric Beust ♔ <[hidden email]>
Nalin,

On Wed, Jun 8, 2011 at 11:01 AM, Nalin Makar <[hidden email]> wrote:
Cedric,

Isn't this the same as the behavior controlled using "configfailurepolicy" option?

I was under the impression that configfailurepolicy was not looking at instances, it was still an "all-or-nothing" skip or continue, am I wrong?

-- 
Cédric

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.
Reply | Threaded
Open this post in threaded view
|

Re: DataProvider, test dependency (but for the same data)

Cédric Beust ♔-2
In reply to this post by Nalin Makar
Hi Nalin,

Maybe I'm wrong, but here is my understanding. Consider an example such as the one posted above, where two instances of the same class are created, and this class has methods that depend on each other.

When the tests pass, you get:

Instance 1: a, b, c
Instance 2: a, b, c

Now suppose that b on instance 2 fails.

Before my change, the run would become:

Instance 1: a, b (pass), c (ckipped)  (note that instance1 was affected by instance2's failure)
Instance 2: a, b (fail), c (skipped)

If you set configfailurepolicy=continue, you will get:

Instance1: a, b (pass), c (pass)
Instance 2: a, b (fail), c (pass)

With the new change, you get:

Instance1: a, b (pass), c (pass) (note that the test run of instance 1 is completely unaffected by what happens with instance 2).
Instance 2: a, b (fail), c (skipped)

I suspect (but I haven't tested yet) that setting configfailurepolicy=continue with the new change will produce:

Instance1: a, b (pass), c (pass)
Instance 2: a, b (fail), c (pass)

Does this make sense?

-- 
Cédric




On Wed, Jun 8, 2011 at 12:57 PM, Nalin Makar <[hidden email]> wrote:
That is an all or nothing approach for skipping vs continuing. 

I figured you were talking about the same. How is this different? You'll also be now looking at each instance and continue running tests for one instance even when a dependency failed for another.

-nalin



2011/6/8 Cédric Beust ♔ <[hidden email]>
Nalin,

On Wed, Jun 8, 2011 at 11:01 AM, Nalin Makar <[hidden email]> wrote:
Cedric,

Isn't this the same as the behavior controlled using "configfailurepolicy" option?

I was under the impression that configfailurepolicy was not looking at instances, it was still an "all-or-nothing" skip or continue, am I wrong?

-- 
Cédric

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.
Reply | Threaded
Open this post in threaded view
|

Re: DataProvider, test dependency (but for the same data)

François Reynaud
In reply to this post by Cédric Beust ♔-2
I'm not against refactoring my tests, it would actually make it easier and I see how it works if all the method share the same data provider. I'm curious to know how i need to update my tests for the "checkserver" thing.

2011/6/8 Cédric Beust ♔ <[hidden email]>
Hi François,
Hi François,

Can you try your tests with the current beta and let me know?

This is a beta, so it's definitely not set in stone and if we catch problems with this change, I'll be happy to hide it behind a configuration parameter.

-- 
Cédric




2011/6/8 François Reynaud <[hidden email]>
Hi Cedric,

I'm not sure I understand the change, but i believe that will impact my tests a lot.

can you clarify what will happen for a test designed that way ?


class Demo {
@Test
public void m1(){
checkServerIsUp()
}

@DataProvider(name=user)
public Object[][] users(){
// returns some users
}

@Test(depednsOnMethos="m1", dataProvider=user)
public void m2(User u){
// test}


@Test(dependsOnMethods=m2)
public void m3(){
//
}


How would a class like that work ? How will testng map instances of the methods with dataprovider and instances without 

thanks
François

On Wed, Jun 8, 2011 at 8:57 PM, Nalin Makar <[hidden email]> wrote:
That is an all or nothing approach for skipping vs continuing. 

I figured you were talking about the same. How is this different? You'll also be now looking at each instance and continue running tests for one instance even when a dependency failed for another.

-nalin



2011/6/8 Cédric Beust ♔ <[hidden email]>
Nalin,

On Wed, Jun 8, 2011 at 11:01 AM, Nalin Makar <[hidden email]> wrote:
Cedric,

Isn't this the same as the behavior controlled using "configfailurepolicy" option?

I was under the impression that configfailurepolicy was not looking at instances, it was still an "all-or-nothing" skip or continue, am I wrong?

-- 
Cédric

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.
Reply | Threaded
Open this post in threaded view
|

Re: DataProvider, test dependency (but for the same data)

Cédric Beust ♔-2

2011/6/8 François Reynaud <[hidden email]>
I'm not against refactoring my tests, it would actually make it easier and I see how it works if all the method share the same data provider. I'm curious to know how i need to update my tests for the "checkserver" thing.

Like I said, the worst that can happen is that before the change, some of your tests used to skip (note: skip, not fail) and now they will start passing. This sounds scary but it's because the failures were not really there in the first place, they were a side effect of the way TestNG works.

In other words, before, if you have:

a, b (fail), c (skip)

then other instances of that same class will all skip their c(), even if their own b() actually passed.

With this change, only that specific instance will show failures, all your other instances will pass.

The reason why I think this change will probably go unnoticed is because most users probably don't have tests that constantly skip and that they ignore: they probably reworked their tests to present the true result. So they won't notice this change.

-- 
Cédric

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.
Reply | Threaded
Open this post in threaded view
|

Re: DataProvider, test dependency (but for the same data)

François Reynaud
Hi Cédric,

just tried the beta, and i don't get it. I think i'm not in reproducing the scenario you're talking about.


in that case, I would have expected 

skipped m3 (failed , false) 
passed m3 ( passed ,  true) 

but instead I have :

SKIPPED: m3

thanks,

Françosi


2011/6/8 Cédric Beust ♔ <[hidden email]>

2011/6/8 François Reynaud <[hidden email]>
I'm not against refactoring my tests, it would actually make it easier and I see how it works if all the method share the same data provider. I'm curious to know how i need to update my tests for the "checkserver" thing.

Like I said, the worst that can happen is that before the change, some of your tests used to skip (note: skip, not fail) and now they will start passing. This sounds scary but it's because the failures were not really there in the first place, they were a side effect of the way TestNG works.

In other words, before, if you have:

a, b (fail), c (skip)

then other instances of that same class will all skip their c(), even if their own b() actually passed.

With this change, only that specific instance will show failures, all your other instances will pass.

The reason why I think this change will probably go unnoticed is because most users probably don't have tests that constantly skip and that they ignore: they probably reworked their tests to present the true result. So they won't notice this change.

-- 
Cédric

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.
Reply | Threaded
Open this post in threaded view
|

Re: DataProvider, test dependency (but for the same data)

Nalin Makar
In reply to this post by Cédric Beust ♔-2
Maybe I am wrong, but I think if you set configfailurepolicy=continue (without your beta changes), you will get:

Instance1: a, b (pass), c (pass)
Instance 2: a, b (fail), c (skip)

And I see the distinction you are talking about. I am just not clear if it's working right now as you are describing it. I should test it.

-nalin



2011/6/8 Cédric Beust ♔ <[hidden email]>
Hi Nalin,

Maybe I'm wrong, but here is my understanding. Consider an example such as the one posted above, where two instances of the same class are created, and this class has methods that depend on each other.

When the tests pass, you get:

Instance 1: a, b, c
Instance 2: a, b, c

Now suppose that b on instance 2 fails.

Before my change, the run would become:

Instance 1: a, b (pass), c (ckipped)  (note that instance1 was affected by instance2's failure)
Instance 2: a, b (fail), c (skipped)

If you set configfailurepolicy=continue, you will get:

Instance1: a, b (pass), c (pass)
Instance 2: a, b (fail), c (pass)

With the new change, you get:

Instance1: a, b (pass), c (pass) (note that the test run of instance 1 is completely unaffected by what happens with instance 2).
Instance 2: a, b (fail), c (skipped)

I suspect (but I haven't tested yet) that setting configfailurepolicy=continue with the new change will produce:

Instance1: a, b (pass), c (pass)
Instance 2: a, b (fail), c (pass)

Does this make sense?

-- 
Cédric




On Wed, Jun 8, 2011 at 12:57 PM, Nalin Makar <[hidden email]> wrote:
That is an all or nothing approach for skipping vs continuing. 

I figured you were talking about the same. How is this different? You'll also be now looking at each instance and continue running tests for one instance even when a dependency failed for another.

-nalin



2011/6/8 Cédric Beust ♔ <[hidden email]>
Nalin,

On Wed, Jun 8, 2011 at 11:01 AM, Nalin Makar <[hidden email]> wrote:
Cedric,

Isn't this the same as the behavior controlled using "configfailurepolicy" option?

I was under the impression that configfailurepolicy was not looking at instances, it was still an "all-or-nothing" skip or continue, am I wrong?

-- 
Cédric

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.
Reply | Threaded
Open this post in threaded view
|

Re: DataProvider, test dependency (but for the same data)

Cédric Beust ♔-2
In reply to this post by Cédric Beust ♔-2
Hi François,

Your test only ever creates one instance, so it's not applicable to this discussion. Having said that, I just noticed that something is indeed not working as expected, I'm looking into it.

-- 
Cédric




2011/6/8 Cédric Beust ♔ <[hidden email]>
Hi Julia,

You're coming across a behavior that has been the standard in TestNG but that I've been meaning to change for a while. The only thing that was holding me back is the fact that it might break existing tests, but I think this behavior makes more sense so I'm just going to go ahead an implement it.

The short answer to your question: please try the beta (http://testng.org/beta ), it should fix your problem.

The longer answer now...

So far, TestNG's dependency mechanism was acting at the class level instead of the instance level. In other words, if method f2() depends on f1() and f1() fails, then f2() is going to be skipped across all the instances. This seemed reasonable to me at the time but more recently, I started realizing that just because a method such as f1() fails on a certain instance doesn't mean it will fail on other instances, so it shouldn't take down all the other classes on the other instances in the process.

So, what could break with this change?

If you are relying on the current behavior, you might start seeing tests that used to fail start passing. Fundamentally, I think these new tests suddenly passing are not lying to you: all the methods and the methods they depend on are successful, so there is little reason to flag any failure there.

I would love to get some feedback on this change, just to make sure I didn't miss anything.

Thanks.

-- 
Cédric




On Wed, Jun 8, 2011 at 4:41 AM, Julia S.S <[hidden email]> wrote:
I'm just starting with TestNG and Selenium

I want to perform a registration test in a collection of webpages.
I've writen a Register class with the following methods

@Test( dataProvider = "WebSites", groups = "launchSite")
public void launchSite(WeboSite webSite)

@Test( dataProvider = "WebSites", groups = "goToRegPage",
      dependsOnGroups = "launchSite")
public void openRegisterPage(WeboSite webSite)

@Test( dataProvider = "WebSites", groups = "register",
  dependsOnGroups = "goToRegPage")
public void enterRegistrationData(BingoSite bingoSite)

So I've made each test dependant on the previous (obviously if you
cannot enter the registration page you cannot register)

What I want now is each webpage result be independant from the
others.

Now openRegisterPage is run for every webpage and if it fails in one
website enterRegistrationData is not run for any of them but should be
run for the rest of websites that passed openRegisterPage

What is the correct way to do this?

Thanks a lot

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.



--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.
Reply | Threaded
Open this post in threaded view
|

Re: DataProvider, test dependency (but for the same data)

François Reynaud
cool, no refactoring :)

2011/6/8 Cédric Beust ♔ <[hidden email]>
Hi François,

Your test only ever creates one instance, so it's not applicable to this discussion. Having said that, I just noticed that something is indeed not working as expected, I'm looking into it.

-- 
Cédric




2011/6/8 Cédric Beust ♔ <[hidden email]>
Hi Julia,

You're coming across a behavior that has been the standard in TestNG but that I've been meaning to change for a while. The only thing that was holding me back is the fact that it might break existing tests, but I think this behavior makes more sense so I'm just going to go ahead an implement it.

The short answer to your question: please try the beta (http://testng.org/beta ), it should fix your problem.

The longer answer now...

So far, TestNG's dependency mechanism was acting at the class level instead of the instance level. In other words, if method f2() depends on f1() and f1() fails, then f2() is going to be skipped across all the instances. This seemed reasonable to me at the time but more recently, I started realizing that just because a method such as f1() fails on a certain instance doesn't mean it will fail on other instances, so it shouldn't take down all the other classes on the other instances in the process.

So, what could break with this change?

If you are relying on the current behavior, you might start seeing tests that used to fail start passing. Fundamentally, I think these new tests suddenly passing are not lying to you: all the methods and the methods they depend on are successful, so there is little reason to flag any failure there.

I would love to get some feedback on this change, just to make sure I didn't miss anything.

Thanks.

-- 
Cédric




On Wed, Jun 8, 2011 at 4:41 AM, Julia S.S <[hidden email]> wrote:
I'm just starting with TestNG and Selenium

I want to perform a registration test in a collection of webpages.
I've writen a Register class with the following methods

@Test( dataProvider = "WebSites", groups = "launchSite")
public void launchSite(WeboSite webSite)

@Test( dataProvider = "WebSites", groups = "goToRegPage",
      dependsOnGroups = "launchSite")
public void openRegisterPage(WeboSite webSite)

@Test( dataProvider = "WebSites", groups = "register",
  dependsOnGroups = "goToRegPage")
public void enterRegistrationData(BingoSite bingoSite)

So I've made each test dependant on the previous (obviously if you
cannot enter the registration page you cannot register)

What I want now is each webpage result be independant from the
others.

Now openRegisterPage is run for every webpage and if it fails in one
website enterRegistrationData is not run for any of them but should be
run for the rest of websites that passed openRegisterPage

What is the correct way to do this?

Thanks a lot

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.



--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.
Reply | Threaded
Open this post in threaded view
|

Re: DataProvider, test dependency (but for the same data)

hithwen
In reply to this post by Cédric Beust ♔-2
Hi Cedric,
I agree with you about new results wont be lying.
I've downloaded beta version of testNG and trying it. The only thing is that I have to keep a Map of dataProvider items and selenium instances, otherwhise, group2 tests will use last instance of group1 tests.

Any better way to do this?

Thanks,
Julia

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.
Reply | Threaded
Open this post in threaded view
|

Re: DataProvider, test dependency (but for the same data)

hithwen
I mean, Ideally for me all tests for one instance of dataprovider would run before changing instance.

2011/6/9 Julia S.S. <[hidden email]>
Hi Cedric,
I agree with you about new results wont be lying.
I've downloaded beta version of testNG and trying it. The only thing is that I have to keep a Map of dataProvider items and selenium instances, otherwhise, group2 tests will use last instance of group1 tests.

Any better way to do this?

Thanks,
Julia

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.
Reply | Threaded
Open this post in threaded view
|

Re: DataProvider, test dependency (but for the same data)

Tamás Kende
Hi all,

I'm facing with this issue for a while, and I created a small test to
reproduce it. I've tried it with the latest beta, but I see no
difference in the results. What did I wrong?

Here is my script:


package org.testngdebug3;

import org.testng.annotations.DataProvider;
import org.testng.annotations.Factory;
import org.testng.annotations.Test;

/**
 * Question: is it possible to prevent the login and modify method
skip
 * if other registration method fails?
 * The current result is:
 * PASSED: register on instance pass
        FAILED: register on instance fail
        SKIPPED: login
        SKIPPED: modify

 * and I expect something like this:
 * PASSED: register on instance pass
  PASSED: login on instance pass
  PASSED: modify on instance pass
        FAILED: register on instance fail
        SKIPPED: login on instance fail
        SKIPPED: modify on instance fail

 * @author tkende
 *
 */
@Test(threadPoolSize=2)
public class FailDependency {

        String setting;

        @Factory(dataProvider="dataProv")
        public FailDependency(String setting){
                this.setting=setting;
        }

        @DataProvider
        public Object[][] dataProv(){
                return new Object[][]{
                        new Object[]{"pass"},
                        new Object[]{"fail"}
                };
        }

        @Test
        public void register(){
                if(setting.equalsIgnoreCase("pass")){
                        return;
                }
                else{
                        throw new AssertionError();
                }
        }

        @Test(dependsOnMethods={"register"})
        public void login(){
                return;
        }

        @Test(dependsOnMethods={"login"})
        public void modify(){
                return;
        }

        @Override
        public String toString() {
                return setting;
        }
}

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.

Reply | Threaded
Open this post in threaded view
|

Re: Re: DataProvider, test dependency (but for the same data)

Cédric Beust ♔-2
Hi Tamás,

Like I said previously, managing failures at the instance level is a tricky problem given TestNG's current architecture, but I'm experimenting with a few solutions. No promise they'll go anywhere, though, but I agree it would be nice to get this to work.

-- 
Cédric




On Thu, Jun 9, 2011 at 10:52 AM, Tamás Kende <[hidden email]> wrote:
Hi all,

I'm facing with this issue for a while, and I created a small test to
reproduce it. I've tried it with the latest beta, but I see no
difference in the results. What did I wrong?

Here is my script:


package org.testngdebug3;

import org.testng.annotations.DataProvider;
import org.testng.annotations.Factory;
import org.testng.annotations.Test;

/**
 * Question: is it possible to prevent the login and modify method
skip
 * if other registration method fails?
 * The current result is:
 *      PASSED: register on instance pass
       FAILED: register on instance fail
       SKIPPED: login
       SKIPPED: modify

 * and I expect something like this:
 *      PASSED: register on instance pass
       PASSED: login on instance pass
       PASSED: modify on instance pass
       FAILED: register on instance fail
       SKIPPED: login on instance fail
       SKIPPED: modify on instance fail

 * @author tkende
 *
 */
@Test(threadPoolSize=2)
public class FailDependency {

       String setting;

       @Factory(dataProvider="dataProv")
       public FailDependency(String setting){
               this.setting=setting;
       }

       @DataProvider
       public Object[][] dataProv(){
               return new Object[][]{
                       new Object[]{"pass"},
                       new Object[]{"fail"}
               };
       }

       @Test
       public void register(){
               if(setting.equalsIgnoreCase("pass")){
                       return;
               }
               else{
                       throw new AssertionError();
               }
       }

       @Test(dependsOnMethods={"register"})
       public void login(){
               return;
       }

       @Test(dependsOnMethods={"login"})
       public void modify(){
               return;
       }

       @Override
       public String toString() {
               return setting;
       }
}

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.


--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.
Reply | Threaded
Open this post in threaded view
|

Re: DataProvider, test dependency (but for the same data)

Tamás Kende
Hi Cedric,

sorry, an oversight... I thought the beta will fix it. :)

On jún. 9, 20:37, Cédric Beust ♔ <[hidden email]> wrote:

> Hi Tamás,
>
> Like I said previously, managing failures at the instance level is a tricky
> problem given TestNG's current architecture, but I'm experimenting with a
> few solutions. No promise they'll go anywhere, though, but I agree it would
> be nice to get this to work.
>
> --
> Cédric
>
>
>
>
>
>
>
> On Thu, Jun 9, 2011 at 10:52 AM, Tamás Kende <[hidden email]> wrote:
> > Hi all,
>
> > I'm facing with this issue for a while, and I created a small test to
> > reproduce it. I've tried it with the latest beta, but I see no
> > difference in the results. What did I wrong?
>
> > Here is my script:
>
> > package org.testngdebug3;
>
> > import org.testng.annotations.DataProvider;
> > import org.testng.annotations.Factory;
> > import org.testng.annotations.Test;
>
> > /**
> >  * Question: is it possible to prevent the login and modify method
> > skip
> >  * if other registration method fails?
> >  * The current result is:
> >  *      PASSED: register on instance pass
> >        FAILED: register on instance fail
> >        SKIPPED: login
> >        SKIPPED: modify
>
> >  * and I expect something like this:
> >  *      PASSED: register on instance pass
> >        PASSED: login on instance pass
> >        PASSED: modify on instance pass
> >        FAILED: register on instance fail
> >        SKIPPED: login on instance fail
> >        SKIPPED: modify on instance fail
>
> >  * @author tkende
> >  *
> >  */
> > @Test(threadPoolSize=2)
> > public class FailDependency {
>
> >        String setting;
>
> >        @Factory(dataProvider="dataProv")
> >        public FailDependency(String setting){
> >                this.setting=setting;
> >        }
>
> >        @DataProvider
> >        public Object[][] dataProv(){
> >                return new Object[][]{
> >                        new Object[]{"pass"},
> >                        new Object[]{"fail"}
> >                };
> >        }
>
> >        @Test
> >        public void register(){
> >                if(setting.equalsIgnoreCase("pass")){
> >                        return;
> >                }
> >                else{
> >                        throw new AssertionError();
> >                }
> >        }
>
> >        @Test(dependsOnMethods={"register"})
> >        public void login(){
> >                return;
> >        }
>
> >        @Test(dependsOnMethods={"login"})
> >        public void modify(){
> >                return;
> >        }
>
> >        @Override
> >        public String toString() {
> >                return setting;
> >        }
> > }
>
> > --
> > You received this message because you are subscribed to the Google Groups
> > "testng-users" group.
> > To post to this group, send email to [hidden email].
> > To unsubscribe from this group, send email to
> > [hidden email].
> > For more options, visit this group at
> >http://groups.google.com/group/testng-users?hl=en.

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.