dependsonmethods not sequencing properly(jumping between classes)

classic Classic list List threaded Threaded
27 messages Options
12
Reply | Threaded
Open this post in threaded view
|

dependsonmethods not sequencing properly(jumping between classes)

goodtest123

Hi,
Hope some one will help me on this.

I have two classes: ClassA and ClassB with the following methods:
ClassA:
TestA1
TestA2 dependsonmethod TestA1
TestA3 dependsOnMethod TestA2
TestA4 dependsOnMethod TestA3

ClassB:
TestB1
TestB2 dependsonmethod TestB1
TestB3 dependsonmethod TestB2
TestB4 dependsonmethod TestB3

Now, when I run TestNG, it sequences the tests like:


TestA1
TestA2
TestB1 (jumped to ClassB's first method)
TestB2
TestB3
TestA3 (jumped back to ClassA's method)
TestA4
TestB4

Can anyone please point out why?


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: dependsonmethods not sequencing properly(jumping between classes)

Cédric Beust ♔
The only guarantee is ordering, not atomicity.  If you want to avoid that interleaving, put your test classes in separate <test> tags.

--
Cedric


On Fri, Jun 27, 2008 at 10:05 AM, goodtest123 <[hidden email]> wrote:

Hi,
Hope some one will help me on this.

I have two classes: ClassA and ClassB with the following methods:
ClassA:
TestA1
TestA2 dependsonmethod TestA1
TestA3 dependsOnMethod TestA2
TestA4 dependsOnMethod TestA3

ClassB:
TestB1
TestB2 dependsonmethod TestB1
TestB3 dependsonmethod TestB2
TestB4 dependsonmethod TestB3

Now, when I run TestNG, it sequences the tests like:


TestA1
TestA2
TestB1 (jumped to ClassB's first method)
TestB2
TestB3
TestA3 (jumped back to ClassA's method)
TestA4
TestB4

Can anyone please point out why?






--
Cédric

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: dependsonmethods not sequencing properly(jumping between classes)

goodtest123

Thanks Cedric for the reply.

I have like 30 test-classes each with 4 or 5 tests and have
dependsOnMethod on them. testng.xml would be unmanagable if I have to
add <test>  for each one of them. Wondering if there is any other
alternative solution or if it is possible to use method interceptors
to order this?

On Jun 27, 1:07 am, Cédric Beust ♔ <[hidden email]> wrote:

> The only guarantee is ordering, not atomicity.  If you want to avoid that
> interleaving, put your test classes in separate <test> tags.
>
> --
> Cedric
>
>
>
> On Fri, Jun 27, 2008 at 10:05 AM, goodtest123 <[hidden email]> wrote:
>
> > Hi,
> > Hope some one will help me on this.
>
> > I have two classes: ClassA and ClassB with the following methods:
> > ClassA:
> > TestA1
> > TestA2 dependsonmethod TestA1
> > TestA3 dependsOnMethod TestA2
> > TestA4 dependsOnMethod TestA3
>
> > ClassB:
> > TestB1
> > TestB2 dependsonmethod TestB1
> > TestB3 dependsonmethod TestB2
> > TestB4 dependsonmethod TestB3
>
> > Now, when I run TestNG, it sequences the tests like:
>
> > TestA1
> > TestA2
> > TestB1 (jumped to ClassB's first method)
> > TestB2
> > TestB3
> > TestA3 (jumped back to ClassA's method)
> > TestA4
> > TestB4
>
> > Can anyone please point out why?
>
> --
> Cédric
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: dependsonmethods not sequencing properly(jumping between classes)

goodtest123

ah, tried to write a method interceptor but its only for methods that
are run randomly :(



On Jun 27, 7:56 am, goodtest123 <[hidden email]> wrote:

> Thanks Cedric for the reply.
>
> I have like 30 test-classes each with 4 or 5 tests and have
> dependsOnMethod on them. testng.xml would be unmanagable if I have to
> add <test>  for each one of them. Wondering if there is any other
> alternative solution or if it is possible to use method interceptors
> to order this?
>
> On Jun 27, 1:07 am, Cédric Beust ♔ <[hidden email]> wrote:
>
> > The only guarantee is ordering, not atomicity.  If you want to avoid that
> > interleaving, put your test classes in separate <test> tags.
>
> > --
> > Cedric
>
> > On Fri, Jun 27, 2008 at 10:05 AM, goodtest123 <[hidden email]> wrote:
>
> > > Hi,
> > > Hope some one will help me on this.
>
> > > I have two classes: ClassA and ClassB with the following methods:
> > > ClassA:
> > > TestA1
> > > TestA2 dependsonmethod TestA1
> > > TestA3 dependsOnMethod TestA2
> > > TestA4 dependsOnMethod TestA3
>
> > > ClassB:
> > > TestB1
> > > TestB2 dependsonmethod TestB1
> > > TestB3 dependsonmethod TestB2
> > > TestB4 dependsonmethod TestB3
>
> > > Now, when I run TestNG, it sequences the tests like:
>
> > > TestA1
> > > TestA2
> > > TestB1 (jumped to ClassB's first method)
> > > TestB2
> > > TestB3
> > > TestA3 (jumped back to ClassA's method)
> > > TestA4
> > > TestB4
>
> > > Can anyone please point out why?
>
> > --
> > Cédric
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: dependsonmethods not sequencing properly(jumping between classes)

Cédric Beust ♔


On Fri, Jun 27, 2008 at 10:19 PM, goodtest123 <[hidden email]> wrote:

ah, tried to write a method interceptor but its only for methods that
are run randomly :(

Yes, what would be the point of setting an order with dependencies and then change that order with method interceptors?

Since TestNG's native dependsOnMethods/Groups doesn't suit your needs, I suggest you stop using them and use an IMethodInterceptor, which will give you full control on the ordering of your methods.

--
Cédric

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: dependsonmethods not sequencing properly(jumping between classes)

goodtest123

thanks Cedric for the reply. I went with having <test> tags as your
earlier suggestion. It turns out its not soo bad.


On Jun 27, 11:44 pm, Cédric Beust ♔ <[hidden email]> wrote:

> On Fri, Jun 27, 2008 at 10:19 PM, goodtest123 <[hidden email]> wrote:
>
> > ah, tried to write a method interceptor but its only for methods that
> > are run randomly :(
>
> Yes, what would be the point of setting an order with dependencies and then
> change that order with method interceptors?
>
> Since TestNG's native dependsOnMethods/Groups doesn't suit your needs, I
> suggest you stop using them and use an IMethodInterceptor, which will give
> you full control on the ordering of your methods.
>
> --
> Cédric
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: dependsonmethods not sequencing properly(jumping between classes)

viet nguyen
I'm too trying to solve this problem.  What did you mean by adding <test> tags?
something like this:

<test>
  <classes>
     <class name="test1"/>
  </classes>
</test

<test>
  <classes>
     <class name="test2"/>
  </classes>
</test




<quote author="goodtest123">

thanks Cedric for the reply. I went with having <test> tags as your
earlier suggestion. It turns out its not soo bad.

Reply | Threaded
Open this post in threaded view
|

Re: dependsonmethods not sequencing properly(jumping between classes)

yanan
Hi,

If you use <test> tags, how to add <groups> tag into testng.xml, add <groups> into every <test> tag?

thanks,
Yanan




viet nguyen wrote
I'm too trying to solve this problem.  What did you mean by adding <test> tags?
something like this:

<test>
  <classes>
     <class name="test1"/>
  </classes>
</test

<test>
  <classes>
     <class name="test2"/>
  </classes>
</test




<quote author="goodtest123">

thanks Cedric for the reply. I went with having <test> tags as your
earlier suggestion. It turns out its not soo bad.
Reply | Threaded
Open this post in threaded view
|

Re: dependsonmethods not sequencing properly(jumping between classes)

Cédric Beust ♔
Hi Yanan,

You specify the groups to run in the <include><groups> tag, not in <test>.

-- 
Cedric


On Thu, Jan 8, 2009 at 5:30 AM, yanan <[hidden email]> wrote:


Hi,

If you use <test> tags, how to add <groups> tag into testng.xml, add
<groups> into every <test> tag?

thanks,
Yanan





I'm too trying to solve this problem.  What did you mean by adding <test>
tags?
something like this:

<test>
 <classes>
    <class name="test1"/>
 </classes>
</test

<test>
 <classes>
    <class name="test2"/>
 </classes>
</test





goodtest123 wrote:
>
>
> thanks Cedric for the reply. I went with having <test> tags as your
> earlier suggestion. It turns out its not soo bad.
>
>
>

--
View this message in context: http://www.nabble.com/dependsonmethods-not-sequencing-properly%28jumping-between-classes%29-tp18150149p21351802.html
Sent from the testng-users mailing list archive at Nabble.com.






--
Cédric



--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: dependsonmethods not sequencing properly(jumping between classes)

yanan
It doesn’t work if I specify groups out side the <test> tag like:
<groups>
<run>
        <exclude name="broken" />
        </run>
</groups>

<test> 
  <classes> 
     <class name="test1"/> 
  </classes> 
</test>

<test> 
  <classes> 
     <class name="test2"/> 
  </classes> 
</test>

But it works if I put the groups for each <test> tag like:
<test> 
<groups>
       <run>
        <exclude name="broken" />
       </run>
</groups>
  <classes> 
     <class name="test1"/> 
  </classes> 
</test>

<test> 
<groups>
       <run>
        <exclude name="broken" />
       </run>
</groups>
  <classes> 
     <class name="test2"/> 
  </classes> 
</test>

Is there any way to put the groups for the test <suite> not every <test>?

Thanks,
Yanan

Cédric Beust ♔ wrote
Hi Yanan,
You specify the groups to run in the <include><groups> tag, not in <test>.

--
Cedric


On Thu, Jan 8, 2009 at 5:30 AM, yanan <yanan58@hotmail.com> wrote:

>
>
> Hi,
>
> If you use <test> tags, how to add <groups> tag into testng.xml, add
> <groups> into every <test> tag?
>
> thanks,
> Yanan
>
>
>
>
>
> I'm too trying to solve this problem.  What did you mean by adding <test>
> tags?
> something like this:
>
> <test>
>  <classes>
>     <class name="test1"/>
>  </classes>
> </test
>
> <test>
>  <classes>
>     <class name="test2"/>
>  </classes>
> </test
>
>
>
>
>
> goodtest123 wrote:
> >
> >
> > thanks Cedric for the reply. I went with having <test> tags as your
> > earlier suggestion. It turns out its not soo bad.
> >
> >
> >
>
> --
> View this message in context:
> http://www.nabble.com/dependsonmethods-not-sequencing-properly%28jumping-between-classes%29-tp18150149p21351802.html
> Sent from the testng-users mailing list archive at Nabble.com.
>
>
> >
>


--
***Cédric
*

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to testng-users@googlegroups.com
To unsubscribe from this group, send email to testng-users+unsubscribe@googlegroups.com
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en
-~----------~----~----~----~------~----~------~--~---
Reply | Threaded
Open this post in threaded view
|

Re: dependsonmethods not sequencing properly(jumping between classes)

Endre Stølsvik-9
In reply to this post by Cédric Beust ♔

I also just got this exact problem. All of a sudden tests fail when
they are run all together, and not when alone. I suddenly realize that
the test-runner _interleaves_ different tests from different classes!

The scenario is like this: I have a database that I take up, clear and
populate, do testing on, then take down. Often I have several
different @Tests: add some entities, take down the system (while
keeping the database populated), take it up again, see if they can be
read properly in. This I do in several classes. I use either @[Before|
After]Class to do the takeup/clear/populate/takedown, or/and use
ordering. Since several of the tests do different things with this
test database, I obviously need the "class sequence/atomicity" to hold
- two different test-sequences running on the same database "at the
same time" (due to the interleaving) is no-go.

Starting to use an xml file, when I went over to Guice due to Spring's
XML hell, and am using annotations all over for testing, is 100% not
an option in any way whatsoever!! ... XML?? I even find it annoying
that there aren't any option to just run all tests in the project,
like it was with JUnit (which was the obvious way that testing in an
IDE should work, IMU). Luckily, this was overcome with the following
small xml file, and then I've never had to see any XML configuration
for my project development again.

<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="Example">
        <test verbose="2" name="All tests in com.example" annotations="JDK5">
                <packages>
                        <package name="com.example.*"></package>
                </packages>
        </test>
</suite>

Why is it like this? Why would one want methods from different classes
to be interleaved absolutely randomly? I thought that one class was
"one suite". Actually, I thought that this logic was a bit what was
the difference between TestNG and JUnit: That TestNG was better suited
for functional testing than JUnit, since the latter is die-hard in its
approach that one method is one stand-alone test.

What are my options? Will this possibly be changed? Why is it a good
idea to have it like this? (For parallelizing, I'd say that this would
have to be configured specifically - not the other way round. Like
@Selfcontained or @Parallelizable or something like that)

Kind regards,
Endre.


On Jan 8, 6:01 pm, Cédric Beust ♔ <[hidden email]> wrote:

> Hi Yanan,
> You specify the groups to run in the <include><groups> tag, not in <test>.
>
> --
> Cedric
>
>
>
> On Thu, Jan 8, 2009 at 5:30 AM, yanan <[hidden email]> wrote:
>
> > Hi,
>
> > If you use <test> tags, how to add <groups> tag into testng.xml, add
> > <groups> into every <test> tag?
>
> > thanks,
> > Yanan
>
> > I'm too trying to solve this problem.  What did you mean by adding <test>
> > tags?
> > something like this:
>
> > <test>
> >  <classes>
> >     <class name="test1"/>
> >  </classes>
> > </test
>
> > <test>
> >  <classes>
> >     <class name="test2"/>
> >  </classes>
> > </test
>
> > goodtest123 wrote:
>
> > > thanks Cedric for the reply. I went with having <test> tags as your
> > > earlier suggestion. It turns out its not soo bad.
>
> > --
> > View this message in context:
> >http://www.nabble.com/dependsonmethods-not-sequencing-properly%28jump...
> > Sent from the testng-users mailing list archive at Nabble.com.
>
> --
> ***Cédric
> *

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: dependsonmethods not sequencing properly(jumping between classes)

Cédric Beust ♔


2009/1/21 Endre Stølsvik <[hidden email]>
I also just got this exact problem. All of a sudden tests fail when
they are run all together, and not when alone. I suddenly realize that
the test-runner _interleaves_ different tests from different classes!

Tests in the same <test> tag will always be run interleaved, if you want them separate, you should probably put them in different <test> tags...

--
Cédric



--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: dependsonmethods not sequencing properly(jumping between classes)

Endre Stølsvik-9

On Jan 21, 5:41 pm, Cédric Beust ♔ <[hidden email]> wrote:
> 2009/1/21 Endre Stølsvik <[hidden email]>
>
> > I also just got this exact problem. All of a sudden tests fail when
> > they are run all together, and not when alone. I suddenly realize that
> > the test-runner _interleaves_ different tests from different classes!
>
> Tests in the same <test> tag will always be run interleaved, if you want
> them separate, you should probably put them in different <test> tags...

What is the point of this? There must be a reason, I assume?

I do not want to have anything to do with XML as configuration when
there exists annotations and pure java. Thus, the "put them in another
<test> tag" is not a solution. Are there no other options? What about
some programmatic (java) based stuff?

Endre
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: dependsonmethods not sequencing properly(jumping between classes)

Endre Stølsvik-9
In reply to this post by Cédric Beust ♔

Is there no other option to have this fixed? I am still wondering why
this would be a good thing - why would one want, when specifically not
having specified that things can be parallelized, tests from one class
to be interleaved with tests from other classes? I even see that they
are run on another thread?

Is there no way to hinder this from happening, bar using a
configuration file?

Is there any interest in having this bug fixed?

Endre.

On Jan 21, 5:41 pm, Cédric Beust ♔ <[hidden email]> wrote:

> 2009/1/21 Endre Stølsvik <[hidden email]>
>
> > I also just got this exact problem. All of a sudden tests fail when
> > they are run all together, and not when alone. I suddenly realize that
> > the test-runner _interleaves_ different tests from different classes!
>
> Tests in the same <test> tag will always be run interleaved, if you want
> them separate, you should probably put them in different <test> tags...
>
> --
> ***Cédric
> *
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: dependsonmethods not sequencing properly(jumping between classes)

Cédric Beust ♔
Well, TestNG already makes its best to try to keep methods within class bounds and so far, it looks like its heuristics was good enough, but you seem to have encountered an example that contradicts this.

Would you mind sending me a zip file containing a few simple test classes along with a testng.xml that reproduces the problem?  I'll take a look. 

--
Cedric


2009/1/30 Endre Stølsvik <[hidden email]>

Is there no other option to have this fixed? I am still wondering why
this would be a good thing - why would one want, when specifically not
having specified that things can be parallelized, tests from one class
to be interleaved with tests from other classes? I even see that they
are run on another thread?

Is there no way to hinder this from happening, bar using a
configuration file?

Is there any interest in having this bug fixed?

Endre.

On Jan 21, 5:41 pm, Cédric Beust ♔ <[hidden email]> wrote:
> 2009/1/21 Endre Stølsvik <[hidden email]>
>
> > I also just got this exact problem. All of a sudden tests fail when
> > they are run all together, and not when alone. I suddenly realize that
> > the test-runner _interleaves_ different tests from different classes!
>
> Tests in the same <test> tag will always be run interleaved, if you want
> them separate, you should probably put them in different <test> tags...
>
> --
> ***Cédric
> *




--
Cédric



--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: dependsonmethods not sequencing properly(jumping between classes)

Endre Stølsvik-9

Thanks!

Here's the xml file. It is basically just to overcome the lack of a
"run all tests in project" option in the Eclipse runner:

<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="All tests in project">
        <test verbose="10" name="All tests com.*" annotations="JDK5">
                <packages>
                        <package name="com.*"></package>
                </packages>
        </test>
</suite>

I don't have a specific test case, as it just suddenly appeared. I
added a @Test-annotated method to an existing class, and all of a
sudden this method was run long after the first @Test method of that
same class, with a whole other class's @Test-method running inbetween.
Very similar to OP's report. This interleaving might actually have
happened before, but this time it caused problems as the other
classes' methods were also employing the test database (clearing it),
thus making the out-of-sequence run into a problem.

Also, I believe I saw that this method was run on some "Test-NG-
runner" thread, instead of "main", which other tests run on. It was
run in sequence, not being parallelized, but on another thread
nevertheless.

I don't quite see what heuristics would be needed to ensure @Tests
from the same class is runnning in sequence, rather than wildly out of
sequence?

(And I fail to see why a configuration file would be needed in any
case for this simple setup. I actually don't see why not all
configuration shouldn't be done with annotations, and possibly
"configuration classes" much like Guice does it.)

Thanks,
Endre.

On Jan 30, 4:57 pm, Cédric Beust ♔ <[hidden email]> wrote:

> Well, TestNG already makes its best to try to keep methods within class
> bounds and so far, it looks like its heuristics was good enough, but you
> seem to have encountered an example that contradicts this.
>
> Would you mind sending me a zip file containing a few simple test classes
> along with a testng.xml that reproduces the problem?  I'll take a look.
>
> --
> Cedric
>
> 2009/1/30 Endre Stølsvik <[hidden email]>
>
>
>
>
>
> > Is there no other option to have this fixed? I am still wondering why
> > this would be a good thing - why would one want, when specifically not
> > having specified that things can be parallelized, tests from one class
> > to be interleaved with tests from other classes? I even see that they
> > are run on another thread?
>
> > Is there no way to hinder this from happening, bar using a
> > configuration file?
>
> > Is there any interest in having this bug fixed?
>
> > Endre.
>
> > On Jan 21, 5:41 pm, Cédric Beust ♔ <[hidden email]> wrote:
> > > 2009/1/21 Endre Stølsvik <[hidden email]>
>
> > > > I also just got this exact problem. All of a sudden tests fail when
> > > > they are run all together, and not when alone. I suddenly realize that
> > > > the test-runner _interleaves_ different tests from different classes!
>
> > > Tests in the same <test> tag will always be run interleaved, if you want
> > > them separate, you should probably put them in different <test> tags...
>
> > > --
> > > ***Cédric
> > > *
>
> --
> ***Cédric
> *
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: dependsonmethods not sequencing properly(jumping between classes)

Cédric Beust ♔
I tried to reproduce the problem and I can't.

I verified that if you are not using any parallelism, the methods are *always* invoked by class affinity. This is what the code does and I observe the expected behavior.  Take a look at TestRunner:121 and see for yourself that the default method interceptor will sort methods by class affinity.

I tested with three classes, A, B and C, each with three methods, a1()/a2()/a3(),...

If I completely turn off parallelism, I see:

PASSED: a3
PASSED: a2
PASSED: a1
PASSED: b3
PASSED: b2
PASSED: b1
PASSED: c1
PASSED: c2
PASSED: c3

If I specify parallel="methods", the order becomes unpredictible, which is expected:

PASSED: b3
PASSED: a3
PASSED: b1
PASSED: c1
PASSED: c2
PASSED: b2
PASSED: a1
PASSED: c3
PASSED: a2

Are you sure you are not using parallelism to run your tests?

--
Cedric



2009/1/30 Endre Stølsvik <[hidden email]>

Thanks!

Here's the xml file. It is basically just to overcome the lack of a
"run all tests in project" option in the Eclipse runner:

<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="All tests in project">
       <test verbose="10" name="All tests com.*" annotations="JDK5">
               <packages>
                       <package name="com.*"></package>
               </packages>
       </test>
</suite>

I don't have a specific test case, as it just suddenly appeared. I
added a @Test-annotated method to an existing class, and all of a
sudden this method was run long after the first @Test method of that
same class, with a whole other class's @Test-method running inbetween.
Very similar to OP's report. This interleaving might actually have
happened before, but this time it caused problems as the other
classes' methods were also employing the test database (clearing it),
thus making the out-of-sequence run into a problem.

Also, I believe I saw that this method was run on some "Test-NG-
runner" thread, instead of "main", which other tests run on. It was
run in sequence, not being parallelized, but on another thread
nevertheless.

I don't quite see what heuristics would be needed to ensure @Tests
from the same class is runnning in sequence, rather than wildly out of
sequence?

(And I fail to see why a configuration file would be needed in any
case for this simple setup. I actually don't see why not all
configuration shouldn't be done with annotations, and possibly
"configuration classes" much like Guice does it.)

Thanks,
Endre.

On Jan 30, 4:57 pm, Cédric Beust ♔ <[hidden email]> wrote:
> Well, TestNG already makes its best to try to keep methods within class
> bounds and so far, it looks like its heuristics was good enough, but you
> seem to have encountered an example that contradicts this.
>
> Would you mind sending me a zip file containing a few simple test classes
> along with a testng.xml that reproduces the problem?  I'll take a look.
>
> --
> Cedric
>
> 2009/1/30 Endre Stølsvik <[hidden email]>
>
>
>
>
>
> > Is there no other option to have this fixed? I am still wondering why
> > this would be a good thing - why would one want, when specifically not
> > having specified that things can be parallelized, tests from one class
> > to be interleaved with tests from other classes? I even see that they
> > are run on another thread?
>
> > Is there no way to hinder this from happening, bar using a
> > configuration file?
>
> > Is there any interest in having this bug fixed?
>
> > Endre.
>
> > On Jan 21, 5:41 pm, Cédric Beust ♔ <[hidden email]> wrote:
> > > 2009/1/21 Endre Stølsvik <[hidden email]>
>
> > > > I also just got this exact problem. All of a sudden tests fail when
> > > > they are run all together, and not when alone. I suddenly realize that
> > > > the test-runner _interleaves_ different tests from different classes!
>
> > > Tests in the same <test> tag will always be run interleaved, if you want
> > > them separate, you should probably put them in different <test> tags...
>
> > > --
> > > ***Cédric
> > > *
>
> --
> ***Cédric
> *




--
Cédric



--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: dependsonmethods not sequencing properly(jumping between classes)

Endre Stølsvik-9

On Fri, Jan 30, 2009 at 19:36, Cédric Beust ♔ <[hidden email]> wrote:

> Are you sure you are not using parallelism to run your tests?

Unless it is on by some default, I am sure about that - the XML file
is what you saw, and I haven't asked for parallelism any other place.

But, I can't reproduce the problem anymore. I had changed the
problematic test to simply have one test-method call the different
parts in sequence instead. When I changed it back to have separate
methods, I cannot any longer reproduce the problem - the tests do run
class-by-class. I do not remember the names of those original methods,
though - if that can have anything whatsoever to do with it. And I was
pretty sure I hadn't changed anything else..

PS: What is the sort-order for the tests - It really seems to be
completely stable utter randomness?!

Regards,
Endre.

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: dependsonmethods not sequencing properly(jumping between classes)

Cédric Beust ♔


On Mon, Feb 2, 2009 at 4:32 PM, Endre Stølsvik <[hidden email]> wrote:
PS: What is the sort-order for the tests - It really seems to be
completely stable utter randomness?!

It just depends on the reflection API.  I could impose an additional order on top of the per-class ordering already in place but I couldn't come up with one that would be better than others...

--
Cédric



--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: dependsonmethods not sequencing properly(jumping between classes)

Endre Stølsvik-9

On Tue, Feb 3, 2009 at 02:06, Cédric Beust ♔ <[hidden email]> wrote:

>
>
> On Mon, Feb 2, 2009 at 4:32 PM, Endre Stølsvik <[hidden email]> wrote:
>>
>> PS: What is the sort-order for the tests - It really seems to be
>> completely stable utter randomness?!
>
> It just depends on the reflection API.  I could impose an additional order
> on top of the per-class ordering already in place but I couldn't come up
> with one that would be better than others...

Alpha? That would at least be _some_ order, predictable even.

I was also thinking about the packages - I'd love if these also was
alpha - then the tests would be run in a predictable order, which
happens to be the order my packages are ordered in Eclipse and pretty
much any listing in the world!

Kind regards,
Endre.

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en
-~----------~----~----~----~------~----~------~--~---

12