Skipping tests with unavailable resources

classic Classic list List threaded Threaded
17 messages Options
Reply | Threaded
Open this post in threaded view
|

Skipping tests with unavailable resources

isterin

So I'm using maven with surefire plugin.  I have a requirement to
basically skip tests that are dependent on various resources that are
not available on a particular build machine.

I know that I can easily use the test dependency mechanism in TestNG
to skip tests that are dependent on a particular test that has failed.
 The problem with that is that Maven fails the build cycle if any
tests fail.

I can use testFailureIgnore, but that basically ignores all tests that
fail, but that might ignore important tests that have to pass in order
to build the application.

Can someone let me know if there is some workaround for this?

If there isn't, I propose one of two solutions...

1.  Allow some sort of a way to semi-fail the test.  Basically, the
test doesn't throw an exception, but though doesn't truly succeed,
therefore the dependencies are not run.  This can be something in the
likes of
Assert.failWithoutException("resource X not found.");
or any other API call that allows to mark the test as not succeeded,
but do not report any failures.

This can also maybe be done is a way of providing some sort of an
annotation to silence the test failure.
@Test(silentFail = true);

2.  Provide a way to stop a particular test suite from running.
Therefore in the @BeforeClass method, if some resource is found to be
not available you can instruct TestNG runner to skip the rest of the
tests in this particular suite.  The granularity of this might go
further, not sure if anyone has requirements for a more granular
mechanism of test skip control.

I'm wondering if there is an API that's already exposed in TestNG that
allows to accomplish this, though I can't seem to find it.

Ilya

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: Skipping tests with unavailable resources

Alexandru Popescu ☀

On 4/10/07, Ilya Sterin <[hidden email]> wrote:

>
> So I'm using maven with surefire plugin.  I have a requirement to
> basically skip tests that are dependent on various resources that are
> not available on a particular build machine.
>
> I know that I can easily use the test dependency mechanism in TestNG
> to skip tests that are dependent on a particular test that has failed.
>  The problem with that is that Maven fails the build cycle if any
> tests fail.
>
> I can use testFailureIgnore, but that basically ignores all tests that
> fail, but that might ignore important tests that have to pass in order
> to build the application.
>
> Can someone let me know if there is some workaround for this?
>
> If there isn't, I propose one of two solutions...
>
> 1.  Allow some sort of a way to semi-fail the test.  Basically, the
> test doesn't throw an exception, but though doesn't truly succeed,
> therefore the dependencies are not run.  This can be something in the
> likes of
> Assert.failWithoutException("resource X not found.");
> or any other API call that allows to mark the test as not succeeded,
> but do not report any failures.
>
> This can also maybe be done is a way of providing some sort of an
> annotation to silence the test failure.
> @Test(silentFail = true);
>
> 2.  Provide a way to stop a particular test suite from running.
> Therefore in the @BeforeClass method, if some resource is found to be
> not available you can instruct TestNG runner to skip the rest of the
> tests in this particular suite.  The granularity of this might go
> further, not sure if anyone has requirements for a more granular
> mechanism of test skip control.
>
> I'm wondering if there is an API that's already exposed in TestNG that
> allows to accomplish this, though I can't seem to find it.
>
> Ilya
>

Pretty interesting scenario. Have you looked at AnnotationTransformer?
I am thinking that you can check the resource existence in there, and
considering that all @Test will pass through that point you will be
able to disable them.

Another approach that may be even cleaner is to use groups (but this
would require to know upfront what resources are/are not available).

If the resource tests are quite simple, then yet another approach is
method selectors with script.

Please let us know what do you think about these approaches.

bests,

./alex
--
.w( the_mindstorm )p.
  TestNG co-founder
EclipseTestNG Creator

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: Skipping tests with unavailable resources

Mark Derricutt
In reply to this post by isterin
Would you not be able to use groups for this?  And include/exclude different groups based upon the build machine?

Maybe have a @BeforeGroup that checks for the resources and fails if unavailble and trigger a skip?

On 4/10/07, Ilya Sterin <[hidden email]> wrote:

So I'm using maven with surefire plugin.  I have a requirement to
basically skip tests that are dependent on various resources that are
not available on a particular build machine.


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: Skipping tests with unavailable resources

isterin
In reply to this post by Alexandru Popescu ☀
Alex, the annotation transformer would work and is actually a pretty elegant solution, but how do I disable the test using the ITest interface?  There doesn't seem to be any concrete way to disable a particular test from running.  Maybe there are some weird work arounds by setting some other annotation to a dependency that doesn't exist?  Either way, it would be nice if there was a way to skip a test explicitly using the ITest interface.  Also, it would be nice to skip a group of tests, otherwise I'd have to basically verify all test names, or have a particular method naming convention.

Any ideas?

Thanks.

Ilya Sterin

On 4/10/07, Alexandru Popescu ☀ <[hidden email]> wrote:

On 4/10/07, Ilya Sterin <[hidden email]> wrote:

>
> So I'm using maven with surefire plugin.  I have a requirement to
> basically skip tests that are dependent on various resources that are
> not available on a particular build machine.
>
> I know that I can easily use the test dependency mechanism in TestNG
> to skip tests that are dependent on a particular test that has failed.
>  The problem with that is that Maven fails the build cycle if any
> tests fail.
>
> I can use testFailureIgnore, but that basically ignores all tests that
> fail, but that might ignore important tests that have to pass in order
> to build the application.
>

> Can someone let me know if there is some workaround for this?
>
> If there isn't, I propose one of two solutions...
>
> 1.  Allow some sort of a way to semi-fail the test.  Basically, the
> test doesn't throw an exception, but though doesn't truly succeed,
> therefore the dependencies are not run.  This can be something in the
> likes of
> Assert.failWithoutException("resource X not found.");
> or any other API call that allows to mark the test as not succeeded,
> but do not report any failures.
>
> This can also maybe be done is a way of providing some sort of an
> annotation to silence the test failure.
> @Test(silentFail = true);
>
> 2.  Provide a way to stop a particular test suite from running.
> Therefore in the @BeforeClass method, if some resource is found to be
> not available you can instruct TestNG runner to skip the rest of the
> tests in this particular suite.  The granularity of this might go
> further, not sure if anyone has requirements for a more granular
> mechanism of test skip control.
>
> I'm wondering if there is an API that's already exposed in TestNG that
> allows to accomplish this, though I can't seem to find it.
>
> Ilya
>

Pretty interesting scenario. Have you looked at AnnotationTransformer?
I am thinking that you can check the resource existence in there, and
considering that all @Test will pass through that point you will be
able to disable them.

Another approach that may be even cleaner is to use groups (but this
would require to know upfront what resources are/are not available).

If the resource tests are quite simple, then yet another approach is
method selectors with script.

Please let us know what do you think about these approaches.

bests,

./alex
--
.w( the_mindstorm )p.
  TestNG co-founder
EclipseTestNG Creator





--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: Skipping tests with unavailable resources

isterin
Ah, I guess I can use the setInvocationCount :-)  Sorry...

Ilya

On 4/10/07, Ilya Sterin <[hidden email]> wrote:

> Alex, the annotation transformer would work and is actually a pretty elegant
> solution, but how do I disable the test using the ITest interface?  There
> doesn't seem to be any concrete way to disable a particular test from
> running.  Maybe there are some weird work arounds by setting some other
> annotation to a dependency that doesn't exist?  Either way, it would be nice
> if there was a way to skip a test explicitly using the ITest interface.
> Also, it would be nice to skip a group of tests, otherwise I'd have to
> basically verify all test names, or have a particular method naming
> convention.
>
> Any ideas?
>
> Thanks.
>
> Ilya Sterin
>
>
> On 4/10/07, Alexandru Popescu ☀
> <[hidden email] > wrote:
> >
> > On 4/10/07, Ilya Sterin <[hidden email] > wrote:
> > >
> > > So I'm using maven with surefire plugin.  I have a requirement to
> > > basically skip tests that are dependent on various resources that are
> > > not available on a particular build machine.
> > >
> > > I know that I can easily use the test dependency mechanism in TestNG
> > > to skip tests that are dependent on a particular test that has failed.
> > >  The problem with that is that Maven fails the build cycle if any
> > > tests fail.
> > >
> > > I can use testFailureIgnore, but that basically ignores all tests that
> > > fail, but that might ignore important tests that have to pass in order
> > > to build the application.
> > >
> > > Can someone let me know if there is some workaround for this?
> > >
> > > If there isn't, I propose one of two solutions...
> > >
> > > 1.  Allow some sort of a way to semi-fail the test.  Basically, the
> > > test doesn't throw an exception, but though doesn't truly succeed,
> > > therefore the dependencies are not run.  This can be something in the
> > > likes of
> > > Assert.failWithoutException("resource X not found.");
> > > or any other API call that allows to mark the test as not succeeded,
> > > but do not report any failures.
> > >
> > > This can also maybe be done is a way of providing some sort of an
> > > annotation to silence the test failure.
> > > @Test(silentFail = true);
> > >
> > > 2.  Provide a way to stop a particular test suite from running.
> > > Therefore in the @BeforeClass method, if some resource is found to be
> > > not available you can instruct TestNG runner to skip the rest of the
> > > tests in this particular suite.  The granularity of this might go
> > > further, not sure if anyone has requirements for a more granular
> > > mechanism of test skip control.
> > >
> > > I'm wondering if there is an API that's already exposed in TestNG that
> > > allows to accomplish this, though I can't seem to find it.
> > >
> > > Ilya
> > >
> >
> > Pretty interesting scenario. Have you looked at AnnotationTransformer?
> > I am thinking that you can check the resource existence in there, and
> > considering that all @Test will pass through that point you will be
> > able to disable them.
> >
> > Another approach that may be even cleaner is to use groups (but this
> > would require to know upfront what resources are/are not available).
> >
> > If the resource tests are quite simple, then yet another approach is
> > method selectors with script.
> >
> > Please let us know what do you think about these approaches.
> >
> > bests,
> >
> > ./alex
> > --
> > .w( the_mindstorm )p.
> >   TestNG co-founder
> > EclipseTestNG Creator
> >
> > > >
> >
>
>

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: Skipping tests with unavailable resources

Alexandru Popescu ☀
On 4/11/07, Ilya Sterin <[hidden email]> wrote:
> Ah, I guess I can use the setInvocationCount :-)  Sorry...
>

He he... this is be a nice idea I wasn't thinking about :-). I
initially thought that there should be a setEnabled(boolean), but I
guess this one is good enough.

bests,

./alex
--
.w( the_mindstorm )p.
  TestNG co-founder
EclipseTestNG Creator

> Ilya
>
> On 4/10/07, Ilya Sterin <[hidden email]> wrote:
> > Alex, the annotation transformer would work and is actually a pretty elegant
> > solution, but how do I disable the test using the ITest interface?  There
> > doesn't seem to be any concrete way to disable a particular test from
> > running.  Maybe there are some weird work arounds by setting some other
> > annotation to a dependency that doesn't exist?  Either way, it would be nice
> > if there was a way to skip a test explicitly using the ITest interface.
> > Also, it would be nice to skip a group of tests, otherwise I'd have to
> > basically verify all test names, or have a particular method naming
> > convention.
> >
> > Any ideas?
> >
> > Thanks.
> >
> > Ilya Sterin
> >
> >
> > On 4/10/07, Alexandru Popescu ☀
> > <[hidden email] > wrote:
> > >
> > > On 4/10/07, Ilya Sterin <[hidden email] > wrote:
> > > >
> > > > So I'm using maven with surefire plugin.  I have a requirement to
> > > > basically skip tests that are dependent on various resources that are
> > > > not available on a particular build machine.
> > > >
> > > > I know that I can easily use the test dependency mechanism in TestNG
> > > > to skip tests that are dependent on a particular test that has failed.
> > > >  The problem with that is that Maven fails the build cycle if any
> > > > tests fail.
> > > >
> > > > I can use testFailureIgnore, but that basically ignores all tests that
> > > > fail, but that might ignore important tests that have to pass in order
> > > > to build the application.
> > > >
> > > > Can someone let me know if there is some workaround for this?
> > > >
> > > > If there isn't, I propose one of two solutions...
> > > >
> > > > 1.  Allow some sort of a way to semi-fail the test.  Basically, the
> > > > test doesn't throw an exception, but though doesn't truly succeed,
> > > > therefore the dependencies are not run.  This can be something in the
> > > > likes of
> > > > Assert.failWithoutException("resource X not found.");
> > > > or any other API call that allows to mark the test as not succeeded,
> > > > but do not report any failures.
> > > >
> > > > This can also maybe be done is a way of providing some sort of an
> > > > annotation to silence the test failure.
> > > > @Test(silentFail = true);
> > > >
> > > > 2.  Provide a way to stop a particular test suite from running.
> > > > Therefore in the @BeforeClass method, if some resource is found to be
> > > > not available you can instruct TestNG runner to skip the rest of the
> > > > tests in this particular suite.  The granularity of this might go
> > > > further, not sure if anyone has requirements for a more granular
> > > > mechanism of test skip control.
> > > >
> > > > I'm wondering if there is an API that's already exposed in TestNG that
> > > > allows to accomplish this, though I can't seem to find it.
> > > >
> > > > Ilya
> > > >
> > >
> > > Pretty interesting scenario. Have you looked at AnnotationTransformer?
> > > I am thinking that you can check the resource existence in there, and
> > > considering that all @Test will pass through that point you will be
> > > able to disable them.
> > >
> > > Another approach that may be even cleaner is to use groups (but this
> > > would require to know upfront what resources are/are not available).
> > >
> > > If the resource tests are quite simple, then yet another approach is
> > > method selectors with script.
> > >
> > > Please let us know what do you think about these approaches.
> > >
> > > bests,
> > >
> > > ./alex
> > > --
> > > .w( the_mindstorm )p.
> > >   TestNG co-founder
> > > EclipseTestNG Creator
> > >
> > > > >
> > >
> >
> >
>
> >
>

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: Skipping tests with unavailable resources

Cédric Beust ♔
In reply to this post by isterin


On 4/10/07, Ilya Sterin <[hidden email]> wrote:
Ah, I guess I can use the setInvocationCount :-)  Sorry...

No need to apologize, Ilya, I hadn't thought about that.

Nice hack :-)

--
Cédric
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: Skipping tests with unavailable resources

isterin
In reply to this post by Alexandru Popescu ☀
This now works great, but when not invoked through maven.
Unfortunately with maven, I'm still stuck with testng 5.1, since any
version 5.3 and above had extensive changes that break the interface
between surefire and testng.  Either way, I followed the threads on
testng, etc..., but still don't see a solution.  I also downloaded the
latest sandbox for the maven/testng integration and compiled, but
still no luck.  After screwing around with fixing various API calls, I
thought I'd ask...

Is anyone still working on this issue?  If yes, is there some possible
ETA for this (Yeah, I know you're doing this for free:-).  If you guys
don't have the time, I'm willing maybe in the next week or so to take
a crack at this, as it's becoming more important on most of my
projects.  We use maven for all project builds (by choice:-), and I'm
stuck with TNG 5.1:-(

Ilya

On 4/10/07, Alexandru Popescu ☀ <[hidden email]> wrote:

> On 4/11/07, Ilya Sterin <[hidden email]> wrote:
> > Ah, I guess I can use the setInvocationCount :-)  Sorry...
> >
>
> He he... this is be a nice idea I wasn't thinking about :-). I
> initially thought that there should be a setEnabled(boolean), but I
> guess this one is good enough.
>
> bests,
>
> ./alex
> --
> .w( the_mindstorm )p.
>   TestNG co-founder
> EclipseTestNG Creator
>
> > Ilya
> >
> > On 4/10/07, Ilya Sterin <[hidden email]> wrote:
> > > Alex, the annotation transformer would work and is actually a pretty elegant
> > > solution, but how do I disable the test using the ITest interface?  There
> > > doesn't seem to be any concrete way to disable a particular test from
> > > running.  Maybe there are some weird work arounds by setting some other
> > > annotation to a dependency that doesn't exist?  Either way, it would be nice
> > > if there was a way to skip a test explicitly using the ITest interface.
> > > Also, it would be nice to skip a group of tests, otherwise I'd have to
> > > basically verify all test names, or have a particular method naming
> > > convention.
> > >
> > > Any ideas?
> > >
> > > Thanks.
> > >
> > > Ilya Sterin
> > >
> > >
> > > On 4/10/07, Alexandru Popescu ☀
> > > <[hidden email] > wrote:
> > > >
> > > > On 4/10/07, Ilya Sterin <[hidden email] > wrote:
> > > > >
> > > > > So I'm using maven with surefire plugin.  I have a requirement to
> > > > > basically skip tests that are dependent on various resources that are
> > > > > not available on a particular build machine.
> > > > >
> > > > > I know that I can easily use the test dependency mechanism in TestNG
> > > > > to skip tests that are dependent on a particular test that has failed.
> > > > >  The problem with that is that Maven fails the build cycle if any
> > > > > tests fail.
> > > > >
> > > > > I can use testFailureIgnore, but that basically ignores all tests that
> > > > > fail, but that might ignore important tests that have to pass in order
> > > > > to build the application.
> > > > >
> > > > > Can someone let me know if there is some workaround for this?
> > > > >
> > > > > If there isn't, I propose one of two solutions...
> > > > >
> > > > > 1.  Allow some sort of a way to semi-fail the test.  Basically, the
> > > > > test doesn't throw an exception, but though doesn't truly succeed,
> > > > > therefore the dependencies are not run.  This can be something in the
> > > > > likes of
> > > > > Assert.failWithoutException("resource X not found.");
> > > > > or any other API call that allows to mark the test as not succeeded,
> > > > > but do not report any failures.
> > > > >
> > > > > This can also maybe be done is a way of providing some sort of an
> > > > > annotation to silence the test failure.
> > > > > @Test(silentFail = true);
> > > > >
> > > > > 2.  Provide a way to stop a particular test suite from running.
> > > > > Therefore in the @BeforeClass method, if some resource is found to be
> > > > > not available you can instruct TestNG runner to skip the rest of the
> > > > > tests in this particular suite.  The granularity of this might go
> > > > > further, not sure if anyone has requirements for a more granular
> > > > > mechanism of test skip control.
> > > > >
> > > > > I'm wondering if there is an API that's already exposed in TestNG that
> > > > > allows to accomplish this, though I can't seem to find it.
> > > > >
> > > > > Ilya
> > > > >
> > > >
> > > > Pretty interesting scenario. Have you looked at AnnotationTransformer?
> > > > I am thinking that you can check the resource existence in there, and
> > > > considering that all @Test will pass through that point you will be
> > > > able to disable them.
> > > >
> > > > Another approach that may be even cleaner is to use groups (but this
> > > > would require to know upfront what resources are/are not available).
> > > >
> > > > If the resource tests are quite simple, then yet another approach is
> > > > method selectors with script.
> > > >
> > > > Please let us know what do you think about these approaches.
> > > >
> > > > bests,
> > > >
> > > > ./alex
> > > > --
> > > > .w( the_mindstorm )p.
> > > >   TestNG co-founder
> > > > EclipseTestNG Creator
> > > >
> > > > > >
> > > >
> > >
> > >
> >
> > >
> >
>
> >
>

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: Skipping tests with unavailable resources

Jessek

The latest compatible up to 5.6 (I think) version isn't in the main
trunk area right now:

http://svn.apache.org/viewvc/maven/sandbox/branches/surefire/surefire-collaboration/

I still haven't committed my local changes for arbitrary property pass
through / 5.6 future updates but the src in there should be good.

I also think the 5.5 jars were published to ibiblio this morning.

On Apr 10, 6:42 pm, "Ilya Sterin" <[hidden email]> wrote:

> This now works great, but when not invoked through maven.
> Unfortunately with maven, I'm still stuck with testng 5.1, since any
> version 5.3 and above had extensive changes that break the interface
> between surefire and testng.  Either way, I followed the threads on
> testng, etc..., but still don't see a solution.  I also downloaded the
> latest sandbox for the maven/testng integration and compiled, but
> still no luck.  After screwing around with fixing various API calls, I
> thought I'd ask...
>
> Is anyone still working on this issue?  If yes, is there some possible
> ETA for this (Yeah, I know you're doing this for free:-).  If you guys
> don't have the time, I'm willing maybe in the next week or so to take
> a crack at this, as it's becoming more important on most of my
> projects.  We use maven for all project builds (by choice:-), and I'm
> stuck with TNG 5.1:-(
>
> Ilya
>
> On 4/10/07, Alexandru Popescu ☀ <[hidden email]> wrote:
>
> > On 4/11/07, Ilya Sterin <[hidden email]> wrote:
> > > Ah, I guess I can use the setInvocationCount :-)  Sorry...
>
> > He he... this is be a nice idea I wasn't thinking about :-). I
> > initially thought that there should be a setEnabled(boolean), but I
> > guess this one is good enough.
>
> > bests,
>
> > ./alex
> > --
> > .w( the_mindstorm )p.
> >   TestNG co-founder
> > EclipseTestNG Creator
>
> > > Ilya
>
> > > On 4/10/07, Ilya Sterin <[hidden email]> wrote:
> > > > Alex, the annotation transformer would work and is actually a pretty elegant
> > > > solution, but how do I disable the test using the ITest interface?  There
> > > > doesn't seem to be any concrete way to disable a particular test from
> > > > running.  Maybe there are some weird work arounds by setting some other
> > > > annotation to a dependency that doesn't exist?  Either way, it would be nice
> > > > if there was a way to skip a test explicitly using the ITest interface.
> > > > Also, it would be nice to skip a group of tests, otherwise I'd have to
> > > > basically verify all test names, or have a particular method naming
> > > > convention.
>
> > > > Any ideas?
>
> > > > Thanks.
>
> > > > Ilya Sterin
>
> > > > On 4/10/07, Alexandru Popescu ☀
> > > > <[hidden email] > wrote:
>
> > > > > On 4/10/07, Ilya Sterin <[hidden email] > wrote:
>
> > > > > > So I'm using maven with surefire plugin.  I have a requirement to
> > > > > > basically skip tests that are dependent on various resources that are
> > > > > > not available on a particular build machine.
>
> > > > > > I know that I can easily use the test dependency mechanism in TestNG
> > > > > > to skip tests that are dependent on a particular test that has failed.
> > > > > >  The problem with that is that Maven fails the build cycle if any
> > > > > > tests fail.
>
> > > > > > I can use testFailureIgnore, but that basically ignores all tests that
> > > > > > fail, but that might ignore important tests that have to pass in order
> > > > > > to build the application.
>
> > > > > > Can someone let me know if there is some workaround for this?
>
> > > > > > If there isn't, I propose one of two solutions...
>
> > > > > > 1.  Allow some sort of a way to semi-fail the test.  Basically, the
> > > > > > test doesn't throw an exception, but though doesn't truly succeed,
> > > > > > therefore the dependencies are not run.  This can be something in the
> > > > > > likes of
> > > > > > Assert.failWithoutException("resource X not found.");
> > > > > > or any other API call that allows to mark the test as not succeeded,
> > > > > > but do not report any failures.
>
> > > > > > This can also maybe be done is a way of providing some sort of an
> > > > > > annotation to silence the test failure.
> > > > > > @Test(silentFail = true);
>
> > > > > > 2.  Provide a way to stop a particular test suite from running.
> > > > > > Therefore in the @BeforeClass method, if some resource is found to be
> > > > > > not available you can instruct TestNG runner to skip the rest of the
> > > > > > tests in this particular suite.  The granularity of this might go
> > > > > > further, not sure if anyone has requirements for a more granular
> > > > > > mechanism of test skip control.
>
> > > > > > I'm wondering if there is an API that's already exposed in TestNG that
> > > > > > allows to accomplish this, though I can't seem to find it.
>
> > > > > > Ilya
>
> > > > > Pretty interesting scenario. Have you looked at AnnotationTransformer?
> > > > > I am thinking that you can check the resource existence in there, and
> > > > > considering that all @Test will pass through that point you will be
> > > > > able to disable them.
>
> > > > > Another approach that may be even cleaner is to use groups (but this
> > > > > would require to know upfront what resources are/are not available).
>
> > > > > If the resource tests are quite simple, then yet another approach is
> > > > > method selectors with script.
>
> > > > > Please let us know what do you think about these approaches.
>
> > > > > bests,
>
> > > > > ./alex
> > > > > --
> > > > > .w( the_mindstorm )p.
> > > > >   TestNG co-founder
> > > > > EclipseTestNG Creator


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: Skipping tests with unavailable resources

isterin
Is this the same as...
https://svn.apache.org/repos/asf/maven/sandbox/branches/surefire/surefire-collaboration/

I checked out the above earlier today and it failed with 5.5 TNG release.  Basically the TestNGExecutor (I think that's the class, can't recall now) was invoking a method that required a string with a boolean.  I think this was due to the testng API changes between the latest releases.  Let me know if the above repo is latest and if you need more detail, I can recompile and rerun.

Thanks.

Ilya

On 4/10/07, [hidden email] <[hidden email]> wrote:

The latest compatible up to 5.6 (I think) version isn't in the main
trunk area right now:

http://svn.apache.org/viewvc/maven/sandbox/branches/surefire/surefire-collaboration/

I still haven't committed my local changes for arbitrary property pass
through / 5.6 future updates but the src in there should be good.

I also think the 5.5 jars were published to ibiblio this morning.

On Apr 10, 6:42 pm, "Ilya Sterin" <[hidden email]> wrote:

> This now works great, but when not invoked through maven.
> Unfortunately with maven, I'm still stuck with testng 5.1, since any
> version 5.3 and above had extensive changes that break the interface
> between surefire and testng.  Either way, I followed the threads on
> testng, etc..., but still don't see a solution.  I also downloaded the
> latest sandbox for the maven/testng integration and compiled, but
> still no luck.  After screwing around with fixing various API calls, I
> thought I'd ask...
>
> Is anyone still working on this issue?  If yes, is there some possible
> ETA for this (Yeah, I know you're doing this for free:-).  If you guys
> don't have the time, I'm willing maybe in the next week or so to take
> a crack at this, as it's becoming more important on most of my
> projects.  We use maven for all project builds (by choice:-), and I'm
> stuck with TNG 5.1:-(
>
> Ilya
>
> On 4/10/07, Alexandru Popescu ☀ <[hidden email]> wrote:
>
> > On 4/11/07, Ilya Sterin <[hidden email]> wrote:
> > > Ah, I guess I can use the setInvocationCount :-)  Sorry...
>
> > He he... this is be a nice idea I wasn't thinking about :-). I
> > initially thought that there should be a setEnabled(boolean), but I
> > guess this one is good enough.
>
> > bests,
>
> > ./alex
> > --
> > .w( the_mindstorm )p.
> >   TestNG co-founder
> > EclipseTestNG Creator
>
> > > Ilya
>
> > > On 4/10/07, Ilya Sterin < [hidden email]> wrote:
> > > > Alex, the annotation transformer would work and is actually a pretty elegant
> > > > solution, but how do I disable the test using the ITest interface?  There
> > > > doesn't seem to be any concrete way to disable a particular test from
> > > > running.  Maybe there are some weird work arounds by setting some other
> > > > annotation to a dependency that doesn't exist?  Either way, it would be nice
> > > > if there was a way to skip a test explicitly using the ITest interface.
> > > > Also, it would be nice to skip a group of tests, otherwise I'd have to
> > > > basically verify all test names, or have a particular method naming
> > > > convention.
>
> > > > Any ideas?
>
> > > > Thanks.
>
> > > > Ilya Sterin
>
> > > > On 4/10/07, Alexandru Popescu ☀
> > > > <[hidden email] > wrote:
>
> > > > > On 4/10/07, Ilya Sterin <[hidden email] > wrote:
>
> > > > > > So I'm using maven with surefire plugin.  I have a requirement to
> > > > > > basically skip tests that are dependent on various resources that are
> > > > > > not available on a particular build machine.
>
> > > > > > I know that I can easily use the test dependency mechanism in TestNG
> > > > > > to skip tests that are dependent on a particular test that has failed.
> > > > > >  The problem with that is that Maven fails the build cycle if any
> > > > > > tests fail.
>
> > > > > > I can use testFailureIgnore, but that basically ignores all tests that
> > > > > > fail, but that might ignore important tests that have to pass in order
> > > > > > to build the application.
>
> > > > > > Can someone let me know if there is some workaround for this?
>
> > > > > > If there isn't, I propose one of two solutions...
>
> > > > > > 1.  Allow some sort of a way to semi-fail the test.  Basically, the
> > > > > > test doesn't throw an exception, but though doesn't truly succeed,
> > > > > > therefore the dependencies are not run.  This can be something in the
> > > > > > likes of
> > > > > > Assert.failWithoutException("resource X not found.");
> > > > > > or any other API call that allows to mark the test as not succeeded,
> > > > > > but do not report any failures.
>
> > > > > > This can also maybe be done is a way of providing some sort of an
> > > > > > annotation to silence the test failure.
> > > > > > @Test(silentFail = true);
>
> > > > > > 2.  Provide a way to stop a particular test suite from running.
> > > > > > Therefore in the @BeforeClass method, if some resource is found to be
> > > > > > not available you can instruct TestNG runner to skip the rest of the
> > > > > > tests in this particular suite.  The granularity of this might go
> > > > > > further, not sure if anyone has requirements for a more granular
> > > > > > mechanism of test skip control.
>
> > > > > > I'm wondering if there is an API that's already exposed in TestNG that
> > > > > > allows to accomplish this, though I can't seem to find it.
>
> > > > > > Ilya
>
> > > > > Pretty interesting scenario. Have you looked at AnnotationTransformer?
> > > > > I am thinking that you can check the resource existence in there, and
> > > > > considering that all @Test will pass through that point you will be
> > > > > able to disable them.
>
> > > > > Another approach that may be even cleaner is to use groups (but this
> > > > > would require to know upfront what resources are/are not available).
>
> > > > > If the resource tests are quite simple, then yet another approach is
> > > > > method selectors with script.
>
> > > > > Please let us know what do you think about these approaches.
>
> > > > > bests,
>
> > > > > ./alex
> > > > > --
> > > > > .w( the_mindstorm )p.
> > > > >   TestNG co-founder
> > > > > EclipseTestNG Creator

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: Skipping tests with unavailable resources

Jessek

Yeah that's the old version running, but the svn repo you pointed out
was correct.

You can actually see the literal fix you mention commented about in
the src:
http://svn.apache.org/viewvc/maven/sandbox/branches/surefire/surefire-collaboration/surefire-providers/surefire-testng/src/main/java/org/apache/maven/surefire/testng/TestNGDirectoryTestSuite.java?view=markup

It gets a little tricky installing this version locally but I'd be
sure you were really executing with this exact version at runtime and
not some other accidentally.

It'll be pushed out officially soon-ish. I expect it has at least one
more weekend of dev to go before it's done. The next version should be
much much more resilient to changes on either side of maven/testng.

On Apr 10, 11:33 pm, "Ilya Sterin" <[hidden email]> wrote:

> Is this the same as...https://svn.apache.org/repos/asf/maven/sandbox/branches/surefire/sure...
>
> I checked out the above earlier today and it failed with 5.5 TNG release.
> Basically the TestNGExecutor (I think that's the class, can't recall now)
> was invoking a method that required a string with a boolean.  I think this
> was due to the testng API changes between the latest releases.  Let me know
> if the above repo is latest and if you need more detail, I can recompile and
> rerun.
>
> Thanks.
>
> Ilya
>
> On 4/10/07, [hidden email] <[hidden email]> wrote:
>
>
>
> > The latest compatible up to 5.6 (I think) version isn't in the main
> > trunk area right now:
>
> >http://svn.apache.org/viewvc/maven/sandbox/branches/surefire/surefire...
>
> > I still haven't committed my local changes for arbitrary property pass
> > through / 5.6 future updates but the src in there should be good.
>
> > I also think the 5.5 jars were published to ibiblio this morning.
>
> > On Apr 10, 6:42 pm, "Ilya Sterin" <[hidden email]> wrote:
> > > This now works great, but when not invoked through maven.
> > > Unfortunately with maven, I'm still stuck with testng 5.1, since any
> > > version 5.3 and above had extensive changes that break the interface
> > > between surefire and testng.  Either way, I followed the threads on
> > > testng, etc..., but still don't see a solution.  I also downloaded the
> > > latest sandbox for the maven/testng integration and compiled, but
> > > still no luck.  After screwing around with fixing various API calls, I
> > > thought I'd ask...
>
> > > Is anyone still working on this issue?  If yes, is there some possible
> > > ETA for this (Yeah, I know you're doing this for free:-).  If you guys
> > > don't have the time, I'm willing maybe in the next week or so to take
> > > a crack at this, as it's becoming more important on most of my
> > > projects.  We use maven for all project builds (by choice:-), and I'm
> > > stuck with TNG 5.1:-(
>
> > > Ilya
>
> > > On 4/10/07, Alexandru Popescu ☀ <[hidden email]>
> > wrote:
>
> > > > On 4/11/07, Ilya Sterin <[hidden email]> wrote:
> > > > > Ah, I guess I can use the setInvocationCount :-)  Sorry...
>
> > > > He he... this is be a nice idea I wasn't thinking about :-). I
> > > > initially thought that there should be a setEnabled(boolean), but I
> > > > guess this one is good enough.
>
> > > > bests,
>
> > > > ./alex
> > > > --
> > > > .w( the_mindstorm )p.
> > > >   TestNG co-founder
> > > > EclipseTestNG Creator
>
> > > > > Ilya
>
> > > > > On 4/10/07, Ilya Sterin <[hidden email]> wrote:
> > > > > > Alex, the annotation transformer would work and is actually a
> > pretty elegant
> > > > > > solution, but how do I disable the test using the ITest
> > interface?  There
> > > > > > doesn't seem to be any concrete way to disable a particular test
> > from
> > > > > > running.  Maybe there are some weird work arounds by setting some
> > other
> > > > > > annotation to a dependency that doesn't exist?  Either way, it
> > would be nice
> > > > > > if there was a way to skip a test explicitly using the ITest
> > interface.
> > > > > > Also, it would be nice to skip a group of tests, otherwise I'd
> > have to
> > > > > > basically verify all test names, or have a particular method
> > naming
> > > > > > convention.
>
> > > > > > Any ideas?
>
> > > > > > Thanks.
>
> > > > > > Ilya Sterin
>
> > > > > > On 4/10/07, Alexandru Popescu ☀
> > > > > > <[hidden email] > wrote:
>
> > > > > > > On 4/10/07, Ilya Sterin <[hidden email] > wrote:
>
> > > > > > > > So I'm using maven with surefire plugin.  I have a requirement
> > to
> > > > > > > > basically skip tests that are dependent on various resources
> > that are
> > > > > > > > not available on a particular build machine.
>
> > > > > > > > I know that I can easily use the test dependency mechanism in
> > TestNG
> > > > > > > > to skip tests that are dependent on a particular test that has
> > failed.
> > > > > > > >  The problem with that is that Maven fails the build cycle if
> > any
> > > > > > > > tests fail.
>
> > > > > > > > I can use testFailureIgnore, but that basically ignores all
> > tests that
> > > > > > > > fail, but that might ignore important tests that have to pass
> > in order
> > > > > > > > to build the application.
>
> > > > > > > > Can someone let me know if there is some workaround for this?
>
> > > > > > > > If there isn't, I propose one of two solutions...
>
> > > > > > > > 1.  Allow some sort of a way to semi-fail the
> > test.  Basically, the
> > > > > > > > test doesn't throw an exception, but though doesn't truly
> > succeed,
> > > > > > > > therefore the dependencies are not run.  This can be something
> > in the
> > > > > > > > likes of
> > > > > > > > Assert.failWithoutException("resource X not found.");
> > > > > > > > or any other API call that allows to mark the test as not
> > succeeded,
> > > > > > > > but do not report any failures.
>
> > > > > > > > This can also maybe be done is a way of providing some sort of
> > an
> > > > > > > > annotation to silence the test failure.
> > > > > > > > @Test(silentFail = true);
>
> > > > > > > > 2.  Provide a way to stop a particular test suite from
> > running.
> > > > > > > > Therefore in the @BeforeClass method, if some resource is
> > found to be
> > > > > > > > not available you can instruct TestNG runner to skip the rest
> > of the
> > > > > > > > tests in this particular suite.  The granularity of this might
> > go
> > > > > > > > further, not sure if anyone has requirements for a more
> > granular
> > > > > > > > mechanism of test skip control.
>
> > > > > > > > I'm wondering if there is an API that's already exposed in
> > TestNG that
> > > > > > > > allows to accomplish this, though I can't seem to find it.
>
> > > > > > > > Ilya
>
> > > > > > > Pretty interesting scenario. Have you looked at
> > AnnotationTransformer?
> > > > > > > I am thinking that you can check the resource existence in
> > there, and
> > > > > > > considering that all @Test will pass through that point you will
> > be
> > > > > > > able to disable them.
>
> > > > > > > Another approach that may be even cleaner is to use groups (but
> > this
> > > > > > > would require to know upfront what resources are/are not
> > available).
>
> > > > > > > If the resource tests are quite simple, then yet another
> > approach is
> > > > > > > method selectors with script.
>
> > > > > > > Please let us know what do you think about these approaches.
>
> > > > > > > bests,
>
> > > > > > > ./alex
> > > > > > > --
> > > > > > > .w( the_mindstorm )p.
> > > > > > >   TestNG co-founder
> > > > > > > EclipseTestNG Creator


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: Skipping tests with unavailable resources

Mike Willekes
In reply to this post by Alexandru Popescu ☀

This workaround works (setting invocation count to 0 simply doesn't
run the test), but does anyone know if there is a way to manipulate
one of the annotations so that it "appears" as though the test was
skipped?

Suppose we have 100 tests, 20 of which are dependent on a particular
resource (attached via USB connection) which may not be present.  I
could write one test method that looks for the resource and then have
the 20 tests depend on that method...  But then I'll have 80 success,
1 fail and 20 skip (which isn't really accurate).

I was hoping to be able to use the Annotation Transformer to somehow
force a test to skip and be reported as such.  The closest I've been
able to come is to set the "dependsOnGroups()" attribute to be a non-
existent group.  Then the test is "skipped" but then the skipped
description is:  java.lang.Throwable: Method foo() depends on
nonexistent group "bar"

Any insight?

Thanks,
Mike



> On 4/11/07, Ilya Sterin <[hidden email]> wrote:
>
> > Ah, I guess I can use the setInvocationCount :-)  Sorry...
>
> He he... this is be a nice idea I wasn't thinking about :-). I
> initially thought that there should be a setEnabled(boolean), but I
> guess this one is good enough.
>
> bests,
>
> ./alex


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: Skipping tests with unavailable resources

Alexandru Popescu ☀

I have just introduced a couple of days ago support for SkipException.
Basically if you throw this exception from your @Test or
@configuration methods they will be marked as skipped.
SkipException can be extended so that smarter functionality can be
achieved. Currently in the core I have created a TimebombSkipException
which behaves like a SkipException until a date limit is reached,
after which the skip just transforms into a failure.

This is currently on SVN trunk (which may not work yet with Maven),
but it will come pretty soon and the next TestNG release will contain
the SkipException stuff.

bests,

./alex
--
.w( the_mindstorm )p.
  TestNG co-founder
EclipseTestNG Creator

On 4/17/07, Mike Willekes <[hidden email]> wrote:

>
> This workaround works (setting invocation count to 0 simply doesn't
> run the test), but does anyone know if there is a way to manipulate
> one of the annotations so that it "appears" as though the test was
> skipped?
>
> Suppose we have 100 tests, 20 of which are dependent on a particular
> resource (attached via USB connection) which may not be present.  I
> could write one test method that looks for the resource and then have
> the 20 tests depend on that method...  But then I'll have 80 success,
> 1 fail and 20 skip (which isn't really accurate).
>
> I was hoping to be able to use the Annotation Transformer to somehow
> force a test to skip and be reported as such.  The closest I've been
> able to come is to set the "dependsOnGroups()" attribute to be a non-
> existent group.  Then the test is "skipped" but then the skipped
> description is:  java.lang.Throwable: Method foo() depends on
> nonexistent group "bar"
>
> Any insight?
>
> Thanks,
> Mike
>
>
>
> > On 4/11/07, Ilya Sterin <[hidden email]> wrote:
> >
> > > Ah, I guess I can use the setInvocationCount :-)  Sorry...
> >
> > He he... this is be a nice idea I wasn't thinking about :-). I
> > initially thought that there should be a setEnabled(boolean), but I
> > guess this one is good enough.
> >
> > bests,
> >
> > ./alex
>
>
> >
>

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: Skipping tests with unavailable resources

Cédric Beust ♔
In reply to this post by Mike Willekes
Hi Mike,

I'm curious:  why do you think 1 fail / 20 skips it not accurate?  What would you expect?

--
Cédric


On 4/16/07, Mike Willekes <[hidden email]> wrote:

This workaround works (setting invocation count to 0 simply doesn't
run the test), but does anyone know if there is a way to manipulate
one of the annotations so that it "appears" as though the test was
skipped?

Suppose we have 100 tests, 20 of which are dependent on a particular
resource (attached via USB connection) which may not be present.  I
could write one test method that looks for the resource and then have
the 20 tests depend on that method...  But then I'll have 80 success,
1 fail and 20 skip (which isn't really accurate).

I was hoping to be able to use the Annotation Transformer to somehow
force a test to skip and be reported as such.  The closest I've been
able to come is to set the "dependsOnGroups()" attribute to be a non-
existent group.  Then the test is "skipped" but then the skipped
description is:  java.lang.Throwable: Method foo() depends on
nonexistent group "bar"

Any insight?

Thanks,
Mike



> On 4/11/07, Ilya Sterin <[hidden email]> wrote:
>
> > Ah, I guess I can use the setInvocationCount :-)  Sorry...
>
> He he... this is be a nice idea I wasn't thinking about :-). I
> initially thought that there should be a setEnabled(boolean), but I
> guess this one is good enough.
>
> bests,
>
> ./alex






--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: Skipping tests with unavailable resources

Mike Willekes

Hi Cédric,

Thanks for the quick reply...  Here's a bit of background:

We are testing a piece of (Java) software that allows a user to
interact with an external development board.  We have (for arguments
sake) a suite of 1000 tests and 40 different board configurations that
we support (i.e. different RAM sizes, peripherals, etc.).  Most of the
tests will run on several different configurations - but no single
test will run on all of them.  Rather then trying to maintain a
massive list of which tests should be run on which configurations, we
are able to query the board directly and build a list of constraints.
Our current test suite is entirely Tcl/tcltest and we really take
advantage of constraints.  Here's a snippet of what one of our current
test runs would look like:

DeviceConfiguration.test
...
++++ ResetDevice SKIPPED: peripheralAvailable
++++ DeviceWakeup SKIPPED: peripheralAvailable & lowPower
++++ RestartDevice SKIPPED: peripheralAvailable

Tests ended at Thu Mar 29 10:36:31 AM Eastern Daylight Time 2007
AllTests.tcl: Total 60 Passed 15 Skipped 45 Failed 0

In this example the DeviceWakeup test has 2 constraints
"peripheralAvailable" and "lowPower".  If those constrains aren't
satisfied, the test will be skipped.  Those constraints are set
conditionally during the "setup" routine which queries the external
board and determines if the specific peripheral is available and if
the device supports a low-power mode.

This allows us the flexibility to easily run the entire suite on a
singe board, or a single test on each of the boards without having to
maintain a complicated configuration file.  The user will also have an
accurate picture of what was Run/Passed/Skipped.

I'll look-into this SkipException - as it sounds promising...

Thanks,
Mike



On Apr 16, 8:59 pm, "Cédric Beust ♔ " <[hidden email]> wrote:
> Hi Mike,
>
> I'm curious:  why do you think 1 fail / 20 skips it not accurate?  What
> would you expect?
>
> --
> Cédric


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: Skipping tests with unavailable resources

Cédric Beust ♔
Hi Mike,

Thanks for the description, it's always nice to hear from real-world scenarios where TestNG is involved.

From your message, it looks like you are not expecting anything from us until you've looked into SkipException, correct?  Let us know otherwise, and also how SkipException works for you.

--
Cedric


On 4/16/07, Mike Willekes <[hidden email]> wrote:

Hi Cédric,

Thanks for the quick reply...  Here's a bit of background:

We are testing a piece of (Java) software that allows a user to
interact with an external development board.  We have (for arguments
sake) a suite of 1000 tests and 40 different board configurations that
we support (i.e. different RAM sizes, peripherals, etc.).  Most of the
tests will run on several different configurations - but no single
test will run on all of them.  Rather then trying to maintain a
massive list of which tests should be run on which configurations, we
are able to query the board directly and build a list of constraints.
Our current test suite is entirely Tcl/tcltest and we really take
advantage of constraints.  Here's a snippet of what one of our current
test runs would look like:

DeviceConfiguration.test
...
++++ ResetDevice                 SKIPPED: peripheralAvailable
++++ DeviceWakeup               SKIPPED: peripheralAvailable & lowPower
++++ RestartDevice              SKIPPED: peripheralAvailable

Tests ended at Thu Mar 29 10:36:31 AM Eastern Daylight Time 2007
AllTests.tcl:   Total   60              Passed 15       Skipped 45      Failed 0

In this example the DeviceWakeup test has 2 constraints
"peripheralAvailable" and "lowPower".  If those constrains aren't
satisfied, the test will be skipped.  Those constraints are set
conditionally during the "setup" routine which queries the external
board and determines if the specific peripheral is available and if
the device supports a low-power mode.

This allows us the flexibility to easily run the entire suite on a
singe board, or a single test on each of the boards without having to
maintain a complicated configuration file.  The user will also have an
accurate picture of what was Run/Passed/Skipped.

I'll look-into this SkipException - as it sounds promising...

Thanks,
Mike



On Apr 16, 8:59 pm, "Cédric Beust ♔ " <[hidden email]> wrote:
> Hi Mike,
>
> I'm curious:  why do you think 1 fail / 20 skips it not accurate?  What
> would you expect?
>
> --
> Cédric







--
Cédric
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: Skipping tests with unavailable resources

Steve Loughran-7
In reply to this post by Mike Willekes

On 17/04/07, Mike Willekes <[hidden email]> wrote:

>
> Hi Cédric,
>
> Thanks for the quick reply...  Here's a bit of background:
>
> We are testing a piece of (Java) software that allows a user to
> interact with an external development board.  We have (for arguments
> sake) a suite of 1000 tests and 40 different board configurations that
> we support (i.e. different RAM sizes, peripherals, etc.).  Most of the
> tests will run on several different configurations - but no single
> test will run on all of them.  Rather then trying to maintain a
> massive list of which tests should be run on which configurations, we
> are able to query the board directly and build a list of constraints.
> Our current test suite is entirely Tcl/tcltest and we really take
> advantage of constraints.  Here's a snippet of what one of our current
> test runs would look like:

So you are using TestNG to test the hardware? Or just the java-level API?

Either way, this is really cool.

I worked on a DVD burning project a few years back, where we used
CppUnit to test our C++ device API, first through the emulation (we'd
compared the sent SCSI commands with the expected ones), then into
production, though its harder to tell what works in that situation. We
had CruiseControl starting Ant, which would immediately exec visual
studio to build everything, then another <exec> for the cppunit runs.
It worked surprisingly well.

one problem was with the early hardware: it is a lot more stateful
than anything else, and you could leave the system in a mess that a
full reset needed to fix. The other was that if the burner locked up
you could (theoretically) start a fire, so we couldnt run tests
unattended. No overnight builds and tests for us...

-steve

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en
-~----------~----~----~----~------~----~------~--~---