[testng-dev] New feature: @Test(skippingExceptions=...)

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

[testng-dev] New feature: @Test(skippingExceptions=...)

Guillaume Juillot
Hi,

I'm using TestNG for some years on functional tests and really like its power for this.
However there's few things I would really appreciate as an user, and @Test(skippingExceptions=...) is one of them.

I already forked TestNG and made the modifications myself. That's the first time I contribute to TestNG so I don't know the "normal" process.
That's why I post here before making the Pull Request.
You can check my changes in this branch https://github.com/gjuillot/testng/tree/skippingExceptions
I'm waiting for your comments before Pull Request.

Here is the description of my modifications.
I added a parameter skippingException to the @Test, similarly to the expectedExceptions.
If an exception in this list is thrown, the test will be marked as skipped.

For example during my tests, I need a remote connection to a device. If the connection fails, an exception is thrown and the test fails also.
The problem is that in this case, the failure is a false status because the connection outage prevents us from deciding if the system under test really does not pass the test.
With skippingExceptions, we can now mark such a test as skipped if a network failure occurs.

--
You received this message because you are subscribed to the Google Groups "testng-dev" group.
To view this discussion on the web visit https://groups.google.com/d/msg/testng-dev/-/bvpaH-wBPQkJ.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/testng-dev?hl=en.
Reply | Threaded
Open this post in threaded view
|

Re: [testng-dev] New feature: @Test(skippingExceptions=...)

Cédric Beust ♔-2
Hi Guillaume,

You can already do this by throwing a SkipException. I'm not sure I see the point in allowing other exception classes to do this, can you explain why SkipException is not enough for you?

Thanks.

-- 
Cédric




On Thu, Jul 5, 2012 at 8:11 AM, Guillaume Juillot <[hidden email]> wrote:
Hi,

I'm using TestNG for some years on functional tests and really like its power for this.
However there's few things I would really appreciate as an user, and @Test(skippingExceptions=...) is one of them.

I already forked TestNG and made the modifications myself. That's the first time I contribute to TestNG so I don't know the "normal" process.
That's why I post here before making the Pull Request.
You can check my changes in this branch https://github.com/gjuillot/testng/tree/skippingExceptions
I'm waiting for your comments before Pull Request.

Here is the description of my modifications.
I added a parameter skippingException to the @Test, similarly to the expectedExceptions.
If an exception in this list is thrown, the test will be marked as skipped.

For example during my tests, I need a remote connection to a device. If the connection fails, an exception is thrown and the test fails also.
The problem is that in this case, the failure is a false status because the connection outage prevents us from deciding if the system under test really does not pass the test.
With skippingExceptions, we can now mark such a test as skipped if a network failure occurs.

--
You received this message because you are subscribed to the Google Groups "testng-dev" group.
To view this discussion on the web visit https://groups.google.com/d/msg/testng-dev/-/bvpaH-wBPQkJ.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/testng-dev?hl=en.

--
You received this message because you are subscribed to the Google Groups "testng-dev" group.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/testng-dev?hl=en.
Reply | Threaded
Open this post in threaded view
|

Re: [testng-dev] New feature: @Test(skippingExceptions=...)

Guillaume Juillot

I know I can force a test to be Skipped by throwing an exception.
What I mean is that sometimes I make a call to an external library, that throws an Exception. In my case, because of that exception I can't conclude on the real result of the test.

For example:

    @Test
    public void skyShouldNotBeGreen() {
        WindowShutter.open(); // may throw a BlockedException
        assertNotEquals(Sky.color(), GREEN);
    }

I need to open the window shutter in order to see the color of the sky. If I can't open that shutter (open() throws a Exception), the test will fail. But that doesn't mean the sky is green.



Le jeudi 5 juillet 2012 17:19:41 UTC+2, Cédric Beust ♔ a écrit :
Hi Guillaume,

You can already do this by throwing a SkipException. I'm not sure I see the point in allowing other exception classes to do this, can you explain why SkipException is not enough for you?

Thanks.

-- 
Cédric




On Thu, Jul 5, 2012 at 8:11 AM, Guillaume Juillot wrote:
Hi,

I'm using TestNG for some years on functional tests and really like its power for this.
However there's few things I would really appreciate as an user, and @Test(skippingExceptions=...) is one of them.

I already forked TestNG and made the modifications myself. That's the first time I contribute to TestNG so I don't know the "normal" process.
That's why I post here before making the Pull Request.
You can check my changes in this branch https://github.com/gjuillot/testng/tree/skippingExceptions
I'm waiting for your comments before Pull Request.

Here is the description of my modifications.
I added a parameter skippingException to the @Test, similarly to the expectedExceptions.
If an exception in this list is thrown, the test will be marked as skipped.

For example during my tests, I need a remote connection to a device. If the connection fails, an exception is thrown and the test fails also.
The problem is that in this case, the failure is a false status because the connection outage prevents us from deciding if the system under test really does not pass the test.
With skippingExceptions, we can now mark such a test as skipped if a network failure occurs.

--
You received this message because you are subscribed to the Google Groups "testng-dev" group.
To view this discussion on the web visit https://groups.google.com/d/msg/testng-dev/-/bvpaH-wBPQkJ.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/testng-dev?hl=en.

--
You received this message because you are subscribed to the Google Groups "testng-dev" group.
To view this discussion on the web visit https://groups.google.com/d/msg/testng-dev/-/rAC5W2jMGRQJ.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/testng-dev?hl=en.
Reply | Threaded
Open this post in threaded view
|

Re: [testng-dev] New feature: @Test(skippingExceptions=...)

Davide Del Vento-3
How about this?

    @Test
    public void skyShouldNotBeGreen() {
        try {
            WindowShutter.open();
        } catch (BlockedException b) {
            throw SkipException("Can't tell, the shutter was blocked");
        }
        assertNotEquals(Sky.color(), GREEN);
    }

On Thu, Jul 5, 2012 at 9:49 AM, Guillaume Juillot <[hidden email]> wrote:

    @Test
    public void skyShouldNotBeGreen() {
        WindowShutter.open(); // may throw a BlockedException
        assertNotEquals(Sky.color(), GREEN);
    }

--
You received this message because you are subscribed to the Google Groups "testng-dev" group.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/testng-dev?hl=en.
Reply | Threaded
Open this post in threaded view
|

Re: [testng-dev] New feature: @Test(skippingExceptions=...)

Cédric Beust ♔-2
Yup, that's exactly what I was going to suggest.

-- 
Cédric




On Thu, Jul 5, 2012 at 9:14 AM, Davide Del Vento <[hidden email]> wrote:
How about this?

    @Test
    public void skyShouldNotBeGreen() {
        try {
            WindowShutter.open();
        } catch (BlockedException b) {
            throw SkipException("Can't tell, the shutter was blocked");
        }
        assertNotEquals(Sky.color(), GREEN);

    }

On Thu, Jul 5, 2012 at 9:49 AM, Guillaume Juillot <[hidden email]> wrote:

    @Test
    public void skyShouldNotBeGreen() {
        WindowShutter.open(); // may throw a BlockedException
        assertNotEquals(Sky.color(), GREEN);
    }

--
You received this message because you are subscribed to the Google Groups "testng-dev" group.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/testng-dev?hl=en.

--
You received this message because you are subscribed to the Google Groups "testng-dev" group.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/testng-dev?hl=en.
Reply | Threaded
Open this post in threaded view
|

Re: [testng-dev] New feature: @Test(skippingExceptions=...)

Guillaume Juillot

That's what I'm doing for the moment but it's boilerplate code, isn't it ?


Le jeudi 5 juillet 2012 19:54:50 UTC+2, Cédric Beust ♔ a écrit :
Yup, that's exactly what I was going to suggest.

-- 
Cédric




On Thu, Jul 5, 2012 at 9:14 AM, Davide Del Vento <[hidden email]> wrote:
How about this?

    @Test
    public void skyShouldNotBeGreen() {
        try {
            WindowShutter.open();
        } catch (BlockedException b) {
            throw SkipException("Can't tell, the shutter was blocked");
        }
        assertNotEquals(Sky.color(), GREEN);

    }

On Thu, Jul 5, 2012 at 9:49 AM, Guillaume Juillot <[hidden email]> wrote:

    @Test
    public void skyShouldNotBeGreen() {
        WindowShutter.open(); // may throw a BlockedException
        assertNotEquals(Sky.color(), GREEN);
    }

--
You received this message because you are subscribed to the Google Groups "testng-dev" group.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/testng-dev?hl=en.

--
You received this message because you are subscribed to the Google Groups "testng-dev" group.
To view this discussion on the web visit https://groups.google.com/d/msg/testng-dev/-/LXFMu0nsZwYJ.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/testng-dev?hl=en.
Reply | Threaded
Open this post in threaded view
|

Re: [testng-dev] New feature: @Test(skippingExceptions=...)

kaname4711
What about the option to skip Exceptions thrown by Testng itsself?

Something like:

Test(groups = { "manual" }, timeOut = 5000, skipExceptions = { org.testng.internal.thread.ThreadTimeoutException.class })

--
You received this message because you are subscribed to the Google Groups "testng-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at http://groups.google.com/group/testng-dev?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|

Re: [testng-dev] New feature: @Test(skippingExceptions=...)

Michael May
I like this idea. It's better than having a try/catch block to catch exceptions just to rethrow a SkipException. Removing this boilerplate code would make the code more readable.

On Friday, February 1, 2013 8:54:39 AM UTC-5, [hidden email] wrote:
What about the option to skip Exceptions thrown by Testng itsself?

Something like:

Test(groups = { "manual" }, timeOut = 5000, skipExceptions = { org.testng.internal.thread.ThreadTimeoutException.class })

--
You received this message because you are subscribed to the Google Groups "testng-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at http://groups.google.com/group/testng-dev.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [testng-dev] New feature: @Test(skippingExceptions=...)

Michael May
In reply to this post by kaname4711
I would use this for SQLTimeoutException's. I would rather tests that are skipped in this way show up as warnings instead of looking like any other skipped tests. I see these as being a different test type.

Test Types:
PASS - tests that pass
SKIP - skip because of a known issue, wrong environment - these tests weren't expected to run
WARN - skip because test timeout, etc; but these tests we're expected to run
FAIL - tests that fail

On Friday, February 1, 2013 8:54:39 AM UTC-5, [hidden email] wrote:
What about the option to skip Exceptions thrown by Testng itsself?

Something like:

Test(groups = { "manual" }, timeOut = 5000, skipExceptions = { org.testng.internal.thread.ThreadTimeoutException.class })

--
You received this message because you are subscribed to the Google Groups "testng-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at http://groups.google.com/group/testng-dev.
For more options, visit https://groups.google.com/d/optout.