Can I mark retried tests as "RETRYING" instead of "SKIP"

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Can I mark retried tests as "RETRYING" instead of "SKIP"

Vishvesh Mulay
I have a framework which at the end of every test case execution pushes all the metadata information about that execution to ELK stack and shows various trends etc on the kibana dashboard.  

This includes all of the metadata including test execution status  which can be one of 1. success 2. failure 3. skip

I have a RetryAnalyzer implementation that retries failed tests 3 times and it marks the failed tests as "SKIP" if they are up for a retry, 

What that does is , once the retryAnalyzer is executed on the failed test, the status gets updated to SKIP and it shows it that way on our dashboard and after retry we show one more execution as either success or failure .

We upload each execution for the same test case as a separate run  to offer more granular metrics and stats...   

Right now,if a test status is "SKIP" it can mean one of 2 things 1. the data provider had an issue and the test was never run or 2. test failed and was retried 

This affects our data quality and I am wondering why testNG doesn't have a separate enum value for tests that are failing currently but are going to be retried... something like "RETRIABLE" or something to that effect.  Or is there a way to do that ? 

Thanks,
Vishvesh

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Can I mark retried tests as "RETRYING" instead of "SKIP"

Krishnan Mahadevan
Why not add an attribute to the ITestResult object within your org.testng.IRetryAnalyzer implementation which explicitly states that this method is being retried.

That way, you should be able to tweak your dashboard in Kibana to reflect this accordingly no ?

Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @ http://wakened-cognition.blogspot.com/
My Technical Scribbings @ http://rationaleemotions.wordpress.com/

On Wed, Feb 14, 2018 at 11:11 AM, Vishvesh Mulay <[hidden email]> wrote:
I have a framework which at the end of every test case execution pushes all the metadata information about that execution to ELK stack and shows various trends etc on the kibana dashboard.  

This includes all of the metadata including test execution status  which can be one of 1. success 2. failure 3. skip

I have a RetryAnalyzer implementation that retries failed tests 3 times and it marks the failed tests as "SKIP" if they are up for a retry, 

What that does is , once the retryAnalyzer is executed on the failed test, the status gets updated to SKIP and it shows it that way on our dashboard and after retry we show one more execution as either success or failure .

We upload each execution for the same test case as a separate run  to offer more granular metrics and stats...   

Right now,if a test status is "SKIP" it can mean one of 2 things 1. the data provider had an issue and the test was never run or 2. test failed and was retried 

This affects our data quality and I am wondering why testNG doesn't have a separate enum value for tests that are failing currently but are going to be retried... something like "RETRIABLE" or something to that effect.  Or is there a way to do that ? 

Thanks,
Vishvesh

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Can I mark retried tests as "RETRYING" instead of "SKIP"

Vishvesh Mulay
That is precisely what we have to do right now.  But my questions is why TestNG is overloading the definition of "SKIP" in this context.   Why not have a separate enum value representing the true status of the "failed but up for retry" tests ?

On Tuesday, February 13, 2018 at 11:44:59 PM UTC-8, Krishnan Mahadevan wrote:
Why not add an attribute to the ITestResult object within your org.testng.IRetryAnalyzer implementation which explicitly states that this method is being retried.

That way, you should be able to tweak your dashboard in Kibana to reflect this accordingly no ?

Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @ <a href="http://wakened-cognition.blogspot.com/" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fwakened-cognition.blogspot.com%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHzOdYJCASIDF_28vQtkp9gnKAYSQ&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fwakened-cognition.blogspot.com%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHzOdYJCASIDF_28vQtkp9gnKAYSQ&#39;;return true;">http://wakened-cognition.blogspot.com/
My Technical Scribbings @ <a href="http://rationaleemotions.wordpress.com/" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Frationaleemotions.wordpress.com%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEcBOKyRn0lpL8LDbBuyAlKdwgXwQ&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Frationaleemotions.wordpress.com%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEcBOKyRn0lpL8LDbBuyAlKdwgXwQ&#39;;return true;">http://rationaleemotions.wordpress.com/

On Wed, Feb 14, 2018 at 11:11 AM, Vishvesh Mulay <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="LgZL2wnICAAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">vishve...@...> wrote:
I have a framework which at the end of every test case execution pushes all the metadata information about that execution to ELK stack and shows various trends etc on the kibana dashboard.  

This includes all of the metadata including test execution status  which can be one of 1. success 2. failure 3. skip

I have a RetryAnalyzer implementation that retries failed tests 3 times and it marks the failed tests as "SKIP" if they are up for a retry, 

What that does is , once the retryAnalyzer is executed on the failed test, the status gets updated to SKIP and it shows it that way on our dashboard and after retry we show one more execution as either success or failure .

We upload each execution for the same test case as a separate run  to offer more granular metrics and stats...   

Right now,if a test status is "SKIP" it can mean one of 2 things 1. the data provider had an issue and the test was never run or 2. test failed and was retried 

This affects our data quality and I am wondering why testNG doesn't have a separate enum value for tests that are failing currently but are going to be retried... something like "RETRIABLE" or something to that effect.  Or is there a way to do that ? 

Thanks,
Vishvesh

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="LgZL2wnICAAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">testng-users...@googlegroups.com.
To post to this group, send email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="LgZL2wnICAAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">testng...@....
Visit this group at <a href="https://groups.google.com/group/testng-users" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/group/testng-users&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/group/testng-users&#39;;return true;">https://groups.google.com/group/testng-users.
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Can I mark retried tests as "RETRYING" instead of "SKIP"

Krishnan Mahadevan
>>>> "why TestNG is overloading the definition of "SKIP" in this context"

What do you mean by this ? TestNG marks a method as skipped under the following conditions :

  1. When a configuration method fails.
  2. When a test method depends on another method and the original method fails, causing the dependent method to be skipped.
  3. When a data provider has a problem (which you have called out)
In the case of a failed method that is being retried, then the previous attempts must be skipped, which is what I am guessing that TestNG is doing here.

Just curious. What exactly are you achieving by posting the failed attempts when there's a retry involved ? I can understand you trying to capture execution information, but these attempts IMO should be skipped, if you are going to be retrying anyway no ?


Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @ http://wakened-cognition.blogspot.com/
My Technical Scribbings @ http://rationaleemotions.wordpress.com/

On Wed, Feb 14, 2018 at 10:12 PM, Vishvesh Mulay <[hidden email]> wrote:
That is precisely what we have to do right now.  But my questions is why TestNG is overloading the definition of "SKIP" in this context.   Why not have a separate enum value representing the true status of the "failed but up for retry" tests ?

On Tuesday, February 13, 2018 at 11:44:59 PM UTC-8, Krishnan Mahadevan wrote:
Why not add an attribute to the ITestResult object within your org.testng.IRetryAnalyzer implementation which explicitly states that this method is being retried.

That way, you should be able to tweak your dashboard in Kibana to reflect this accordingly no ?

Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @ http://wakened-cognition.blogspot.com/
My Technical Scribbings @ http://rationaleemotions.wordpress.com/

On Wed, Feb 14, 2018 at 11:11 AM, Vishvesh Mulay <[hidden email]> wrote:
I have a framework which at the end of every test case execution pushes all the metadata information about that execution to ELK stack and shows various trends etc on the kibana dashboard.  

This includes all of the metadata including test execution status  which can be one of 1. success 2. failure 3. skip

I have a RetryAnalyzer implementation that retries failed tests 3 times and it marks the failed tests as "SKIP" if they are up for a retry, 

What that does is , once the retryAnalyzer is executed on the failed test, the status gets updated to SKIP and it shows it that way on our dashboard and after retry we show one more execution as either success or failure .

We upload each execution for the same test case as a separate run  to offer more granular metrics and stats...   

Right now,if a test status is "SKIP" it can mean one of 2 things 1. the data provider had an issue and the test was never run or 2. test failed and was retried 

This affects our data quality and I am wondering why testNG doesn't have a separate enum value for tests that are failing currently but are going to be retried... something like "RETRIABLE" or something to that effect.  Or is there a way to do that ? 

Thanks,
Vishvesh

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to testng-users...@googlegroups.com.
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Can I mark retried tests as "RETRYING" instead of "SKIP"

Vishvesh Mulay
First of all, Thanks for your prompt responses ! 

but these attempts IMO should be skipped, if you are going to be retrying anyway no

We have literally hundreds of test cases and each test case execution time is quite significant (These are E2E functional test cases) So we want to minimize the retries by fixing the underlying inconsistencies.   To get the clear idea of what types of tests are inconsistent (so in turn have to be retried) we need to post execution info for EACH test case run.   That is one of the main reasons why we are posting info even for the skipped ones.

What do you mean by this ? 

The 3 conditions you mention below are checked before the actual test case is executed and the test is skipped BEFORE it is run. 

The retry scenario is different where we are marking the status of the test case as skipped AFTER it is executed.  And that is where it gets a little confusing. 

My understanding of status is that it represents "execution" status of that test. That's why we have success, failure and skip   as the possible values.   So in my view,  skipped is different from "failed but up for retry" 

Does that make sense ?

On Wednesday, February 14, 2018 at 8:58:32 AM UTC-8, Krishnan Mahadevan wrote:
>>>> "why TestNG is overloading the definition of "SKIP" in this context"

What do you mean by this ? TestNG marks a method as skipped under the following conditions :

  1. When a configuration method fails.
  2. When a test method depends on another method and the original method fails, causing the dependent method to be skipped.
  3. When a data provider has a problem (which you have called out)
In the case of a failed method that is being retried, then the previous attempts must be skipped, which is what I am guessing that TestNG is doing here.

Just curious. What exactly are you achieving by posting the failed attempts when there's a retry involved ? I can understand you trying to capture execution information, but these attempts IMO should be skipped, if you are going to be retrying anyway no ?


Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @ <a href="http://wakened-cognition.blogspot.com/" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fwakened-cognition.blogspot.com%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHzOdYJCASIDF_28vQtkp9gnKAYSQ&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fwakened-cognition.blogspot.com%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHzOdYJCASIDF_28vQtkp9gnKAYSQ&#39;;return true;">http://wakened-cognition.blogspot.com/
My Technical Scribbings @ <a href="http://rationaleemotions.wordpress.com/" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Frationaleemotions.wordpress.com%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEcBOKyRn0lpL8LDbBuyAlKdwgXwQ&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Frationaleemotions.wordpress.com%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEcBOKyRn0lpL8LDbBuyAlKdwgXwQ&#39;;return true;">http://rationaleemotions.wordpress.com/

On Wed, Feb 14, 2018 at 10:12 PM, Vishvesh Mulay <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="EagFAD_mCAAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">vishve...@...> wrote:
That is precisely what we have to do right now.  But my questions is why TestNG is overloading the definition of "SKIP" in this context.   Why not have a separate enum value representing the true status of the "failed but up for retry" tests ?

On Tuesday, February 13, 2018 at 11:44:59 PM UTC-8, Krishnan Mahadevan wrote:
Why not add an attribute to the ITestResult object within your org.testng.IRetryAnalyzer implementation which explicitly states that this method is being retried.

That way, you should be able to tweak your dashboard in Kibana to reflect this accordingly no ?

Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @ <a href="http://wakened-cognition.blogspot.com/" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fwakened-cognition.blogspot.com%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHzOdYJCASIDF_28vQtkp9gnKAYSQ&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fwakened-cognition.blogspot.com%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHzOdYJCASIDF_28vQtkp9gnKAYSQ&#39;;return true;">http://wakened-cognition.blogspot.com/
My Technical Scribbings @ <a href="http://rationaleemotions.wordpress.com/" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Frationaleemotions.wordpress.com%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEcBOKyRn0lpL8LDbBuyAlKdwgXwQ&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Frationaleemotions.wordpress.com%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEcBOKyRn0lpL8LDbBuyAlKdwgXwQ&#39;;return true;">http://rationaleemotions.wordpress.com/

On Wed, Feb 14, 2018 at 11:11 AM, Vishvesh Mulay <[hidden email]> wrote:
I have a framework which at the end of every test case execution pushes all the metadata information about that execution to ELK stack and shows various trends etc on the kibana dashboard.  

This includes all of the metadata including test execution status  which can be one of 1. success 2. failure 3. skip

I have a RetryAnalyzer implementation that retries failed tests 3 times and it marks the failed tests as "SKIP" if they are up for a retry, 

What that does is , once the retryAnalyzer is executed on the failed test, the status gets updated to SKIP and it shows it that way on our dashboard and after retry we show one more execution as either success or failure .

We upload each execution for the same test case as a separate run  to offer more granular metrics and stats...   

Right now,if a test status is "SKIP" it can mean one of 2 things 1. the data provider had an issue and the test was never run or 2. test failed and was retried 

This affects our data quality and I am wondering why testNG doesn't have a separate enum value for tests that are failing currently but are going to be retried... something like "RETRIABLE" or something to that effect.  Or is there a way to do that ? 

Thanks,
Vishvesh

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to testng-users...@googlegroups.com.
To post to this group, send email to [hidden email].
Visit this group at <a href="https://groups.google.com/group/testng-users" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/group/testng-users&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/group/testng-users&#39;;return true;">https://groups.google.com/group/testng-users.
For more options, visit <a href="https://groups.google.com/d/optout" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="EagFAD_mCAAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">testng-users...@googlegroups.com.
To post to this group, send email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="EagFAD_mCAAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">testng...@....
Visit this group at <a href="https://groups.google.com/group/testng-users" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/group/testng-users&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/group/testng-users&#39;;return true;">https://groups.google.com/group/testng-users.
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Can I mark retried tests as "RETRYING" instead of "SKIP"

Krishnan Mahadevan

Cedric/Julien,

What do you think?

 

Vishvesh,

Yep. That makes sense and sounds like a fair ask IMO.

You can start off by filing this as an issue on our GitHub page here.

 

We also encourage contributions to TestNG via pull requests.

 

So in-case you want to give a shot at fixing this as well, you can refer here to get started with contributing to TestNG.

 

Thanks & Regards

Krishnan Mahadevan

 

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"

My Scribblings @ http://wakened-cognition.blogspot.com/

My Technical Scribbings @ http://rationaleemotions.wordpress.com/

Reply-To: <[hidden email]>
Date: Wednesday, February 14, 2018 at 10:46 PM
To: testng-users <[hidden email]>
Subject: Re: [testng-users] Can I mark retried tests as "RETRYING" instead of "SKIP"

 

 

but these attempts IMO should be skipped, if you are going to be retrying anyway no

 

We have literally hundreds of test cases and each test case execution time is quite significant (These are E2E functional test cases) So we want to minimize the retries by fixing the underlying inconsistencies.   To get the clear idea of what types of tests are inconsistent (so in turn have to be retried) we need to post execution info for EACH test case run.   That is one of the main reasons why we are posting info even for the skipped ones.

 

What do you mean by this ? 

 

The 3 conditions you mention below are checked before the actual test case is executed and the test is skipped BEFORE it is run. 

 

The retry scenario is different where we are marking the status of the test case as skipped AFTER it is executed.  And that is where it gets a little confusing. 

 

My understanding of status is that it represents "execution" status of that test. That's why we have success, failure and skip   as the possible values.   So in my view,  skipped is different from "failed but up for retry" 

 

Does that make sense ?


On Wednesday, February 14, 2018 at 8:58:32 AM UTC-8, Krishnan Mahadevan wrote:

>>>> "why TestNG is overloading the definition of "SKIP" in this context"

 

What do you mean by this ? TestNG marks a method as skipped under the following conditions :

 

  1. When a configuration method fails.
  2. When a test method depends on another method and the original method fails, causing the dependent method to be skipped.
  3. When a data provider has a problem (which you have called out)

In the case of a failed method that is being retried, then the previous attempts must be skipped, which is what I am guessing that TestNG is doing here.

 

Just curious. What exactly are you achieving by posting the failed attempts when there's a retry involved ? I can understand you trying to capture execution information, but these attempts IMO should be skipped, if you are going to be retrying anyway no ?

 


Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @
http://wakened-cognition.blogspot.com/

My Technical Scribbings @ http://rationaleemotions.wordpress.com/

 

On Wed, Feb 14, 2018 at 10:12 PM, Vishvesh Mulay <<a href="javascript:" target="_blank">vishve...@...> wrote:

That is precisely what we have to do right now.  But my questions is why TestNG is overloading the definition of "SKIP" in this context.   Why not have a separate enum value representing the true status of the "failed but up for retry" tests ?

On Tuesday, February 13, 2018 at 11:44:59 PM UTC-8, Krishnan Mahadevan wrote:

Why not add an attribute to the ITestResult object within your org.testng.IRetryAnalyzer implementation which explicitly states that this method is being retried.

 

That way, you should be able to tweak your dashboard in Kibana to reflect this accordingly no ?


Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @
http://wakened-cognition.blogspot.com/

My Technical Scribbings @ http://rationaleemotions.wordpress.com/

 

On Wed, Feb 14, 2018 at 11:11 AM, Vishvesh Mulay <[hidden email]> wrote:

I have a framework which at the end of every test case execution pushes all the metadata information about that execution to ELK stack and shows various trends etc on the kibana dashboard.  

 

This includes all of the metadata including test execution status  which can be one of 1. success 2. failure 3. skip

 

I have a RetryAnalyzer implementation that retries failed tests 3 times and it marks the failed tests as "SKIP" if they are up for a retry, 

 

What that does is , once the retryAnalyzer is executed on the failed test, the status gets updated to SKIP and it shows it that way on our dashboard and after retry we show one more execution as either success or failure .

 

We upload each execution for the same test case as a separate run  to offer more granular metrics and stats...   

 

Right now,if a test status is "SKIP" it can mean one of 2 things 1. the data provider had an issue and the test was never run or 2. test failed and was retried 

 

This affects our data quality and I am wondering why testNG doesn't have a separate enum value for tests that are failing currently but are going to be retried... something like "RETRIABLE" or something to that effect.  Or is there a way to do that ? 

 

Thanks,

Vishvesh

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at
https://groups.google.com/group/testng-users.
For more options, visit
https://groups.google.com/d/optout.

 

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to
<a href="javascript:" target="_blank">testng-users...@....
To post to this group, send email to
<a href="javascript:" target="_blank">testng...@....
Visit this group at
https://groups.google.com/group/testng-users.
For more options, visit
https://groups.google.com/d/optout.

 

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to
[hidden email].
To post to this group, send email to
[hidden email].
Visit this group at
https://groups.google.com/group/testng-users.
For more options, visit
https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.