[testng-dev] Possibility to retry tests until FAILED (not success status)

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

[testng-dev] Possibility to retry tests until FAILED (not success status)

Katya L
We need to have a possibility to run tests until test will be failed (because we have flaky tests and want to run a test until it failed and then investigate: read logs etc) Invocation count is not a workaround.
1st problem: hardcoded FAILURE test status in condition for retry: 
boolean willRetry = retryAnalyzer != null && status == ITestResult.FAILURE && failure.instances != null && retryAnalyzer.retry(testResult);

In old TestNG versions we had a chance to avoid it by the following way:

class RetryUntilFailed : IRetryAnalyzer {

companion object {
private var retryCount = 0
private var maxRetryCount = 1000
}

override fun retry(result: ITestResult?): Boolean {
if (result!!.getAttribute("RETRY") == "NO")
return false
else if (result.getAttribute("RETRY") == "YES" && retryCount < maxRetryCount) {
retryCount++
return true
}
return false
}
}

class RetryUntilFailedListener : IInvokedMethodListener
{
override fun beforeInvocation(method: IInvokedMethod?, testResult: ITestResult?) {

}

override fun afterInvocation(method: IInvokedMethod?, testResult: ITestResult?) {
if (method!!.isTestMethod) {
if (testResult!!.status == ITestResult.SUCCESS) {
println("SUCCESS. NEED TO RETRY")
testResult.status = ITestResult.FAILURE //because retry works only if (testResult.getStatus() == ITestResult.FAILURE)
testResult.setAttribute("RETRY", "YES")
}
else if (testResult.status == ITestResult.FAILURE)
testResult.setAttribute("RETRY", "NO")
}
}

}


But in the latest TestNG it doesn't work because handleInvocationResults() is invoked before afterInvocation() and no chance to override test status before retrying logic

finally {
...
handleInvocationResults(tm, results, expectedExceptionClasses, failureContext);
...
// Run invokedMethodListeners after updating TestResult
runInvokedMethodListeners(AFTER_INVOCATION, invokedMethod, testResult);
...
}



Any ideas? 
Thanks in advance









--
You received this message because you are subscribed to the Google Groups "testng-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-dev.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

[testng-dev] Re: Possibility to retry tests until FAILED (not success status)

Julien Herr
Hi,

I'm not sure the regression is so recent. Could you tell us which is the last version working as you expect?
And I can't remember if the new behavior is expected or not.

Could you open an issue on GitHub? It is easier to follow issue there.

Julien

Le mercredi 30 août 2017 21:35:33 UTC+2, Katya L a écrit :
We need to have a possibility to run tests until test will be failed (because we have flaky tests and want to run a test until it failed and then investigate: read logs etc) Invocation count is not a workaround.
1st problem: hardcoded FAILURE test status in condition for retry: 
boolean willRetry = retryAnalyzer != null && status == ITestResult.FAILURE && failure.instances != null && retryAnalyzer.retry(testResult);

In old TestNG versions we had a chance to avoid it by the following way:

class RetryUntilFailed : IRetryAnalyzer {

companion object {
private var retryCount = 0
private var maxRetryCount = 1000
}

override fun retry(result: ITestResult?): Boolean {
if (result!!.getAttribute("RETRY") == "NO")
return false
else if (result.getAttribute("RETRY") == "YES" && retryCount < maxRetryCount) {
retryCount++
return true
}
return false
}
}

class RetryUntilFailedListener : IInvokedMethodListener
{
override fun beforeInvocation(method: IInvokedMethod?, testResult: ITestResult?) {

}

override fun afterInvocation(method: IInvokedMethod?, testResult: ITestResult?) {
if (method!!.isTestMethod) {
if (testResult!!.status == ITestResult.SUCCESS) {
println("SUCCESS. NEED TO RETRY")
testResult.status = ITestResult.FAILURE //because retry works only if (testResult.getStatus() == ITestResult.FAILURE)
testResult.setAttribute("RETRY", "YES")
}
else if (testResult.status == ITestResult.FAILURE)
testResult.setAttribute("RETRY", "NO")
}
}

}


But in the latest TestNG it doesn't work because handleInvocationResults() is invoked before afterInvocation() and no chance to override test status before retrying logic

finally {
...
handleInvocationResults(tm, results, expectedExceptionClasses, failureContext);
...
// Run invokedMethodListeners after updating TestResult
runInvokedMethodListeners(AFTER_INVOCATION, invokedMethod, testResult);
...
}



Any ideas? 
Thanks in advance









--
You received this message because you are subscribed to the Google Groups "testng-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-dev.
For more options, visit https://groups.google.com/d/optout.