Customizing failure reporting in TestNG

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Customizing failure reporting in TestNG

abhijeetvaikar
I have created a basic playground project that contains:

A testLogin.java file that contains:
a. testng package imports (org.testng.*)
b. selenium webdriver imports (org.openqa.selenium.*)
c. 5 test-methods with testng annotations:

@Test(groups={"init"}) public void openURL()
Contains webdriver code to initiate the webdriver and open a chrome >instance with a given url.

@Test(dependsOnGroups={"init"}) public void testLogin()
Contains webdriver code to:
1. Locate username password text-input elements, enter the username password from a properties file.
2. Locate the "log in" button and click the button to log-in
3. Manage a login-forcefully scenario if someone else has already logged in using the credentials.

@Test(dependsOnMethods={"testLogin"}) public void testPatientsScheduleList()
Contains webdriver code to check if any patients have been scheduled. If yes, then fetch the names and display in console.

@Test() public void testLogout()
Contains webdriver code to locate the logout button and click on the button to logout of the app.

@AfterTest() public void closeConnection()
Contains webdriver code to dispose the webdriver object and close the chrome instance.

Currently I am simply running the test script wrapped as testng methods from ANT and a testng-xslt report gets generated.

Issues:

1. Performing validations against every line of code of webdriver script in a test method:

I know:
1. Selenium webdriver script contains API methods (findElement() and others.) that throw exceptions as a result of a default assertion/validation they perform. These exceptions show up in the generated report when a test-method fails.
2. TestNG provides Assert class that has many assertion methods but I have not yet figured out how can i use them to perform validation/assertions against every line of code of webdriver script. I tried adding assertion methods after every line of webdriver script code. What appeared in the output was just an AssertionError exception for a testmethod.

2. Failing a certain test method which gets passed due to try.. catch block.
If I use a try catch block around a set of 2 or more test drive script steps, and if a test-case fails in any of the steps (script line) then the try..catch block handles it thereby showing the test-method as "passed" in the execution report, which actually failed.

3. Creating a custom report which will show desired test execution results and not stack-traces!
When I execute the above script, a testng-xslt report gets generated that contains pass/fail status of each test method in a test-suite (configured in testng.xml). The test-results only give me whether a test-method has passed or failed and provides an exception's stack-trace which really doesn't provide any helpful information. I don't want such abstract level of test execution results but something like:

Name | Started | Duration | What-really-went-wrong (Failure)


Can anyone please suggest/ give some pointers regarding:
1. How can I perform validation/assertion against every line of code of webdriver script in a test-method without writing asserts after every script line?
2. How can I fail a certain test method which gets passed due to try catch block?
3. How can I customize the failure reporting so that I can send a failure result like "Expected element "button" with id "bnt12" but did not find the element at step 3 of test-method" to testng's reporting utility?
4. In the testng-xslt report I want to display where exactly in the test-method a failure occurred. So for example if my test-method fails because of a webelement = driver.findElement() at line 3 of a test-method, I want to display this issue in the test-report in the "What-really-went-wrong" column. I have read about testng testlisteners TestListenerAdapter / ITestListener/ IReporter but I don't understand how to use them after checking testng's javadocs.
5. Also, I have to implement PageObject pattern once I am done with customizing the test report. Where would be the right place to perform assertions in a page-object pattern? Should assertions be written in the page object test methods or in the higher level test methods that will use the PageObject classes?

P.S: I am completely new to testng framework and webdriver scripting. Please bear with any technical mistakes or observation errors if any in the post.
Reply | Threaded
Open this post in threaded view
|

Re: Customizing failure reporting in TestNG

Krishnan Mahadevan
Answers INLINE

On Thursday, January 17, 2013, abhijeetvaikar wrote:
I have created a basic playground project that contains:

A testLogin.java file that contains:
a. testng package imports (org.testng.*)
b. selenium webdriver imports (org.openqa.selenium.*)
c. 5 test-methods with testng annotations:

@Test(groups={"init"}) public void openURL()
Contains webdriver code to initiate the webdriver and open a chrome
>instance with a given url.

@Test(dependsOnGroups={"init"}) public void testLogin()
Contains webdriver code to:
1. Locate username password text-input elements, enter the username password
from a properties file.
2. Locate the "log in" button and click the button to log-in
3. Manage a login-forcefully scenario if someone else has already logged in
using the credentials.

@Test(dependsOnMethods={"testLogin"}) public void testPatientsScheduleList()
Contains webdriver code to check if any patients have been scheduled. If
yes, then fetch the names and display in console.

@Test() public void testLogout()
Contains webdriver code to locate the logout button and click on the button
to logout of the app.

@AfterTest() public void closeConnection()
Contains webdriver code to dispose the webdriver object and close the chrome
instance.

Currently I am simply running the test script wrapped as testng methods from
ANT and a testng-xslt report gets generated.

*Issues:*

*1. Performing validations against every line of code of webdriver script in
a test method:*

*I know:*
1. Selenium webdriver script contains API methods (findElement() and
others.) that throw exceptions as a result of a default assertion/validation
they perform. These exceptions show up in the generated report when a
test-method fails.
2. TestNG provides Assert class that has many assertion methods but I have
not yet figured out how can i use them to perform validation/assertions
against every line of code of webdriver script. I tried adding assertion
methods after every line of webdriver script code. What appeared in the
output was just an AssertionError exception for a testmethod.

*2. Failing a certain test method which gets passed due to try.. catch
block. *
If I use a try catch block around a set of 2 or more test drive script
steps, and if a test-case fails in any of the steps (script line) then the
try..catch block handles it thereby showing the test-method as "passed" in
the execution report, which actually failed.

*3. Creating a custom report which will show desired test execution results
and not stack-traces!*
When I execute the above script, a testng-xslt report gets generated that
contains pass/fail status of each test method in a test-suite (configured in
testng.xml). The test-results only give me whether a test-method has passed
or failed and provides an exception's stack-trace which really doesn't
provide any helpful information. I don't want such abstract level of test
execution results but something like:

Name | Started | Duration | What-really-went-wrong (Failure)


*Can anyone please suggest/ give some pointers regarding:*
1. How can I perform validation/assertion against every line of code of
webdriver script in a test-method without writing asserts after every script
line?

>>>>> Take a look at EventFiringWebDriver. That is what you need 
 
 
2. How can I fail a certain test method which gets passed due to try catch
block?

>>> Trigger a Assert.fail() in the catch Block.  
3. How can I customize the failure reporting so that I can send a failure
result like "Expected element "button" with id "bnt12" but did not find the
element at step 3 of test-method" to testng's reporting utility?

>>> Explore TestNG listeners. Start of by implementing IReporter and plug-in your desired way of reporting by writing the logic. 

 
4. In the testng-xslt report I want to display where exactly in the
test-method a failure occurred. So for example if my test-method fails
because of a webelement = driver.findElement() at line 3 of a test-method, I
want to display this issue in the test-report in the
"What-really-went-wrong" column. I have read about testng testlisteners
TestListenerAdapter / ITestListener/ IReporter but I don't understand how to
use them after checking testng's javadocs.

 
 
5. Also, I have to implement PageObject pattern once I am done with
customizing the test report. Where would be the right place to perform
assertions in a page-object pattern? Should assertions be written in the
page object test methods or in the higher level test methods that will use
the PageObject classes?

>>>Not a TestNG question. Please post this on webdriver/selenium-users forum.  

P.S: I am completely new to testng framework and webdriver scripting. Please
bear with any technical mistakes or observation errors if any in the post.


>>> you are better off seggregating your questions into TestNG and Selenium related and post them in the relevant forums than clubbing everything into ONE thread and posting it in just the testng forum.  


--
View this message in context: http://testng.1065351.n5.nabble.com/Customizing-failure-reporting-in-TestNG-tp17564.html
Sent from the testng-users mailing list archive at Nabble.com.

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to <a href="javascript:;" onclick="_e(event, &#39;cvml&#39;, &#39;testng-users@googlegroups.com&#39;)">testng-users@....
To unsubscribe from this group, send email to <a href="javascript:;" onclick="_e(event, &#39;cvml&#39;, &#39;testng-users+unsubscribe@googlegroups.com&#39;)">testng-users+unsubscribe@....
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.



--
Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @ http://wakened-cognition.blogspot.com/

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.