@BeforeClass method not run (or completed) before tests run

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

@BeforeClass method not run (or completed) before tests run

Jeff-351
I'm seeing what appears to be an issue with my multi-threaded test methods running before the @BeforeClass setup() method has completed (or even run).

I've attached a screen shot where I catch the NullPointerException.  

The ordering *should be* (I thought):

  1. Run @BeforeClass setup()
    1. sets urlPath using parameter passed to setup()
    2. setup() also creates the instance of the Apache HTTP client stored as 'client'
  2. Run @Test test_getSettings()
    1. construct the value for 'url'.
      1. NOTE: The url value has a 'null' for 'urlPath'
    2. Make HTTP request using 'client' 
      1. NOTE: At this point 'client' is null causing NullPointerException
When I catch the NullPointerException and inspect the value for 'urlPath' as well as 'client', they are valid, but 'url' was constructed with a null value for 'urlPath'.  This suggests that either the setup() method didn't run, or (due to the threading) has not completed before the test method runs.

Shouldn't @BeforeClass methods run and complete before @Test methods even when tests are run in parallel?  Or am I missing something dead simple and just need another week off?

Thanks!


Screenshot:

Inline image 1

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: @BeforeClass method not run (or completed) before tests run

Jeff-351
I added some additional print statements and watched the output and have verified that a number of test methods start running before the setup completes (at least based on the output ordering).

-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Running MalwareFeedbackApi.Test_Settings_v1
Starting...
Running setup.
Running MalwareFeedbackApi.Test_Settings_v1.testGetSettingsSerialCertificate() on Thread [21]
Running MalwareFeedbackApi.Test_Settings_v1.test_getSettings() on Thread [23]
Running MalwareFeedbackApi.Test_Settings_v1.test_NoVersion() on Thread [22]
    client: null
Done with setup.
Running MalwareFeedbackApi.Test_Settings_v1.testGetSettingsBadSerial() on Thread [20]

On Wed, Jan 3, 2018 at 10:53 AM, Jeff <[hidden email]> wrote:
I'm seeing what appears to be an issue with my multi-threaded test methods running before the @BeforeClass setup() method has completed (or even run).

I've attached a screen shot where I catch the NullPointerException.  

The ordering *should be* (I thought):

  1. Run @BeforeClass setup()
    1. sets urlPath using parameter passed to setup()
    2. setup() also creates the instance of the Apache HTTP client stored as 'client'
  2. Run @Test test_getSettings()
    1. construct the value for 'url'.
      1. NOTE: The url value has a 'null' for 'urlPath'
    2. Make HTTP request using 'client' 
      1. NOTE: At this point 'client' is null causing NullPointerException
When I catch the NullPointerException and inspect the value for 'urlPath' as well as 'client', they are valid, but 'url' was constructed with a null value for 'urlPath'.  This suggests that either the setup() method didn't run, or (due to the threading) has not completed before the test method runs.

Shouldn't @BeforeClass methods run and complete before @Test methods even when tests are run in parallel?  Or am I missing something dead simple and just need another week off?

Thanks!


Screenshot:

Inline image 1

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: @BeforeClass method not run (or completed) before tests run

Todd Bradley-2
Interesting. Normally I'd discount the order of output from a multithreaded application, since we know it's not reliable. But your debugger output is hard to explain. What does super.setup() look like? What are you doing with the ITestContext? It seems like you don't need it. Also, what results do you see when you run this from outside your IDE? It's possible your IDE's test runner is broken.


Todd. 

On Wed, Jan 3, 2018 at 11:25 AM, Jeff <[hidden email]> wrote:
I added some additional print statements and watched the output and have verified that a number of test methods start running before the setup completes (at least based on the output ordering).

-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Running MalwareFeedbackApi.Test_Settings_v1
Starting...
Running setup.
Running MalwareFeedbackApi.Test_Settings_v1.testGetSettingsSerialCertificate() on Thread [21]
Running MalwareFeedbackApi.Test_Settings_v1.test_getSettings() on Thread [23]
Running MalwareFeedbackApi.Test_Settings_v1.test_NoVersion() on Thread [22]
    client: null
Done with setup.
Running MalwareFeedbackApi.Test_Settings_v1.testGetSettingsBadSerial() on Thread [20]

On Wed, Jan 3, 2018 at 10:53 AM, Jeff <[hidden email]> wrote:
I'm seeing what appears to be an issue with my multi-threaded test methods running before the @BeforeClass setup() method has completed (or even run).

I've attached a screen shot where I catch the NullPointerException.  

The ordering *should be* (I thought):

  1. Run @BeforeClass setup()
    1. sets urlPath using parameter passed to setup()
    2. setup() also creates the instance of the Apache HTTP client stored as 'client'
  2. Run @Test test_getSettings()
    1. construct the value for 'url'.
      1. NOTE: The url value has a 'null' for 'urlPath'
    2. Make HTTP request using 'client' 
      1. NOTE: At this point 'client' is null causing NullPointerException
When I catch the NullPointerException and inspect the value for 'urlPath' as well as 'client', they are valid, but 'url' was constructed with a null value for 'urlPath'.  This suggests that either the setup() method didn't run, or (due to the threading) has not completed before the test method runs.

Shouldn't @BeforeClass methods run and complete before @Test methods even when tests are run in parallel?  Or am I missing something dead simple and just need another week off?

Thanks!


Screenshot:

Inline image 1

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: @BeforeClass method not run (or completed) before tests run

Jeff-351
Thanks for taking the time to respond!  

The super.setup() does the following:
  • loads test configuration properties
  • instantiates the HttpClient 
  • sets the urlPath
  • connects to the target server via SSH (net.schmizz.sshj.SSHClient)
    • stops tomcat
    • restores the default configuration (in the case the previous test terminated abnormally)
    • reads default config into memory for the tests to use
    • cleans out all log files
    • restarts tomcat server
  • configures the AmazonSQSAsyncClient instance
  • Using iTestContext 
    • calls ctx.getSuite().getXmlSuite()
    • retrieves suiteName for use in debug messages
    • sets suite.setThreadCount
    • sets suite.setDataProviderThreadCount
The calls on the XmlSuite to update the thread counts were added (if I recall correctly...its been a while) because it seemed to be the only way I could get the number of test threads I wanted to take effect when using Maven and debugging in NetBeans.  I'm sure my current solution could be improved.

As for running the tests, I'm using NetBeans 8.2 but the result is the same running the Maven command from the console.  I thought that maybe my testng.xml file was causing havoc, so I removed all references to the xml file in the POM file and ran the command:

mvn test -Pwp-acc -Dtest=Test_Settings_v1

with output:

[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running MalwareFeedbackApi.Test_Settings_v1
Starting...
Running setup.
Running MalwareFeedbackApi.Test_Settings_v1.test_NoVersion() on Thread [20]
Running MalwareFeedbackApi.Test_Settings_v1.test_getSettings() on Thread [21]
    client: null
    url: http://malwarefeedback-acc01.mycompany.com:8080null/1
Running MalwareFeedbackApi.Test_Settings_v1.testGetSettingsSerialCertificate() on Thread [19]
Done with setup.
Running MalwareFeedbackApi.Test_Settings_v1.testGetSettingsBadSerial() on Thread [18]

I'm in the process of adding a mechanism to set a 'setupComplete' variable when setup is done.  Each test will have to monitor the status at the beginning and wait for setup to complete.  I don't know how else to get this working.

-Jeff

On Wed, Jan 3, 2018 at 12:12 PM, Todd Bradley <[hidden email]> wrote:
Interesting. Normally I'd discount the order of output from a multithreaded application, since we know it's not reliable. But your debugger output is hard to explain. What does super.setup() look like? What are you doing with the ITestContext? It seems like you don't need it. Also, what results do you see when you run this from outside your IDE? It's possible your IDE's test runner is broken.


Todd. 

On Wed, Jan 3, 2018 at 11:25 AM, Jeff <[hidden email]> wrote:
I added some additional print statements and watched the output and have verified that a number of test methods start running before the setup completes (at least based on the output ordering).

-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Running MalwareFeedbackApi.Test_Settings_v1
Starting...
Running setup.
Running MalwareFeedbackApi.Test_Settings_v1.testGetSettingsSerialCertificate() on Thread [21]
Running MalwareFeedbackApi.Test_Settings_v1.test_getSettings() on Thread [23]
Running MalwareFeedbackApi.Test_Settings_v1.test_NoVersion() on Thread [22]
    client: null
Done with setup.
Running MalwareFeedbackApi.Test_Settings_v1.testGetSettingsBadSerial() on Thread [20]

On Wed, Jan 3, 2018 at 10:53 AM, Jeff <[hidden email]> wrote:
I'm seeing what appears to be an issue with my multi-threaded test methods running before the @BeforeClass setup() method has completed (or even run).

I've attached a screen shot where I catch the NullPointerException.  

The ordering *should be* (I thought):

  1. Run @BeforeClass setup()
    1. sets urlPath using parameter passed to setup()
    2. setup() also creates the instance of the Apache HTTP client stored as 'client'
  2. Run @Test test_getSettings()
    1. construct the value for 'url'.
      1. NOTE: The url value has a 'null' for 'urlPath'
    2. Make HTTP request using 'client' 
      1. NOTE: At this point 'client' is null causing NullPointerException
When I catch the NullPointerException and inspect the value for 'urlPath' as well as 'client', they are valid, but 'url' was constructed with a null value for 'urlPath'.  This suggests that either the setup() method didn't run, or (due to the threading) has not completed before the test method runs.

Shouldn't @BeforeClass methods run and complete before @Test methods even when tests are run in parallel?  Or am I missing something dead simple and just need another week off?

Thanks!


Screenshot:

Inline image 1

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: @BeforeClass method not run (or completed) before tests run

Krishnan Mahadevan

Jeff,

 

That looks like a straight forward bug to me.

 

I have logged an issue for tracking this : https://github.com/cbeust/testng/issues/1660

 

I was able to build a very simple test case, which can be executed to reproduce the problem that you are seeing.

 

I believe that all Test methods should start running only after all of the configuration methods have run to completion. At-least that’s my basic understanding. I attribute parallel execution to only test methods and not to parallel configurations.

 

Please add yourself as a watcher to this issue so that you can keep an eye on it. I will try to get to this issue during this weekend.

 

Thanks & Regards

Krishnan Mahadevan

 

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"

My Scribblings @ http://wakened-cognition.blogspot.com/

My Technical Scribbings @ http://rationaleemotions.wordpress.com/

 

From: <[hidden email]> on behalf of Jeff <[hidden email]>
Reply-To: <[hidden email]>
Date: Thursday, January 4, 2018 at 1:24 AM
To: <[hidden email]>
Subject: Re: [testng-users] Re: @BeforeClass method not run (or completed) before tests run

 

Thanks for taking the time to respond!  

 

The super.setup() does the following:

  • loads test configuration properties
  • instantiates the HttpClient 
  • sets the urlPath
  • connects to the target server via SSH (net.schmizz.sshj.SSHClient)
    • stops tomcat
    • restores the default configuration (in the case the previous test terminated abnormally)
    • reads default config into memory for the tests to use
    • cleans out all log files
    • restarts tomcat server
  • configures the AmazonSQSAsyncClient instance
  • Using iTestContext 
    • calls ctx.getSuite().getXmlSuite()
    • retrieves suiteName for use in debug messages
    • sets suite.setThreadCount
    • sets suite.setDataProviderThreadCount

The calls on the XmlSuite to update the thread counts were added (if I recall correctly...its been a while) because it seemed to be the only way I could get the number of test threads I wanted to take effect when using Maven and debugging in NetBeans.  I'm sure my current solution could be improved.

 

As for running the tests, I'm using NetBeans 8.2 but the result is the same running the Maven command from the console.  I thought that maybe my testng.xml file was causing havoc, so I removed all references to the xml file in the POM file and ran the command:

 

mvn test -Pwp-acc -Dtest=Test_Settings_v1

 

with output:

 

[INFO] -------------------------------------------------------

[INFO]  T E S T S

[INFO] -------------------------------------------------------

[INFO] Running MalwareFeedbackApi.Test_Settings_v1

Starting...

Running setup.

Running MalwareFeedbackApi.Test_Settings_v1.test_NoVersion() on Thread [20]

Running MalwareFeedbackApi.Test_Settings_v1.test_getSettings() on Thread [21]

    client: null

    url: http://malwarefeedback-acc01.mycompany.com:8080null/1

Running MalwareFeedbackApi.Test_Settings_v1.testGetSettingsSerialCertificate() on Thread [19]

Done with setup.

Running MalwareFeedbackApi.Test_Settings_v1.testGetSettingsBadSerial() on Thread [18]

 

I'm in the process of adding a mechanism to set a 'setupComplete' variable when setup is done.  Each test will have to monitor the status at the beginning and wait for setup to complete.  I don't know how else to get this working.

 

-Jeff

 

On Wed, Jan 3, 2018 at 12:12 PM, Todd Bradley <[hidden email]> wrote:

Interesting. Normally I'd discount the order of output from a multithreaded application, since we know it's not reliable. But your debugger output is hard to explain. What does super.setup() look like? What are you doing with the ITestContext? It seems like you don't need it. Also, what results do you see when you run this from outside your IDE? It's possible your IDE's test runner is broken.

 

 

Todd. 

 

On Wed, Jan 3, 2018 at 11:25 AM, Jeff <[hidden email]> wrote:

I added some additional print statements and watched the output and have verified that a number of test methods start running before the setup completes (at least based on the output ordering).

 

-------------------------------------------------------

 T E S T S

-------------------------------------------------------

Running MalwareFeedbackApi.Test_Settings_v1

Starting...

Running setup.

Running MalwareFeedbackApi.Test_Settings_v1.testGetSettingsSerialCertificate() on Thread [21]

Running MalwareFeedbackApi.Test_Settings_v1.test_getSettings() on Thread [23]

Running MalwareFeedbackApi.Test_Settings_v1.test_NoVersion() on Thread [22]

    client: null

Done with setup.

Running MalwareFeedbackApi.Test_Settings_v1.testGetSettingsBadSerial() on Thread [20]

 

On Wed, Jan 3, 2018 at 10:53 AM, Jeff <[hidden email]> wrote:

I'm seeing what appears to be an issue with my multi-threaded test methods running before the @BeforeClass setup() method has completed (or even run).

 

I've attached a screen shot where I catch the NullPointerException.  

 

The ordering *should be* (I thought):

 

  1. Run @BeforeClass setup()
    1. sets urlPath using parameter passed to setup()
    2. setup() also creates the instance of the Apache HTTP client stored as 'client'
  1. Run @Test test_getSettings()
    1. construct the value for 'url'.
      1. NOTE: The url value has a 'null' for 'urlPath'
    1. Make HTTP request using 'client' 
      1. NOTE: At this point 'client' is null causing NullPointerException

When I catch the NullPointerException and inspect the value for 'urlPath' as well as 'client', they are valid, but 'url' was constructed with a null value for 'urlPath'.  This suggests that either the setup() method didn't run, or (due to the threading) has not completed before the test method runs.

 

Shouldn't @BeforeClass methods run and complete before @Test methods even when tests are run in parallel?  Or am I missing something dead simple and just need another week off?

 

Thanks!

 

 

Screenshot:

 

nline image 1

 

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.

 

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.

 

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: @BeforeClass method not run (or completed) before tests run

Jeff-351
Thank you so much for the quick response and validation that I am not losing my mind!  I was beating my head over the holiday due to this issue thinking I was doing something terribly wrong.  This particular test suite I'm working on is one of my most complex and it's been difficult to get things working right.

Regards,

Jeff


On Wed, Jan 3, 2018 at 8:06 PM, Krishnan Mahadevan <[hidden email]> wrote:

Jeff,

 

That looks like a straight forward bug to me.

 

I have logged an issue for tracking this : https://github.com/cbeust/testng/issues/1660

 

I was able to build a very simple test case, which can be executed to reproduce the problem that you are seeing.

 

I believe that all Test methods should start running only after all of the configuration methods have run to completion. At-least that’s my basic understanding. I attribute parallel execution to only test methods and not to parallel configurations.

 

Please add yourself as a watcher to this issue so that you can keep an eye on it. I will try to get to this issue during this weekend.

 

Thanks & Regards

Krishnan Mahadevan

 

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"

My Scribblings @ http://wakened-cognition.blogspot.com/

My Technical Scribbings @ http://rationaleemotions.wordpress.com/

 

From: <[hidden email]> on behalf of Jeff <[hidden email]>
Reply-To: <[hidden email]>
Date: Thursday, January 4, 2018 at 1:24 AM
To: <[hidden email]>
Subject: Re: [testng-users] Re: @BeforeClass method not run (or completed) before tests run

 

Thanks for taking the time to respond!  

 

The super.setup() does the following:

  • loads test configuration properties
  • instantiates the HttpClient 
  • sets the urlPath
  • connects to the target server via SSH (net.schmizz.sshj.SSHClient)
    • stops tomcat
    • restores the default configuration (in the case the previous test terminated abnormally)
    • reads default config into memory for the tests to use
    • cleans out all log files
    • restarts tomcat server
  • configures the AmazonSQSAsyncClient instance
  • Using iTestContext 
    • calls ctx.getSuite().getXmlSuite()
    • retrieves suiteName for use in debug messages
    • sets suite.setThreadCount
    • sets suite.setDataProviderThreadCount

The calls on the XmlSuite to update the thread counts were added (if I recall correctly...its been a while) because it seemed to be the only way I could get the number of test threads I wanted to take effect when using Maven and debugging in NetBeans.  I'm sure my current solution could be improved.

 

As for running the tests, I'm using NetBeans 8.2 but the result is the same running the Maven command from the console.  I thought that maybe my testng.xml file was causing havoc, so I removed all references to the xml file in the POM file and ran the command:

 

mvn test -Pwp-acc -Dtest=Test_Settings_v1

 

with output:

 

[INFO] -------------------------------------------------------

[INFO]  T E S T S

[INFO] -------------------------------------------------------

[INFO] Running MalwareFeedbackApi.Test_Settings_v1

Starting...

Running setup.

Running MalwareFeedbackApi.Test_Settings_v1.test_NoVersion() on Thread [20]

Running MalwareFeedbackApi.Test_Settings_v1.test_getSettings() on Thread [21]

    client: null

    url: http://malwarefeedback-acc01.mycompany.com:8080null/1

Running MalwareFeedbackApi.Test_Settings_v1.testGetSettingsSerialCertificate() on Thread [19]

Done with setup.

Running MalwareFeedbackApi.Test_Settings_v1.testGetSettingsBadSerial() on Thread [18]

 

I'm in the process of adding a mechanism to set a 'setupComplete' variable when setup is done.  Each test will have to monitor the status at the beginning and wait for setup to complete.  I don't know how else to get this working.

 

-Jeff

 

On Wed, Jan 3, 2018 at 12:12 PM, Todd Bradley <[hidden email]> wrote:

Interesting. Normally I'd discount the order of output from a multithreaded application, since we know it's not reliable. But your debugger output is hard to explain. What does super.setup() look like? What are you doing with the ITestContext? It seems like you don't need it. Also, what results do you see when you run this from outside your IDE? It's possible your IDE's test runner is broken.

 

 

Todd. 

 

On Wed, Jan 3, 2018 at 11:25 AM, Jeff <[hidden email]> wrote:

I added some additional print statements and watched the output and have verified that a number of test methods start running before the setup completes (at least based on the output ordering).

 

-------------------------------------------------------

 T E S T S

-------------------------------------------------------

Running MalwareFeedbackApi.Test_Settings_v1

Starting...

Running setup.

Running MalwareFeedbackApi.Test_Settings_v1.testGetSettingsSerialCertificate() on Thread [21]

Running MalwareFeedbackApi.Test_Settings_v1.test_getSettings() on Thread [23]

Running MalwareFeedbackApi.Test_Settings_v1.test_NoVersion() on Thread [22]

    client: null

Done with setup.

Running MalwareFeedbackApi.Test_Settings_v1.testGetSettingsBadSerial() on Thread [20]

 

On Wed, Jan 3, 2018 at 10:53 AM, Jeff <[hidden email]> wrote:

I'm seeing what appears to be an issue with my multi-threaded test methods running before the @BeforeClass setup() method has completed (or even run).

 

I've attached a screen shot where I catch the NullPointerException.  

 

The ordering *should be* (I thought):

 

  1. Run @BeforeClass setup()
    1. sets urlPath using parameter passed to setup()
    2. setup() also creates the instance of the Apache HTTP client stored as 'client'
  1. Run @Test test_getSettings()
    1. construct the value for 'url'.
      1. NOTE: The url value has a 'null' for 'urlPath'
    1. Make HTTP request using 'client' 
      1. NOTE: At this point 'client' is null causing NullPointerException

When I catch the NullPointerException and inspect the value for 'urlPath' as well as 'client', they are valid, but 'url' was constructed with a null value for 'urlPath'.  This suggests that either the setup() method didn't run, or (due to the threading) has not completed before the test method runs.

 

Shouldn't @BeforeClass methods run and complete before @Test methods even when tests are run in parallel?  Or am I missing something dead simple and just need another week off?

 

Thanks!

 

 

Screenshot:

 

nline image 1

 

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.

 

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.

 

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: @BeforeClass method not run (or completed) before tests run

Krishnan Mahadevan

No probs Jeff. It would also be great if you could quickly validate the same in the issue and confirm if it matches your scenario.


On Thu, Jan 4, 2018, 09:15 Jeff <[hidden email]> wrote:
Thank you so much for the quick response and validation that I am not losing my mind!  I was beating my head over the holiday due to this issue thinking I was doing something terribly wrong.  This particular test suite I'm working on is one of my most complex and it's been difficult to get things working right.

Regards,

Jeff


On Wed, Jan 3, 2018 at 8:06 PM, Krishnan Mahadevan <[hidden email]> wrote:

Jeff,

 

That looks like a straight forward bug to me.

 

I have logged an issue for tracking this : https://github.com/cbeust/testng/issues/1660

 

I was able to build a very simple test case, which can be executed to reproduce the problem that you are seeing.

 

I believe that all Test methods should start running only after all of the configuration methods have run to completion. At-least that’s my basic understanding. I attribute parallel execution to only test methods and not to parallel configurations.

 

Please add yourself as a watcher to this issue so that you can keep an eye on it. I will try to get to this issue during this weekend.

 

Thanks & Regards

Krishnan Mahadevan

 

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"

My Scribblings @ http://wakened-cognition.blogspot.com/

My Technical Scribbings @ http://rationaleemotions.wordpress.com/

 

From: <[hidden email]> on behalf of Jeff <[hidden email]>
Reply-To: <[hidden email]>
Date: Thursday, January 4, 2018 at 1:24 AM
To: <[hidden email]>
Subject: Re: [testng-users] Re: @BeforeClass method not run (or completed) before tests run

 

Thanks for taking the time to respond!  

 

The super.setup() does the following:

  • loads test configuration properties
  • instantiates the HttpClient 
  • sets the urlPath
  • connects to the target server via SSH (net.schmizz.sshj.SSHClient)
    • stops tomcat
    • restores the default configuration (in the case the previous test terminated abnormally)
    • reads default config into memory for the tests to use
    • cleans out all log files
    • restarts tomcat server
  • configures the AmazonSQSAsyncClient instance
  • Using iTestContext 
    • calls ctx.getSuite().getXmlSuite()
    • retrieves suiteName for use in debug messages
    • sets suite.setThreadCount
    • sets suite.setDataProviderThreadCount

The calls on the XmlSuite to update the thread counts were added (if I recall correctly...its been a while) because it seemed to be the only way I could get the number of test threads I wanted to take effect when using Maven and debugging in NetBeans.  I'm sure my current solution could be improved.

 

As for running the tests, I'm using NetBeans 8.2 but the result is the same running the Maven command from the console.  I thought that maybe my testng.xml file was causing havoc, so I removed all references to the xml file in the POM file and ran the command:

 

mvn test -Pwp-acc -Dtest=Test_Settings_v1

 

with output:

 

[INFO] -------------------------------------------------------

[INFO]  T E S T S

[INFO] -------------------------------------------------------

[INFO] Running MalwareFeedbackApi.Test_Settings_v1

Starting...

Running setup.

Running MalwareFeedbackApi.Test_Settings_v1.test_NoVersion() on Thread [20]

Running MalwareFeedbackApi.Test_Settings_v1.test_getSettings() on Thread [21]

    client: null

    url: http://malwarefeedback-acc01.mycompany.com:8080null/1

Running MalwareFeedbackApi.Test_Settings_v1.testGetSettingsSerialCertificate() on Thread [19]

Done with setup.

Running MalwareFeedbackApi.Test_Settings_v1.testGetSettingsBadSerial() on Thread [18]

 

I'm in the process of adding a mechanism to set a 'setupComplete' variable when setup is done.  Each test will have to monitor the status at the beginning and wait for setup to complete.  I don't know how else to get this working.

 

-Jeff

 

On Wed, Jan 3, 2018 at 12:12 PM, Todd Bradley <[hidden email]> wrote:

Interesting. Normally I'd discount the order of output from a multithreaded application, since we know it's not reliable. But your debugger output is hard to explain. What does super.setup() look like? What are you doing with the ITestContext? It seems like you don't need it. Also, what results do you see when you run this from outside your IDE? It's possible your IDE's test runner is broken.

 

 

Todd. 

 

On Wed, Jan 3, 2018 at 11:25 AM, Jeff <[hidden email]> wrote:

I added some additional print statements and watched the output and have verified that a number of test methods start running before the setup completes (at least based on the output ordering).

 

-------------------------------------------------------

 T E S T S

-------------------------------------------------------

Running MalwareFeedbackApi.Test_Settings_v1

Starting...

Running setup.

Running MalwareFeedbackApi.Test_Settings_v1.testGetSettingsSerialCertificate() on Thread [21]

Running MalwareFeedbackApi.Test_Settings_v1.test_getSettings() on Thread [23]

Running MalwareFeedbackApi.Test_Settings_v1.test_NoVersion() on Thread [22]

    client: null

Done with setup.

Running MalwareFeedbackApi.Test_Settings_v1.testGetSettingsBadSerial() on Thread [20]

 

On Wed, Jan 3, 2018 at 10:53 AM, Jeff <[hidden email]> wrote:

I'm seeing what appears to be an issue with my multi-threaded test methods running before the @BeforeClass setup() method has completed (or even run).

 

I've attached a screen shot where I catch the NullPointerException.  

 

The ordering *should be* (I thought):

 

  1. Run @BeforeClass setup()
    1. sets urlPath using parameter passed to setup()
    2. setup() also creates the instance of the Apache HTTP client stored as 'client'
  1. Run @Test test_getSettings()
    1. construct the value for 'url'.
      1. NOTE: The url value has a 'null' for 'urlPath'
    1. Make HTTP request using 'client' 
      1. NOTE: At this point 'client' is null causing NullPointerException

When I catch the NullPointerException and inspect the value for 'urlPath' as well as 'client', they are valid, but 'url' was constructed with a null value for 'urlPath'.  This suggests that either the setup() method didn't run, or (due to the threading) has not completed before the test method runs.

 

Shouldn't @BeforeClass methods run and complete before @Test methods even when tests are run in parallel?  Or am I missing something dead simple and just need another week off?

 

Thanks!

 

 

Screenshot:

 

 

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.

 

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.

 

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.
--

Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @ http://wakened-cognition.blogspot.com/

My Technical Scribbings @ http://rationaleemotions.wordpress.com/

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.

image001.png (81K) Download Attachment