[testng-dev] Probable race while running TestNG with multiple xml suites (maven, Surefire)

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

[testng-dev] Probable race while running TestNG with multiple xml suites (maven, Surefire)

Nikolay Shustov
Hi,
I would like apologize in advance for not attaching the reproducible test case - I do have one but:
1. The complete project code is in commercial domain so I am unable to share it.
2. I could not make a smaller test case that would reproduce the issue.

I am open to suggestion for collecting logs, stack traces etc. for somebody who would like to look into that problem.

Synopsis:

I run TestNG tests from maven, using Surefire plugin. I use TestNG XML configuration file that includes multiple suite configurations.
When I run the tests from maven, I detect that my @BeforeSuite method is called only once, for the very first suite and is not called for the next one.
Thus, my second suite of tests fails as @BeforeSuite method contains an important logic that initializes the test environment.
TestNG is configured to run tests in non-parallel mode.
This issue has been mostly intermittent but in my environment, I found set of steps that reproduces is in 99.9%.

Details:

Tests are run using XML configuration file that looks like the following:

testng.xml:

<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd" >

<suite name="My-Suite-Name" verbose="2" >
  <suite-files>
    <suite-file path="offline.xml"/>
    <suite-file path="server.xml"/>
    <suite-file path="online.xml"/>
    <suite-file path="online-encrypted.xml"/>
  </suite-files>
</suite>

offline.xml:

<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd" >

<suite name="my-offline" verbose="2" >
  <test name="all-standalone">
    <packages>
      <package name="com.blah1.blah2.*"/>
    </packages>
    <groups>
      <run>
        <include name="my-offline"/>
      </run>
    </groups>
 </test>
</suite>

online.xml:

<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd" >

<suite name="my-online" verbose="2" >
  <parameter name="PARAMETER1" value="value1"/>
  <test name="all-my-online">
    <packages>
      <package name="com.blah1.blah2.*"/>
    </packages>
    <groups>
      <run>
        <include name="my-group1"/>
        <exclude name="my-group2"/>
        <exclude name="my-offline"/>
      </run>
    </groups>
 </test>
</suite>

In my test infrastructure, I have the @BeforeSuite method that is supposed to initalize the test environment.

@Parameters({
"PARAMETER1",
"PARAMETER2",
"PARAMETER3",
"PARAMETER4",
"PARAMETER5"})
@BeforeSuite(alwaysRun = true)
protected void suiteSetUp(
@Optional() String parameter1,
@Optional() String parameter2,
@Optional() String parameter3,
@Optional() String parameter4,
@Optional() String parameter5)
{
Map<String, String> parameters = new HashMap<>();
parameters.put("PARAMETER1", parameter1);
parameters.put("PARAMETER2", parameter2);
parameters.put("PARAMETER3", parameter3);
parameters.put("PARAMETER4", parameter4);
parameters.put("PARAMETER5", parameter5);

setupSuite(parameters);
}

All the tests I have belong to multiple test groups.
Test suites XML configuration are designed to setup various set of parameters for the groups of the tests, described in each configuration.

What happens is that tests from "my-offline" tests suite run OK, but then TestNG continues with tests that belong to the "my-online" suite, without having @BeforeSuite method run again (with proper parameters setup in XML configuration).
"my-online" set of tests fails then, as the logic invoked from @BeforeSuite method is supposed to set tests environment (run a few external processes etc.)
I can detect that @BeforeSuite was not run on the second time as my tests environment initialization logic prints out the diagnosis info which differ for every XML TestNG configuration.

Special conditions, when it is reproducible:

1. I have to run  "mvn clean install -e -pl myproject -am -B -fae -Dmaven.javadoc.skip=true".
   Running "mvn test <rest of options>" does not reproduce the issue.
   Compiling the classes in the separate step "mvn compile" then "mvn install" does not allow to reproduce the issue.

2. Build includes ~2500 classes. With smaller codebase I have troubles reproducing the issue.

3. Environment:
Linux <hostname skipped> 2.6.32-358.el6.x86_64 #1 SMP Tue Jan 29 11:47:41 EST 2013 x86_64 x86_64 x86_64 GNU/Linux

Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                2
On-line CPU(s) list:   0,1
Thread(s) per core:    1
Core(s) per socket:    1
Socket(s):             2
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 44
Stepping:              2
CPU MHz:               3332.386
BogoMIPS:              6664.77
Hypervisor vendor:     VMware
Virtualization type:   full
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              12288K
NUMA node0 CPU(s):     0,1

4Gb of memory

Found workarounds:

When I modify my @BeforeSuite method like the following, the problem goes away:

@Parameters({
"PARAMETER1",
"PARAMETER2",
"PARAMETER3",
"PARAMETER4",
"PARAMETER5"})
@BeforeSuite(alwaysRun = true)
protected void suiteSetUp(
ITestContext testContext, // added
@Optional() String parameter1,
@Optional() String parameter2,
@Optional() String parameter3,
@Optional() String parameter4,
@Optional() String parameter5)
{
// added test suite name printout
if(testContext.getCurrentXmlTest() != null)
System.out.printf("Running test suite %1$s%n", testContext.getCurrentXmlTest().getSuite().getName());

Map<String, String> parameters = new HashMap<>();
parameters.put("PARAMETER1", parameter1);
parameters.put("PARAMETER2", parameter2);
parameters.put("PARAMETER3", parameter3);
parameters.put("PARAMETER4", parameter4);
parameters.put("PARAMETER5", parameter5);

setupSuite(parameters);
}

Note that with the following modification, the problem is still reproducible:


@Parameters({
"PARAMETER1",
"PARAMETER2",
"PARAMETER3",
"PARAMETER4",
"PARAMETER5"})
@BeforeSuite(alwaysRun = true)
protected void suiteSetUp(
ITestContext testContext, // added
@Optional() String parameter1,
@Optional() String parameter2,
@Optional() String parameter3,
@Optional() String parameter4,
@Optional() String parameter5)
{
// added test suite name printout
if(testContext.getCurrentXmlTest() != null); // we do not print test suite name here
// System.out.printf("Running test suite %1$s%n", testContext.getCurrentXmlTest().getSuite().getName());

Map<String, String> parameters = new HashMap<>();
parameters.put("PARAMETER1", parameter1);
parameters.put("PARAMETER2", parameter2);
parameters.put("PARAMETER3", parameter3);
parameters.put("PARAMETER4", parameter4);
parameters.put("PARAMETER5", parameter5);

setupSuite(parameters);
}

...which makes me strongly suspect a race condition on testContext.getCurrentXmlTest().getSuite() logic.
getSuite() apparenty just returns m_suite and it appears to me that something is wrong with getCurrentXmlTest() .

Thank you in advance for looking into it.

- Nikolay

--
You received this message because you are subscribed to the Google Groups "testng-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at http://groups.google.com/group/testng-dev.
For more options, visit https://groups.google.com/d/optout.