[testng-dev] New feature: deferred execution of factory constructors

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

[testng-dev] New feature: deferred execution of factory constructors

Alex
Hi,

In an earlier thread (https://groups.google.com/forum/?fromgroups=#!topic/testng-users/3mGX6xChv64), I brought up the issue where @Factory constructors are always executed even if its classes test methods are completely excluded.  This is an undesirable behavior as running certain data providers (at least, for what we're testing) can be very expensive, and in some cases not work in certain environments.  I would like to be able to exclude these as easily as I can exclude the test cases themselves.  To do this, I propose a two-pass test method discovery process:

1.  Enumerate all test classes and methods using default ctors, and non-ctor factories 
2.  From the list of test methods that returns, build a list of test classes that actually get used
3.  Subtract that list from list of all test classes
4.  Repeat step 1, but expanding the ctor factories this time

This requires two small changes to TestRunner and TestNGClassFinder.  The class finder accepts a flag indicating whether or not this is a test method discovery pass or not.  If the flag is set, we ignore any @Factory constructors on the object.

In TestNGClassFinder.ctor, 

if (factoryMethod != null) {

becomes:

if (factoryMethod != null && factoryMethod.getEnabled() && (m_discoveryMode == false || factoryMethod.getConstructor() == null)) {

(...where m_discoveryMode is passed into the ctor.)

To use this, the TestRunner will need to call what formerly was initMethods twice:

private void initMethods() {
    //  1st pass: find all TestNG methods from all test classes.
    ClassInfoMap classMap = new ClassInfoMap(m_testClassesFromXml);
    scanForTestNGMethods(classMap, true);

    ... create list of test classes that are actually used from m_allTestMethods ...

    //  2nd pass: initialize all test classes that have included test methods.
    classMap = new ClassInfoMap(testClassSubset);
    scanForTestNGMethods(classMap, false);
}

The method scanForTestNGMethods is what initMethods is now, except that the classMap and discovery flag get sent in as parameters.

All existing TestNG unit tests pass, and my own test suites are now properly excluding unnecessary factories.  Also, while I would've expected this to have the nasty side effect of double-execution of non-ctor factories, there seems to be some clever caching going on behind the scenes that prevents that, so this change shouldn't cause any feature or functionality regressions.

What do you guys think?  Is this change this easy?  Or am I missing something here?  I understand that this won't (and probably never can) work for non-ctor factories as we'll often have no clue what those factories are even going to produce until they are actually executed, but this should work nicely for the factory-ctor case.

-Alex

--
You received this message because you are subscribed to the Google Groups "testng-dev" group.
To view this discussion on the web visit https://groups.google.com/d/msg/testng-dev/-/JwlLnHeDSwwJ.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/testng-dev?hl=en.
Reply | Threaded
Open this post in threaded view
|

Re: [testng-dev] New feature: deferred execution of factory constructors

Cédric Beust ♔-2
Hi Alex,

Before commenting, if it's not too much trouble, could you please send me a pull request? This way I can review your change in its entirety, which should give me a good idea of what's going on.

Thanks!


-- 
Cédric



On Mon, Jan 7, 2013 at 2:04 PM, Alex <[hidden email]> wrote:
Hi,

In an earlier thread (https://groups.google.com/forum/?fromgroups=#!topic/testng-users/3mGX6xChv64), I brought up the issue where @Factory constructors are always executed even if its classes test methods are completely excluded.  This is an undesirable behavior as running certain data providers (at least, for what we're testing) can be very expensive, and in some cases not work in certain environments.  I would like to be able to exclude these as easily as I can exclude the test cases themselves.  To do this, I propose a two-pass test method discovery process:

1.  Enumerate all test classes and methods using default ctors, and non-ctor factories 
2.  From the list of test methods that returns, build a list of test classes that actually get used
3.  Subtract that list from list of all test classes
4.  Repeat step 1, but expanding the ctor factories this time

This requires two small changes to TestRunner and TestNGClassFinder.  The class finder accepts a flag indicating whether or not this is a test method discovery pass or not.  If the flag is set, we ignore any @Factory constructors on the object.

In TestNGClassFinder.ctor, 

if (factoryMethod != null) {

becomes:

if (factoryMethod != null && factoryMethod.getEnabled() && (m_discoveryMode == false || factoryMethod.getConstructor() == null)) {

(...where m_discoveryMode is passed into the ctor.)

To use this, the TestRunner will need to call what formerly was initMethods twice:

private void initMethods() {
    //  1st pass: find all TestNG methods from all test classes.
    ClassInfoMap classMap = new ClassInfoMap(m_testClassesFromXml);
    scanForTestNGMethods(classMap, true);

    ... create list of test classes that are actually used from m_allTestMethods ...

    //  2nd pass: initialize all test classes that have included test methods.
    classMap = new ClassInfoMap(testClassSubset);
    scanForTestNGMethods(classMap, false);
}

The method scanForTestNGMethods is what initMethods is now, except that the classMap and discovery flag get sent in as parameters.

All existing TestNG unit tests pass, and my own test suites are now properly excluding unnecessary factories.  Also, while I would've expected this to have the nasty side effect of double-execution of non-ctor factories, there seems to be some clever caching going on behind the scenes that prevents that, so this change shouldn't cause any feature or functionality regressions.

What do you guys think?  Is this change this easy?  Or am I missing something here?  I understand that this won't (and probably never can) work for non-ctor factories as we'll often have no clue what those factories are even going to produce until they are actually executed, but this should work nicely for the factory-ctor case.

-Alex

--
You received this message because you are subscribed to the Google Groups "testng-dev" group.
To view this discussion on the web visit https://groups.google.com/d/msg/testng-dev/-/JwlLnHeDSwwJ.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/testng-dev?hl=en.

--
You received this message because you are subscribed to the Google Groups "testng-dev" group.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/testng-dev?hl=en.