Several Issues - TestNG not useful as-is by Paul Murray

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Several Issues - TestNG not useful as-is by Paul Murray

Alexandru Popescu ☀

[quote]
Mr Beust,

I am having all sorts of trouble getting TestNG to do what I want it to do.

I want to prompt a user for some parameters indicating what tests the
user wants to run, then I want to programattically invoke TestNG and
run them.

The way to do this seems to be to construct an XmlSuite object and call
    TestNG ng = new TestNG();
    ng.setXmlSuites(Arrays.asList(new XmlSuite[] {suite}));
    ng.run();

Currently, I find the test classes by scanning the file directory
where the classes get put by the appserver (JBoss unpacks all the ears
and drop the class files onto disk somewhere). This leads to my first
problem.

1 - there is an interface named ITestClassFinder which finds test
classes, but there seems to be no way to plug an implementation of
this interface into the testing framework. The only place it is used
is in TestRunner, where the TestNGClassFinder method is hard-coded in:

    ITestClassFinder testClassFinder= new
TestNGClassFinder(Utils.xmlClassesToClasses(m_testClassesFromXml),
                                             null,
                                             m_xmlTest,
                                             m_annotationFinder);

There's not much point abstracting out an interface if it is not
pluggable. At the very least, it would be nice if the TestRunner class
had a overridable method to get the test class finder. At present,
this line is buried in amongst a stack of other code. Even if I did
extendTestRunner and override initMethods() (which I can't do anyway
because it's private), it's not clear how I would plug that into the
TestNG class and use it.

TestRunner seems "tied" to the XmlClass and XmlPackage - there is
nothing in the framework to which I can "give" some ITestNGMethod
objects and say "run these, and any of their prereqs, in the correct
order please"

2 - the only way I found this information was by loading the source
into Eclipse and searching it. The testNG javadoc at
http://testng.org/doc/javadocs/index.html?org/testng/ITestClassFinder.html
was not compiled with the -use flag set.

So, at present I find all the test classes (ie: those with a @Test
annotation) and construct an XMLObject specifying all of those test
classes. I load those objects with the method and group regular
expressions specified by the user.

3 - if you specify that something "dependsongroups", does that mean
that it depends on other test *classes* being run first? My impression
is that this is not the case: that the framework treats all test
classes as being complete in themselves. This is fine, but it might be
nice to say so.

I am finding, however, that it is not working very well. Specifically,

4 - if I specify an includesMethods, and one of the XmlClass does not
have any methods of that name, then the runner runs all the test
methodsin the class - that is - it treats it as if I had not specified
any includesMethods at all.

I could deal with this by being careful, when building the xml, not to
include any classes that do not contain any methods specified by the
user as includesMethods parameters. But this kind of defeats the
purpose of having a framework to do this stuff. And anyway, if I do my
own regular expression math on method name to confirm that a test
class has at least one matching method - can I *really* be sure that
your framework will do the match in exactly the same way?

So, I attempted to implement a method selector of my own. However:
5 - the documentation does not tell me how the business of building
the prerequisite execution graph interacts with method selectors. If I
implement an IMethodSelector, does that selector have to do the job of
figuring out what prerequisites need to be run? If it does, how does
it specify dependeny order?
6 - a suite has a collection of method selectors. Is the sequence of
selectors significant? Must a method match all selectors to be run, or
is it run if it matche any selector?

7 - In any case, I find that it just doesn't work.

When I run this code:
    XmlSuite suite = new XmlSuite();
    XmlTest tst = new XmlTest(suite);
    tst.getXmlClasses().add(new XmlClass(FooTest.class));
    tst.getXmlClasses().add(new XmlClass(BarTest.class));

    XmlMethodSelector methselect = new XmlMethodSelector();
    methselect.setName(MyMethodSelector.class.getName());
    suite.getMethodSelectors().add(new XmlMethodSelector() {});

    TestNG ng = new TestNG();
    ng.setXmlSuites(Arrays.asList(new XmlSuite[] {suite}));
    ng.run();

I find that the methods in MyMethodSelector are nott invoked, and I get this:

[TestNG] Reporter org.testng.reporters.SuiteHTMLReporter@785d65 failed
java.lang.NullPointerException
at java.util.Hashtable.put(Hashtable.java:396)
at java.util.Properties.setProperty(Properties.java:128)
at org.testng.xml.XmlMethodSelector.toXml(XmlMethodSelector.java:82)
at org.testng.xml.XmlSuite.toXml(XmlSuite.java:390)
at org.testng.reporters.SuiteHTMLReporter.generateXmlFile(SuiteHTMLReporter.java:79)


Why it's attempting to run a "HTML Reporter", and where the output is
going, I have no idea. I certainly didn't ask it to do any such thing.
I don't know why this reporter is attempting to convert the XmlSuite
into text, and I don't know why that line is blowing up when it tries
to do it, because it shouldn't be being executed at all - it's in the
else clause of a
    if(null != m_className) {
and I did set the classname.

Oh - and in the XmlMethodSelector class, the name of the getter and
setter for m_className do not match.

And - note that you add a test to a suite by specifying it in the
consructor, but you add a class to a test by calling add on the
getXmlClasses(). I don't know why they are different.

.....

At this point, I was just about ready to toss the framework altogether
and just use the annotations. But then I came across the IHookable
interface and realised that it would probably be too much work.

.....

Ok. What else? Oh yes, that's right. dependsOnMethod.

Let's say I have a test class

@Test
public class BarTest
{
  public void testA (){}

  @Test(dependsOnMethods= {"testA"})
  public void testB (){}
}

If I run it like so:
    XmlSuite suite = new XmlSuite();
    XmlTest tst = new XmlTest(suite);

    XmlClass barClass = new XmlClass(BarTest.class);
    tst.getXmlClasses().add(barClass);

    barClass.getIncludedMethods().add("testB");

    TestNG ng = new TestNG();
    ng.setXmlSuites(Arrays.asList(new XmlSuite[] {suite}));
    ng.run();

It crashes with a BarTest.testB() is depending on nonexistent method
testA. (oh, and note that in this message the first method gets
parenthesis after the name but the second one doesn't).

Now - if I say "run test B", I mean "run test B .... and whatever else
you might have to to run it". If I had specifically excluded testA,
then this message might be more to the point. But that's not the case.
I could, of course, write my own stuff to figure out prerequisites -
but that kind of defeats the purpose of having a framework which is
supposed to do that stuff for you.

....

In general, the semantics of TestNG are not well specified
(included/excluded packages, classes, groups, and methods). There are
all these interfaces floating around, but there is nothing to plug
them into. The engine classes directly use the classes in the xml
package rather than being written in terms of ITest, ITestClass etc.
There seems to be no way to say "I want to run these methods and any
of their prerequisites". And if there is a way to do it, the only way
to find out seems to be to trawl through the source code.

There seem to be classes that do the work of finding prerequisites,
but beats me how to invoke them and then hand those results to the
engine that runs the tests. If you ask me, this framework would
benefit from an old-fashioned data flow diagram.

My project is keen on TestNG, and the group selection stuff seems
cool. But frankly, it all seems a little bitsy and unfinished.
[/quote]

--~--~---------~--~----~------------~-------~--~----~
 You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: Several Issues - TestNG not useful as-is by Paul Murray

Alexandru Popescu ☀

On 1/12/07, Alexandru Popescu <[hidden email]> wrote:

> [quote]
> Mr Beust,
>
> I am having all sorts of trouble getting TestNG to do what I want it to do.
>
> I want to prompt a user for some parameters indicating what tests the
> user wants to run, then I want to programattically invoke TestNG and
> run them.
>
> The way to do this seems to be to construct an XmlSuite object and call
>     TestNG ng = new TestNG();
>     ng.setXmlSuites(Arrays.asList(new XmlSuite[] {suite}));
>     ng.run();
>
> Currently, I find the test classes by scanning the file directory
> where the classes get put by the appserver (JBoss unpacks all the ears
> and drop the class files onto disk somewhere). This leads to my first
> problem.
>
> 1 - there is an interface named ITestClassFinder which finds test
> classes, but there seems to be no way to plug an implementation of
> this interface into the testing framework. The only place it is used
> is in TestRunner, where the TestNGClassFinder method is hard-coded in:
>
>     ITestClassFinder testClassFinder= new
> TestNGClassFinder(Utils.xmlClassesToClasses(m_testClassesFromXml),
>                                              null,
>                                              m_xmlTest,
>                                              m_annotationFinder);
>
> There's not much point abstracting out an interface if it is not
> pluggable. At the very least, it would be nice if the TestRunner class
> had a overridable method to get the test class finder. At present,
> this line is buried in amongst a stack of other code. Even if I did
> extendTestRunner and override initMethods() (which I can't do anyway
> because it's private), it's not clear how I would plug that into the
> TestNG class and use it.
>
> TestRunner seems "tied" to the XmlClass and XmlPackage - there is
> nothing in the framework to which I can "give" some ITestNGMethod
> objects and say "run these, and any of their prereqs, in the correct
> order please"
>
> 2 - the only way I found this information was by loading the source
> into Eclipse and searching it. The testNG javadoc at
> http://testng.org/doc/javadocs/index.html?org/testng/ITestClassFinder.html
> was not compiled with the -use flag set.
>
> So, at present I find all the test classes (ie: those with a @Test
> annotation) and construct an XMLObject specifying all of those test
> classes. I load those objects with the method and group regular
> expressions specified by the user.
>
> [trimmed /]
> [/quote]
>

Do you need to manually search for those classes? Otherwise it should
be enough to provide TestNG with the directory and package names and
TestNG will take care of finding your tests.

Those interfaces were mostly designed for internal usage and for
allowing integration with JUnit. However, if you need this detailed
kind of filtering and XmlSelectors are not enough, then I guess we can
further investigate the possibilities.

I will try to address the other points later.

./alex
--
.w( the_mindstorm )p.
  TestNG co-founder
EclipseTestNG Creator

--~--~---------~--~----~------------~-------~--~----~
 You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: Several Issues - TestNG not useful as-is by Paul Murray

Cédric Beust ♔
In reply to this post by Alexandru Popescu ☀
Hi Paul,

First of all, thanks for taking the time to take the time to write all this detailed feedback.

My first impression is that you are trying to do a lot of things that TestNG wasn't designed to support, so I'm not really surprised to hear about your difficulties.  Still, I'm sorry TestNG is causing you trouble, and I hope we can resolve your issues.

First nitpick:  you are saying that there is no point in using an interface if it's not exposed to users.  I disagree with this assertion since this interface is used internally (it has two implementations as far as I can recall).

Anyway, before tackling your points about making TestNG more extensible than it is now, I'd like to make sure that you really need all this, so my first question to you is:  what exactly prevents you from letting TestNG find your classes for you?

Couldn't you achieve this goal either with testng.xml or by using something like **/*.class in build.xml?

[Note: I'm pasting this answer in JIRA, but from now on, let's continue the discussion on the mailing-list until we come to a resolution]

--
Cedric

On 1/11/07, Alexandru Popescu <[hidden email]> wrote:

[quote]
Mr Beust,

I am having all sorts of trouble getting TestNG to do what I want it to do.

I want to prompt a user for some parameters indicating what tests the
user wants to run, then I want to programattically invoke TestNG and
run them.

The way to do this seems to be to construct an XmlSuite object and call
    TestNG ng = new TestNG();
    ng.setXmlSuites(Arrays.asList(new XmlSuite[] {suite}));
    ng.run();

Currently, I find the test classes by scanning the file directory
where the classes get put by the appserver (JBoss unpacks all the ears
and drop the class files onto disk somewhere). This leads to my first
problem.

1 - there is an interface named ITestClassFinder which finds test
classes, but there seems to be no way to plug an implementation of
this interface into the testing framework. The only place it is used
is in TestRunner, where the TestNGClassFinder method is hard-coded in:

    ITestClassFinder testClassFinder= new
TestNGClassFinder(Utils.xmlClassesToClasses (m_testClassesFromXml),
                                             null,
                                             m_xmlTest,
                                             m_annotationFinder);

There's not much point abstracting out an interface if it is not
pluggable. At the very least, it would be nice if the TestRunner class
had a overridable method to get the test class finder. At present,
this line is buried in amongst a stack of other code. Even if I did
extendTestRunner and override initMethods() (which I can't do anyway
because it's private), it's not clear how I would plug that into the
TestNG class and use it.

TestRunner seems "tied" to the XmlClass and XmlPackage - there is
nothing in the framework to which I can "give" some ITestNGMethod
objects and say "run these, and any of their prereqs, in the correct
order please"

2 - the only way I found this information was by loading the source
into Eclipse and searching it. The testNG javadoc at
http://testng.org/doc/javadocs/index.html?org/testng/ITestClassFinder.html
was not compiled with the -use flag set.

So, at present I find all the test classes (ie: those with a @Test
annotation) and construct an XMLObject specifying all of those test
classes. I load those objects with the method and group regular
expressions specified by the user.

3 - if you specify that something "dependsongroups", does that mean
that it depends on other test *classes* being run first? My impression
is that this is not the case: that the framework treats all test
classes as being complete in themselves. This is fine, but it might be
nice to say so.

I am finding, however, that it is not working very well. Specifically,

4 - if I specify an includesMethods, and one of the XmlClass does not
have any methods of that name, then the runner runs all the test
methodsin the class - that is - it treats it as if I had not specified
any includesMethods at all.

I could deal with this by being careful, when building the xml, not to
include any classes that do not contain any methods specified by the
user as includesMethods parameters. But this kind of defeats the
purpose of having a framework to do this stuff. And anyway, if I do my
own regular expression math on method name to confirm that a test
class has at least one matching method - can I *really* be sure that
your framework will do the match in exactly the same way?

So, I attempted to implement a method selector of my own. However:
5 - the documentation does not tell me how the business of building
the prerequisite execution graph interacts with method selectors. If I
implement an IMethodSelector, does that selector have to do the job of
figuring out what prerequisites need to be run? If it does, how does
it specify dependeny order?
6 - a suite has a collection of method selectors. Is the sequence of
selectors significant? Must a method match all selectors to be run, or
is it run if it matche any selector?

7 - In any case, I find that it just doesn't work.

When I run this code:
    XmlSuite suite = new XmlSuite();
    XmlTest tst = new XmlTest(suite);
    tst.getXmlClasses().add(new XmlClass(FooTest.class));
    tst.getXmlClasses().add(new XmlClass(BarTest.class));

    XmlMethodSelector methselect = new XmlMethodSelector();
    methselect.setName(MyMethodSelector.class.getName());
    suite.getMethodSelectors().add(new XmlMethodSelector() {});

    TestNG ng = new TestNG();
    ng.setXmlSuites(Arrays.asList(new XmlSuite[] {suite}));
    ng.run();

I find that the methods in MyMethodSelector are nott invoked, and I get this:

[TestNG] Reporter org.testng.reporters.SuiteHTMLReporter@785d65 failed
java.lang.NullPointerException
at java.util.Hashtable.put(Hashtable.java:396)
at java.util.Properties.setProperty(Properties.java:128)
at org.testng.xml.XmlMethodSelector.toXml(XmlMethodSelector.java:82)
at org.testng.xml.XmlSuite.toXml(XmlSuite.java :390)
at org.testng.reporters.SuiteHTMLReporter.generateXmlFile(SuiteHTMLReporter.java:79)


Why it's attempting to run a "HTML Reporter", and where the output is
going, I have no idea. I certainly didn't ask it to do any such thing.
I don't know why this reporter is attempting to convert the XmlSuite
into text, and I don't know why that line is blowing up when it tries
to do it, because it shouldn't be being executed at all - it's in the
else clause of a
    if(null != m_className) {
and I did set the classname.

Oh - and in the XmlMethodSelector class, the name of the getter and
setter for m_className do not match.

And - note that you add a test to a suite by specifying it in the
consructor, but you add a class to a test by calling add on the
getXmlClasses(). I don't know why they are different.

.....

At this point, I was just about ready to toss the framework altogether
and just use the annotations. But then I came across the IHookable
interface and realised that it would probably be too much work.

.....

Ok. What else? Oh yes, that's right. dependsOnMethod.

Let's say I have a test class

@Test
public class BarTest
{
  public void testA (){}

  @Test(dependsOnMethods= {"testA"})
  public void testB (){}
}

If I run it like so:
    XmlSuite suite = new XmlSuite();
    XmlTest tst = new XmlTest(suite);

    XmlClass barClass = new XmlClass(BarTest.class);
    tst.getXmlClasses().add(barClass);

    barClass.getIncludedMethods().add("testB");

    TestNG ng = new TestNG();
    ng.setXmlSuites(Arrays.asList(new XmlSuite[] {suite}));
    ng.run();

It crashes with a BarTest.testB() is depending on nonexistent method
testA. (oh, and note that in this message the first method gets
parenthesis after the name but the second one doesn't).

Now - if I say "run test B", I mean "run test B .... and whatever else
you might have to to run it". If I had specifically excluded testA,
then this message might be more to the point. But that's not the case.
I could, of course, write my own stuff to figure out prerequisites -
but that kind of defeats the purpose of having a framework which is
supposed to do that stuff for you.

....

In general, the semantics of TestNG are not well specified
(included/excluded packages, classes, groups, and methods). There are
all these interfaces floating around, but there is nothing to plug
them into. The engine classes directly use the classes in the xml
package rather than being written in terms of ITest, ITestClass etc.
There seems to be no way to say "I want to run these methods and any
of their prerequisites". And if there is a way to do it, the only way
to find out seems to be to trawl through the source code.

There seem to be classes that do the work of finding prerequisites,
but beats me how to invoke them and then hand those results to the
engine that runs the tests. If you ask me, this framework would
benefit from an old-fashioned data flow diagram.

My project is keen on TestNG, and the group selection stuff seems
cool. But frankly, it all seems a little bitsy and unfinished.
[/quote]




--
Cédric
--~--~---------~--~----~------------~-------~--~----~
 You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: Several Issues - TestNG not useful as-is by Paul Murray

Steve Loughran-7
On 12/01/07, Cédric Beust ♔ <[hidden email]> wrote:

> Hi Paul,
>
> First of all, thanks for taking the time to take the time to write all this
> detailed feedback.
>
> My first impression is that you are trying to do a lot of things that TestNG
> wasn't designed to support, so I'm not really surprised to hear about your
> difficulties.  Still, I'm sorry TestNG is causing you trouble, and I hope we
> can resolve your issues.
>
> First nitpick:  you are saying that there is no point in using an interface
> if it's not exposed to users.  I disagree with this assertion since this
> interface is used internally (it has two implementations as far as I can
> recall).

the whole public/private thing is complex in OSS code. You can't keep
the private stuff secret, but there's a really good reason for not
allowing people to call it. Once you make it public you've made it
part of your API and are constrained to keeping it stable and
backwards compatible forever.

The Ant codebase has nearly everything public with the consequence
maintenance problems, whereas Xom has everything but the official API
private. If you want something inside exposed, you have to argue your
case with Elliotte Harold, and more often than not get turned down if
there already is a solution to the problem.



Actually, that gives me a thought. Maybe I need an @unstable attribute
which marks something as not guaranteed to last.


@unstable(expires="2007",comment="for rigged demos only") public class
something { ... }


>
> Anyway, before tackling your points about making TestNG more extensible than
> it is now, I'd like to make sure that you really need all this, so my first
> question to you is:  what exactly prevents you from letting TestNG find your
> classes for you?
>
> Couldn't you achieve this goal either with testng.xml or by using something
> like **/*.class in build.xml?
>

And <input> to get user input; there's a also project somewhere on
sourceforge to actually bring up AWT or swing forms from inside a
build. http://antforms.sourceforge.net/.

Given you can embed ant inproc fairly easily, you may want to go that
way, at the expense of a few more MB of JAR files


-steve

--~--~---------~--~----~------------~-------~--~----~
 You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en
-~----------~----~----~----~------~----~------~--~---