Negative feedback

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Negative feedback

Laurent Caillette

Hi all,

Some feedback after a recent frustrating experience with TestNG 5.5:

1. There is no built-in type system helping to know which parameters
an annotated method does accept. This problem is inherent to
Annotations but intensive usage of Annotations make the problem spread
on the framework.

2. My test requires some setup which depends on the test name: I need
to create a scratch directory for the test, using the test short class
name and the test method name. This is also useful for logging the
beginning of the test.
A method annotated with @DataProvider may receive a Method parameter
(which is cool) but the @BeforeMethod cannot receive the parameters
generated by the DataProvider so the fixture is called from the test
itself (bad).

3. Annotations bring an ugly syntax.

4. Expected exceptions and groups are cool.


This is globally frustrating because I've spent sometime to try TestNG
and I find it less suitable than JUnit-3.8 for my very simple needs.


The API of my dreams would look like:

public class MyTest implements Testable, Setupable, GroupMember {
  public void testSomething() { ... }
  public Iterable< String > getGroups() { ... }
  public void setUp( ITestContext context ) { ... }
  public void tearDown( ITestContext context ) { ... }
}

Well, it's very JUnit3ish and I understand it's quite not in the
spirit of TestNG API.

I let the steam out on them (TestNG taking collateral damages) on my
blog, where you can find some code examples illustrating points 1 to
4.
http://www.jroller.com/page/softmotion?
entry=annotations_suck_a_lot_and


Regards,

c.


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: Negative feedback

Cédric Beust ♔
Salut Laurent, and sorry to hear about your bad experience.  I hope we can do something about your bad experience!

On 1/28/07, Laurent Caillette <[hidden email]> wrote:

Hi all,

Some feedback after a recent frustrating experience with TestNG 5.5:

1. There is no built-in type system helping to know which parameters
an annotated method does accept. This problem is inherent to
Annotations but intensive usage of Annotations make the problem spread
on the framework.

I'm not sure what you mean:  both Eclipse and IDEA give you full completion on the attribute of the annotations.  Can you be more specific?

2. My test requires some setup which depends on the test name: I need
to create a scratch directory for the test, using the test short class
name and the test method name. This is also useful for logging the
beginning of the test.
A method annotated with @DataProvider may receive a Method parameter
(which is cool) but the @BeforeMethod cannot receive the parameters
generated by the DataProvider so the fixture is called from the test
itself (bad).

I was following until the "so" part :-)

Can you explain what you mean by "the fixture is called from the test itself"?

It's true that configuration methods (@Before and @After) do not support data providers because we've never really found a need for this:  between the possibility to pass the parameters directly to the method and the fact that values stored in field instances are preserved by TestNG, it's never been a problem to receive context in configuration methods.  Can you show us an example of what you are trying to achieve?

3. Annotations bring an ugly syntax.

I'd argue that this is a personal opinion that's not relevant to the current discussion :-)

4. Expected exceptions and groups are cool.


This is globally frustrating because I've spent sometime to try TestNG
and I find it less suitable than JUnit-3.8 for my very simple needs.


The API of my dreams would look like:

public class MyTest implements Testable, Setupable, GroupMember {
  public void testSomething() { ... }
  public Iterable< String > getGroups() { ... }
  public void setUp( ITestContext context ) { ... }
  public void tearDown( ITestContext context ) { ... }
}

I understand what you are saying for the last two methods:  would it help you if you could declare an ITestContext in a configuration method and TestNG would automatically pass you this object?

I don't really understand the getGroups() method, though, or is this method there just because you don't like the annotation syntax?  Would these groups apply to all the public test methods?  How do you define groups for individual methods?

Anyway, I'd like to keep the "anti annotation" discussion separate from your other concerns...

Looking forward to hearing from you.

--
Cédric
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: Negative feedback

Laurent Caillette

Hi Cedric,

So Internet brought two French-speaking guys talking in English!
Do you know I remember you since Amiga times?


>> 1. There is no built-in type system helping to know which
>> parameters an annotated method does accept.
>> This problem is inherent to Annotations but intensive usage
>> of Annotations make the problem spread on the framework.

> I'm not sure what you mean: both Eclipse and IDEA give you full
> completion on the attribute of the annotations.
> Can you be more specific?

I mean when you type (the | shows where the caret is):
  @DataProvider( name = "nameProvider" )
  public String[][] createData( |

IDEA (nor any IDE) can't tell you which parameters the createData
method may support as it does not reference any static definition.
This problem is intrisic to Annotations.


>> 2. My test requires some setup which depends on the test name:
>> I need to create a scratch directory for the test, using the
>> test short class name and the test method name.
>> This is also useful for logging the beginning of the test.
>> A method annotated with @DataProvider may receive a Method
>> parameter (which is cool) but the @BeforeMethod cannot receive
>> the parameters generated by the DataProvider so the fixture is
>> called from the test itself (bad).

> I was following until the "so" part :-)
> Can you explain what you mean by "the fixture is called from the
> test itself"?

I have to write something like:

  @Test( dataProvider = "nameProvider" )
  public void method1( String testName ) {
    callTheFixture( testName ) ;
    // test the stuff...
  }

I have to copy "callTheFixture( testName )" at the beginning
of each test, while it would be logical to have this in a
@BeforeMethod-annotated method.

Staying in an Annotation-eager spirit one could prefer:

  @Test( dataProvider = "nameProvider" )
  public void method1( String testName ) {
    // test the stuff...
  }

  @BeforeMethod( dataProvider = "nameProvider" )
  public void setup( String testName ) {
    callTheFixture( testName ) ;
  }

But I don't like it anyways because there is no statically-
verifiable way to ensure consistency between parameter types.


> would it help you if you could declare an ITestContext
> in a configuration method and TestNG would automatically
> pass you this object?

I admit that supporting this would weaken my arguments a bit
and possibly help some other users:
  @BeforeMethod
  public void setup( ITestContext context ) { ... }


> Anyway, I'd like to keep the "anti annotation" discussion separate
> from your other concerns...

That can't be easy since TestNG relies heavily on them!

TestNG has some really cool features (groups being my
favourites as we had some tests with a UI robot that had to
be skipped in a headless environment) but too many of them
are implemented in a way which loses static typing.
Maybe Annotations drove you thinking that way, or maybe you
don't care about static typing at all.
But since TestNG supports at least two declaration
systems (Annotations + XML), static typing support could be
an interesting extension. While I'm fed up with my job plus an
OSS project for now I cant spend a few hours thinking
about the API (I would switch to dev list then).


Regards,

c.



--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: Negative feedback

Cédric Beust ♔


On 1/30/07, Laurent Caillette <[hidden email]> wrote:



>> 2. My test requires some setup which depends on the test name:
>> I need to create a scratch directory for the test, using the
>> test short class name and the test method name.
>> This is also useful for logging the beginning of the test.
>> A method annotated with @DataProvider may receive a Method
>> parameter (which is cool) but the @BeforeMethod cannot receive
>> the parameters generated by the DataProvider so the fixture is
>> called from the test itself (bad).

> I was following until the "so" part :-)
> Can you explain what you mean by "the fixture is called from the
> test itself"?

I have to write something like:

  @Test( dataProvider = "nameProvider" )
  public void method1( String testName ) {
    callTheFixture( testName ) ;
    // test the stuff...
  }

I have to copy "callTheFixture( testName )" at the beginning
of each test, while it would be logical to have this in a
@BeforeMethod-annotated method.

Staying in an Annotation-eager spirit one could prefer:

  @Test( dataProvider = "nameProvider" )
  public void method1( String testName ) {
    // test the stuff...
  }

  @BeforeMethod( dataProvider = "nameProvider" )
  public void setup( String testName ) {
    callTheFixture( testName ) ;
  }

But I don't like it anyways because there is no statically-
verifiable way to ensure consistency between parameter types.

I see.  I actually think this is a reasonable request, and probably the first convincing example that DataProviders could be extended to Configuration methods.


TestNG has some really cool features (groups being my
favourites as we had some tests with a UI robot that had to
be skipped in a headless environment) but too many of them
are implemented in a way which loses static typing.
Maybe Annotations drove you thinking that way, or maybe you
don't care about static typing at all.

Oooh, now you've struck a nerve :-)

I firmly believe that static typing is the only sane way to build large-scale software, so you're definitely preaching to the choir here.

Here is my claim:  TestNG is as statically safe as is possible given the limitations of the JDK Annotation API. 

If you can suggest ways in which this claim is not true, I will definitely be interested in hearing them:  I  want TestNG to be as safe as possible.  I am not very fond of the Object[][] signature of data providers and the lack of type safety that comes with it, but I haven't been able to find a better way so far, so any suggestion will be welcome.

--
Cédric
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: Negative feedback

Laurent Caillette
In reply to this post by Cédric Beust ♔

Hi Cedric,

So Internet brought two French-speaking guys talking in English!
Do you know I remember you since Amiga times?


>> 1. There is no built-in type system helping to know which
>> parameters an annotated method does accept.
>> This problem is inherent to Annotations but intensive usage
>> of Annotations make the problem spread on the framework.

> I'm not sure what you mean: both Eclipse and IDEA give you full
> completion on the attribute of the annotations.
> Can you be more specific?

I mean when you type (the | shows where the caret is):
  @DataProvider( name = "nameProvider" )
  public String[][] createData( |

IDEA (nor any IDE) can't tell you which parameters the createData
method may support as it does not reference any static definition.
This problem is intrisic to Annotations.


>> 2. My test requires some setup which depends on the test name:
>> I need to create a scratch directory for the test, using the
>> test short class name and the test method name.
>> This is also useful for logging the beginning of the test.
>> A method annotated with @DataProvider may receive a Method
>> parameter (which is cool) but the @BeforeMethod cannot receive
>> the parameters generated by the DataProvider so the fixture is
>> called from the test itself (bad).

> I was following until the "so" part :-)
> Can you explain what you mean by "the fixture is called from the
> test itself"?

I have to write something like:

  @Test( dataProvider = "nameProvider" )
  public void method1( String testName ) {
    callTheFixture( testName ) ;
    // test the stuff...
  }

I have to copy "callTheFixture( testName )" at the beginning
of each test, while it would be logical to have this in a
@BeforeMethod-annotated method.

Staying in an Annotation-eager spirit one could prefer:

  @Test( dataProvider = "nameProvider" )
  public void method1( String testName ) {
    // test the stuff...
  }

  @BeforeMethod( dataProvider = "nameProvider" )
  public void setup( String testName ) {
    callTheFixture( testName ) ;
  }

But I don't like it anyways because there is no statically-
verifiable way to ensure consistency between parameter types.


> would it help you if you could declare an ITestContext
> in a configuration method and TestNG would automatically
> pass you this object?

I admit that supporting this would weaken my arguments a bit
and possibly help some other users:
  @BeforeMethod
  public void setup( ITestContext context ) { ... }


> Anyway, I'd like to keep the "anti annotation" discussion separate
> from your other concerns...

That can't be easy since TestNG relies heavily on them!

TestNG has some really cool features (groups being my
favourites as we had some tests with a UI robot that had to
be skipped in a headless environment) but too many of them
are implemented in a way which loses static typing.
Maybe Annotations drove you thinking that way, or maybe you
don't care about static typing at all.
But since TestNG supports at least two declaration
systems (Annotations + XML), static typing support could be
an interesting extension. While I'm fed up with my job plus an
OSS project for now I cant spend a few hours thinking
about the API (I would switch to dev list then).


Regards,

c.



--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: Negative feedback

Mark Derricutt
In reply to this post by Laurent Caillette
On 1/31/07, Laurent Caillette <[hidden email]> wrote:

> I'm not sure what you mean: both Eclipse and IDEA give you full
> completion on the attribute of the annotations.
> Can you be more specific?

I mean when you type (the | shows where the caret is):
  @DataProvider( name = "nameProvider" )
  public String[][] createData( |

IDEA (nor any IDE) can't tell you which parameters the createData
method may support as it does not reference any static definition.
This problem is intrisic to Annotations.

On this side, I was looking the other day at the syntax completion APIs for plugin authors (for IDEA) with the crazy thought of doing just this - completion of discovered data providers, groups, properties (from found suite.xml files or plugin defaults).  So far I've not really looked at this beyond initial thoughts thou.

If anyone heres already done some IDEA plugin/completion stuff and want to provide a patch that'd be great...

Mark

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: Negative feedback

Laurent Caillette
In reply to this post by Cédric Beust ♔

Hi Cedric,

Sorry for the duplicate post, I couldn't see the first one for hours
so I thought Google ate it.

I just posted a draft for a statically-typed testing framework on my
blog :
http://jroller.com/page/softmotion?entry=testing_framework_with_static_typing

Ok it's quite verbose (surprised?). Maybe you'll find one idea or two
to pick up.

Regards,

c.


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: Negative feedback

Steve Loughran-7

On 01/02/07, Laurent Caillette <[hidden email]> wrote:

>
> Hi Cedric,
>
> Sorry for the duplicate post, I couldn't see the first one for hours
> so I thought Google ate it.
>
> I just posted a draft for a statically-typed testing framework on my
> blog :
> http://jroller.com/page/softmotion?entry=testing_framework_with_static_typing
>
> Ok it's quite verbose (surprised?). Maybe you'll find one idea or two
> to pick up.

"I think that JUnit authors had a pretty good idea to instantiate the
test object before each call to a test* method. "

aah, there is some more to the junit internals than that. JUnit
instantiates *every* test object before running any of them, and even
though it calls setup and teardown before and after, you don't know
that full GC has kicked in before the next method. Ant <junit> lets
you run each test class in its own VM, but nobody has done the same
for every method as the time hit generally outweighs the isolation.

-steve

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en
-~----------~----~----~----~------~----~------~--~---