Code Coverage in TestNG IDEA Plugin

classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

Code Coverage in TestNG IDEA Plugin

Mark Derricutt
Hey all,

I've just been trying to track down reasons why the code coverage support in the IDEA plugin mysteriously stopped working sometime in the past 3-4 months (sadly the old javaforge repository no longer exists to compare :().

Anyway - I can't seem to spot anything that jumps out at me that might have changed, except for the RemoteTestNG runner (or what it invokes, as I believe coverage stopped working before we had the shared RemoteTestNG class).

I was wondering if anything changed on the inside of TestNG which may or may not cause problems with the coverage support (implemented as a -javaagent process)?

I have a few other things I'm going to try to see if I can spot whats playing up but thought I'd ask here as well...

Mark

--
http://www.talios.com
http://www.flickr.com/photos/talios
--~--~---------~--~----~------------~-------~--~----~
 You received this message because you are subscribed to the Google Groups "testng-dev" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/testng-dev?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: Code Coverage in TestNG IDEA Plugin

Alexandru Popescu ☀

On 12/11/06, Mark Derricutt <[hidden email]> wrote:

> Hey all,
>
> I've just been trying to track down reasons why the code coverage support in
> the IDEA plugin mysteriously stopped working sometime in the past 3-4 months
> (sadly the old javaforge repository no longer exists to compare :().
>
> Anyway - I can't seem to spot anything that jumps out at me that might have
> changed, except for the RemoteTestNG runner (or what it invokes, as I
> believe coverage stopped working before we had the shared RemoteTestNG
> class).
>
> I was wondering if anything changed on the inside of TestNG which may or may
> not cause problems with the coverage support (implemented as a -javaagent
> process)?
>
> I have a few other things I'm going to try to see if I can spot whats
> playing up but thought I'd ask here as well...
>

Frankly speaking I cannot see any good reasons why this would be
caused by the above mentioned change.... but as everybody knows
sometimes weird things are still happening. Please keep us posted.

./alex
--
.w( the_mindstorm )p.
  TestNG co-founder
EclipseTestNG Creator

> Mark
>
> --
> http://www.talios.com
> http://www.flickr.com/photos/talios
>  >
>

--~--~---------~--~----~------------~-------~--~----~
 You received this message because you are subscribed to the Google Groups "testng-dev" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/testng-dev?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: Code Coverage in TestNG IDEA Plugin

Daniel Dyer
In reply to this post by Mark Derricutt

Don't know if this is at all relevant, but code coverage for TestNG
does not work with TeamCity either (I have tried with 5.1 and 5.4).

In the case of TeamCity, the problem seems to be a TestNGException,
when running the tests, with the following root cause:

Caused by: java.lang.NoClassDefFoundError: com/vladium/emma/rt/RT

So it would appear that the bytecode injected by EMMA cannot find the
EMMA runtime classes.  This could easily be a bug in TeamCity, but I
thought I'd mention it in case it is related.


--~--~---------~--~----~------------~-------~--~----~
 You received this message because you are subscribed to the Google Groups "testng-dev" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/testng-dev?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: Code Coverage in TestNG IDEA Plugin

Cédric Beust ♔
Hi Daniel,

Now that TestNG is officially supported by JetBrains themselves, I would definitely bring this up to their attention...

--
Cedric


On 12/12/06, Daniel Dyer <[hidden email]> wrote:

Don't know if this is at all relevant, but code coverage for TestNG
does not work with TeamCity either (I have tried with 5.1 and 5.4).

In the case of TeamCity, the problem seems to be a TestNGException,
when running the tests, with the following root cause:

Caused by: java.lang.NoClassDefFoundError: com/vladium/emma/rt/RT

So it would appear that the bytecode injected by EMMA cannot find the
EMMA runtime classes.  This could easily be a bug in TeamCity, but I
thought I'd mention it in case it is related.







--
Cédric
--~--~---------~--~----~------------~-------~--~----~
 You received this message because you are subscribed to the Google Groups "testng-dev" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/testng-dev?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: Code Coverage in TestNG IDEA Plugin

Alexandru Popescu ☀
Thinking about this more I might see a possible problem:

- code coverage tools may need to use a special class loader to weave
the tests in order to gather the required data
- TestNG loads the test classes through reflection mechanism

If these 2 steps are using different classloader hierarchies then I
think there will be problems.

Can you help me with more details about what code coverage solution is
used in IDEA? Maybe I can further investigate the problem, but I would
definitely need to dig in the code to see what exactly are they doing.
If the problem comes from my above supposition then fixing it would be
quite tricky, but I may have a couple of ideas.

./alex
--
.w( the_mindstorm )p.
  TestNG co-founder
EclipseTestNG Creator


On 12/12/06, Cédric Beust ♔ <[hidden email]> wrote:

> Hi Daniel,
>
> Now that TestNG is officially supported by JetBrains themselves, I would
> definitely bring this up to their attention...
>
> --
> Cedric
>
>
> On 12/12/06, Daniel Dyer <[hidden email]> wrote:
> >
> > Don't know if this is at all relevant, but code coverage for TestNG
> > does not work with TeamCity either (I have tried with 5.1 and 5.4).
> >
> > In the case of TeamCity, the problem seems to be a TestNGException,
> > when running the tests, with the following root cause:
> >
> > Caused by: java.lang.NoClassDefFoundError: com/vladium/emma/rt/RT
> >
> > So it would appear that the bytecode injected by EMMA cannot find the
> > EMMA runtime classes.  This could easily be a bug in TeamCity, but I
> > thought I'd mention it in case it is related.
> >
> >
> >
> >
> >
>
>
>
> --
> Cédric
>
>  >
>

--~--~---------~--~----~------------~-------~--~----~
 You received this message because you are subscribed to the Google Groups "testng-dev" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/testng-dev?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: Code Coverage in TestNG IDEA Plugin

Daniel Dyer
In reply to this post by Cédric Beust ♔

Already have:

http://www.jetbrains.net/jira/browse/TW-1305


--~--~---------~--~----~------------~-------~--~----~
 You received this message because you are subscribed to the Google Groups "testng-dev" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/testng-dev?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: Code Coverage in TestNG IDEA Plugin

Mark Derricutt
In reply to this post by Daniel Dyer
Now that you mention it I do recall seeing that exception some time ago.  I'll set my verbosity back to 9 and see what I get here.  I just tried adding emma.jar to the project class path but that didn't help at all.

I'm talking to Eugene Vigdorchik (who wrote the emma integration) whose going to look into this - I'll send him towards TW-1305 as well..

Mark

On 12/13/06, Daniel Dyer <[hidden email]> wrote:

Caused by: java.lang.NoClassDefFoundError: com/vladium/emma/rt/RT


--~--~---------~--~----~------------~-------~--~----~
 You received this message because you are subscribed to the Google Groups "testng-dev" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/testng-dev?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: Code Coverage in TestNG IDEA Plugin

Daniel Dyer
In reply to this post by Alexandru Popescu ☀


Alexandru Popescu wrote:

> Can you help me with more details about what code coverage solution is
> used in IDEA? Maybe I can further investigate the problem, but I would
> definitely need to dig in the code to see what exactly are they doing.
> If the problem comes from my above supposition then fixing it would be
> quite tricky, but I may have a couple of ideas.
>

IDEA and TeamCity both use EMMA (http://emma.sourceforge.net/).  I
guess the first step would be to take IDEA/TeamCity out of the equation
and see if we can get EMMA to work with TestNG from Ant.


--~--~---------~--~----~------------~-------~--~----~
 You received this message because you are subscribed to the Google Groups "testng-dev" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/testng-dev?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: Code Coverage in TestNG IDEA Plugin

Alexandru Popescu ☀

On 12/13/06, Daniel Dyer <[hidden email]> wrote:

>
>
> Alexandru Popescu wrote:
>
> > Can you help me with more details about what code coverage solution is
> > used in IDEA? Maybe I can further investigate the problem, but I would
> > definitely need to dig in the code to see what exactly are they doing.
> > If the problem comes from my above supposition then fixing it would be
> > quite tricky, but I may have a couple of ideas.
> >
>
> IDEA and TeamCity both use EMMA (http://emma.sourceforge.net/).  I
> guess the first step would be to take IDEA/TeamCity out of the equation
> and see if we can get EMMA to work with TestNG from Ant.
>

I can definitely look into this, but I wouldn't like to duplicate the
effort :-).

./alex
--
.w( the_mindstorm )p.
  TestNG co-founder
EclipseTestNG Creator

>
> >
>

--~--~---------~--~----~------------~-------~--~----~
 You received this message because you are subscribed to the Google Groups "testng-dev" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/testng-dev?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: Code Coverage in TestNG IDEA Plugin

Eran-2
Hi Alexandru,

On 12/13/06, Alexandru Popescu <[hidden email]> wrote:

On 12/13/06, Daniel Dyer <[hidden email]> wrote:

>
>
> Alexandru Popescu wrote:
>
> > Can you help me with more details about what code coverage solution is
> > used in IDEA? Maybe I can further investigate the problem, but I would
> > definitely need to dig in the code to see what exactly are they doing.
> > If the problem comes from my above supposition then fixing it would be
> > quite tricky, but I may have a couple of ideas.
> >
>
> IDEA and TeamCity both use EMMA (http://emma.sourceforge.net/).  I
> guess the first step would be to take IDEA/TeamCity out of the equation
> and see if we can get EMMA to work with TestNG from Ant.
>

I can definitely look into this, but I wouldn't like to duplicate the
effort :-).

Don't trouble yourself. I can verify that Emma and TestNG play together nicely when invoked from Ant.

Regards,
Eran

./alex
--
.w( the_mindstorm )p.
  TestNG co-founder
EclipseTestNG Creator

>
> >
>





--~--~---------~--~----~------------~-------~--~----~
 You received this message because you are subscribed to the Google Groups "testng-dev" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/testng-dev?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: Code Coverage in TestNG IDEA Plugin

Daniel Dyer

Kirill at JetBrains has added this comment to the issue:

"The problem is that to run instrumented classes for TestNG we should
have added emma.jar to class path (like we do for JUnit tests).
I'll try to investigate the problem in the nearest time."

So it sounds like it can be fixed at their end, for TeamCity at least.
This doesn't explain why it doesn't work for IDEA though, unless the
same problem exists there.


--~--~---------~--~----~------------~-------~--~----~
 You received this message because you are subscribed to the Google Groups "testng-dev" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/testng-dev?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: Code Coverage in TestNG IDEA Plugin

Mark Derricutt
I saw that - I tried adding emma.jar to my project libraries the other day but no joy, so there may be a combination of that and something else...

On 12/14/06, Daniel Dyer <[hidden email]> wrote:
So it sounds like it can be fixed at their end, for TeamCity at least.
This doesn't explain why it doesn't work for IDEA though, unless the
same problem exists there.


--~--~---------~--~----~------------~-------~--~----~
 You received this message because you are subscribed to the Google Groups "testng-dev" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/testng-dev?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: Code Coverage in TestNG IDEA Plugin

Mark Derricutt
I just received a patch from Eugene at Jetbrains which fixes the coverage problem, so will be making a new plugin release shortly (which also includes TestNG 5.4).

On 12/14/06, Mark Derricutt <[hidden email]> wrote:
I saw that - I tried adding emma.jar to my project libraries the other day but no joy, so there may be a combination of that and something else...


On 12/14/06, Daniel Dyer <[hidden email]> wrote:
So it sounds like it can be fixed at their end, for TeamCity at least.
This doesn't explain why it doesn't work for IDEA though, unless the
same problem exists there.




--
http://www.talios.com
http://www.flickr.com/photos/talios
--~--~---------~--~----~------------~-------~--~----~
 You received this message because you are subscribed to the Google Groups "testng-dev" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/testng-dev?hl=en
-~----------~----~----~----~------~----~------~--~---