What if I have a batch of data need to be tested

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

What if I have a batch of data need to be tested

aaron-53

Hello,
I am a professional java-developer working for a famous international
company.
I am now looking for a suitable testing enviroment that could have
enough functionalities to be adequate on my case.
Below is some points in my case,
1> We have a bunch of data need to be tested each day.
2> We need some kind of control like a workflow. I can tell the testing
env what to do if some test case fail to execute.
So, does TestNG have such functionalites? If if doesn't now, do you
have any plan for that in the future?

Thanks!
Aaron


--~--~---------~--~----~------------~-------~--~----~
 You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: What if I have a batch of data need to be tested

Alexandru Popescu ☀

Hi Aaron!

Unfortunately the details you are including are not enough to allow me
to answer how TestNG can help you. However, I would encourage you to:

1/ detail a bit what your intentions are
2/ read the public available documentation for TestNG and see if it
fits your scenario.

Please see also my inlined comments.

On 12/29/06, aaron <[hidden email]> wrote:
>
> Hello,
> I am a professional java-developer working for a famous international
> company.
> I am now looking for a suitable testing enviroment that could have
> enough functionalities to be adequate on my case.
> Below is some points in my case,
> 1> We have a bunch of data need to be tested each day.

Where is this data living? How do you load it? etc.

> 2> We need some kind of control like a workflow. I can tell the testing
> env what to do if some test case fail to execute.

What is the relation between a test case and the "bunch of data". Do
you mean that you want to test your code against some real data each
day or as you prior formulated: test the data?

You can always create a workflow in your java code, so I am not sure
what are your special requirements from a testing framework.

HTH,

./alex
--
.w( the_mindstorm )p.
  TestNG co-founder
EclipseTestNG Creator

> So, does TestNG have such functionalites? If if doesn't now, do you
> have any plan for that in the future?
>
> Thanks!
> Aaron
>
>
> >
>

--~--~---------~--~----~------------~-------~--~----~
 You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: What if I have a batch of data need to be tested

Mark Derricutt
On 12/29/06, Alexandru Popescu <[hidden email]> wrote:

> 2> We need some kind of control like a workflow. I can tell the testing
> env what to do if some test case fail to execute.

You can always create a workflow in your java code, so I am not sure
what are your special requirements from a testing framework.

The original question was quite vague, but a custom listener would work here, failing that the return code from the TestNG class in a shell script :)

Mark


--
http://www.talios.com
http://www.flickr.com/photos/talios
--~--~---------~--~----~------------~-------~--~----~
 You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: What if I have a batch of data need to be tested

Steve Loughran-7
In reply to this post by aaron-53

On 29/12/06, aaron <[hidden email]> wrote:

>
> Hello,
> I am a professional java-developer working for a famous international
> company.
> I am now looking for a suitable testing enviroment that could have
> enough functionalities to be adequate on my case.
> Below is some points in my case,
> 1> We have a bunch of data need to be tested each day.
> 2> We need some kind of control like a workflow. I can tell the testing
> env what to do if some test case fail to execute.
> So, does TestNG have such functionalites? If if doesn't now, do you
> have any plan for that in the future?

This is more a task for your test running environment, rather than the
test framework itself.

1. use your favourite build tool to run the tests & create the reports
(e..g ant, maven, make)

2. use a continuous integration server to check out the app, rebuild
it and run the tests, emailing reports.

There's a lot of options for #2. I've found Luntbuild and Anthill
easiest to get started. Cruise control need too much nurturing for my
liking, and I havent played with Continuuum or whatever Sun use on
java.net.

You can set luntbuild up to run builds on a schedule (like a big test
every night), and run other scripts/build files first do things like
set up the database. Dont host it on your development machine; use a
spare box or a dedicated vmware/xen image.

-steve

--~--~---------~--~----~------------~-------~--~----~
 You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: What if I have a batch of data need to be tested

aaron-53
In reply to this post by Alexandru Popescu ☀

Hello Alex,
Sorry for the obscure part. I will try to explain it more precisely.

1> Those data were mostly living in a database.
2> The relation between the code to be tested and those data restored
in datatbase is just as what the doc describes in the DataProvider
chapter. The database contains most values need to be supplied to test.
For instance as below,

    /* Assume this function is within a bank system to process account
bills.
    /* @AccInfo, an info object of the account owener, which contains
some basic properites, such as a)credit level,
    /* b)owener catagory, c)sex, d)country, e)favourite...
    /* @AcctBill, a bill, which contains some bill properties, such as
a)credit/deposite, b)money...
    */
    process(AcctInfo, AcctBill) /**/

As described above, each AcctBill object contains a reference to an
AcctInfo, and both of them have a dozen important properties
approximately. To make this case tested thoroughly, because we want as
best as we can to achieve condition coverage or even path coverage, we
need to create hundreds of AcctBill objects. And the most annoying
thing is, we have hundreds of this sort of methods need to be tested.
This must be a catastrophe.

Actually, I find TestNG a better testing framework than some others.
And that's why I spend time to write comments here.

However, frankly speaking, it seems to me that some features are rather
toys than really helpful & useful tools. Let's talk a little deep into
DataProvider. What it benefits us? I think the only thing could be the
time saved to generate data. (Is my understanding right?) Then I wanna
to ask how much time could be saved through this. I don't think it is
quit much. Let me make a contrast to figure this out.

a> With DataProvider,
a.1> Creates a DataProvider function.
a.2> Write java code to generate data.
a.3> Write annotation to indicate which exactly the matching
dataprovider is.

b> Without DataProvider,
b.1> Creates a code block or helper function to return data generated.
b.2> Write java code to generate data.
b.3> Loop those data
b.4> Invoke testing method.

As the contrast above, the first two steps have no difference with or
without dataprovider. Hence, the total difference is only the matter of
the latter one or two steps. But how much time does it take for a
experienced developer to write a loop block to invoke a method? Couple
of seconds may be the tiny benefit to earn from this.

What if the dataprovider feature can absorb data from other data
sources, such as a database or Excel. Hundreds of seconds at least
would be saved for each case at least.

So, from my perspective, a step further this would be a wonderful tool,
otherwise, it is rather a toy.

Aaron


--~--~---------~--~----~------------~-------~--~----~
 You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: What if I have a batch of data need to be tested

Alexandru Popescu ☀

Hi Aaron,

I will split my answer in 2 big parts:

1/
> What it benefits us? I think the only thing could be the
> time saved to generate data. (Is my understanding right?) Then I wanna
> to ask how much time could be saved through this. I don't think it is
> quit much.

In big lines the above is both true and false. Now let me detail: I
agree that almost anybody can write a piece of code that is testing a
part of his app. But this is just the top of the iceberg imo, because
after this your will need to also develop: a way to automatically call
that code (otherwise you will most probably use a main method),
eventually integrate this with a CI tool, next you will need to write
code for generating reports, and maybe at some point you will also
need some integration with your IDE. After all this work (and call
them costs for your company), you will need to maintain and evolve it.
Over time, when you developers will join the team you will have to
teach them about the in-house tool you have developed. So, does it
save time? Maybe yes, maybe no. You are the only one able to answer
this question for your special case.

2/ @DataProviders

The second part of the answer will be related to the particular
technical question you are raising about @DataProviders. As far as I
understand, your point is that you would like to have special
@DataProviders that would be able to fetch data from different data
sources. While at first glance this makes sense, there are tones of
questions that we would need to answer before providing out of the box
something like this. I will just enumerate a few of them:
- should the @DataProvider fetch the data transactionally or not?
- should the @DataProvider load bulk data or fetch the data incrementally?
- should the @DataProvider use pure JDBC or use some ORM solution? If
using an ORM solution then which one?
- if using a 3rd party tool for loading the data, then what versions
should we support by default? etc.

A solution that meets everybody's expectations would require lots of
work (it is somehow a combinatoric number of solutions that would need
to be developed in order to be sure that we are addressing any
particular case, and this only if we were able to identify all
possible scenarios). So, concluding: does this make any sense? And the
answer is imo clear: no. Almost every app that will need testing will
have its custom way to answer the above questions, and those that I
have missed. And I guess my point is correct: you are saying that for
you, knowing your application it is somehow tedious to develop
in-house such a solution, then what about us, which are external to
your app and we would need to have the most extreme generical solution
to meet all possible expectations?

Finally to address the last question/statement: is it a toy? I don't
know. For me it is not and I guess it is not for other TestNG users.
In case it is a toy for you then you might need something else. Or if
you can tell us what you are missing, then maybe we can do something
about it. Or you can develop the extensions you think are important
and contribute them back to the project.

HTH,

./alex
--
.w( the_mindstorm )p.
  TestNG co-founder
EclipseTestNG Creator



On 12/30/06, aaron <[hidden email]> wrote:

>
> Hello Alex,
> Sorry for the obscure part. I will try to explain it more precisely.
>
> 1> Those data were mostly living in a database.
> 2> The relation between the code to be tested and those data restored
> in datatbase is just as what the doc describes in the DataProvider
> chapter. The database contains most values need to be supplied to test.
> For instance as below,
>
>     /* Assume this function is within a bank system to process account
> bills.
>     /* @AccInfo, an info object of the account owener, which contains
> some basic properites, such as a)credit level,
>     /* b)owener catagory, c)sex, d)country, e)favourite...
>     /* @AcctBill, a bill, which contains some bill properties, such as
> a)credit/deposite, b)money...
>     */
>     process(AcctInfo, AcctBill) /**/
>
> As described above, each AcctBill object contains a reference to an
> AcctInfo, and both of them have a dozen important properties
> approximately. To make this case tested thoroughly, because we want as
> best as we can to achieve condition coverage or even path coverage, we
> need to create hundreds of AcctBill objects. And the most annoying
> thing is, we have hundreds of this sort of methods need to be tested.
> This must be a catastrophe.
>
> Actually, I find TestNG a better testing framework than some others.
> And that's why I spend time to write comments here.
>
> However, frankly speaking, it seems to me that some features are rather
> toys than really helpful & useful tools. Let's talk a little deep into
> DataProvider. What it benefits us? I think the only thing could be the
> time saved to generate data. (Is my understanding right?) Then I wanna
> to ask how much time could be saved through this. I don't think it is
> quit much. Let me make a contrast to figure this out.
>
> a> With DataProvider,
> a.1> Creates a DataProvider function.
> a.2> Write java code to generate data.
> a.3> Write annotation to indicate which exactly the matching
> dataprovider is.
>
> b> Without DataProvider,
> b.1> Creates a code block or helper function to return data generated.
> b.2> Write java code to generate data.
> b.3> Loop those data
> b.4> Invoke testing method.
>
> As the contrast above, the first two steps have no difference with or
> without dataprovider. Hence, the total difference is only the matter of
> the latter one or two steps. But how much time does it take for a
> experienced developer to write a loop block to invoke a method? Couple
> of seconds may be the tiny benefit to earn from this.
>
> What if the dataprovider feature can absorb data from other data
> sources, such as a database or Excel. Hundreds of seconds at least
> would be saved for each case at least.
>
> So, from my perspective, a step further this would be a wonderful tool,
> otherwise, it is rather a toy.
>
> Aaron
>
>
> >
>

--~--~---------~--~----~------------~-------~--~----~
 You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: What if I have a batch of data need to be tested

Steve Loughran-7

On 30/12/06, Alexandru Popescu <[hidden email]> wrote:
>

> 2/ @DataProviders
>
> The second part of the answer will be related to the particular
> technical question you are raising about @DataProviders. As far as I
> understand, your point is that you would like to have special
> @DataProviders that would be able to fetch data from different data
> sources. While at first glance this makes sense, there are tones of
> questions that we would need to answer before providing out of the box
> something like this. I will just enumerate a few of them:
> - should the @DataProvider fetch the data transactionally or not?
> - should the @DataProvider load bulk data or fetch the data incrementally?
> - should the @DataProvider use pure JDBC or use some ORM solution? If
> using an ORM solution then which one?
> - if using a 3rd party tool for loading the data, then what versions
> should we support by default? etc.
>
> A solution that meets everybody's expectations would require lots of
> work (it is somehow a combinatoric number of solutions that would need
> to be developed in order to be sure that we are addressing any
> particular case, and this only if we were able to identify all
> possible scenarios). So, concluding: does this make any sense? And the
> answer is imo clear: no. Almost every app that will need testing will
> have its custom way to answer the above questions, and those that I
> have missed. And I guess my point is correct: you are saying that for
> you, knowing your application it is somehow tedious to develop
> in-house such a solution, then what about us, which are external to
> your app and we would need to have the most extreme generical solution
> to meet all possible expectations?

And you have to test it :)

>
> Finally to address the last question/statement: is it a toy? I don't
> know. For me it is not and I guess it is not for other TestNG users.
> In case it is a toy for you then you might need something else. Or if
> you can tell us what you are missing, then maybe we can do something
> about it. Or you can develop the extensions you think are important
> and contribute them back to the project.
>

Generating realistic test data is a hard problem.

It's not just a data provider problem, as I can write code to populate
my db from a CSV text file on demand, the db is then snapshotted in
some way for instant setup before a test run. If you work this way you
can adapt to changes in the db schema fairly rapidly, and still have
fast setup of the db. You can build up an hsqldb image for fast test
runs, mysql and derby images for proper transactions and concurrency,
and oracle & SQL server for emulating the production machine.
[Normally use whatever persistence API I'm working with (eg hibernate,
ejb3), but I'm thinking of doing a smartfrog component to populate
tables via with JDBC. This would separate DB setup code from the test
code and test interop better. If I use ejb3 to write and then read the
test data, I am ensuring round trips work, but I am not ensuring that
java.util.UUID types are being converted to strings in the database]

Creating that CSV of test data is hard. Do you make up data that looks
good, think up data that may break the app, or use real data collected
from user info, in which case -especially if you are working in the
financial sector- you need to think about data security. Using real
customer data in your test runs gives you realistic data (especially
for performance testing), but suddenly all test reports and stack
traces become sensitive.

I dont think this stuff is the role of the unit test framework.
Database setup and population does need to be part of the test
infrastructure. Creating good test data, that's really hard.

One talk on this topic:
http://video.google.com/videoplay?docid=6552390366458794081&q=london+test+automation+conference

-steve

--~--~---------~--~----~------------~-------~--~----~
 You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: What if I have a batch of data need to be tested

aaron-53

Hello Steve,
Your suggestion is constructive. :-)

About secure problem, I think it is not that secure. There are many
easy ways to solve that.
1> You may choose some old or useless data to test.
2> You can collect some of the typical secure data a little first. Then
modify the sensitive parts of data to test. E.g., name, country,
money...

Aha, to me it is not a problem. :)

Aaron


--~--~---------~--~----~------------~-------~--~----~
 You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en
-~----------~----~----~----~------~----~------~--~---