TestNG & Selenium: Separate tests into “groups”, run ordered inside each group

Question

We use TestNG and Selenium WebDriver to test our web application.

Now our problem is that we often have several tests that need to run in a certain order, e.g.:

Answer 1

Try with depends on group along with depends on method. Add all methods in same class in one group. For example

@Test(groups={"cls1","other"}) 
public void cls1test1(){
}
@Test(groups={"cls1","other"}, dependsOnMethods="cls1test1", alwaysrun=true) 
public void cls1test2(){
}

In class 2
@Test(groups={"cls2","other"}, dependsOnGroups="cls1", alwaysrun=true) 
public void cls2test1(){
}
@Test(groups={"cls2","other"}, dependsOnMethods="cls2test1", dependsOnGroups="cls1", alwaysrun=true) 
public void cls2test2(){
}

Answer 2

You are mixing "functionality" and "test". Separating them will solve your problem.

For example, create a helper class/method that executes the steps to log in, then call that class/method in your Login test and all other tests that require the user to be logged in.

Your other tests do not actually need to rely on your Login "Test", just the login class/method.

If later back-end modifications introduce a bug in the login process, all of the tests which rely on the Login helper class/method will still fail as expected.

Update:

Turns out this already has a name, the Page Object pattern. Here is a page with Java examples of using this pattern:

http://code.google.com/p/selenium/wiki/PageObjects

Answer 3

There is an easy (whilst hacky) workaround for this if you are comfortable with your first approach:

At first we just put tests that belong together into one class, and used dependsOnMethods to make them run in the right order. This used to work in TestNG V5, but in V6 TestNG will sometimes interleave tests from different classes (while respecting the ordering imposed by dependsOnMethods). There does not seem to be a way to tell TestNG "Always run tests from one class together".

We had a similar problem: we need our tests to be run class-wise because we couldn't guarantee the test classes not interfering with each other.

This is what we did: Put a

@Test( dependsOnGroups= { "dummyGroupToMakeTestNGTreatThisAsDependentClass" } )

Annotation on an Abstract Test Class or Interface that all your Tests inherit from. This will put all your methods in the "first group" (group as described in this paragraph, not TestNG-groups). Inside the groups the ordering is class-wise. Thanks to Cedric Beust, he provided a very quick answer for this.

Edit: The group dummyGroupToMakeTestNGTreatThisAsDependentClass actually has to exist, but you can just add a dummy test case for that purpose..