Though unit tests have always recommended to be independent of each other, advent of selenium+TestNG has given birth to dependent integration/end to end tests. TestNG Author Cédric Beust had written about it few years ago. When I had begun writing selenium tests I used to launch browser once in one test class and test methods were usually dependent on one another. Hence 2nd test method would begin test run on same browser + application state where 1st test method stopped. Launching just one browser for five or more test methods in a class is big time saving is not it? Well not really, for the reasons I am going to describe in this post. Since then I have moved on to launching browser per test method but yet give in “charm” of dependent test methods at times. to So here are the few pros and cons for dependent tests and -
Pros -
- Similar steps of execution don’t have to be repeated in multiple tests when steps constitutes a test method in themselves
- If login fails then only login test should fail and not other tests which are dependent on login feature. Tests which are dependent on login feature should be should be skipped and no time should be spent on execution of these test. This is a big saving on time when running end to end test.
There is a compelling example of dependent tests provided on the post I linked earlier. The example goes as -
Say I want to test a JDBC driver I have written by running 100 queries and verifying the results. Say also that for whatever reason, establishing a database connection and ensuring the sample data is there takes a fairly long time (say 30 seconds) relative to the query time (0.1 ms).
Under the JUnit philosophy, I have two choices
1) Establish the DB connection in the setUp() method and have a test* method for each of the 100 queries. GOOD: fine-grained test cases BAD: the tests take nearly an hour to run.
2) Do everything in a single test* method. GOOD: the test takes less than a minute to run. BAD: coarse-grained test cases (just one)
Are you really prepared to argue that I must choose between these approaches simply to get a warm fuzzy feeling that my test methods are independent? As Mr. Boyens quite rightly points out above, this does nothing more than allow you to be sloppy in writing your tests. Fortunately, though, I have other options:
3) Store the DB connection in a static variable in my test class and initialize it only during the first test case. GOOD: fine-grained tests AND they run in 60 seconds BAD: it’s a gross hack but JUnit gives me no choice.
4) Use TestNG and do the DB setup in a separate test case on which all the other tests depend. GOOD: even more fine-grained test cases that run in seconds and without a hack. BAD: I still find myself arguing with people who inexplicably insist on the inherent evil of dependent test methods.
The bottom line is that when you’re testing real world systems, you simply have to be able to use the postconditions of some tests as preconditions for others. It is naive to insist that every single test start from scratch in building up the state of the system under test. And you’re a lot better off if your test framework realizes this and has formal mechanisms to help you cope with it.
This approach seems good to me but not the implementation. Instead of creating a dependent method I would have created a setup method applicable for only this test class and have it run only once before first test method in class. I described a similar approach later in the article though in that example setup method is run before each test method and not just once.
Cons -
- Dependent tests make debugging difficult as you can see from following examples -
- Test 4 dependent on Test 3 dependent on Test 2 dependent on Test 1
Now it is not easy to read one test and understand what it is doing, what interactions it is testing and what internal states it verifies
- What if you just want to run one/many failed tests. Here in dependent tests would be run firsts and assertion carried out in the dependent tests would slow down the overall test execution
- One test failure would hide another test failures. Consider that your login test asserts that after login user sees
>. What if element locator of > is changed hence login test fails. If you are using TestNG then all the tests dependents on login test would be skipped. This does not seem right. The failure in login test is not because of application bug but changes which are to be fixed on element locator. But herein dependent test would also stop further test execution and may hide more defects (if any) down the test execution line. - Now you can argue what if element locator of login textbox changed, now login test as well as any other test which is dependent in login feature would fail. Do you see my point how this is an endless argument.
So how do I not repeat the same steps / page object methods in multiple test method? How about moving them to set up method like following -
This approach also gives an indication of cohesiveness of test methods. If test methods have different setup methods then probably they don’t belong to same test class? Or am I wrong in my assumption?
Whether you go ahead with dependent test would eventually boil down to if you and your team sees value in it. If you any way decide to use dependent tests then my suggestion would be to not use more than one level of dependency
What do you think of dependent tests? Would you write dependent tests?