Skip to main content

Should selenium tests be always independent? The endless debate !!

Though unit tests have always recommended to be independent of each other, advent of selenium+TestNG has given birth to dependent integration/end to end tests. TestNG Author C├ędric Beust had written about it few years ago. When I had begun writing selenium tests I used to launch browser once in one test class and test methods were usually dependent on one another. Hence 2nd test method would begin test run on same browser + application state where 1st test method stopped. Launching just one browser for five or more test methods in a class is big time saving is not it? Well not really, for the reasons I am going to describe in this post. Since then I have moved on to launching browser per test method but yet give in “charm” of dependent test methods at times. to So here are the few pros and cons for dependent tests and -

Pros -

  • Similar steps of execution don’t have to be repeated in multiple tests when steps constitutes a test method in themselves

  • If login fails then only login test should fail and not other tests which are dependent on login feature. Tests which are dependent on login feature should be should be skipped and no time should be spent on execution of these test. This is a big saving on time when running end to end test.

There is a compelling example of dependent tests provided on the post I linked earlier. The example goes as -

Say I want to test a JDBC driver I have written by running 100 queries and verifying the results. Say also that for whatever reason, establishing a database connection and ensuring the sample data is there takes a fairly long time (say 30 seconds) relative to the query time (0.1 ms).
Under the JUnit philosophy, I have two choices
1) Establish the DB connection in the setUp() method and have a test* method for each of the 100 queries. GOOD: fine-grained test cases BAD: the tests take nearly an hour to run.
2) Do everything in a single test* method. GOOD: the test takes less than a minute to run. BAD: coarse-grained test cases (just one)
Are you really prepared to argue that I must choose between these approaches simply to get a warm fuzzy feeling that my test methods are independent? As Mr. Boyens quite rightly points out above, this does nothing more than allow you to be sloppy in writing your tests. Fortunately, though, I have other options:
3) Store the DB connection in a static variable in my test class and initialize it only during the first test case. GOOD: fine-grained tests AND they run in 60 seconds BAD: it’s a gross hack but JUnit gives me no choice.
4) Use TestNG and do the DB setup in a separate test case on which all the other tests depend. GOOD: even more fine-grained test cases that run in seconds and without a hack. BAD: I still find myself arguing with people who inexplicably insist on the inherent evil of dependent test methods.
The bottom line is that when you’re testing real world systems, you simply have to be able to use the postconditions of some tests as preconditions for others. It is naive to insist that every single test start from scratch in building up the state of the system under test. And you’re a lot better off if your test framework realizes this and has formal mechanisms to help you cope with it.

This approach seems good to me but not the implementation. Instead of creating a dependent method I would have created a setup method applicable for only this test class and have it run only once before first test method in class. I described a similar approach later in the article though in that example setup method is run before each test method and not just once.

Cons -

  • Dependent tests make debugging difficult as you can see from following examples -
    • Test 4 dependent on Test 3 dependent on Test 2 dependent on Test 1
Now it is not easy to read one test and understand what it is doing, what interactions it is testing and what internal states it verifies

  • What if you just want to run one/many failed tests. Here in dependent tests would be run firsts and assertion carried out in the dependent tests would slow down the overall test execution

  • One test failure would hide another test failures. Consider that your login test asserts that after login user sees >. What if element locator of > is changed hence login test fails. If you are using TestNG then all the tests dependents on login test would be skipped. This does not seem right. The failure in login test is not because of application bug but changes which are to be fixed on element locator. But herein dependent test would also stop further test execution and may hide more defects (if any) down the test execution line.
    • Now you can argue what if element locator of login textbox changed, now login test as well as any other test which is dependent in login feature would fail. Do  you see my point how this is an endless argument.

So how do I not repeat the same steps / page object methods in multiple test method? How about moving them to set up method like following -

Screenshot from 2016-05-30 11:47:00.png

This approach also gives an indication of cohesiveness of test methods. If test methods have different setup methods then probably they don’t belong to same test class? Or am I wrong in my assumption?

Whether you go ahead with dependent test would eventually boil down to if you and your team sees value in it. If you any way decide to use dependent tests then my suggestion would be to not use more than one level of dependency

What do you think of dependent tests? Would you write dependent tests?

Popular posts from this blog

Distributed Load Testing with JMeter

Distributed Testing with JMeter When one JMeter client is not able to offer amount of threads required for load testing then distributed testing is used. In distributed testing - One instance of JMeter client can control number of JMeter instances and collect data from them Test plan does not need to be copied to each server, the client sends it to all servers note - JMeter will run all the threads on all the servers, hence 100 threads on 5 JMeter server would pump 500 threads in total. If many server instances are used, the client JMeter can become overloaded and so the client network connection. This has been improved in latest versions of JMeter by switching to Stripped modes, but you should always check that your client is not overloaded When Client (master) and Server (slave) nodes are on same network (no SSH required) Configure Client Node Herein client is referred as the machine controlling test execution on other JMeter nodes. This is also referred

Appium and android mobile app automation

Next appium and Android mobile app automation video tutoria l is live. If you are new to appium then please check - appium-tutorial This video tutorial covers - Start vysor (Just for this session and not mobile automation :)) Start appium and start appium inspector Desired Capabilities platformName - Android deviceName - L2N0219828001013 (as seen on "adb devices") Saved Capability Sets Start Session Scan app elements using appium inspector Get appPackage and appActivity using "APK info" app Install "APK info" app and open app whose appPackage and appActivity are required i.e. calculator Check top section of app icon is app package is app activity testng.xml file settings for running Android app tests Test details and CalculatorScreen class View beautiful STF test report  

Verify email confirmation using Selenium

Note: If you are new to java and selenium then start with selenium java training videos .     Email confirmation seems to be integral part of any registration process. I came across an application which lets you provide your email address. You can follow the sign up link in you mail and then complete the registration process. Lets consider we provide GMail address for it. Now if were to use only Selenium then we would have to follow following steps - Launch GMail using Selenium; Some how search for new mail in the list of available mails; Some how click on it; Parse the mail message; Get the registration link; Follow up with registration process What do you think of an approach in which you can