Have you tried step by step reporting Selenium Tests Framework? Grab free selenium step by step reporting framework today!

Wednesday, February 8, 2012

Is it sane to compare Selenium with QTP?


I have seen many articles which compare Selenium (1 or 2) with QTP (or other commercial tools). But these articles miss very basic point that Selenium is a library available in variety of language while QTP has entire gamut of tools with it. So if you were to really compare QTP with Selenium then comparison should be based on the library of languages which is offered either of these tools, which in my opinion Selenium wins outright over QTP.
My only intention of writing this post is not claim Selenium victory of QTP but to demonstrate how badly Selenium is compared with QTP at times. There might be things QTP would be good at doing (I left QTP long ago hence can not name any) but to disregard Selenium on the factors I listed herein is highly biased.
  • Object spying - Sure, you have QTP recorder which does it for you but don't forget that you can test object locator using Selenium IDE and it is not limited to using only Selenium IDE. Firebug with Firefox can be used for same while Chrome has built in development tool bar to test element locator. In case you don't know you can use $$("cssLocator") to test css locator and $x("xpathLocator") to test xPath in both FF and chrome to test application objects. Can you use QTP to test object locators in FF and Chrome?
  • Debugging code - So QTP comes in with built-in editor and Selenium is defeated in this aspect. Is it? Given the variety of languages Selenium can be used with, you have far greater language editor options with Selenium than with QTP. You have umpteen options with editors like Intellij,  Eclipse, Visual Studio than the one available with QTP. When I was working with QTP I could not even conveniently rename test methods spanning across multiple files. Such re factoring is child's play with professional editors you use with Selenium
  • Recovery Scenarios - This is one of the biggest cheat point of QTP. Add recovery scenarios and tool takes care of working around the update windows. First of all if you add recovery scenario then it slows down the speed of test execution as there would be one listener always looking for such events, which means more recovery scenario and be ready for more slower test execution. Why should you bother about those automatic update windows in your test environment? Should not you proof your test environment against such update windows? If it is windows or FF then disable automatic updates. And best if you can not manage your test environment then out source it to some else like - Sauce Labs or Testing Bot
  • Testing non browser applications - Well, when did Selenium claim that it can tests window based applications. Selenium is browser automation framework. Comparing Selenium and QTP on this front is like comparing apples with oranges.
  • Great object Repository of QTP - So QTP can store objects in an external location, all you need to do is to update object repo when your object location changes. And then you find that the easiest way to port QTP scripts from one machine to another is to use Descriptive Programming which indeed means that you should do away with your beloved object repository of QTP. Now coming to absence of object repository mechanism in Selenium. Have you heard of properties files or Page Factory and guess what, you you don't have to do away with any thing to be able to port your Selenium tests from one machine to another.
  • No built-in Report with Selenium - QTP generates nice test reports and with Selenium - none. Really? When you pick up a language for Selenium you would be using a framework in that language and most probably the framework will provide you the reporting capabilities. For example there is ant task to generate test report in JUnit while TestNG has buit-in reporting mechanism. Google it and you would find similar options in language of your choice.
  • QTP and QC integration- You bought QTP for test automation now buy QC for test management. What with Selenium Test Management tools. None. There are indeed options - xStudio is one good option for small Selenium teams looking for test management with out spending any extra money. And their commercial options are cheaper than  QC. If you are looking to port your test results to Test Management tool then TestLink might be worth a try. In this case you don't have to pay at all.

6 comments:

  1. i'm working on qtp i taught qtp to be the only good testing tool but after reading this post i would like to say qtp is one of the testing tool.

    ReplyDelete
  2. Good write up, Many aspects to consider when selecting an auto tool. Selenium has many strong features as mentioned and best of all its free.

    ReplyDelete
  3. I find this post very slanted and inaccurate. For example, your first question: Can you test object locators in FF and chrome in QTP. Well, yes you can. And since both CSS and Xpath are available, in addition to all the other recognition properties in QTP why was that made a central point?

    And your point about the object repository is down right bizarre? Why on earth would you need to use descriptive programming to move your test from one machine to another? We have ours running on a Jenkins grid, with our tests stored in Git without the need for resorting to descriptive programming. Some simple regex's is probably all you needed for a property or two.

    One last point I want to make. Given the report in Qtp which will show you a screenshot or video, expected result, and where the test was when it failed vs a unit test Xml report, I find it hard to believe you think Selenium is on par with QuickTest.

    If you are going to do a review at least make it fair and do your homework about what you are reviewing. Both tools have their strong points and are good for a specific type of job.

    ReplyDelete
    Replies
    1. Let me answer your observations-

      I suppose you are referring to testing object locator in FF and Chrome using QTP by running the test, is not it? With Selenium you don't have to. Use FF and Chrome console to test your locator. Not really a selenium feature and so the point, why compare object spying of QTP with Selenium.

      About object repo, keeping descriptive programming aside, how slow it behaves when it begins to grow in size. Using object repo and you are committing binary files to your git. If there is a conflict in object repo committed by two team members and here your git goes for toss resolving conflicts in binary file.

      About Test Reporting, if built in reporting of TestNG is not enough then Use SauceLab or Testing bot and you have videos of your Selenium test execution on cloud.

      Talking about review, you got it wrong here. It is not review, it is about how Selenium should not be compared with QTP and it is has to be then it could be on some thing else but not on what I mentioned in post. There is no comparison. Having worked on both QTP and Selenium. I prefer Selenium may be because of its language support, may be because it is open source.

      b/w I appreciate when I see comments with identity of poster. Else it is difficult to see if comments are from a Test Engineer or from a QTP marketing guy. Any way thanks for your comments.

      Delete
  4. There is one big difference between Selenium and QTP.QTP has made all essential test automation features simpler for the user while an average Selenium user is on his journey to make automation easier for him.

    ReplyDelete

ShareThis