Fwd: [Work] Automated testing -- Tools discussion

[Michelle D'Souza]

An interesting summary of the research Kasper has done into automated  
testing tools.

Michelle



Begin forwarded message:

> From: Kasper Galschiot Markus <kasper2 at markus.dk>
> Date: April 7, 2010 7:39:53 AM GMT-04:00
> To: CollectionSpace Work list <work at lists.collectionspace.org>, obara.justin at gmail.com
> Subject: [Work] Automated testing -- Tools discussion
>
> I have been browsing around in the quite extensive amount of free  
> web test automation tools around. Everyone seems to have their own  
> preferences, and there are pros and cons for each tool. Below I'll  
> try summarizing my observations (not experience, but what I have  
> read). In the very bottom, I posted my personal opinion of choice.
> I'm sending this mail to the work list, because any experience,  
> rumors, comments, suggestions, etc. would be highly appreciated.  
> Also comments on the importance of: speed of tests, reality of  
> tests, ease of programming, etc, as well as suggestions for other  
> tools I should look into would be appreciated.
> ----
> Selenium: Probably the best known tool.
> Pros:
> It automates/scripts real browsers, which is an advantage as you get  
> close to reality.
> Well established and documented
> GUI record tool for firefox, making it easy to write test (for all  
> browsers)
> Support multiple programming languages (for writing the tests)
> Has been praised for being the easiest to use/get into
> Supports ant
> Has grid functionality to distribute the tests (to a certain extend)
> Cons:
> Events are simulated, which is sometimes buggy (based on rather old  
> articles, and justins (fluid) experiences X months ago, might have  
> changed).
> Needs actual running windows+mac machines to run against
> Slow compared to emulated browser approach (can be somewhat remedied  
> by running tests in parallel)
> Disregards JS errors (unless they affect the asserts)!!!!
> Potential problem with waiting for pages to load (think this has  
> changed)
> Canoo Webtest: Also widely known tool.
> Pros:
> Do not need specific OS, as browsers are emulated (uses HtmlUnit)
> Record tool for writing easy tests
> Well established/documented
> Supports ant
> Easy to integrate/setup continuous testing
> Very good reports
> Fast
>
> Cons:
> Does not use real browsers, but emulates them using HtmlUnit. This  
> means some errors might potentially be not be caught.
> A bit steeper learning curve than Selenium
> Javascript support not as good as in "normal" browser
> Doesn't accept (too) badly formed html (Dont think this will be a  
> problem)
>
> Doh.Robot: A smaller project, brought to my attention by Justin.
>
> Pros:
> Does not simulate mouse/keys, but makes actual system calls using  
> java's robot
> As close as can be to real users (except for embedded scripts)
> Cons:
> Not too well documented/established
> Unsure if updated or discontinued
> requires code to be inserted on the page
>
> Sahi:
>
> Pros:
> GUI record for writing tests
> Cons:
> Small community
> Underdeveloped
> Confusing Interface
>
> Watir:
>
> Pros:
> Multi browser (& OS) support
> Has a rich API
> Has a ‘Simple’ class (for non-tech users)
> Watij & Watin (Java & .NET)
> Cons:
>
> Have to learn Ruby (unless you choose Watij or Watin)
> Every browser requires a different library
>
> Selenium 2:
>
> Pros:
> Has both standard selenium and HtmlUnit available for testing
> Circumvent browser restrictions by using whichever mechanism is most  
> appropriate to control the browser. For Firefox, this means that  
> WebDriver is implemented as an extension. For IE, WebDriver makes  
> use of IE's Automation controls, etc.
> WebDriver can make use of facilities offered by the OS (allows to  
> get closer to a real user, similar to robot i guess?!)
> Cons:
> Alpha state
> Sparse documentation, and probably somewhat buggy
> Steeper learning curve due to amount of features and lack of docs
> ---
> From what I can tell, Selenium and Canoo Webtest seems to be the  
> most appealing choices, despite their shortcommings. It seems that  
> Webtest receives less criticism than Selenium, but it is hard to  
> tell whether this is due to Selenium being used/compared to more  
> often. Generally, both people using Selenium and Webtest are  
> satisfied. The others; Doh Robot, Sahi and Watir all have their  
> advantages (except for Sahi :( ), but for various reasons didn't  
> appeal to me (too little documentation, steep learning curve,  
> lousier version of Webtest/Selenium). There is also the option of  
> Selenium 2, but the problem with this is that it is still in alpha,  
> and the documentation is rather sparse. Also, it might be kind of  
> buggy.
> In many ways I am split between Selenium and Webtest, but I guess I  
> am leaning towards Selenium. There are two main reasons for this:
> Using real browsers is a lot closer to reality, which I weigh high
> (some of) the shortcommings of Selenium will be fixed in Selenium 2.  
> I take it that our tests written for Selenium will be compatible  
> with Selenium 2. When S2 will be at a decent stage it is easy to  
> make the switch.
> The major problem with Selenium is the need of access to windows/mac  
> machines (windows is doable, using Vms, I hate mac :).
> Another major problem is the speed. This can hopefully be remedied  
> using Selenium Grid.
> As a final note, I guess I should mention that I might be a bit  
> biased towards Selenium, since I knew that first, have read more of  
> the documentation, etc.
> _______________________________________________
> Work mailing list
> Work at lists.collectionspace.org
> http://lists.collectionspace.org/mailman/listinfo/work_lists.collectionspace.org

------------------------------------------------------
Michelle D'Souza
Software Developer, Fluid Project
Adaptive Technology Resource Centre

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://fluidproject.org/pipermail/fluid-work/attachments/20100407/31c8b574/attachment.html>