Without considering language, web automation has a variety of choices available that are open source.
GEB – http://www.gebish.org
Watij – http://watij.com
Watij – http://watij.com
Watir-webdriver – http://watirwebdriver.com
I tend to omit Java choices right away, due to Java being so code bloated. I think for QA, having code that can be written in one line, as opposed to 20 lines of multiple closures and complexity is far easier to manage and easier to create. For me, that makes the choice between Groovy and Ruby. I didn’t include Python as I haven’t had much use of it. But there’s another language choice with it’s own frameworks.
If you were to google: groovy web automation, you will find almost all the results point to GEB. GEB is a framework that includes the test automation (using Spock), as well as the browser automation (via web driver.) You write familiar Groovy code to kick off tests that will be written in a BDD format to automate a browser through a series of steps.
My Problem with GEB, is that I’ve found it incredibly lacking. If you simply follow the directions, it doesn’t work as designed. It’s over thought and very convoluted. People praise GEB for it’s design, yet it’s functionality isn’t really talked about.
When we adopted GEB at eHarmony, as our automated software solution – it started (as expected) with growing pains, but we never got out of the growing pains. The pains got worse and worse.
Real World Example
One real world example with GEB at eHarmony, was that it was initially working “ok” with Firefox 3.6 (the current Firefox version at the time.) When Firefox 3.6 jumped to version 4, GEB no longer worked. The GEB framework required a specific version of Spock, which required a specific version of Groovy (at that time, I believe it was 1.7) But with the new web driver that was out to support Firefox 4, GEB could no longer talk to the web driver API. We had to upgrade the dependancies, which broke GEB. So we stayed on ver 3.6 of FF.
Firefox went from 4, to 5, to 6, to 7…. and at last, GEB updated. It was now compatible with web driver to support Firefox 7. But in a day or two, Firefox was on version 8. Then 9, 10… You guessed it. GEB no longer worked with these new web drivers required for the newer browser versions. This cat and mouse game continued till version 14 of Firefox.
Meanwhile, in the Watir-webdriver world – Firefox compatibility was always maintained.
Another Real World Example with GEB was the poor choice of Spock. Spock is a poor mans Cucumber. While it does some things well, it misses the mark on one concept of BDD – keeping the team no the same page. Spock simplifies much, but the GEB use of Spock still has code in the Spock tests themselves. This had a direct result of our product managers not reading our QA Automation tests to verify we had met all their required testing requirements. They would ignore, overlook and dismiss the code. The code put up a wall between the non tech savvy business units and the tests themselves. In Cucumber, the tests always remain human language. The code driving the tests is kept separate in a step_definition.rb file.
If you like Groovy, then this is a great simplistic way to create web driver instances to manage your browser automation.
Some people love the Page/Object model.
The use of JQuery is very useful in structuring tests to click on element id/names, etc.
Assertions are automatic in the Then statements.
Static closures can be set up on the Page Object so when it’s instantiated it does checks on basic element validation.
The GEB community is lacking, this is part of the main problem with GEB. When a change is needed, it takes weeks or even months to get a fix.
The documentation is suffering from faults and incorrect data. If you follow the documentation for 0.9.2 (the latest version of GEB at the time of this writing), it tells you how you can call a browser inline and specify a driver… doing something like:
def browser = Browser(driver: FireFoxDriver) OR by doing def browser = Browser(new FireFoxDriver)
According to the documentation these two statements do the same thing. But it looks like if we run that, we’ll be creating two browsers. We’re instantiating the browser, then calling a new browser of a specific driver. In fact that’s what happens. When run, you’ll get one browser (i.e Firefox, then another Firefox/chrome/etc)
I had use cases of setting a browser to full screen, or constraining a browser in size. This is doable, but each time I do this with GEB inline (Or overriding a browser config), it would barf up the browser I want (the way I want it) then proceed to open a second browser in the default profile!! The second browser was used and the first ignored.
GEB’s cross browser compatibility is also a problem. While the site says it’s compatible with all but Safari, it really is difficult to get it working with other browsers.
Spock isn’t so hot. Grails developers would slap me across the face for writing that… but in the QA world, Spock is a pale image of Cucumber. Spock adds a BDD style of test construction. One of the goals of BDD is to simplify tests so that all aspects of a company can be productive. This means that anyone can read and understand the tests. Cucumber achieves this goal by removing code from the BDD steps. Spock however does not. In the Spock world you have the Given/When/Then followed by methods, objects, etc. To a developer, it’s fine. But show that to a marketing, business unit member and they’ll just shrug and look the other way. It obstructs them from contributing in planning meetings and in going through the tests to make sure we have coverage over their goals. How do I know this? Because I saw it first hand at eHarmony.
Another alternative out there in Groovy/Java is Watij. It’s a spin off of the idea behind Watir. It’s a wonderful idea… but the framework hasn’t been touched in 4 years. 4 YEARS! So leave it. No point in working with something that won’t be compatible with the latest browsers.
Watir is a web automation framework written in Ruby. It comes in two flavors: Watir and Watir-webdriver. The later is the more often used, as it has full browser support in IE, Firefox, Chrome and Safari.
Creating a browser instance in Watir-webdriver goes like this:
browser = Watir::Browser.new :ff
Then to use the browser, you do something like:
Very easy to read and write.
Real World Example
I discovered Watir-webdriver, while working at eHarmony. After two years of frustration with GEB, I offered this solution to management. I told my direct manager at the time (the 4th manager I had during my stay there) how in 10min I had full browser support and automation support… and he said to me, “I really want to give GEB a better chance…” and then trailed off on “do we really need to automate in browsers anyway? Why not go headless…”
I couldn’t believe it. QA has to sign off on each regression, on each build going out in all browsers supported. The goal of automation is to take the burden off manual testing (i.e. regression.) If automation doesn’t take off any of the burden of the cross-browser testing, then the QA team is still burdened with all those tasks! What’s the point then? To validate API’s? He said, “yes, let’s just do api validation.” It was so discouraging.
Why was he saying all this? Because of the problems of GEB. GEB couldn’t be used in IE, had problems in new versions of Firefox, and we couldn’t figure out how to automate in Safari. Basically we were stuck with GEB and it’s lack of community support… so the management team was ready to throw in the towel on browser automation and focus automation on headless drivers.
They never left GEB, but I left eHarmony. I went to a telecom company that had both a Web App, as well as a telecom component. By the end of my first day on the job, I had a automation framework built around Watir. It was already compatible with IE, FF, Chrome. I added Safari support about 2 weeks later. That was my first day on the job. GEB took 2 different teams months of work and set up and bug fighting to get working….
In the next few weeks at my new job, I was able to get parallel tests set up using Selenium Grid. Whereas our GEB framework at eHarmony took 6-9 hours to finish, I could finish tests in minutes.
Watir is by far the easiest web automation tool to set up. It’s super simple and easy to maintain. Reading watir is very logical.
Watir works out of the box in almost every browser.
Watir can easily add Safari support with just a little bit of work.
Watir is compatible with Selenium Grid for parallel test runs.
The community behind Watir is very rapid in response to changes in Browsers, Webdriver, language updates and so forth. The community already has solutions to the most common (and even rare) cases you may encounter.
Watir works well with Cucumber… Which is similar in scope to Spock, but vastly superior. With Cucumber we don’t introduce code into the test. The test remains high level and readable by all team members. The code that drives the test is in separate files called step_definition files.
Watir is Ruby based, and if you have a tool that calls a web browser (i.e. a monitoring tool), then you can’t use the Java/Grails frameworks that have more support behind them in that regard. You are forced into the Rails paradigm which is somewhat lacking.
In my years of automation testing and tool building, I’m torn on language. I prefer building web applications in Grails. But when I’m building web automation solutions, I’ve found nothing to beat Watir.
The sheer community behind it, makes it a step ahead of most frameworks. There’s nothing out there keeps as up to date as the Watir-webdriver community. Not only is the support outstanding but the sheer ease of use of the tool is another big win. It just works. There’s no hassle. None. How easy is it to set up watir?
gem install watir-webdriver
You’re all set to write watir code. Add in complexity of setting up different browsers, it works. when you do:
browserChrome = Watir::Browser.new :chrome
it works! If you want to create a special profile where chrome launches in one test full screen and in another not full screen, it’s easily doable. If you want to close popups, easy. Record screenshots, piece of cake. Execute command lines, simple.
I never have issues with Watir.
While I love Groovy and Grails for development, I have to concede that in the end there is nothing as good as Watir.
But Watir is Ruby based and my App is Java…
So what? When we are at the level of Browser Acceptance Tests, we aren’t going to be calling private methods. We’re already testing at the front end. There’s no need for this type of thinking that we need to write the automation framework in the same language as the application.
What about Flash, Who Wins?
When we talk about Flash or non browser components that the web driver api won’t talk to easily, then we are probably going to use Sikuli. Sikuli is a Java Jar…. as such it would seem a clear winner in the Groovy framework. However, there’s a lot of support in the JRuby world to integrate Sikuli over. Yes it works. I use JRuby in Sikuli, I also use it in Groovy. The draw back on JRuby is that JRuby requires Ruby 1.9.3…. and if you’re using the latest Rails, you’ll need Ruby 2.*
So in the case of using Sikuli, Groovy does have the upper hand.
Outside of potentially needing Sikuli and if you aren’t considering building a monitoring tool of some sort, Ruby & Watir is a clear winner.
When I think about making a monitoring tool that calls the production environment, does VOIP testing, or audio analysis… I will probably always choose to use Grails. But when I’m tasked with writing a browser automation framework, I’ll probably always start with Ruby and Watir.