Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

For business or tester facing tests on projects (not unit tests), I've typically used FitNesse, but I've dabbled with RSpec and I'm interested in learning Selenium and Robot.

Do you have experience in these (or other) acceptance testing tools? What are the pros and cons?

For FitNesse, I can think of the following pros and cons initially.

PROS:

  • Wiki based editing, which most people understand
  • Lots of support and plugins for different languages
  • Completely self contained, you just download and go

CONS:

  • Refactoring the tests can be a little clunky
  • You need to know (or document) the various properties that each method in your test fixture expects. There's no "intellisense" or IDE to do this for you that I'm aware of.
share|improve this question

closed as not constructive by MichaelT, Eric King, Jalayn, Martijn Pieters, ChrisF Apr 2 '13 at 21:19

As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.If this question can be reworded to fit the rules in the help center, please edit the question.

add comment

3 Answers

up vote 7 down vote accepted

We use Robot Framework extensively and Fitnesse a little.

Fitnesse I think has an edge if you like the wiki interface that it exposes to people writing the tests - it's particularly good if your data consists of tables of primitives with expected inputs and outputs. Once you get beyond the basics though it gets clunkier and clunkier.

Robot Framework is extremely good if you're testing workflows either directly or indirectly (e.g. if you're testing through the GUI and need to reach a particular position in a system with no test hooks).

It works around the idea of keywords which can be composed from other keywords - which allows you to organise your tests into layers of abstraction, but in a way which people from a non-programming can grasp and leverage. There are a number of libraries out there (for example a wrapper around Selenium for web testing and others for Java and Windows apps, telnet, etc).

I've found that we could get testers with no previous test automation experience editing and maintaining tests with no or little developer support with a couple days (aside from initial setup) and within a week or so they were creating new tests in a maintainable manner by themselves. I don't think any of the other frameworks I've seen really allow that without a lot of developer support.

I don't have direct experience of the other acceptance testing frameworks out there - but from what I've seen from brief dabbles plus experience reports at conferences I think if it's primarily developers (or testers/analysts with moderate to good coding skills) who will be creating and maintaining the tests then probably one of the other frameworks could be a good choice. Cucumber (or its variations) is talked about a lot for example. I really liked the look of something I saw at CITCON London 2010 by YouDevise which has apparently just been released as open source - I think this is it.

I don't think there's a one-size-fits-all solution - different tools are good for different contexts and mixes of users.

share|improve this answer
add comment

We use Cucumber with Selenium where I'm working, and it's been really helpful for making it clear to all parties what exactly a feature is going to entail.

Keeping the underlying code organized is another challenge, though.

share|improve this answer
    
I'm doing the same, except since I'm working with .NET, I'm using SpecFlow which is a .NET port of Cucumber. Once you learn it and get good at keeping the underlying code organized (it takes some time), it is a brilliant way to write acceptance tests. –  Pete Feb 9 '11 at 13:12
add comment

I've used FitNesse and Selenium (essentially FitNesse driving Selenium).

When working with Selenium, I think Selenium test scripts should definitely be written by developers, with testers defining the acceptance criteria in a different tool (such as FitNesse). I've seen the mess that can be made out of selenium-only test scripts, which define long processes and reference elements using long xpaths which will break if the page layout changes slightly.

Some cons of Selenium, for which effort needs to be taken to avoid, include:

  • Can be brittle, so changing something simple on the ui can break lots of tests
  • There's a need to put explicit "waits" in tests to wait for things to happen in the ui, which can be brittle as well
share|improve this answer
add comment

Not the answer you're looking for? Browse other questions tagged or ask your own question.