Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

I've been spending the last week or so learning selenium and building a series of web tests for a website we're about to launch. it's been great to learn, and I've picked up some xpath and css location techniques.

the problem for me though, is seeing little changes break the tests - any change to a div, an id, or some autoid number that helps identify widgets breaks any number of tests - it just seems to be very brittle.

so have you written selenium (or other similar) tests, and how do you deal with the brittle nature of the tests (or how do you stop them being brittle), and what sort of tests do you use selenium for?

share|improve this question
    
If little changes are breaking the tests, then the tests are not being run after the changes are made - which is the whole point of the tests. Also, developer awareness about the (existence of the) tests seems to be lacking. –  talonx Nov 14 '10 at 10:18
    
@talonx - it's not enough just to run this sort of test, if a small change results in a big change to the suite then it's going to be problematic regardless of whether they're run by the developer, the ci box or the testers –  FinnNk Nov 14 '10 at 11:41
    
@FinnNK: Agree - small changes shouldn't result in big changes. That means the test cases were written by someone who's not in touch with the developers making the changes - smells like a communication gap. –  talonx Nov 14 '10 at 12:15
    
@talonx: UI tests are inherently brittle and take much longer time to run than unit-tests. They are a special challenge, so I wouldn't jump to conclusions about developers in @Sam's organization. –  azheglov Nov 14 '10 at 13:22
1  
I've changed the title - hopefully a little more representative of the question and a little less negative. –  Jon Hopkins Nov 14 '10 at 17:19
show 2 more comments

5 Answers 5

up vote 6 down vote accepted

The purpose of Selenium is to create UI-driven integration tests.

Integration tests verify that all components of your system work correctly when deployed together. Integration tests are not a sufficient test strategy and complement other test strategies having a different focus, for example unit-testing and acceptance testing.

UI-driven tests are inherently brittle, although Selenium and Watir are a step up from the early days of record-and-playback tools. There are several ways to deal with this problem - here's is a compilation of advice from some world-class experts:

Don't try to get all your test coverage from this type of tests. Robert C. Martin argues that your code coverage by integration tests should be about 20%. It is simply impractical to test all paths of execution when the input is several application layers away.

Get most of the test coverage from unit- and acceptance tests. Look for links to Gojko Adzic's articles in FinnNk's answer. Adzic argued repeatedly about testing business logic through acceptance tests and bypassing UI.

But some amount of UI-driven tests still needs to be written. This is where you need some practical advice in addition to "don't test your business logic via UI." I'd recommend Patrick Wilson-Welsh's blog as the starting point.

share|improve this answer
add comment

The most important thing when creating tests like this once you get past a trivial number is the idea of symmetric change - a small change in the code should result in a small change in the test suite.

For example, let's say you collect someone's name with two text boxes in 100 tests. If you write those tests niavely (or maybe are using record-playback) then you will have 100 tests to change. If instead you abstract out an 'enter name' step and use that in your tests instead of directly using selenium then you only have 1 place to make your change. It'll depend on your context how many layers of abstraction you use - but using abstraction will go a long way to making your tests maintainable.

A second important thing is to make sure your selectors are based on the thing which is most important and least likely to change. Sometimes this will be an id or a class or it could be the text in or surrounding an element. Always prefer the simplest selector (e.g. an id) but sometimes to achieve this you will have to use something like xpath.

Gojko Adzic has lots of great advice on his blog when it comes to this sort of testing - check out his Sine of Death and how to avoid it in particular.

share|improve this answer
    
good tips FinnNk - I've abstracted common methods (login, look for sidebar widgets etc) - so the damage is reduced when tests do change. Unfortunately we're using a propriety system that creates some of the widgets, and creates id's for widgets based on their position in the dom, so whenever something is moved, tests are broken. –  Sam J Nov 14 '10 at 23:56
1  
Usually you can find some way of identifying an element - perhaps some text that element contains, or maybe it's always has the same relative position to something else. Finally, as a last resort, you can also define custom locator strategies for locating elements using selenium which gives you full control (e.g. if you're looking for elements with the name base name and some arbitary stuff added to the end, as in your case or in asp.net webforms). –  FinnNk Nov 15 '10 at 7:57
add comment

Patrick Wilson Welsh has written a bit about this. I can't begin to re-create his work here, but check it out:

http://patrickwilsonwelsh.com/?p=332

share|improve this answer
add comment

Testing ensures that code behaves as expected. If your tests break on a regular basis, then the tests are either not testing the right thing or the developers do not care about the tests.

Personally I think that if the presence of a <div> tag breaks a test then the test is needlessly strictly dependent on the surrounding tags, and a more lenient approach should be taken.

share|improve this answer
add comment

Patrick Wilson provides good starting point, as others suggested.
I would also suggest reading my answer to similar question here.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.