Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

We are currently investigating automated user interface testing (we currently do automated unit and integration testing).

We've looked at Selenium and Telerik and have settled on the latter as the tool of choice due to its much more flexible recorder - and we don't really want testers writing too much code.

However, I am trying to understand the overall benefit. What are peoples' views and what sort of things work well and what doesn't?

Our system is under constant development and we regularly release new versions of our (web based) platform.

So far the main benefit we can see is for regression testing, especially across multiple client deployments of our platform.

Really looking for other people's views. We "think" it is the right thing to do but in an already busy schedule are looking for some additional insight.

share|improve this question
2  
Doesn't the term "automated testing" imply the problem it's trying to solve? // OTOH, if you're inquiring about the ROI attached to "automated testing", that's a different question... –  Jim G. Jul 12 '11 at 15:30
add comment

11 Answers 11

up vote 12 down vote accepted

When my team implemented automated UI testing a lot of great things happened.

First, the QA team became much more efficient at testing the application as well as more proficient with the application. The lead QA said that he was able to bring new QA members up to speed quickly by introducing them to the test suites for the UI.

Second, the quality of QA tickets that came back to the Dev team were better. Instead of 'Page broke when I clicked Submit button' we got the exact case that failed so we could see what was input into the form. The QA team also took it a step further by checking all cases that failed and tested other scenarios around that page to give us a better view of what happened.

Third, the QA team had more time. With this extra time, they were able to sit in on more design meetings. This in turn allowed them to be writing the new test suite cases at the same time as the Devs were coding those new features.

Also, the stress testing that the test suite we used was worth it's weight in gold. It honestly helped me sleep better at night knowing that our app could take pretty much anything thrown at it. We found quite a few pages that bucked under pressure that we were able to fix before go live. Just perfect.

The last thing that we found was that with some tweaks by the QA team, we could also do some SQL injection testing on our app. We found some vulnerabilities that we were able to get fixed up quickly.

The setup of the UI test suite took a good amount of time. But, once it was there it became a central part of our development process.

share|improve this answer
    
+1 for explaining steps to recreate failed test being intrinsic in the process (your second point) –  IThasTheAnswer Jul 12 '11 at 15:45
    
One issue: Doesn't UI unit testing block potential changes in the UI [locks you in]... although I upvoted because you described the benefit in a way where the overall runtime of the application is being monitored by the unit tests rather than an individual component of the system being tested. –  monksy Oct 5 '11 at 14:27
    
@monksy - The test suite that we used (I can't remember it's name for the life of me) wasn't co-ordinate based. It was smart enough to use the element ids. So long as we gave all our UI elements names, and kept those names through design revisions the test cases still worked. We paid a pretty penny for that software, but we felt that feature was worth it. –  Tyanna Oct 5 '11 at 20:59
    
@Tyanna Trust me on this... it was. I've tried to automate UI testing [for regressive testing] based on location. That doesn't work, and is quite frustrating. Btu I was refering to moving components arround, changing out the views/ui, and themeable UIs –  monksy Oct 5 '11 at 23:21
add comment

and we don't really want testers writing too much code

We took the opposite approach. We wanted the testers writing code.

Here's the workflow we started to adopt. It's not easy to do this because management doesn't absolutely depend on automated testing of the front-end. They're willing to settle for "close-enough".

  1. User stories.

  2. Operational concept. How the story would likely work. Design review.

  3. Screen sketch: UI design. How it would look.

  4. Selenium Scripts. If the scripts all work, we're done with the release.

  5. Coding and testing until the script works.

Automated testing is the only way to demonstrate that the functionality exists.

Manual testing is error-prone and subject to management override: "it's good enough, those failing tests don't really matter as much as releasing this on time."

"Any program feature without an automated test simply doesn't exist."

Visual presentation is another story. Manual testing of a visual layout is an exceptional case because it may involve either esthetic judgement or looking at specific (small) issues on a very large and complex screenful of pixels.

share|improve this answer
2  
"testers write automated checks". That's how testers should do their jobs. "testers ever get a chance to test" doesn't make much sense to me. Can you explain what this could mean? –  S.Lott Jul 12 '11 at 14:32
3  
@S.Lott: presumably manual testing. Automated testing is nice, but not everything. It can't spot many unexpected error modes (such as layout problem). And I'd say that the fundamentalism displayed in the last two sentences is counterproductive. –  Michael Borgwardt Jul 12 '11 at 14:38
7  
Automated testing is the only way to demonstrate that the functionality exists. No it isn't. Exploratory testing or manually executed tests demonstrates the functionality exists. It's not as good as automated testing, but automated testing isn't the only way to test. –  StuperUser Jul 12 '11 at 14:41
1  
@S.Lott - Michael and StuperUser had it right. Manual and prefereably exploratory testing. –  Lyndon Vrooman Jul 12 '11 at 14:46
1  
-1 for the fundamentalism, as Michael put it. See joelonsoftware.com/items/2007/12/03.html for an explanation of just how ridiculous that attitude is when taken to its logical conclusion. –  Mason Wheeler Jul 12 '11 at 18:08
show 4 more comments

Automated UI tests are the real integration tests. They test the entire system in the way it's actually used when it's live. That makes them the most meaningful tests. However, they also tend to be the most brittle, and the slowest to execute.

Keep an eye on the cost/benefit ratio (with brittleness being a part of the cost) and don't hestitate to have some things that are tested only manually (but make sure they are tested). And if at all possible, make it possible for developers to run specific parts of the UI test suite against their locally running version of the app, so that they can benefit from the tests during development.

Having the tests run automatically on a build server (at least once a day) is an absolute must, of course.

we don't really want testers writing too much code.

IMO this is a pipe dream. Creating automated tests is writing code. Recording functionality can help you write some of that code faster and get started more quickly with writing it manually (and slow you down terribly if you miss the point where writing code manually becomes faster), but ultimately writing code manually is what you will end up doing a lot. Better hope your testing framework supports it well and hasn't had its development focussed too much on the (very sellable) pipe dream of allowing people who can't write code to produce automated tests.

share|improve this answer
add comment

So far the main benefit we can see is for regression testing, especially across multiple client deployments of our platform.

Automating your regression testing is a good thing. This frees up your testers to do more interesting work - be this adding more automated tests, stress testing your application, or any number of other things.

Also, by making it automated you can get your developers to run the tests and hence forestall problems only being discovered later in the process.

share|improve this answer
    
What experience have you had that maintaining automated regression tests frees up testers to do more interesting work? I know this is the theory but if it takes days to write or modify the tests versus just doing the manual testing then it might not work out effective. –  IThasTheAnswer Jul 12 '11 at 15:40
    
@Tunic - We're using Silverlight in our current project and I'm writing some tests at the moment that check the bindings between the XAML and the view model C# code. This is going to mean that our testers don't have to check that the values they enter are correctly formatted etc. It's early days yet, but looks promising. –  ChrisF Jul 12 '11 at 15:43
add comment

You're right that regression is a huge one. Also -

  • if your tests are written modularly, you can get more bang for the buck by mixing and matching test sets

  • we've reused automated test scripts for data load so that we don't have to kludge a database to do large size testing

  • performance test

  • multi thread tests

  • on web systems - swapping between browsers and swapping between OSes. With browser consistency issues, hitting as wide a base as possible is a huge thing.

Things to skip - especially in web systems, watch out for cases where elements of your display are created with dynamic, changing ids - often automated test scripts don't handle this well, and you may need some serious redesign to update this.

share|improve this answer
    
+1 for your first point. Absolutely critical for a successful test automation suite! –  Lyndon Vrooman Jul 12 '11 at 14:52
    
Yes, agree first point. Have considered second and third points actually but I think this is where Telerik falls down. Selenium scripts (ableit simple ones) can be used by BroswerMob –  IThasTheAnswer Jul 12 '11 at 15:47
add comment

We've looked at Selenium and Telerik and have settled on the latter as the tool of choice due to its much more flexible recorder

I'm not sure how much you've looked into it. There certainly are other options as well. Have you looked into Watir, WatiN, Sikuli to name a few?

and we don't really want testers writing too much code.

I feel bad for the people who have to maintain these scripts. Most often, without code that can be easily modified, scripts become fragile and it begins to take longer to modify the script than it does to re-record it, which wastes even more time.

However, I am trying to understand the overall benefit. What are peoples' views and what sort of things work well and what doesn't?

Test automation is a beautiful thing when done correctly. It saves time on regression tests/checks so as to give your testers more time to do what they do best, test. Don't believe for a moment though that it is a silver bullet. Automation scripts require significant time to develop if the application already exists but the tests don't, and require constant updating with each release. Automated tests are also a great way for new people on the team to see how the system is supposed to behave. Also, make sure that your testers get to decide what needs to be automated. If it's a small check that doesn't take much to check, is very monotonous, and easy to automate, start with that. Always start with the checks that gain the most through automation, and work from there.

So far the main benefit we can see is for regression testing, especially across multiple client deployments of our platform.

It is the main benefit, and if set up correctly, can test most of the browsers that you would need with a small configuration change.

We "think" it is the right thing to do but in an already busy schedule are looking for some additional insight.

As I stated earlier, test automation takes considerable efforts, however, when done correctly, I haven't met a team yet who said "I wish we hadn't set up our test automation."

share|improve this answer
    
+1 especially for "I feel bad for the people who have to maintain these scripts". Well-designed code is a key part of writing maintainable UI tests, especially with a frequently-changing UI. If the OP's testers can't use Page Objects or reuse code, I would seriously advise the OP to consider only automating stable UI (if there is any). –  Ethel Evans Jul 12 '11 at 17:31
add comment

It makes "Expert Testing" (similar to "Exploratory testing", but carried out by end users or members of the team with a great deal of business knowledge) easier to carry out, record results, measure and automate.

share|improve this answer
add comment

I come at this from a different background. At my former employers, we developed commercial automated testing tools (QALoad, QARun, TestPartner, SilkTest, SilkPerfomer).

We saw automated UI testing as filling two roles:

  1. Full regression testing

  2. Automated setup of testing environments

We heavily leaned on the tools to perform regression tests on a nightly basis. We simply didn't have the man power to test all the buttons and dialogs to verify we didn't break anything between the UI and the business logic.

For more important tests, we found that a single person could spin up several VMs and use scripts to get to the point of a real test. It let them focus on the important bits and not try and follow a 24-step test case.

The only problem with automated testing was the habit of dumping too many tests onto the box without any kind of supervision to eliminate duplicate or unnecessary tests. Every now and then we'd have to go in and prune things back so the suite could complete in under 12 hours.

share|improve this answer
add comment

Just one example: accurately measuring the duration of webpage rendering

Using automation tests, it is far easier to test the web browser's performance. To measure the maximum response time you are likely to accept, just set a constant in your test scripts, and/or pass it as an function parameter, e.g in this pseudocode: $sel->wait_for_page_to_load($mypage, $maxtime).

Doing cross-browser testing with low values can be quite enlightening.

The alternative would be to have employees make timing measurements with a stopwatch.

share|improve this answer
add comment

Testing determines many different things. Many of these test can be automated, to allow for the removal of drudgery, and to get more done. To determine if your tests can be automated, you first need to see if the question they ask is appropriate for it.

  • Are you determining if a component functions according to spec?
  • Do you want to test all of the different possible inputs and outputs?
  • stress test the component?
  • Or are you trying to test that "it works"?

Most of these can be automated, because they are mechanical in nature. The new function accepts inputs, so what happens when we throw random data at it? But some, like testing if the system works, requires someone to actually use it. If they don't, you will never know if your users' expectations are the same as the program's. Until, that is, the system 'breaks'.

share|improve this answer
add comment

Automated testing, of any kind, provides for regression testing; by running the test that used to work, you verify it still works (or doesn't) regardless of whatever else you have added. This is true regardless of whether the test is an integration test (which usually don't touch the UI) or an AAT (which usually require the UI).

Automated UI testing allows for the system to be tested as if a user were clicking buttons. Such tests can thus be used to verify navigation through the system, correctness of labels and/or messages, performance and/or load times in a particular test environment, etc etc. The primary goal is to reduce the QA guy's time spent clicking buttons, much like integration and unit tests do for the programmer. He can set up one test one time (usually by recording his own mouse clicks and data entries into a script), and once the test works properly all he should have to do to verify the correctness of the system under test is run it again. Some frameworks, such as Selenium, allow for tests to be migrated between browsers, allowing for testing of several environments in which the site should work properly.

Without automated testing, you are limited by the number and speed of your QA testers; they must literally have hands-on the system, testing that your new feature meets requirements and (just as importantly) that you didn't break anything that was already there.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.