What problem does automated user interface testing solve?

Question

We are currently investigating automated user interface testing (we currently do automated unit and integration testing).

We've looked at Selenium and Telerik and have settled on the latter as the tool of choice due to its much more flexible recorder - and we don't really want testers writing too much code.

However, I am trying to understand the overall benefit. What are peoples' views and what sort of things work well and what doesn't?

Our system is under constant development and we regularly release new versions of our (web based) platform.

So far the main benefit we can see is for regression testing, especially across multiple client deployments of our platform.

Really looking for other people's views. We "think" it is the right thing to do but in an already busy schedule are looking for some additional insight.

Answer 1

When my team implemented automated UI testing a lot of great things happened.

First, the QA team became much more efficient at testing the application as well as more proficient with the application. The lead QA said that he was able to bring new QA members up to speed quickly by introducing them to the test suites for the UI.

Second, the quality of QA tickets that came back to the Dev team were better. Instead of 'Page broke when I clicked Submit button' we got the exact case that failed so we could see what was input into the form. The QA team also took it a step further by checking all cases that failed and tested other scenarios around that page to give us a better view of what happened.

Third, the QA team had more time. With this extra time, they were able to sit in on more design meetings. This in turn allowed them to be writing the new test suite cases at the same time as the Devs were coding those new features.

Also, the stress testing that the test suite we used was worth it's weight in gold. It honestly helped me sleep better at night knowing that our app could take pretty much anything thrown at it. We found quite a few pages that bucked under pressure that we were able to fix before go live. Just perfect.

The last thing that we found was that with some tweaks by the QA team, we could also do some SQL injection testing on our app. We found some vulnerabilities that we were able to get fixed up quickly.

The setup of the UI test suite took a good amount of time. But, once it was there it became a central part of our development process.

Answer 2

Automated UI tests are the real integration tests. They test the entire system in the way it's actually used when it's live. That makes them the most meaningful tests. However, they also tend to be the most brittle, and the slowest to execute.

Keep an eye on the cost/benefit ratio (with brittleness being a part of the cost) and don't hestitate to have some things that are tested only manually (but make sure they are tested). And if at all possible, make it possible for developers to run specific parts of the UI test suite against their locally running version of the app, so that they can benefit from the tests during development.

Having the tests run automatically on a build server (at least once a day) is an absolute must, of course.

we don't really want testers writing too much code.

IMO this is a pipe dream. Creating automated tests is writing code. Recording functionality can help you write some of that code faster and get started more quickly with writing it manually (and slow you down terribly if you miss the point where writing code manually becomes faster), but ultimately writing code manually is what you will end up doing a lot. Better hope your testing framework supports it well and hasn't had its development focussed too much on the (very sellable) pipe dream of allowing people who can't write code to produce automated tests.

Answer 3

and we don't really want testers writing too much code

We took the opposite approach. We wanted the testers writing code.

Here's the workflow we started to adopt. It's not easy to do this because management doesn't absolutely depend on automated testing of the front-end. They're willing to settle for "close-enough".

1. User stories.

2. Operational concept. How the story would likely work. Design review.

3. Screen sketch: UI design. How it would look.

4. Selenium Scripts. If the scripts all work, we're done with the release.

5. Coding and testing until the script works.

Automated testing is the only way to demonstrate that the functionality exists.

Manual testing is error-prone and subject to management override: "it's good enough, those failing tests don't really matter as much as releasing this on time."

"Any program feature without an automated test simply doesn't exist."

Visual presentation is another story. Manual testing of a visual layout is an exceptional case because it may involve either esthetic judgement or looking at specific (small) issues on a very large and complex screenful of pixels.

Answer 4

So far the main benefit we can see is for regression testing, especially across multiple client deployments of our platform.

Automating your regression testing is a good thing. This frees up your testers to do more interesting work - be this adding more automated tests, stress testing your application, or any number of other things.

Also, by making it automated you can get your developers to run the tests and hence forestall problems only being discovered later in the process.

Answer 5

We've looked at Selenium and Telerik and have settled on the latter as the tool of choice due to its much more flexible recorder

I'm not sure how much you've looked into it. There certainly are other options as well. Have you looked into Watir, WatiN, Sikuli to name a few?

and we don't really want testers writing too much code.

I feel bad for the people who have to maintain these scripts. Most often, without code that can be easily modified, scripts become fragile and it begins to take longer to modify the script than it does to re-record it, which wastes even more time.

However, I am trying to understand the overall benefit. What are peoples' views and what sort of things work well and what doesn't?

Test automation is a beautiful thing when done correctly. It saves time on regression tests/checks so as to give your testers more time to do what they do best, test. Don't believe for a moment though that it is a silver bullet. Automation scripts require significant time to develop if the application already exists but the tests don't, and require constant updating with each release. Automated tests are also a great way for new people on the team to see how the system is supposed to behave. Also, make sure that your testers get to decide what needs to be automated. If it's a small check that doesn't take much to check, is very monotonous, and easy to automate, start with that. Always start with the checks that gain the most through automation, and work from there.

So far the main benefit we can see is for regression testing, especially across multiple client deployments of our platform.

It is the main benefit, and if set up correctly, can test most of the browsers that you would need with a small configuration change.

We "think" it is the right thing to do but in an already busy schedule are looking for some additional insight.

As I stated earlier, test automation takes considerable efforts, however, when done correctly, I haven't met a team yet who said "I wish we hadn't set up our test automation."

Answer 6

You're right that regression is a huge one. Also -

• if your tests are written modularly, you can get more bang for the buck by mixing and matching test sets

• we've reused automated test scripts for data load so that we don't have to kludge a database to do large size testing

• performance test

• multi thread tests

• on web systems - swapping between browsers and swapping between OSes. With browser consistency issues, hitting as wide a base as possible is a huge thing.

Things to skip - especially in web systems, watch out for cases where elements of your display are created with dynamic, changing ids - often automated test scripts don't handle this well, and you may need some serious redesign to update this.

Answer 7

Just one example: accurately measuring the duration of webpage rendering

Using automation tests, it is far easier to test the web browser's performance. To measure the maximum response time you are likely to accept, just set a constant in your test scripts, and/or pass it as an function parameter, e.g in this pseudocode: $sel->wait_for_page_to_load($mypage, $maxtime).

Doing cross-browser testing with low values can be quite enlightening.

The alternative would be to have employees make timing measurements with a stopwatch.

Answer 8

Automated User Interface Testing solves the ability to:

• rapidly repeat testing of a large number of components
• remember to test a large number of functions each time
• compare runs and run times of test suites as the application grows
• set up runs with hundreds of different inputs and variable conditions
• enable people who did not write the test to run it and see any visual issues
• allow end-users to see the application being used in a quick and easy way
• distribute test UI runs to a network, remote server or service
• start volume testing using parallel machines.

However, as others have noted:

Telerik...

the tool of choice due to its much more flexible recorder

is a red-flag for many of us.

Scripts recorded this way tend to not be a long-term solution because:

• database/object ID tend to change from case to case
• manually recorded scripts often rely on page layout tags that frequently change
• common actions will need to be written over and over again instead of allowing re-use (see SitePrism & PageObject approach)
• sometimes you need to use tools such as xpath to grab additional information based on current page information. A simple recorded script won't do this.
• developers and testers who write code will not be encouraged to use CSS classes, ID's and HTML5 data-attributes, which are practices that will lead to more robust, maintainable tests.

Telerik does have some advantages that should be considered though:

• aimed at mobile clients
• built-out tools to manage growth
• handles Android, iOS and Windows Phone

An approach that can help bridge the gaps is to record the initial script using the tools page recorder but then change the script to use ID's, classes and data- attributes so that it will last over time. This is an approach that I have actually used with the firefox selenium plugin.

Answer 9

It makes "Expert Testing" (similar to "Exploratory testing", but carried out by end users or members of the team with a great deal of business knowledge) easier to carry out, record results, measure and automate.

Answer 10

I come at this from a different background. At my former employers, we developed commercial automated testing tools (QALoad, QARun, TestPartner, SilkTest, SilkPerfomer).

We saw automated UI testing as filling two roles:

1. Full regression testing

2. Automated setup of testing environments

We heavily leaned on the tools to perform regression tests on a nightly basis. We simply didn't have the man power to test all the buttons and dialogs to verify we didn't break anything between the UI and the business logic.

For more important tests, we found that a single person could spin up several VMs and use scripts to get to the point of a real test. It let them focus on the important bits and not try and follow a 24-step test case.

The only problem with automated testing was the habit of dumping too many tests onto the box without any kind of supervision to eliminate duplicate or unnecessary tests. Every now and then we'd have to go in and prune things back so the suite could complete in under 12 hours.

Answer 11

Automated testing, of any kind, provides for regression testing; by running the test that used to work, you verify it still works (or doesn't) regardless of whatever else you have added. This is true regardless of whether the test is an integration test (which usually don't touch the UI) or an AAT (which usually require the UI).

Automated UI testing allows for the system to be tested as if a user were clicking buttons. Such tests can thus be used to verify navigation through the system, correctness of labels and/or messages, performance and/or load times in a particular test environment, etc etc. The primary goal is to reduce the QA guy's time spent clicking buttons, much like integration and unit tests do for the programmer. He can set up one test one time (usually by recording his own mouse clicks and data entries into a script), and once the test works properly all he should have to do to verify the correctness of the system under test is run it again. Some frameworks, such as Selenium, allow for tests to be migrated between browsers, allowing for testing of several environments in which the site should work properly.

Without automated testing, you are limited by the number and speed of your QA testers; they must literally have hands-on the system, testing that your new feature meets requirements and (just as importantly) that you didn't break anything that was already there.

Answer 12

Testing determines many different things. Many of these test can be automated, to allow for the removal of drudgery, and to get more done. To determine if your tests can be automated, you first need to see if the question they ask is appropriate for it.

• Are you determining if a component functions according to spec?
• Do you want to test all of the different possible inputs and outputs?
• stress test the component?
• Or are you trying to test that "it works"?

Most of these can be automated, because they are mechanical in nature. The new function accepts inputs, so what happens when we throw random data at it? But some, like testing if the system works, requires someone to actually use it. If they don't, you will never know if your users' expectations are the same as the program's. Until, that is, the system 'breaks'.

Answer 13

In my experience, automated user interface testing covers many gaps including:

• Lack of documentation (example: using the automated test runner to demo the existing functionality)
• Outdated requirements due to scope creep (example: identifying the gap between requirements and code by capturing the screen during test runs)
• High turnover of developers and testers (example: reverse engineering legacy JavaScript by capturing the screen during test runs with the developer tool open)
• Identifying violations of standard naming conventions via XPath regression tests (example: searching all DOM attribute nodes for camel cased names)
• Recognizing security holes which only a computer can discover (example: logout from one tab while simultaneously submitting a form in the other)

Answer 14

I would like to share our team’s experience. We have been using our own UI testing tool, Screenster, to test ours and our customers’ web apps. It has proven itself to be a helpful alternative to Selenium for visual/CSS testing tasks. Screenster is a test automation tool which performs screenshot-based comparison of different versions of your web pages. First it creates a visual baseline for a page, taking a screenshot for each user action. During the next run it takes a new screenshot at each step, compares it with the one from baseline and highlights differences.

Summing it up, Screenster has the following advantages: Visual baseline: screenshots are captured for each user step during test recording Screenshot based comparison: Screenster compares images captured during a playback to those from the baseline and highlights all differences Smart CSS selectors: tester can select CSS elements on the screenshots and perform actions with them - e.g. mark them as ignore regions to exclude from further comparison