Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

The basic gist of most Agile methods is that a feature is not "done" until it's been developed, tested, and in many cases released. This is supposed to happen in quick turnaround chunks of time such as "Sprints" in the Scrum process.

A common part of Agile is also TDD, which states that tests are done first.

My team works on a GUI program that does a lot of specific drawing and such. In order to provide tests, the testing team needs to be able to work with something that at least attempts to perform the things they are trying to test. We've found no way around this problem.

I can very much see where they are coming from because if I was trying to write software that targeted some basically mysterious interface I'd have a very hard time. Although we have behavior fairly well specified, the exact process of interacting with various UI elements when it comes to automation seems to be too unique to a feature to allow testers to write automated scripts to drive something that does not exist. Even if we could, a lot of things end up turning up later as having been missing from the specification.

One thing we considered doing was having the testers write test "scripts" that are more like a set of steps that must be performed, as described from a use-case perspective, so that they can be "automated" by a human being. This can then be performed by the developer(s) writing the feature and/or verified by someone else. When the testers later get an opportunity they automate the "script" for regression purposes mainly. This didn't end up catching on in the team though.

The testing part of the team is actually falling behind us by quite a margin. This is one reason why the apparently extra time of developing a "script" for a human being to perform just did not happen....they're under a crunch to keep up with us developers. If we waited for them, we'd get nothing done. It's not their fault really, they're a bottle neck but they're doing what they should be and working as fast as possible. The process itself seems to be set up against them.

Very often we end up having to go back a month or more in what we've done to fix bugs that the testers have finally gotten to checking. It's an ugly truth that I'd like to do something about.

So what do other teams do to solve this fail cascade? How can we get testers ahead of us and how can we make it so that there's actually time for them to write tests for the features we do in a sprint without making us sit and twiddle our thumbs in the meantime? As it's currently going, in order to get a feature "done", using agile definitions, would be to have developers work for 1 week, then testers work the second week, and developers hopefully being able to fix all the bugs they come up with in the last couple days. That's just not going to happen, even if I agreed it was a reasonable solution. I need better ideas...

share|improve this question
add comment

7 Answers

first, get rid of the split between 'testers' and 'developers'. Everyone tests

second, in TDD, the developers code the tests before they code the feature/story

What you have described is not TDD [it may be Scrum though; Scrum is a project-management methodology independent of the development methodology; Scrum is not relevant to your problem]

Scenarios where automated testing is impossible are exceedingly rare. Scenarios where automated testing is difficult, expensive, or inconvenient are much more common - but it is precisely these scenarios where automated testing is needed the most.

From the vague description, I assume the software is drawing something on the screen. If what is being drawn is determined by data, formulas, or functional steps, then at least write automated tests that test to the level of the data/formulas/functional steps. If the screen output is deterministic (the same steps result in the same drawing output every time) then test manually once, take a screenshot, and let future tests compare the output to the verified screenshot. If the screen output is nondeterministic and not governed by data, formulas, or functional steps, then you're in that rare area where automated testing may be impossible. But I doubt it.

I'm guessing that the only reason testing has not been automated so far is that the developers don't care about it. In TDD, the developers do the testing, so they feel the pain of the boring repetition of testing the same 62-step process a hundred times until all the bugs are gone. Nothing will get an automated testing solution developed faster than making the developers do the testing.

share|improve this answer
    
I'm speaking specifically of acceptance tests, not unit tests. At any rate, I'm not going to have the opportunity to alter the departmental structure of the company. Need a solution that works with both developers and "developers in test". That said, how would you do your screenshot example before there was something to screenshot? There's actually a LOT of our automated tests that do this, but before anything like that can be automated there has to be something to take a screenshot of. –  Crazy Eddie Jun 20 '11 at 2:49
    
@Crazy Eddie: TDD defines tests for user stories which define the features of the application. The terminology of 'unit test' is often used in this context, but it does not have the same limited meaning that it would in a purely unit-testing methodology (where code coverage would matter, for example). TDD tests features, and scales up to the acceptance-testing level. The use of the term 'unit testing' in the literature is a regrettable point of confusion for many people. As for the screenshot, you have to run the thing successfully once so that approach is only good for regression testing. –  Steven A. Lowe Jun 20 '11 at 2:52
    
@Crazy Eddie: with a name like Crazy Eddie, I would expect you to be the champion of challenging pointless inefficient cycles of failure ;-) [yes i read your blog]. Seriously though, nothing is going to change because the developers feel no pain. Let them feel the pain - or maybe offer them some kind of reward for helping - and they will come up with a solution for automated testing, and/or starting writing code that is easier to test. –  Steven A. Lowe Jun 20 '11 at 2:56
    
The difference that I see as being most important from my perspective is that they require different areas of expertise. A unit test can, and often is, written in the same language as the application bits they test. Automated acceptance testing on the other hand makes use of whatever language the particular software used to drive the UI responds to. They require different architectures as well, as at least our testers have developed huge libraries of reusable components they adapt for specific needs. Just like a team can involve DB experts vs. code, it seems fine to have test experts. –  Crazy Eddie Jun 20 '11 at 3:00
    
I guess this solves the issue of keeping developers busy while tests are done though. Even if it takes them longer for any given task, it could be worthwhile. I may very well not get the buy in I need though and I imagine the problem to just keep resurfacing until we can figure out how to get the tests written ahead of time or at least in sync. I still don't see any way to solve this issue as the requirement of having something there to drive seems to be a prerequisite of the automation problem. –  Crazy Eddie Jun 20 '11 at 3:05
show 8 more comments

My teams found that TDD wasn't adequate for designing GUI tests. All of the automated GUI level testing frameworks we used required code to be written before the test. So we switched to Behavior Driven Development. Sure, the tests aren't automated, but it did give us a way to measure if the UI was "done-done" from the beginning.

share|improve this answer
add comment

We find ATDD for the GUI to difficult/expensive.

We usually do the front-end first. Since we write webapps it's usually in HTML/CSS/JS, and then we get sign-off on the the look, flow, etc. These will eventually get translatedinto templates with the appropriate parts replaced with dynamic bits.

Once the UI mockup is complete we write tests that mimic a web request. XPath is used to assert the data exists in the correct HTML tags.

We find this style of testing gives us good value for the time we spend. We still assert the data is returned, and some general structure about the html. Since we already worked out the look beforehand through the mockup stage we don't worry about trying to assert pixel placement. We just look at the page as we develop both as part of development as well as a double check.

For GUI non-web dev I would probably try add some hooks to make the front end scriptable. I would probably also mockup the UI first as well, even if on paper.

share|improve this answer
add comment
 > Acceptance tests done first…how can this be accomplished?

I prefer a bdd-tdd-style of development as described in this article: Behavior-Driven Development with SpecFlow and WatiN.

The Example uses .net for development with SpecFlow + NUnit + WaitN. If you are developing with (j)ruby/java you can try cucumber + junit + waitr

share|improve this answer
add comment

One thing we considered doing was having the testers write test "scripts" that are more like a set of steps that must be performed, as described from a use-case perspective, so that they can be "automated" by a human being. This can then be performed by the developer(s) writing the feature and/or verified by someone else. When the testers later get an opportunity they automate the "script" for regression purposes mainly. This didn't end up catching on in the team though.

This is more or less what we do. Every feature is developed in parallell with the test case (script) and no feature is 'done' until all three steps is done: development, test case and testing. All test cases are written from the requirements by testers and checked with the developers, so we all have a common understanding of the problem. Like your 'every other week' suggestion, except that when the developers are working on the feature the testers are working on test cases and when the testers are testing the developers are investigating the next feature.

I think the biggest problem you have is that

[t]he testing part of the team is actually falling behind us by quite a margin. [...] If we waited for them, we'd get nothing done

Since the test team is so far behind I really think you ought to wait for them. I'm sure there are something productive you could do - such as develop some features that are easy to test but takes a lot of time to develop. It can also help to take test time into consideration when planing a sprint, neither developers or testers should be overworked or idle.

share|improve this answer
add comment

Acceptance test-driven development (distinct from, but complementary to test-driven development) is a great way to ensure that you have the requirements nailed before you begin coding.

On my team we (product owner, QA analysts and developers) sit together and use FitNesse two write acceptance tests. These sessions are usually driven by QA but we all participate. These won't run straight away since some development effort is needed to add the fixtures (the glue between the wiki pages and the production code). These tests form the acceptance criteria for a story. FitNesse is designed to interact with the layer below the UI.

As developers we then work to make these acceptance tests pass. We use TDD for this, but TDD should not be confused with ATDD.

Once the FitNesse pages turn green we are almost done we still have some work to do (exploratory testing, usually lead by QA, and so forth).

share|improve this answer
    
I see how unit tests and acceptance tests shouldn't be confused. Not that Uncle Bob is a god, but in "Agile Principles, Patterns and Practices in C#" he includes acceptance testing (and uses FitNesse) as a part of TDD and doesn't make a distinction and doesn't even mention ATDD. –  JeffO Feb 7 at 12:55
add comment

Maybe writing coded UI tests can help you out. If you are working with Visual studio 2010 you can use an add-in called "coded ui test" for recording and coding a very specific script for interacting with the UI of your app. Recording an interaction into a script introduces an extra level of abstraction which guards you aganist small changes in the UI. However introducing this form of testing also introduces the burden of maintaining the tests.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.