Tag Info

Hot answers tagged

25

Having just started trying automated tests in our team, the biggest disadvantage I've seen is that it's very difficult to apply to legacy code that wasn't designed with automated testing in mind. It would undoubtedly improve our code in the long term, but the level of refactoring needed for automated testing while retaining our sanity is a very high barrier ...


21

You pretty much nailed the most important ones. I have a few minor additions, plus the disadvantage of tests actually succeeding - when you don't really want them to (see below). Development time: With test-driven development this is already calculated in for unit tests, but you still need integration and system tests, which may need automation code as ...


17

Perhaps the most important disadvantage is ... tests are production code. Every test you write adds code to your codebase that needs to be maintained and supported. Failing to do so leads to tests you don't believe the results of, so you have no other choice. Don't get me wrong - I'm a big advocate of automated testing. But everything has a cost, and ...


16

Copy-pasted and then edited test cases are often fine. Tests should have as few external dependencies as possible, and be as straightforward as possible. Test cases tend to change with time, and previously almost identical test cases may suddenly diverge. Updating one test case without having to worry about breaking other cases is a Good Thing. Of course, ...


14

I wouldn't say these are entirely applicable disadvantages, but the few I hit most are: Time taken to set up the test in a complex enterprise application. Handling old tests that fail incorrectly, in other words, the system has evolved and now the tests are wrong. False confidence from patchy or unknown test coverage. Test coverage that is patchy can ...


8

Presumably you have some sort of architecture that connects buttons and other widgets to actions -- click on save and the save function should be called, etc. Assuming you have good test coverage of the actions themselves via unit or integration tests, the goal of automated UI testing is to insure that the widgets are all making proper calls to the ...


7

Tell your manager you'd like to automate some of the tests that you're responsible for, work the extra hours and do it. Use an open source tool (there are a bazillion of them here) Then demonstrate the value of them to the team. Nothing wins arguments better than demonstrated success.


7

I'd say the main problem with them is that they can provide a false sense of security. Just because you have unit tests it doesn't mean they're actually doing anything and that includes properly testing the requirements. In addition automated tests can also include bugs themselves, so that brings the question over whether unit tests need testing themselves ...


7

Repetition is the root of all evil That is right! Repetition is the root of all evil. Probably it was Knuth saying in his book “Premature optimization is the root of all evil”, but I think it’s repetition. Whenever you look at a program or you’re writing one and you discover some kind of repetition: Remove it! Kill it immediately… whatever but get rid of ...


7

An important rule when doing repeatable tests (automated or not) is to make sure that the test data is always in the same state before the tests start. This should be ideally assured for every single test of your test suite, so you can interchange the order of any tests at any time. How you achieve this is up to you. For your scenario, it may make sense to ...


5

Simulate the failure / exceptional situation. Your mock objects are crash test dummies and you already know what can possibly go wrong, it shouldn't be hard to emulate it (you know because you've already prepared for it, if you don't, well, you'll soon find out ;). And... it's always amazingly fun to physically unplug your database server while your ...


5

It's still pretty bad to cut and paste. There are a few problems. Your tests may well be brittle, because you are vulnerable to something that requires a change in all that copy-and-pasted code. Will you have to rewrite all the tests? If you can't encapsulate the logic into helper methods outside your tests, you can't write tests of those helper methods ...


5

At the end of a sprint, whatever you deliver should be of production quality, that's one of the pillars of scrum. If it's not done at the end of the sprint, it could drag on for ages as you find new stuff to add and new stuff to fix. Get it done and get it out the door. If you need a reason: because you want that feedback only customer usage can get you. ...


4

Though automation testing has many advantages, it has its own disadvantages too. Some of the disadvantages are: Proficiency is required to write the automation test scripts. Debugging the test script is major issue. If any error is present in the test script, sometimes it may lead to deadly consequences. Test maintenance is costly in case of playback ...


4

An important rule to test everything that is database related is to completely isolate it from the rest of your application. The ports and adapters architecture is a really good example. The database is regarded as an external plugin through an adapter to your application. The same goes with all the 3rd party subsystems. For testing how your app would ...


4

Ruby is a fine choice for a general purpose scripting language on Linux, OS X, etc. But I recently needed to write some scripts that would work on every version of Windows from 2000 through 7. VBScript didn't have the functionality I needed (talking to SOAP web services for example) and PowerShell isn't available for Windows 2000, so I looked outside the ...


4

First of all, it is not very clear if you compare floating point numbers with exact values in your unit tests. If you do so, this is wrong and you should rely on approximate comparison of floating point numbers [1]. That being said, if your computations are sensitive to factors such as the number of threads involved, the precision of numbers used and the ...


3

I used to agree with you. But then, over time, I found that every change I made (particularly DI changes in unit tests) required numerous tests to change and that was cumbersome. Now I subscribe to the school of DRY, even when writing tests. For GUI testing, you may want to look at the PageObject pattern to reduce repeated code.


3

I have to agree with Matthew Flynn. Generally your best bet here is to just go and start dealing with the problem either in your spare time or during regular work hours if it significantly reduces your workload. At a minimum boosting automation is going to save your own personnel time in the long run and hopefully prevent a few 12 am calls. I'll add in ...


3

I recently asked a question about tests in game development - this is BTW how I knew about this one. The answers there pointed some curious, specific disadvantages: It is costly to do when your code should be highly coupled. It is difficult to do when you have to be aware of the various hardware platforms, when you should analyze output to the user and the ...


3

You can take a look at the tests that Scala itself is using at the moment. To be honest, I had never seen the kind of paired test you mention. The scope of ScalaCheck is pretty simple: if there's a rule, property, invariant or condition your code must observe, then ScalaCheck is a great fit for it. I'll state this in reverse to make it even clearer. The ...


3

For coverity, openSSL is https://scan.coverity.com/projects/294 According to Andy Chou, it is not something they currently detect. I think it would take whole-program tracking of allocation sizes, which I'd be surprised if any current C/C++ checker could do without massive false positives. Maybe some of the ultra-expensive aerospace tools like Polyspace ...


2

Firstly, you should not use profiling to determine performance. Profiling is intended to be used in order to identify the parts of your program which take the most time. When measuring performance, we only care about the speed of the whole program taken together. Profiling in that case only skews results. Instead, performance should be tested by benchmarks. ...


2

I want to add one more, a false sense of security. Beyond very small well defined problem sets, it is not possible to create comprehensive tests. There may and often will still be bugs in our software that the automated tests simply don't test for. When the automated tests pass we all too often assume there are no bugs in the code.


2

I would recommend picking up XUnit patterns. I used to have the exact same problem until I started leveraging that book. The Object Mother pattern sounds like it would be the most helpful for your scenario. Like someone else mentioned, properly encapsulating this setup code might be onerous, but having to change it in all the places you copy and pasted is ...


2

That is an interesting question because the database is usually the part that is faked during application unit testing. Hopefully the logic of the database engine itself is well tested by the provider but of course the queries, schema, and stored procedures are code that needs to be tested and protected against regression. This is often left to integration ...


2

Should people try and limit repitition when they can - yes. But the payoff depends on the situation. This could go back to the 'best practice' debate. But the question is what is best for you in this situation. There are exceptions to every rule. A couple things I would ask are: 1) How likely is it that this functionality being tested in the UAT will ...


2

There are unit test generators. For example, in the .NET world, something like Microsoft Pex could do this. For example, Microsoft Pex tries based on code analysis all possible values as arguments for a method. Some arguments are expected to let the method throw an exception. Such things can automatically tests created for. Static values like an empty ...


2

How do you eat an elephant? One bite at a time. It sounds you are looking for a "magic bullet" to solve all of your problems at once, and that's IMHO part of the problem. There is not magic bullet, this is hard work, and you can only solve it step-by-step. For example, you told us that you have already some fuzzy or statistical methods in place to compare ...


1

In the past, a test case usually referred to a function / method. However, in our days, a Test Case usually refers to a "Test", meaning a class containing a set of methods which are strongly related to each-other from a testing point of view. Methods in these Test Case classes are called Test Methods. Considering this, the inclusion would be. Test Method ...



Only top voted, non community-wiki answers of a minimum length are eligible