Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

I've been involved with many projects in several companies because I've been a developer for a long time and I'm a contractor.

I estimate that less than 20% of projects are methodically tested. With methodically tested I mean any testing beyond ad-hoc no plan testing.

I also estimate that less than 10% of projects are thoroughly methodically tested where they have dedicated testers as part of the team, test plan document, where developers write automated tests and then they also track test coverage and measure results.

Two questions

  1. What are your percentage estimates about this issue?
  2. What's your professional experience regarding software testing?

Additional note

Since methodical testing question may get quite biased answers (people like to brag about being superior to others) I encourage other developers (those that're not exposed to methodical testing) to provide their answer as well, because otherwise it will look like testing is being done everywhere... except at your company.

share|improve this question
    
Great question, thanks for taking the time to reformulate! –  user8 Sep 15 '10 at 9:56
    
@Mark Trapp: Thanks. I kindda think it's quite basic but I may ask a few more (based on previous set of questions) –  Robert Koritnik Sep 15 '10 at 12:36
add comment

14 Answers 14

up vote 7 down vote accepted

The pattern I have seen with testing over my career shows a strong correspondence with the risk of failure in a project. Big projects are more likely to be tested than small ones, mission critical applications are more likely to be tested than one off marketing web sites, in house systems are less likely to be tested than public facing ones.

That said there are still projects that have been excessively tested and those that have not been tested enough, but these are the minority.

share|improve this answer
    
I've edited my question slightly. Could you provide your estimates as well? –  Robert Koritnik Sep 15 '10 at 12:44
    
Almost all the projects (80%+) I have been involved with have been methodically tested, but then I have almost exclusivly done corporate mission critical applications. –  Martin Brown Jan 10 '11 at 12:32
    
I work for pharmaceutical corporation. I would say that 80% of applications are tested by professional testers and developers. Those 20% are low risk applications like promotional presentation on iPad. but even this one is ad-hoc checked by someone. –  yoosiba Feb 20 '11 at 19:43
add comment

Everything we produce gets completely tested. If our internal QA team is overloaded, we have an offshore team that tests the projects. They're not as good as our internal team but that's a different topic.

share|improve this answer
    
I've edited my question slightly. Could you provide your estimates of the development professional market. Because your projects get scrutinised. And are your tests automated or not? –  Robert Koritnik Sep 15 '10 at 12:46
    
Who is this offshore team? Would you give them a recomendation? –  Martin Brown Jan 10 '11 at 12:33
    
Usually big companies have R&D Centers in other countries(Mostly in Asia). Where the offshore development is done. The objective of offshore development is to reduce development cost (some part of it). –  Nips Apr 27 '11 at 14:18
add comment

The three companies I have worked for during the last 15 years all had unit tests which were run automatically.

At two of those companies I pushed for introducing them.

share|improve this answer
    
I edited my question slightly. Could you provide your estimates about software testing on professional market as well? –  Robert Koritnik Sep 15 '10 at 12:47
    
@Robert: I haven't been job-hopping enough to give a general estimate. From what little I know, things are becoming better. But then, it had been me who was pushing for automatic unit tests in two out of the three cases I told you about... –  sbi Sep 15 '10 at 18:23
    
But you do talk to other developers in other companies, don't you? –  Robert Koritnik Sep 15 '10 at 19:00
add comment

In the last 9 years, i've basically only met acceptance/regression tests. There were only a few unit tests.

share|improve this answer
    
I've edited my question slightly. Could you provide your estimates of software testing in professional market as well? –  Robert Koritnik Sep 15 '10 at 12:48
add comment

Yes.

The amount of testing is in proportion to the reliability required of the app, as well as the maturity of the programmer culture.

Web sites are quite often walking bug-holes (broken links are a defect).

Video games are often buggy.

Windows (finally) is fairly reliable.

Routers are very reliable

Hospital monitors "don't break"

Note that the fiscal cost of failure is also correlated to reliability.

share|improve this answer
1  
I strongly disagree with you - you've never seen a router fail? Do Xbox, Playstation and Wii games lock-up? Ever had a blue-screen or 'Application is not responding' in Windows? –  JBRWilkinson Jan 21 '12 at 9:33
add comment

In 10 years I never worked on a project with formal code testing.

In my current job we only have functional testing.

The problem is that no one in the management even know about code testing. Test department don't even know about code testing, they just follow the high level specification and verify if we comply from a behavioral/functional standpoint.

We don't have a qualified software leader that forces us to code well. The result is spaghetti code, lot of regressions, missed schedules and so on...

share|improve this answer
    
Thak you for your honest answer. What are your estimates (check my edited question)? –  Robert Koritnik Sep 15 '10 at 12:47
    
My estimate for Italy is under 10% of formal code testing. Perhaps almost only mission-critical code. –  Lorenzo Sep 15 '10 at 12:56
    
I've worked in Ireland, England, Scotland and Slovenia and as it seems Italy doesn't seem any different. –  Robert Koritnik Sep 15 '10 at 17:54
add comment

We are a mid sized offshore company in South Asia. However, we always do USA based projects and directly work with requirements sent from USA company.

We apply methodical testing on every application we build. Perhaps, the quality of testing isn't up to standard, but we do employ them.

share|improve this answer
    
Would those tests be automated or mainly manual? I've edited my question slightly. Could you provide your estimates of software testing in professional market as well? –  Robert Koritnik Sep 15 '10 at 12:49
    
Most of our testing is done manually. –  Shamim Hafiz Sep 30 '10 at 10:31
add comment

As much as the purist in me doesn't want to accept that there has to be some risk management built into the decision for how rigorously you test or whether you do formalized testing at all. For internal apps, which I suspect are a large % of programming projects, the cost of releasing a bug then quickly patching it after it is noticed can sometimes be outweighed by the cost of a full testing team. Of course it depends on the app and the potential cost of failures.

That said, I don't think risk management planning is the reason for the lack of formalized testing. I think it is more a result of non-technical managers not understanding the value it provides and only see the cost.

share|improve this answer
1  
I hear what you are saying, but it's hard to justify. Studies have shown that the longer a bug goes unnoticed the more it costs, exponentially. The cost of bugs making it to the customer is enormous, and patching them often causes new bugs, if a unit testing framework is not in place (a likely scenario when this kind of "patch and fix" mentality is present). Consequently, judicious use of testing tools and methodologies can be shown to be far less expensive than patch and fix. –  Robert Harvey Sep 15 '10 at 17:38
2  
I'm growing more and more skeptical of that dogma, especially how it is universally applied. I'm sure it is true some of the time, but not all bugs are created equally, nor are all apps. I find it very hard to swallow that a bug in an internal app used by 10 people is significantly more expensive to fix if it is found during unit testing versus in a patch release. It may be exponentially more embarrassing, but not more costly unless you ignore the real costs of the time a tester spends finding the bug. –  JohnFx Sep 15 '10 at 18:00
1  
I also wonder if those statistics weren't more applicable to the massive projects (e.g. Operating Systems) that they were created from and don't translate to the CRUD type apps most of us build for a living. –  JohnFx Sep 15 '10 at 18:02
add comment

My sample is very small to deduce percentages from, but here goes anyway.

One was a fabless chip + firmware company, which did fanatical testing. 24/7 automated tests on tens of installations, each testing tens of units in parallel. Software teams dedicated to developing testing software. Hardware teams dedicated to building test rigs. Compatibility testing against tens of competitors. Heck, they even bought a multi-million dollar chip tester installation to develop and debug some of the tests that the fabs run when the chips leave the foundry.

Another one was a bank. This one is a completely different environment: no product releases, but lots and lots of in-house software to keep running continuously. These guys tested the cr*p out of every single change they made. They had very strict separation of DEV/QA/PROD environments, automated regression testing, mandatory QA testing signed off by end-users before releasing into production, etc.

So yeah, people do do methodical testing. But as you can tell I've never worked at a place that ships your typical GUI software for the typical computer user.

share|improve this answer
add comment

I currently write embedded firmware for a small startup company making wireless medical devices. We are required to do rigorous testing, and have a completely separate quality department headed by someone who reports directly to the CEO. I have never had my code so thoroughly tested before by separate testers (the only time that compares, is when I was working on satellite TV systems about 15 years ago.)

Our test results get submitted to the FDA (so far we have gotten two FDA clearances -- each submittal was around 500 pages long). Both our development and testing methodologies are both subject to periodic auditing.

So it is not only the big companies that do lots of formal testing.

Note -- in my 25+ years of contract programming/consulting, I have also worked for many companies that did virtually no formal testing. Most of them are not around anymore.

share|improve this answer
    
I'm also at a medical device company, and the introduction to GMP (Good Manufacturing Practices, FDA-speak for a controlled design/test process) was quite eye-openIng for me. It's made me a better engineer (and docbook expert unfortunately) –  Bill Gribble Jan 21 '12 at 20:09
add comment

Almost every company I have been did methodical testing. My current company has some basic unit style tests and that is not sufficient. We have had some quality issues due to this. I highly recommend independent thorough testing on any project that is going to be used by anyone besides yourself. The money spent will be well worth it. Applications that don't work don't get used. That goes for internally facing as well as externally facing.

share|improve this answer
    
I've edited my question slightly. Could you provide your estimates of software testing in professional market as well? –  Robert Koritnik Sep 15 '10 at 12:49
    
@Robert: I don't understand your question "estimates of software testing". Are you asking my opinion on how many companies test? My estimate would be maybe 90% or more based on what I've seen with my own eyes. Testing is a common part of professional development. –  Bryan Oakley Jan 20 '12 at 18:49
add comment

In the last twenty or so years of my career through eight or so companies, I've never worked on a project that didn't do testing. The amount of testing differed at each company, but every professional development project I've ever worked on did formal testing. This applies equally to both small and middle sized companies (where "small" means less than 10 employees, and "middle sized" means a couple thousand employees or less).

Some companies didn't have much automated testing, some didn't have much manual testing, but they had at least one or the other.

share|improve this answer
add comment

It depends on the needs of the customer. In a contract situation there are probably acceptance tests. In-house is usually a slapjob with little testing. Consumer stuff is usually highly covered in frequent functionality but rough around the edges.

share|improve this answer
add comment

Short answer: Yes

Long answer:

  1. I do not have a good estimate for the first category (it is probably some distance from zero, but how much?), but my experience actually corroborates with your second estimate. It is hard to give meaningful percentages since the amount and type of testing depends on the kind of application being developed, and available timeframe and as well as the skillset of the developers and how the project is run. In practice the most important hurdle for the developers would be the acceptance test since that is an important milestone for billing purposes. But it is also the time where the unexpected may happen (more requirements) and the developers may be pressured to deliver and get by with whatever adhoc testing that is possible and timely (at this stage), on top of time needed for troubleshooting and overcoming the unexpected.

  2. I have been through a variety of projects with different combination of factors mentioned above:

    • no formal unit tests, only integration tests and mostly adhoc tests

    • very formal ranging from unit tests to detailed test plans involving dedicated QA resources, automated testing (as conducted by the testers with their own set of tools) and code coverage reports. But these are not always meaningful to the developers as much as to the managers

On the individual level I seek to maintain an understanding of my options when it comes to writing proper tests appropriate for the technology I am dealing with, and exercise them at my own discretion. Basically the things that are actually meaningful and beneficial to my work, and not so much cranking out numbers.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.