Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

This article shows that for Android apps, software developers use between 30 and 50 different devices for compatibility tests.

I tried to find information about tests of applications on "classic" computers, for example PC games or graphics-oriented business applications: how many different hardware configurations are typically tested in this area?

share|improve this question

closed as too broad by gnat, MichaelT, GlenH7, Dynamic, Bart van Ingen Schenau Jul 9 at 16:17

There are either too many possible answers, or good answers would be too long for this format. Please add details to narrow the answer set or to isolate an issue that can be answered in a few paragraphs.If this question can be reworded to fit the rules in the help center, please edit the question.

    
Why 90%? Why major brand/off-the-shelf? Etc... This goal is just as subjective, you've just masked it. And are you testing the hardware, or the software? There are infinite variations in hardware, which is what your 90% of off-the-shelf systems would be trying to test. –  GalacticCowboy Mar 6 '13 at 16:38
    
@GalacticCowboy agreed, deleted. –  mjn Mar 6 '13 at 16:43
add comment

5 Answers

Enough to gain sufficient confidence that your software will work on the desired platforms.

You don't test every device, of course; you decide how much coverage of your target audience you want, and then structure your tests around the devices you think the vast majority of your audience has.

Unless your software requires exotic hardware (CUDA, perhaps, or multi-channel sound cards), it's easier to test software on PC platforms. Your concerns have more to do with "system requirements" (things like processor speed, RAM and hard disk space) than they do with differences between computers, and software frameworks such as .NET are designed to run on almost any device that is PC compatible.

share|improve this answer
    
Confidence sounds subjective - if the goal is 90% coverage of existing major brand "off-the shelve" PC systems, how many different systems would this be approximately? –  mjn Mar 6 '13 at 16:35
1  
Most of the market share articles on the Internet (that you don't have to pay for) show the top five or so vendors, which gets you about 55% market share. I would imagine that testing on these five platforms would be more than enough coverage. It wouldn't surprise me if the vast majority of individual (sole) developers tested their software on two machines, their desktop and their laptop. Your mileage may vary; if you're programming high-performance games, it might be a different story. –  Robert Harvey Mar 6 '13 at 16:42
1  
However, with games, you still have abstraction layers that you can rest on - the hardware is designed around DirectX or OpenGL, so you just program and test against those. You shouldn't have to compare a PNY GTX 660Ti against an XFX Radeon 7850, because they both fully support DX11. –  GalacticCowboy Mar 6 '13 at 16:46
add comment

We make a .Net obfuscation product. Because we are quite conservative and strive for compatibility, we must test on a lot of different platforms.

Some platforms:

  • Windows XP 32bit
  • Windows Vista 32 and 64bit
  • Windows 7 32 and 64bit
  • Windows 8 32 and 64 bit

And then we also have to test for different versions of .Net

  • .Net 3.5
  • .Net 3.5 SP1
  • .Net 4.0
  • .Net 4.5

and then we have to test that the outputted programs must work on an even more broad set of versions.

However, I'd say you should do general QA testing (like after each feature is implemented) on a few different "popular" set of systems and then before a release do much more broad(but not as deep) testing across all platforms you plan to support. Virtual Machines, a server to run them on, and an MSDN subscription is your friend :)

share|improve this answer
    
VMs are great for applications on different OSes. But games or graphics-oriented business applications could need tests on many different hardware configurations - different graphics and sound cards, CPU speeds etc. –  mjn Mar 6 '13 at 16:39
    
Ah, didn't see the game-development tag. Yea, I'm not sure then. @mjn I'd suspect testing on major ATI and Nvidia cards(and Intel graphics if it can run it) is required, and ensure it works on 32bit and 64bit OSs.. I'd do more testing beyond that, but since it requires actual hardware, that can get expensive. If you're trying to save some money, get a loyal number of beta testers. Then you'll see some of the "out in the wild" configurations –  Earlz Mar 6 '13 at 16:44
add comment

We had various Java and Flash-based apps that we tested (so more simple than a game or fully installed app), and we had ~15 testing machines with (iirc) ~20-30 VMs on each for testing.

When you get to that point, you're really testing for regressions - it becomes unfeasible to run every manual test against every hardware/software configuration.

share|improve this answer
add comment

It depends enormously on the guarantees you want to provide to your customers.
If you don't advertise anything, then it should be sufficient to test only on a single platform.
If you state that it works on platforms X, Y and Z, then you better have the test reports to prove it.

And this is where it gets interesting. With PC applications, people tend to be far more accepting of bugs or incompatibilities than with other electronic devices, such as mobile phones. This is reflected in the testing effort and for that reason, major app developers test their app with such a large number of devices, just so they can account for any possible incompatibility.

As an anecdote, I have worked for a company making car-kits (so you can use your mobile phone in a car). Those car-kits get tested against just about every mobile phone that is on the market, because people just don't accept it if their phone does not interoperate flawlessly with their car-kit.

share|improve this answer
add comment

This is where automated testing is the only manageable approach. It has a lot of overhead but in the end it is worth it to get the coverage you are looking for. For the sake of brevity, you develop the automated tests for maximum code coverage. And then set up a schedule for testing which alternates the different architectures you want to test. You can choose the schedule depending on the number of architecture options you want to cover. I normally just do amd64, ARM, and x86. I also include languages(en-us, ja-jp, etc) for international audiences. But I am working on a Desktop app. Hope that helps!

share|improve this answer
add comment

Not the answer you're looking for? Browse other questions tagged or ask your own question.