Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

I'm currently writing a load test for a cache which should test how a cache will react to persistent requests. A colleague and I had differing opinions on how this load testing should be performed.

I believe that a load test should be as random as possible. It should model real-world load as much as possible, and the way towards that is randomality. So I have created this random test as follows:

  • Test data is held in spreadsheets and is loaded into TestRunner objects at startup
    • The test data is not random
  • The load test will run 10 random TestRunners in individual Threads at the same time
  • The object returned by the cache will be tested to make sure it is sensible, it is not thoroughly tested
    • Any tests that fail will be output at the end and each test has a unique ID to easily find failed tests
  • At random intervals, the cache will be cleared to model the real-world requirement of the cache being cleared at any time.
  • The load test will run for a configurable amount of time

My colleague's idea of what a load test should do is:

  • Test data is held in spreadsheets and is loaded into TestRunner objects at startup
  • All TestRunner objects are run in a sequential manner
  • Each time the load test is run, it will run the TestRunner objects in the same order

Which methodology do you feel would produce the most reliable load test?

I personally think the random test will produce a more reliable load test as it will model real-world usage. It is not known what order requests will come in when this is released to production, so it should be tested with that unknown element taken into account. However, running all tests in the same sequence each time will make any failures reproducable, which is important during testing.

share|improve this question
add comment

2 Answers 2

up vote 2 down vote accepted

Do you have a way to reset the data once the test is run (if this is even necessary)?

If so, what about running the non-random test first - to look for failures when run that way (and possible anomalies caused by the data itself)? Then, after resetting (if necessary), run the random tests to reflect the real world load.

share|improve this answer
    
Best of both worlds. Could be the way to go –  Richard Sep 29 '10 at 16:18
add comment

Multi-thread tests are a pain. It might simulate a real-world scenario but will only tell you that 'There is a problem'. You will not be able to reproduce the test run and, most of the times, this means that you will not be able to find the bug.

The random execution model will reveal strange bugs that don't arise on normal usage - otherwise you would know about them just by 'playing' with the product. That's why the chance of reproducing the same bug with another random run is very small (unless you have infinite time :) ).

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.