Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

I've got a simple authentication solution, where a user can log in with either email or username.

Since I want to show the little feature in the acceptance tests but don't want to pollute them, I thought I'll introduce some randomness. Do you think this is bad or good?

step "Fill in my credentials" do
    fill_in :user_login, with: [@user.email, @user.name].sample
    fill_in :user_password, with: @user.password
end
share|improve this question
    
en.wikipedia.org/wiki/Fuzz_testing –  jk. Apr 30 '13 at 7:11

4 Answers 4

up vote 2 down vote accepted

I would avoid randomness in your automated tests. You want all of your automated tests to be predictable. Trying random inputs is certainly not predictable. That way if one of your tests fail, you will be able to more quickly determine why.

Instead, spend some time thinking about cases that might deserve testing and test those explicitly. Thinking about your code like this is more effective than using random tests and hoping that they catch cases you didn't think of.

share|improve this answer
2  
If add(x, y) = x + y and we test add() by asserting add(x, y) == add(y, x), then we can use random x and y just fine. Random inputs don't have to be unpredictable, but you do have to make sure and show (or log) the test case if it's random and something breaks. QuickCheck is a testing library that was written precisely to help with testing using random values. –  Michael Shaw Mar 30 '13 at 2:14
1  
If your test is only dealing with a small number of randomly generated values per test, recording every random value is sufficient to allow reproduction of the test results. If this is unwieldy saving the seed is often sufficient to allow the test to be reproduced. Of course, this sort of testing can become difficult if working with randomized algorithms, unless they accept a random number generator as a parameter. All that being said, randomness is inappropriate unless you are testing so many values that you have no choice. –  Brian Mar 30 '13 at 15:36

Let's suppose that it works for email, but not name.

Now your acceptance test will fail half the time. Just re-running the test will potentially change the result. That'll make it harder to notice errors and track down the problem with it occurs. It'll be way easier if you just write two tests, one for each login method.

share|improve this answer

I won't state randomness is a bad idea.

[Sav07] tested his algorithm implementation for binary search by writing a group of test which included some "theories" (hypotheses) about how the search algorithm should perform.

He then ran the "thorough" group of tests with a lot of random fixtures.

In Ruby:

require "test/unit"
require "binary_search" # binary_search gem: 
                        # <https://github.com/tyler/binary_search>

class BinarySearchTestTheories < Test::Unit::TestCase
    def before_all
       @rand = Random.new()
    end

    def test_theories
        experiments = 1000
        max_value = 1073741824 # Fixnum max value

        while (experiments-- > 0) do
            test_array = generateRandomSortedArray

            target_must_exist = [true, false].sample
            if target_must_exist
                target = test_array[@rand.rand(test_array.length)]
            elsif
                target = @rand.integer(max_value)
            end

            return_value = test_array.binary_search { |v| target <=> v }

            assert_theory1(test_array, target, return_value)
            assert_theory2(test_array, target, return_value)
            assert_theory3(test_array, target, return_value)
            assert_theory4(test_array, target, return_value)
        end
    end

    # Theories
    def assert_theory1(arr, target, expectation) ... end
    def assert_theory2(arr, target, expectation) ... end
    def assert_theory3(arr, target, expectation) ... end
    def assert_theory4(arr, target, expectation) ... end

    # Helper
    def does_array_contain_target?(arr, target) ... end
    def target_position(arr, target) ... end
end

I'm not certain his "test theories" qualify as acceptance tests---but they clearly are beyond unit or "boundary" tests.

My conclusion: random acceptance tests are useful for predictable algorithms. Sampling email and user name fixtures seems trivial enough.

[Sav07] Alberto Savoia: "Beautiful Tests", in: Andy Oram and Greg Wilson (eds.): Beautiful Code, Beijing: O'Reilly, 2007.

See the full article at: http://cdn.ttgtmedia.com/searchSoftwareQuality/downloads/BeautifulTests.pdf

share|improve this answer
    
Somehow the answers start to look like my question ;) There doesnt seem to be a general solution to this kind of answer. –  wintersolutions Apr 30 '13 at 11:03

I sometimes write tests that depend on random occurrences for whether they succeed or not. I've learned that there are a few caveats, as described by other answers:

  • You do not ideally want a test that only fails sometimes. If you break your code, you want to find out when you integrate your change, so you know which change to undo.
  • If there are only a small number of inputs that cause failure, you need to know which inputs they are in order to fix the problem.

The former problem can be mostly solved, as long as you are willing to accept a slow test suite, by running the test many times. The latter problem requires you to log any random factors that may cause the failure.

These techniques also apply to situations where the randomness is unavoidable, e.g. you are testing for potential timing-related conditions in a multithreaded program.

As long as you are careful, it can be done. But the best thing to do is to isolate potential failure conditions and write a predictable test that always fails for them prior to fixing the problem, IMHO.

share|improve this answer
1  
Well, once you catch a failure via randomness, that failure should be turned into its own test, either with nonrandom parameters or a hard-coded seed. In the OP's case, may as well just start by doing that, so no randomness. –  Brian Mar 30 '13 at 15:37

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.