Unit Testing with functions that return random results

Question

I don\'t think that this is specific to a language or framework, but I am using xUnit.net and C#.

I have a function that returns a random date in a certain range. I

Answer 1

Methods that do not exhibit a deterministic behavior cannot be properly unit-tested,as the results will differ from one execution to another. One way to get around this is to seed the random number generator with a fixed value for the unit test. You can also extract the randomness of the date generation class (and thus applying the Single Responsibility Principle), and inject known values for the unit-tests.

Answer 2

I don't think Unit testing is meant for this. You can use Unit testing for functions that return a stochastic value, but use a fixed seed, in which case in a way they are not stochastic, so to speak, for random seed, I dont think Unit testing is what you want, for example for RNGs what you mean to have is a system test, in which you run the RNG many many times and look at the distribution or moments of it.

Answer 3

I would recommend overriding the random function. I am unit testing in PHP so I write this code:

// If we are unit testing, then...
if (defined('UNIT_TESTING') && UNIT_TESTING)
{
   // ...make our my_rand() function deterministic to aid testing.
   function my_rand($min, $max)
   {
      return $GLOBALS['random_table'][$min][$max];
   }
}
else
{
   // ...else make our my_rand() function truly random.
   function my_rand($min = 0, $max = PHP_INT_MAX)
   {
      if ($max === PHP_INT_MAX)
      {
         $max = getrandmax();
      }
      return rand($min, $max);
   }
}

I then set the random_table as I require it per test.

Testing the true randomness of a random function is a separate test altogether. I would avoid testing the randomness in unit tests and would instead do separate tests and google the true randomness of the random function in the programming language you are using. Non-deterministic tests (if any at all) should be left out of unit tests. Maybe have a separate suite for those tests, that requires human input or much longer running times to minimise the chances of a fail that is really a pass.

Answer 4

I think there are three different aspects of this problem that you test.

The first one: is my algorithm the right one? That is, given a properly-functioning random-number generator, will it produce dates that are randomly distributed across the range?

The second one: does the algorithm handle edge cases properly? That is, when the random number generator produces the highest or lowest allowable values, does anything break?

The third one: is my implementation of the algorithm working? That is, given a known list of pseudo-random inputs, is it producing the expected list of pseudo-random dates?

The first two things aren't something I'd build into the unit-testing suite. They're something I'd prove out while designing the system. I'd probably do this by writing a test harness that generated a zillion dates and performed a chi-square test, as daniel.rikowski suggested. I'd also make sure this test harness didn't terminate until it handled both of the edge cases (assuming that my range of random numbers is small enough that I can get away with this). And I'd document this, so that anyone coming along and trying to improve the algorithm would know that that's a breaking change.

The last one is something I'd make a unit test for. I need to know that nothing has crept into the code that breaks its implementation of this algorithm. The first sign I'll get when that happens is that the test will fail. Then I'll go back to the code and find out that someone else thought that they were fixing something and broke it instead. If someone did fix the algorithm, it'd be on them to fix this test too.

Answer 5

Depending on how your function creates the random date, you may also want to check for illegal dates: impossible leap years, or the 31st day of a 30-day month.

Answer 6

Normaly I use exactly your suggested approach: Control the Random generator. Initialize it for test with a default seed (or replace him by a proxy returning numbers which fit my testcases), so I have deterministic/testable behaviour.