Although I like unit and integration testing there was always an itch. Yes! and itch I could not really scratch with plain tests. A test in the classic sense needs a set of examples, data which will be used to perform a specific case. It will always be the same data, regardless of how many times a test is started. It would be nice to have a way to define a property of the code under test but change the data on each run!
Let’s look into a simple toy function:
public final class Lists {
public static <A> List<A> reverse(final List<A> original) {
final int size = original.size();
final List<A> reversed = new ArrayList<>(size);
// going from the last element to the first of the original list
for (int i = size - 1; i >= 0; i--) {
// thus adding the elements in the reversed order
reversed.add(original.get(i));
}
return reversed;
}
}
The function as the name suggests, takes a list and returns another with elements in a reverse order. Basic stuff!
What should be tested for this function?
We now can create unit tests that check these scenarios. For each scenario we need to come up with some example data. If in our unit tests we will check a three element list, does it mean the algorithm will work for a five element list? how about 100 element list?
@DisplayName("Lists tests")
public class ListReverseExampleBasedApproachTests {
@Nested
@DisplayName("Reverse on")
public class ReverseTests {
@Nested
@DisplayName("an empty list")
public class EmptyList {
@Test
public void should_return_an_empty_list() {
// given:
final List<String> tested = Collections.emptyList();
// when:
final List<String> result = Lists.reverse(tested);
// then:
assertTrue(result.isEmpty());
}
}
The reverse function above has a generic argument, meaning we can pass different lists to it but our unit tests that are using example based approach need to specify the type. If reversing List
@Nested
@DisplayName("a multiple element list")
public class MultipleElementList {
@Test
public void should_return_list_of_the_same_size() {
// given:
final List<String> tested = Arrays.asList("This", "is", "a", "test");
// when:
final List<String> result = Lists.reverse(tested);
// then:
assertEquals(tested.size(), result.size());
}
@Test
public void should_return_list_with_the_same_elements() {
// given:
final List<String> tested = Arrays.asList("This", "is", "a", "test");
// when:
final List<String> result = Lists.reverse(tested);
// then:
assertTrue(result.containsAll(tested));
assertTrue(tested.containsAll(result));
}
@Test
public void should_have_order_of_elements_reversed() {
// given:
final List<String> tested = Arrays.asList("This", "is", "a", "test");
// when:
final List<String> result = Lists.reverse(tested);
// then:
assertEquals(tested.get(0), result.get(3));
assertEquals(tested.get(1), result.get(2));
assertEquals(tested.get(2), result.get(1));
assertEquals(tested.get(3), result.get(0));
}
}
As you see our doubts may only increase and will never be satisfied by mere example based approach. We need something better!
Let’s start with stating the main property of the function:
Reversing a reversed list should result with a list with the original order of elements
@Test
public void reverse_reversed_equals_original() {
// given:
final List<String> tested = Arrays.asList("This", "is", "a", "test");
// when:
final List<String> result = reverse(reverse(tested));
// then:
assertEquals(tested, result);
}
What do we need?
First let’s write a function that returns an integer from a given range:
private static int randInt(final Random seed, final int min, final int max) {
return seed.nextInt((max - min) + 1) + min;
}
Then let’s create a function that returns a List of random size filled with random data:
private static List<String> getRandomList(final Random seed) {
// Here we limit our list to be of maximum size 100
// this is not very useful as we might be hiding bugs
// what if errors occur for Lists of sizes > 100?
// but for the simplicity of the example we will leave
// it as such
final int randomSize = randInt(seed, 0, 100);
final List<String> randomizedList = new ArrayList<>(randomSize);
for (int i = 0; i < randomSize; i++) {
randomizedList.add(String.valueOf(randInt(seed, 0, 100)));
}
return randomizedList;
}
Now we need to define the property in some convenient way. Let’s define and interface that will describe our property:
public interface Property<A> {
boolean check(A value);
}
Now let’s use this newly created interface to describe our reverse property.
public static <A> Property<List<A>> reverseOfReversedEqualsOriginal() {
return original -> reverse(reverse(original)).equals(original);
}
Our property is generic, why not data generation? Let’s have an interface for it similar to the property. Let’s feed it with a random seed, so we can control the data generation.
public interface Generator<A> {
A generate(Random seed);
}
Let’s prepare a generator for our example using our getRandomList function that we’ve created.
public Generator<List<String>> randomStringListsGenerator() {
return seed -> {
final int randomSize = randInt(seed, 0, 100);
final List<String> randomizedList = new ArrayList<>(randomSize);
for (int i = 0; i < randomSize; i++) {
randomizedList.add(String.valueOf(randInt(seed, 0, 100)));
}
return randomizedList;
};
}
Let’s use all the interfaces to creates a mini framework for testing properties:
private static <A> void quickCheck(final long seedValue,
final int numberOfTries,
final Property<A> property,
final Generator<A> generator) {
final Random seed = new Random(seedValue);
for (int i = 0; i < numberOfTries; i++) {
// Let's generate the test example
final A tested = generator.generate(seed);
// and try it against our property
final boolean result = property.check(tested);
// if the property finds a problem with our algorithm
// let's log information about the seed and example
if (!result) {
final StringBuilder builder =
new StringBuilder()
.append("Property test failed")
.append("\nSeed value: ")
.append(seedValue)
.append("\nExample data: ")
.append(tested);
throw new AssertionError(builder);
}
}
}
Now we need to use all of the created elements to get a test.
@Test
public void custom_reverse_property_test() {
// Let's try 1000 times to find a counterexample
final int numberOfTries = 1000;
// this value will help us with reproducing the issue
// if any is found
final long seedValue = new Random().nextLong();
// Let's prepare our property, it's nice that it does not
// rely on the type but is fully generic
final Property<List<String>> reverseProperty =
reverseOfReversedEqualsOriginal();
final Generator<List<String>> generator =
randomStringListsGenerator();
quickCheck(seedValue, numberOfTries, reverseProperty, generator);
}
How can we know if this little framework works correctly? Let’s introduce a property that will immediately fail.
private static <A> Property<List<A>> reverseWrongProperty() {
// this is obviously wrong as reversed list cannot be equal to the original
return original -> reverse(original).equals(original);
}
Using this property in the test fails the test.
@Test
public void custom_reverse_property_fail_test() {
final int numberOfTries = 1000;
final Property<List<String>> reverseProperty =
reverseWrongProperty();
final Generator<List<String>> generator =
randomStringListsGenerator();
final long seedValue = new Random().nextLong();
quickCheck(seedValue, numberOfTries, reverseProperty, generator);
}
Running this test gives us a failure:
java.lang.AssertionError: Property test failed
Seed value: -2569510089704470893
Example data: [40, 15, 20, 30, 36, 35, 55, 99, 89, 93, 67, 27, 31, 95, 26, 6, 84, 23, 92]
Let’s use a seed that was used in the failing example and run a test with it to reproduce the results.
@Test
public void custom_reverse_property_from_seed() {
final int numberOfTries = 1000;
final Property<List<String>> reverseProperty =
reverseWrongProperty();
final Generator<List<String>> generator =
randomStringListsGenerator();
/* this time we want to repeat the failing case: */
final long seedValue = -2569510089704470893L;
quickCheck(seedValue, numberOfTries, reverseProperty, generator);
}
What else do we need?
There are a few things that would a nice thing to have:
In the implementation above although we can repeat the tests with the same random number generator thanks to the seed value, we cannot just repeat the one failing example that our “framework” has found. To be completely honest we can because we can just copy the example data that was printed but we would have to write another tests where we could pass this example data to, there is no easy and quick way to do it.
In the example seed -2569510089704470893 the list that caused the property to fail was rather small but it could be much much shorter instead. The wrong property fails for a list as short as [1, 2]. Our “framework” does not have a capability of shrinking the failing example unfortunately.