The need for shrinkers
At the heart of property-based testing are generators and shrinkers. We generate several input values to see if the functionality holds for all of them, but shrink any input value to their simplest representation if it fails. That is what shrinkers do: they try with several iterations to come up with the simplest input value that fails the test. This is needed because figuring out where the implementation is wrong is harder when dealing with complex and/or big values. This will make more sense later on in this post.
FsCheck is a great library for property-based testing in the .NET framework. It already brings several generators and shrinkers for any built-in primitive values (string, int, array) which means that in a lot of cases, there won’t be a need for custom shrinking. I only find that the more complex the input that you want to generate, the more necessary it becomes to think about shrinking.
Tag example
Here is a simple example to get us started. Imagine you have a `Tag` domain model that represents a series of alphanumeric characters which is preceded by a hashtag (i.e. #a1B23C). We can create such a domain model with ease if we use the FPrimitive library:
👀 Note that the tag should not be longer than 20 characters and only contain alphanumeric characters after the required hashtag.
Imagine that two tags are equal, regardless of their character’s case (lower, higher). The current model does not handle this, so let’s see if we can test-drive this missing link by test properties.
Tag generator
Generating Tags can be done by generating a controlled list of alphanumeric characters.
This is not the main point of this post, so we’ll go over it quickly. Just know that this will generate several Tag models for us.
Properties
To verify that the Tag model is missing equalization functionality, we can create a property that checks if any generated Tag is equal to the same Tag but with lower casing:
👀 Note that the previously defined custom Tag generated should be assigned to the FsCheck arbitrary for this to work.
When we run the test, we will have an error similar to this:
Someone with an eye for detail might see that the casing is indeed different here, but it is not immediately clear from the test output what is wrong.
Shrinkers
We can easily fix the test output by writing a custom shrinker. A shrinker simplifies the generated input. In this case, we should create the most simple Tag instance that is still valid according to our domain rules.
FsCheck shrinkers are actually just functions that take in a single generated input value and should return a sequence of many smaller input values. In our case, we can shrink by removing a character from the tag value. This is what I came up with:
👀 Note that we have a failsafe that checks if the Tag is only two characters long, as two characters are the simplest representation of this model.
If we use this shrinker (with Arb.fromGenShrink Gen.tag Shrink.tag) we get the following test error output:
🎉 Now it is far clearer that the character casing is the reason these two Tags are not equal.
Restaurant 'fully booked' example
While the Tag example seems trivial and is not ‘real’, it does provide a great first step in the power of shrinkers. To go a step further, we can look at a real example where we need to generate complex inputs.
I’m recycling the restaurant example from a previous post. The simple restaurant implementation can book reservations two weeks in advance with a maximum capacity of 20 people each day. To check whether the restaurant can reject a reservation when it is ‘fully booked’, I’ve written a property that generates only a set of reservations that fully book the restaurant in the next two weeks.
Imagine there is a fault in the restaurant code, allowing 21 instead of 20 reservations on each day. A failure without shrinking looks like this:
👀 Notice that multiple reservations could happen on the same day. Because of this, it’s hard to detect from this input what went wrong.
I wrote a small shrinker function that takes in a list of current reservations and returns a sequence of many current reservations, each with one less reservation. At the very end, we get the most simple ‘fully booked’ reservation list, which is 14 reservations all with a quantity of 20.
If we now run the property with this specific shrinker, we get something like this:
Because the quantity of each reservation is simplified for the ‘fully booked’ scenario, you can quickly see the reservation on the 23rd with a single quantity, which should be rejected as the restaurant is already fully booked on that date (see last entry).
Conclusion
Shrinkers have been a peculiar topic for me for a long time. This is especially because the built-in shrinking functionality already does a fairly good job. It’s only when you want to tweak the input, test specific scenarios, or are dealing with domain models that have their own ‘simplest version’ that the real power of shrinkers surfaces. It’s clear to me that each specific structured input will need to have a shrinker for a workable test environment.
Shrinkers, just like generators, are closely related to the system you are trying to test. You learn more about your domain if you spend enough time in your test environment. Edge-cases that would otherwise only pop up during bug reports (or until the next malicious attack) now become visible just by running the tests.
Thanks for reading!
Stijn
Subscribe to our RSS feed