Property-Based Testing with C#

Introduction

Property-Based Testing is about generalizing the input so we can make statements about the output; without specifying exactly what the input or output should be, only should look like.

I’m not going to give you the full-introduction because there are already so much good resources about this topic, (also in different languages).

• An Introduction to Property-Based Testing
• Property-Based Testing for Better Code
• FsCheck Learning Resources
• …

But what I will do, is give you an introduction to FsCheck in a C# environment. FsCheck is written in F# but has a C#-friendly API. I’m going to use the FsCheck.Xunit package for this blog post.

FsCheck

For a full-introduction of FsCheck itself, I highly recommend the documentation of FsCheck; with a good explanation about the framework. Although they give a good idea of how the framework is built, I find it hard to find examples of how it can be used concretely; especially if you’re using the xUnit variant.

Several blog posts are using F# to make properties with FsCheck, but with C# the posts are rather rare…

Fact to Property

Let’s start from the xUnit example they present on their documentation:

If you know xUnit, you know that ‘Fact’ is the way xUnit marks methods as test-methods and the static class ‘Assert’ is used to assert on the result.

Now, I’ll give you the same example but written with some FsCheck properties:

What are the differences?

• The ‘Fact’ attribute is changed to ‘Property’ attribute
• Return type is ‘Property’ instead of ‘void’
• ‘Assert’ class isn’t used, but the condition is returned and transformed by the ‘ToProperty()’ call to a ‘Property’
• The inputs of the method under test aren’t hard-coded anymore

This last difference is probably the most important one.
I highly recommend you read the resources if you haven’t heard about PDT because I won’t list all the benefits of Property-Based Testing. I hope that you see that by using this approach, I can’t maliciously fake the actual implementation anymore, while in the first example I could have done this.

We’ve added two parameters to the test method that FsCheck will fill-in for us with some random values. This will contain negative, zero and positive values all in the range of an Int32 data type. All valid integers so to say. FsCheck will, by default, run 100 tests with random values for the inputs of the test.

FsCheck has several extension methods on boolean values, like the one above. Let’s look at some more.

Conditional & Lazy Properties

Sometimes, we want to restrict the input to make sure you’re always end up with the same output. A simple example is the mathematical division. You can’t divide by zero, so to have the same result we must make sure that the given input isn’t below zero.

What’s different?

• We added the ‘When()’ call to specify that we can’t divide by zero (this makes sure we don’t have to call ‘ToProperty()’ again)
• We extracted the method, which we wanted to test, in its own delegate. Note that FsCheck has extension methods on any delegate that returns a boolean.

That is a good example of the Conditional Property; but why do we need to extract the call to ‘Divide’? Because otherwise FsCheck will evaluate this immediately (even with ‘y’ being zero) which would result in a ‘DivideByZeroException’ and FsCheck will treat any exception being thrown as a test failure. That’s why.

By extracting this, we’re telling FsCheck that we’re only interested in the results IF the condition holds. In our case: ‘y’ must be zero.
That’s convenient!

With this simple example, we’ve shown how we express conditions in our properties to make sure we’re always in a given set of inputs, and shown how we can create Lazy Properties which are useful to only evaluate the test if the condition we’ve set holds. This also can be useful if the actual test takes some time and we don’t want to lose time while evaluating a test which result isn’t of interest for us.

Exception Properties

In functional programming, I’ll try not to use any exceptions in my code; but in imperative languages this is the way to express something went wrong. We also write tests that trigger those exceptions that we throw by giving invalid inputs.

The xUnit package also has some methods on the Assert class called “Throws”, “ThrowsAny”, … How can we express this in FsCheck?

The documentation says that this isn’t actually supported in C# (you can see it at the lower-case method); but writing it this way works.

Observed Properties

The best possible alternative for this feature, is the ‘usermessage’ you can send with the ‘Assert’ class in the xUnit package. We send a string with the assert so we can later see which assertion has failed.

FsCheck takes this a step further.

Trival Properties

FsCheck has a way to count the cases for which a condition is met. In our previous example, can we count how many generated values are negative values?

In our test output, we can see that the positive and negative values are almost split in half:

Ok, passed 100 tests (47% trivial).

Try to run them again and see how this test output change.

Classified Properties

Sometimes, we want more than one condition to check about our input and maybe add some custom message for each category of input. According to me this is the closest thing to the ‘Assert’’s ‘usermessage’.

FsCheck has a way to express this by classifying properties.

In our output, we’re now seeing:

Ok, passed 100 tests.
63% Smaller than ‘1000’.
37% Smaller than ‘1000’, Bigger than ’10’.

See, how the categories can also be combined and are shown to the user in a friendly way.

Collected Properties

We’ve seen some examples how we can express some categories for our test inputs by specifying conditions on them and giving them a name. But sometimes we’re just interested in the actual input value itself and how it changes during the test run.

This will result in this test output:

Ok, passed 100 tests.
8% “Values together: 0”.
5% “Values together: 8”.
5% “Values together: 1”.
4% “Values together: 3”.
4% “Values together: -12”.
3% “Values together: 38”.
3% “Values together: 2”.
3% “Values together: -4”.
3% “Values together: -14”.
3% “Values together: -1”.
2% “Values together: 9”.
2% “Values together: 7”.
2% “Values together: 5”.
2% “Values together: 32”.
2% “Values together: 21”.
…

This way, we can clearly see how the test inputs changes over time.

Combined Observation Properties

As a final observation property, we can also combine several of the previous observed properties into one property that combines all the results:

This will result in this test output:

Ok, passed 100 tests.
7% “Values together: 3”, Smaller than ‘1000’.
5% “Values together: 2”, Smaller than ‘1000’.
5% “Values together: 0”, Smaller than ‘1000’.
4% “Values together: 13”, Smaller than ‘1000’.
4% “Values together: 1”, Smaller than ‘1000’.
3% “Values together: -8”, Smaller than ‘1000’.
3% “Values together: -4”, Smaller than ‘1000’.
3% “Values together: -15”, Smaller than ‘1000’.
3% “Values together: -12”, Smaller than ‘1000’.
2% “Values together: 9”, Smaller than ‘1000’.
2% “Values together: 8”, Smaller than ‘1000’.
2% “Values together: 7”, Smaller than ‘1000’.
2% “Values together: 27”, Smaller than ‘1000’, Bigger than ’10’.
2% “Values together: 22”, Smaller than ‘1000’.
2% “Values together: 1”, Smaller than ‘1000’, Bigger than ’10’.
2% “Values together: -56”, Smaller than ‘1000’.
2% “Values together: -3”, Smaller than ‘1000’.
2% “Values together: -11”, Smaller than ‘1000’.
2% “Values together: -10”, Smaller than ‘1000’.
…

Combined Properties

The previous properties all had the same thing in common: they are  testing a single condition. What if we want to test multiple conditions? And how do we distinguish each property from one another?

Sometimes, we want to test two conditions. Combining them in a ‘AND’ expression. FsCheck also has this as extension method:

We can also add a ‘Label’ which is the same as the ‘usermessage’ in the ‘Assert’ class in the xUnit package: it pops up when the condition isn’t met.

By using this approach, we always know which property has failed.

Note that I now use a ‘NonNegative’ type instead of a regular int. This is also part of the FsCheck framework and allows me to make sure I always get a positive integer without specifying it in a Conditional Property. As you have seen, FsCheck will try any value that acts as a valid type; so, if I would add a condition to my property stating that I want a positive integer, I’ll get roughly the half of the test runs. This way, by using the ‘NonNegative’ I’m sure that I still get my 100 test runs without skewing the input so much I get merely any test runs.

Of course, we can also combine our properties in an ‘OR’ expression with the extension method ‘Or()’.

 Types

We’ve already seen an example with the previous properties where I used the ‘NonNegative’ type. FsCheck has several types that you can use to stricken our input. Some interesting ones:

• PositiveInt represent an integer bigger than zero
• NonZeroInt represent an integer which isn’t zero
• NonNegativeInt represent an integer which isn’t below zero
• IntWithMinMax represent an integer that can contain the int.Min and int.Max values
• NonNull<T> wraps a type to prevent null being generated
• NonEmptyArray<T> represent an array which isn’t empty
• NonEmptySet<T> represent a set which isn’t empty
• NonEmptyString represent a string which isn’t empty
• StringNoNulls represent a string without null characters (‘00’)
• NormalFloat represent a float which isn’t infinite or NaN
• Interval represent an integer interval
• IPv4Address represents an IPv4 Address
• IPv6Address represents an IPv6 Address
• …

And many more… Don’t hesitate to come up with your own generic types that you can contribute to FsCheck!

We can also generate our own domain models with invalid and valid ones and use FsCheck to generate them for use; but that would lead us to another topic about generators.

Conclusion

In this post, we’ve seen how Property-Based Testing isn’t just a functional concept but an idea we can use in any language. FsCheck is inspired from the QuickCheck variant in Haskell, there’s also the ScalaCheck in Scala, JavaQuickCheck for Java, ClojureCheck for Clojure, JSVerify for JavaScript, … and so many more.

I’m not saying that you can abandon all your Example-Based Tests. Just like I stated in the beginning of this post: people think in examples. So, I think the combination of Example-Based Tests and Property-Based Tests is the sweet spot. By examples we can show the next person concrete ways of how to use your API and with properties you ensure that the implementation is the right one tested with any boundary conditions.

Thanks for reading!