The need for property-based testing
Example-based testing is great for humans because we think in examples. This does not mean that it is enough to simply have a stable and secure system that handles all edge cases. Property-based testing has been around for quite some time, but it still has not found its way into current project development. While this is great for unit testing, it could be even better for integration testing.
Just like we need unit testing to drive the design and refactor code, we need integration tests as regression for change on a bigger scale. The shrinking capabilities that come by default in any property-based testing system are a great way to catch interaction bugs that the unit tests failed to discover.
The current state of FsCheck and Expecto
There is one catch with the current state of FsCheck. Because integration tests interact with external resources, they have a lot of asynchronous calls. This is a problem, as the current version of FsCheck still has no (stable) version of this support. Expecto recently released a -fscheck3 version of their testing tool, which makes this preview version of FsCheck available to us.
The possible future of property-based integration tests
The property-based testing in integration tests is different from regular unit tests because it affects the environment. It is therefore a good approach to think about the maximum number of tests we should run for each scenario. While it is valuable to have 100+ unit tests all working on finding edge cases, we might not have that number of resources or time to do it for each integration test run. If we place the maximum test count to one, it will run our integration tests like they are regular tests, except when encountering a failure when it will attempt to shrink the inputs and try again. Therefore, only failures will result in multiple runs of the same test.
Partial property-based tests
Because the built-in testProperty functions in Expecto describe a full test, we can’t use them directly in the partial test list of the testFixtureAsync function. Fortunately, Expecto is flexible enough for us to write our own test property as a partial test.
Here is the property creation code that takes in an arbitrary:
👀 Notice that we wrap this in an asynchronous operation so that we can use it as a partial test. The alternative would be to add it to every partial test yourself.
With this in place, we can set up our test fixture and contact our system like before. This example recycles [a shrinking model from a previous post] that uses a custom shrinker to simplify a `Tag` model (ex. #a1bc23); and a composite test fixture from another post.
If you alter anything in the test, it will try again with a simpler tag.
Conclusion
In this post, we looked at how we can use property-based testing inside integration tests. Integration tests are often overlooked, and property-based testing approaches are definitely not being used frequently in these scenarios. However, they can bring much to the table when it comes to automated defect localization. Unit tests will discover edge cases at a lower level, but integration tests will guard edge cases during regression with a broader scope.
FsCheck is not yet ready to fully support this as a stable version, but we are getting there. This has been especially true since Expecto has started to support this in its own preview package.
Consider this the next time you write new integration tests, or are bug-hunting an existing system.
Thanks for reading,
Stijn
Subscribe to our RSS feed