In this post, we cover test fixtures written with the F# Expecto library, because the library is one of the most flexible .NET libraries for writing tests. However, any ideas or patterns are/should be easily transferable to another framework or language.
Single test fixtures
Some integration tests are so simple that they only require a single test fixture. If this is the case, we can create them within the test body itself. In this case, proving enough test context (adding to the test body) surpasses the need to increase test ignorance (removing from the test body).
Here is a simple example of a test in need of a single environment variable.
👀 Notice that even though the test fixture is part of the actual test body, the teardown is ‘hidden’ within the dispose function of the test fixture. This way of working is no different from how a C# environment will set up its test fixtures. An alternative would be to place the teardown in a finally block, but that would only increase duplication.
Multiple test fixtures with setup function
Integration tests do not stand alone. In many cases, you want to test a series of scenarios that verify different kinds of inputs with different kinds of environments. Because of this, test fixtures quickly become a refactoring opportunity.
F# Expecto uses setup functions (with signature: ('a -> Async) -> Async) to control how the test fixture interacts with the test. It allows for a value to be passed to the test so that it can interact with the test fixture. An interesting point to mention is that the value that the test interacts with does not have to be the test fixture itself. A good example is a temporary API. The test fixture could be to set up and teardown the API application, but the test interacts with the API through an HTTP client.
Using this approach is useful when you do not have a lot of test fixtures and/or want full control over the order of which each test sets up the fixture in relationship with when the test is run.
👀 Notice here, though, that because you have full control, you are also responsible for any exceptions (assertion or otherwise) that were thrown during the test. You will need to catch this and make sure that you tear down the test fixture before you rethrow the exception. This is could easily be forgotten which makes this approach besides super powerful, also rather dangerous if it is not handled with care.
Combining test fixtures
Another problem that I see with only using the previous approach, is that an exception with the test fixture setup, does not correspond with a teardown. This is particularly a problem when the test has multiple test fixtures. A failure in one of the fixture’s setup or teardown should not block any other test fixtures in any way. It is always possible that a test fixture fails for some reason like latency or network unavailability, but it is the job of the developer to make sure that this does not affect any other setup or teardown.
In the past, I made sure that each test fixture was automatically torn down when something happened in its setup, but I wanted a more reusable and flexible approach that took the burden from the test writer.
I came up with the following type of abstract test fixtures.
💡 Another tip when working with multiple test fixtures is to use logging for when a test fixture is set up and when it is torn down. This helps with defect localization and clearly shows what fixture is created before another.
👀 Note that the exception thrown in the test itself (partial), is also rethrown at a later stage – after the teardown. Expecto is smart enough to distinguish assertion exceptions from other ones to let the test either fail or error.
The next question is how to add additional test fixtures without compromising a previous one. The two functions below demonstrate how new test fixtures can be added. One function does not require any additional value to be passed to the test body, while the other allows for an expanded value to be passed.
For instance, in the first case, consider a scenario where we set a temporary environment variable. Here, our main focus is on having the test fixture available, without needing to consider any interaction points, such as an HTTP client for our temporary API.
Once you have created a reliable Fixture from your test fixture, you can combine them with ease, and call apply to transform the collected test fixtures into an Expecto setup function.
💡 Try to explicitly let one of the test fixtures or the test itself fail and notice that it does not affect how the test fixtures are set up or tear down.
The interaction value for the test is now a tuple, but with the help of isomorphic lenses, we could map this value to anything we want.
Conclusion
The challenge with integration testing often arises from an emphasis solely on the interaction between components and the resulting value to be asserted. That is fine during test development, but we should not stop there. To create reliable, repeatable integration tests, we should spend a fair amount of time on the fixtures used during the tests.
One of the tricks is to think in ‘do/undo’ patterns. For every action taken in the test, create an undo action so that after the test is finished we end up with the same environment as before.
This post went over some of the pitfalls of using multiple test fixtures in a single integration test, and how we can make sure that any ‘do’ or ‘undo’ of a test fixture does not affect the ‘do’ and ‘undo’ of another fixture.
Thanks for reading!
Stijn
Subscribe to our RSS feed