Disposable fixtures
One of the first things to realize when writing integration tests is that the system you want to test is not as simple to set up as other lower-level kinds of tests (such as unit tests). Setting up and tearing down your system is crucial and the cornerstone of any good integration test suite, providing stability and correctness throughout. It also allows the test writer to use the system under test as a first-citizen in the test code, giving the writer much more space to come up with an integration test.
C# IDisposable interfaces are a great way to express your system (called ‘fixture’ from a test perspective). Expecto does even better, relying not on the object interfaces, but using the best tool available in F#: functions.
The Expecto function testFixture takes in a factory function for your fixture and a series of ‘partial tests’. These tests are not ‘complete’ tests, as they are not created with the built-in test functions. Here is an example:
This shows that the factory function withMemoryStream actually takes in the entire test body (f). With a simple function injection, we have full control over when and where the fixture(s) are created and how they should be disposed of afterwards. The test body that uses the fixture is a tuple list of the test name and the test body functions that use the fixture. This is a simple example of how test fixtures can be used. There could, however, be a problem with using this in integration tests, as fixtures in integration tests are usually set up/torn down asynchronously. Luckily, Expecto is extensible enough to implement an async version in no time:
This small plumbing function opens up a whole set of possible integration tests. Imagine you want to verify a certain function composition in a Giraffe web API application. Each integration test should have its own Giraffe test fixture and a HTTP client to interact with the fixture. Setting up such an application requires asynchronous execution.
Here is an example of what such a test fixture function could look like:
The withGiraffe test fixture factory function will set up a web API. Instead of sending a memory stream like in the previous example, it will then send a HttpClient to the test function (f) that is initialized with the URL where the Giraffe application is running. Note that this test fixture does not include exception handling or logging — that is for later.
This factory function will become clear in this example:
Note that the passed client already is initialized with a base URL, so we only need to concern ourselves with the route we want to test. This is an excellent example of test ignorance: the test should only know and contain enough to run the test — nothing more or less.
Sequenced running
Integration test suites are typically run sequentially. If remote connections are made to a single Azure resource, for example, parallel runs could cause unstable test results. The test fixture could also take a while to set up, or the costs of setting up/tearing down could be too high. We re-created the Giraffe application for each integration test in the previous example, but this many not always be possible for your test fixture.
First, let’s look at how we can make our test suite run sequentially. Fortunately, Expecto supports this easily by passing Sequenced alongside the CLI arguments in the test startup code:
The second part is how we can make a singleton of a test fixture instead of re-creating it. Other test frameworks need special concepts like ‘test collections’ and ‘class fixtures’ to fit the test fixture into the tests. Here again is where Expecto shines. The tests are just functions and running them is just a function that is very compossible and extensible. Imagine a scenario where you would want a subset of tests to have a singleton test fixture. The first thing that comes to mind is that we could pass this singleton test fixture to those tests as a parameter. In other test frameworks, you first have to define your fixture in the language of your framework, while in Expecto you just write your fixture.
The following example shows how an API endpoint is called before the tests and how the result of this API call is passed as a ‘singleton test fixture’ to the tests that need it:
When I experimented and saw that I could write this purely from functions and composition, I realized that Expecto really is the best testing tool I have come across in a long time. With minimal effort, the code makes use of Expecto’s functionality to read all the tests from the assembly, combining them with any other test you’re setting up. In this case, this is any additional test needing a singleton test fixture.
Logging
Integration tests usually combine several systems together in a single test, hence the name ‘integration’. Integrating systems in a test environment and on test build servers is not always an obvious process, and without logging it can become quite a hurdle to pinpoint the exact problem of why a certain test fails. When setting up one or more test fixtures it might be a good idea to add some logging, to fully understand the order in which test fixtures are set up/torn down. It can also provide additional context when an integration test fails, pointing the test reader towards the inner cause of the test failure. This can either be the test fixture itself, or some other component. Logging is therefore highly recommended for integration test suites.
Expecto has logging built-in and provides a way to write custom log messages during the execution of the test. In our previous example, we used a Giraffe Web App as a test fixture and interacted with this fixture in our tests. Some additional logging could help future test readers to see what the HTTP request/response is of the client:
The GET -> /route log messages are an example how the interaction between systems can be logged while testing the systems. The logging is based on the hierarchical logging system Logary, so arguments can be passed easily. Note that there is a small duplication here. Luckily, Expecto is extensible enough to refactor and use the test framework in your own way. Nothing can stop you from embedding the logging within the provided HTTP client. Have fun experimenting!
💡 Expecto can also be configured to write more diagnostic trace information. This automatically includes the start/stop of each test. This is great for integration tests, as it clearly separates one test from another. If a single test uses multiple test fixtures and/or interacts with multiple systems, then this feature is critical.
Conclusion
Integration testing requires a bit more attention than regular (example-based) unit or property-based tests. Integration tests require you to interact with systems which not always in your control. It is the the tester’s job to set up such a test fixture system that can mimic a real system while still providing a stable test outcome. If this test system also includes logging, than the test suite is mature enough to be expanded with more integration tests.
Test maintenance is extremely important as like all software: tests change. Expecto is a great testing library for integration testing as it is built upon simple F# functions and still providing enough extension points to adapt it to a fully working integration test suite. To learn more about the extensibility of Expecto, see my other post: The Beautiful F# Expecto Paradigm: Think in Tests, Not Frameworks.
Thanks for reading!
Stijn
Subscribe to our RSS feed