Recently, we updated the Invictus Framework testing strategy by introducing continuous deployment and continuous integration testing. All deployments happened manually until this point and testing was not part of the process. Releasing a package to an environment corresponds with deploying the Azure Functions packages to a specific Azure resource group. Now, we have introduced a new test release environment where these release packages can be deployed and against which tests can be run. This post will go over some interesting points that occurred during the migration process of this updated testing strategy.
Merge self-contained and remote integration testing
One of the first goals we wanted to achieve during this exercise was the ability to run the integration tests both locally on your development machine and remotely on a build server. This is a big selling point for people who are hesitant to write tests, as well as people wanting fast test feedback. Providing the ability to run integration tests (and in fact any environment-related tests) locally lowers the threshold for people to write tests, as they do not have to deploy their changes to get a test report. This is in line with the ‘fast feedback loop’ and ‘test-driven’ mindsets that will bring more quality to any project.
We accomplished this by running the Azure Functions applications self-contained when the tests are run on a local development machine, and using the deployed applications when the tests are run remotely. The dependent Azure resources are available in the Dev release environment for local development. Remote tests will by default use the resources in the Test or QA environment.
🔗 There is an entire post dedicated to this if you want to learn more about increasing your motivation to write tests in projects. The trick behind this mindset is to completely hide any reference in the test body as to whether the test is running locally or remotely, which is a good example of ‘test ignorance’.
Continuous deployment towards Test release environment
There were some changes to be made to continuously deploy any source changes. The current release package only contains the required deploy artifacts and any additional custom scripts to complete the deployment — the actual source code was missing. So why do we need the source code? Because we need to have access to them when we want to run the latest tests. We could use and have experimented with a dedicated build that provides us with these sources, but that not only scatters the continuous flow (as we would need two release packages) but also unnecessarily complicates things.
The release was also not yet triggered on any source change. In the diagram, this is one of the first actions in the flow but in practice, it is one of the last steps in completing the chain.
Continuous integration testing
The new release package also has some additional scripts to set up our test environment. The release pipeline was already equipped to update any necessary access tokens and other secrets or configuration values in a separate Azure Key vault and referenced variable groups. The missing piece was client-related resources and any other values/secrets that were not needed for post-deployment but were needed for testing.
Before the integration tests could be run on the Test release environment, we needed to run an additional script that retrieves all these necessary values/secrets and sets up any client-related resources (think about the client Transco database). This information is set as pipeline variables so that they can be used as any other replacement variable for the tokens in our appsettings.json file. Due to their temporary lifespan, they are directly usable and do not affect any existing variable group.
🚩 There is also a dedicated variable group where we placed any public configuration values that we can use during testing. These are mostly names of common resources that were not available in the environment-related variable groups. The local variants of these values are placed in a Git-ignored appsettings.{env}.local.json file – same as the secrets.
🔗There is more on environment-related integration testing and test configuration in this dedicated blog post.
Conclusion
Setting up a stable and usable test suite is critical for the quality of any project. It is the only proof that anything you write actually works. Continuous integration testing is a fundamental part of this as it will block any change that does not validates against the described tests. Unit tests alone cannot do this because of their small scope and limited assertion capabilities.
The Invictus Framework project collects several Azure Functions that each represent an integration concept. It uses backend Azure resources which are automatically populated during the release. Adding the integration tests as a necessary part of this release is not only useful for development – it is mandatory to maintain quality.
Thanks for reading,
Stijn
Subscribe to our RSS feed