Ordered testing with XUnit, NUnit and MSTest part 6: NUnit implementation revised part 2
We have previously implemented some ordered testing with NUnit. However, it required forking NUnit to make the ordering possible.. This made the implementation pretty useless, since you’Il be maintaining your own fork forever. In this blog post we’re going to find a better solution and implement it. The code of this blog post can be found on GitHub. Disclaimer: This code will have rough edges, and may not work for you, kill you cat or blow up in your face.
This post is part of a series of posts on ordered testing:
- Ordered testing with XUnit, NUnit and MSTest part 1: The setup
- Ordered testing with XUnit, NUnit and MSTest part 2: MSTest
- Ordered testing with XUnit, NUnit and MSTest part 3: NUnit
- Ordered testing with XUnit, NUnit and MSTest part 4: XUnit
- Ordered testing with XUnit, NUnit and MSTest part 5: NUnit implementation revised
- Ordered testing with XUnit, NUnit and MSTest part 6: NUnit implementation revised part 2
A new look at DefaultTestAssemblyBuilder
If you’ve read the previous blog post, you’ve seen that NUnit uses the
DefaultTestAssemblyBuilder to set up the hierarchy of tests. The
DefaultTestAssemblyBuilder follows these steps to create the test hierarchy:
- Find all the test fixture types
- Create a namespace-based hierarchy
- Apply any attributes to the root of the hierarchy (the test assembly) which decorate the test assembly and implement
We can create an attribute to hook in at step 3 and apply our ordering logic. In that case we just modify the existing hierarchy, instead of creating a new one. Let’s do that!
Define the attribute:
In the attribute
IApplyToTest implementation we call the
OrderedTreeBuilder we used earlier. We change the
OrderedTreeBuilder a bit so it now uses the existing hierarchy, instead of creating a new one.
And we add a method to finalize creating the test hierarchy:
Apply the attribute in the target assembly:
And we’re done!
Running the tests
To run the tests, we can simply pass our test assembly to the NUnit runner and the tests will be executed in the correct order.
Test Files BankAccountApp.NUnitTests.Integration\bin\BankAccountApp.NUnitTests.Integration.dll => BankAccountApp.NUnitTests.Integration.ChildNamespace.ChildTestFixtureWithoutOrder.Test1 => BankAccountApp.NUnitTests.Integration.ChildNamespace.ChildTestFixtureWithoutOrder.Test2 => BankAccountApp.NUnitTests.Integration.TestFixtureWithoutOrder.Test1 => BankAccountApp.NUnitTests.Integration.TestFixtureWithoutOrder.Test2 => BankAccountApp.NUnitTests.Integration.PreIntegrationTest.PreIntegrationTest_FirstStep => BankAccountApp.NUnitTests.Integration.PreIntegrationTest.PreIntegrationTest_SecondStep => BankAccountApp.NUnitTests.Integration.IntegrationTest.NewAccount_AccountRepository_CanSaveAccount => BankAccountApp.NUnitTests.Integration.IntegrationTest.ExistingAccount_AccountRepository_CanRetrieveSavedAccount => BankAccountApp.NUnitTests.Integration.IntegrationTest.ExistingAccount_AccountRepository_CanDeleteSavedAccount => BankAccountApp.NUnitTests.Integration.IntegrationTest.NonExistingAccount_AccountRepository_GetThrows => BankAccountApp.NUnitTests.Integration.PostIntegrationTest.PostIntegrationTest_FirstStep => BankAccountApp.NUnitTests.Integration.PostIntegrationTest.PostIntegrationTest_SecondStep
Resharper test runner: Resharper appears to use its own methods to discover tests in the solution, so the ordered test collections don’t show up at all. The hierarchy is not correctly displayed. It is understandable, because it would then need to load the NUnit assemblies, and that would add complexity. But that is as far as the issues go: The tests are executed in the correct order and I could confirm the attribute I added is indeed invoked by Resharper.
Visual Studio runner: The display of tests in the test explorer is a bit… sad. Currently there are only a few tests implemented, so it is probably completely unreadable when you have more than a dozen tests.
Console runner: Shows (and executes) the tests in the correct order. The output is shown earlier in this post.
NUnit GUI runner: The NUnit GUI test runner is still under construction, but already in a working state. When loading the tests in the NUnit GUI, the tree of tests, just like intented is nicely displayed. The tests are also run in the correct order.
The same limitations as in the last implementation apply.
We’ve seen how we can use a different approach to get test ordering in NUnit, without using a fork. Still, there is the question of whether you want to use this approach. This approach relies on some non-contractual behaviour of the
DefaultTestAssemblyBuilder to do it’s work: Applying the
Also, just like it said in the last post, keep in mind that there is currently being discussed of implementing test ordering in NUnit beyond test methods. Before a working implementation reached NUnit, it may be a while though, these people put their spare time in developing open-source projects!
What are your thoughts?